Automatic ship handling system and ship control method

Abstract

This invention provides an automated ship handling system and ship control method that can perform appropriate ship handling according to the situation. [Solution] The automatic ship handling system (2) includes a ship (1) with a propeller (20), a steering device (30), a position sensor (46), a camera (44) that takes images of the area around the ship, and a controller (41) that performs automatic ship handling control to control the propeller and steering device for automatic ship handling from the departure point to the destination. The automatic ship handling control includes camera ship handling control, which controls the propeller and steering device by performing recognition processing on images acquired by the camera, and position sensor ship handling control, which controls the propeller and steering device based on position information acquired by the position sensor. If the position sensor fails within a predetermined water area where automatic ship handling is possible using only camera ship handling control, the controller directs the ship to a predetermined target position within the predetermined water area where the failure occurred using camera ship handling control.

Description

Technical Field

[0001] This invention relates to an automatic ship control system and a ship control method.

Background Art

[0002] Patent Document 1 discloses a ship control system including an automatic control device that automatically controls a ship autonomously without a crew. The automatic control device receives, as a steering command, information necessary for navigation such as a destination, a course, and a ship speed by wireless communication, and automatically controls the ship autonomously while detecting surrounding ships and obstacles with a ship radar device. When an abnormal steering state occurs due to hacking or malfunction of the automatic control device, an emergency command is issued by wireless communication from the original ship operator. When the ship control system receives the emergency command, it stops power supply to the automatic control device, brings the ship to an emergency stop, and then performs hovering control to keep the hull stationary.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, when an abnormal steering state occurs, it follows an emergency command by wireless communication from the original ship operator. However, depending on the situation, there may be cases where appropriate steering can be performed based on the autonomous judgment of the automatic ship control system, so there is room for improvement.

[0005] Therefore, an embodiment of this invention is to provide an automatic ship control system and a ship control method that can perform appropriate steering according to the situation.

Means for Solving the Problems

[0006] One embodiment of this invention provides an automatic ship handling system comprising: a propulsion system for propelling a ship; a steering device for changing the ship's course; a position sensor for acquiring the ship's position information; a camera for imaging the area around the ship; and a controller that performs automatic ship handling control for controlling the propulsion system and the steering device for automatic ship handling from a point of departure to a destination. The automatic ship handling control includes camera ship handling control, which performs recognition processing on images acquired by the camera and controls the propulsion system and the steering device based on the recognition processing; and position sensor ship handling control, which controls the propulsion system and the steering device based on position information acquired by the position sensor. If the position sensor fails within a predetermined water area where automatic ship handling is possible using only the camera ship handling control, the controller uses the camera ship handling control to direct the ship towards a predetermined target position within the predetermined water area where the failure occurred.

[0007] With this configuration, even if a position sensor malfunctions within a predetermined water area where automatic navigation using camera control is possible, automatic navigation using camera control can continue, directing the vessel towards a predetermined target position. Therefore, the automatic navigation system can make autonomous decisions to perform appropriate maneuvers according to the situation.

[0008] In one embodiment of this invention, the predetermined target position is the departure point or the destination.

[0009] In one embodiment of this invention, the controller performs the position sensor-based ship handling control outside the predetermined water area.

[0010] In one embodiment of this invention, the controller further performs obstacle avoidance control, which controls the propulsion system and the steering device to avoid obstacles outside the predetermined water area based on recognition processing of images acquired by the camera.

[0011] In one embodiment of this invention, if the last position information acquired immediately before the position sensor fails represents a position outside the predetermined water area, the controller performs a stop control to stop the vessel.

[0012] If the position sensor malfunctions, it may become impossible to properly control the ship's maneuvering using the position sensor. Therefore, if the position sensor malfunctions outside the designated water area, it is appropriate to stop the ship. Accordingly, with the above configuration, the automatic ship handling system can make autonomous decisions to perform appropriate maneuvers according to the situation.

[0013] In one embodiment of this invention, the automatic ship steering system further includes a direction sensor that acquires the direction information of the ship. If the position sensor fails outside the predetermined water area, the controller performs direction sensor ship steering control, which uses the direction information acquired by the direction sensor to control the propulsion system and the steering device to approach the predetermined water area.

[0014] This configuration allows the vessel to approach a designated area through direction sensor-based maneuvering control in the event of a position sensor failure. Once the vessel enters the designated area, the controller may switch to camera-based maneuvering control or perform stopping control. In this way, the automated maneuvering system can perform appropriate maneuvers according to the situation through autonomous decision-making.

[0015] In one embodiment of this invention, the controller executes the direction sensor ship handling control when the last position information acquired immediately before the position sensor fails represents a position within a predetermined distance from the predetermined water area, and when the last position information represents a position outside the predetermined distance from the predetermined water area, the controller does not execute the direction sensor ship handling control and instead executes a stop control to bring the ship to a stop.

[0016] This configuration allows for the appropriate selection of direction sensor-based maneuvering control or stopping control depending on the distance from a predetermined water area. In this way, the automated maneuvering system can perform appropriate maneuvers according to the situation through autonomous decision-making.

[0017] In one embodiment of this invention, the automatic ship handling system further includes a communication device for communicating with a remote ship handling base to remotely operate the propulsion system and the steering device. The controller notifies the remote ship handling base of status information of the automatic ship handling system via the communication device and performs remote control of the propulsion system and the steering device based on remote control signals received from the remote ship handling base via the communication device.

[0018] This configuration allows the status of a vessel in an automated navigation state to be monitored at a remote navigation base, and enables remote control of the vessel (remote navigation) when necessary.

[0019] In one embodiment of this invention, the status information of the automatic ship handling system includes stop control information indicating whether the controller is performing stop control to stop the ship.

[0020] One embodiment of this invention provides a vessel comprising a hull and the aforementioned automatic navigation system installed on the hull.

[0021] One embodiment of the present invention provides a ship control method for controlling a propulsion machine for propelling a ship and a steering device for changing the course of the ship by a controller. The method includes an automatic steering step of executing, by the controller, automatic steering control for controlling the propulsion machine and the steering device for automatic steering from a departure point to a destination based on an output signal of a position sensor for acquiring position information of the ship and an output signal of a camera for imaging the surroundings of the ship. The automatic steering control includes camera steering control for performing recognition processing on an image acquired by the camera and controlling the propulsion machine and the steering device based on the recognition processing, and position sensor steering control for controlling the propulsion machine and the steering device based on position information acquired by the position sensor. The ship control method further includes a camera steering step of directing the ship to a predetermined target position within the predetermined water area where the failure has occurred by the camera steering control when the position sensor fails within a predetermined water area where automatic steering is possible only by the camera steering control.

[0022] In one embodiment of the present invention, the predetermined target position is the departure point or the destination.

[0023] In one embodiment of the present invention, in the automatic steering step, the controller executes the position sensor steering control outside the predetermined water area.

[0024] In one embodiment of the present invention, in the automatic steering step, the controller further executes obstacle avoidance control for controlling the propulsion machine and the steering device to avoid obstacles based on recognition processing on an image acquired by the camera outside the predetermined water area.

[0025] In one embodiment of the present invention, the method further includes a stopping step of executing, by the controller, stopping control for stopping the ship when the last position information acquired immediately before the failure of the position sensor represents a position outside the predetermined water area.

[0026] In one embodiment of the present invention, when the position sensor fails outside the predetermined water area, the controller controls the propulsion device and the steering device so as to approach the predetermined water area by using the azimuth information of the ship acquired by the azimuth sensor, and further includes an azimuth sensor navigation control step of performing azimuth sensor navigation control.

[0027] In one embodiment of the present invention, the method executes the azimuth sensor navigation control step when the last position information acquired immediately before the position sensor fails represents a position within a predetermined distance range from the predetermined water area. When the last position information represents a position outside the predetermined distance range from the predetermined water area, the method further includes a stopping step of executing stopping control for stopping the ship by the controller without executing the azimuth sensor navigation control step.

[0028] In one embodiment of the present invention, the controller further controls a communication device for communicating with a remote navigation base in order to remotely operate the propulsion device and the steering device. The ship control method further includes a remote control step of notifying the remote navigation base of the state information of the ship by the controller through the communication device and controlling the propulsion device and the steering device based on a remote operation signal received from the remote navigation base through the communication device.

[0029] In one embodiment of the present invention, the state information of the ship includes stopping control information indicating whether the controller is executing stopping control for stopping the ship.

[0030] Note that the controller does not necessarily have to be a physically single device, and may be constituted by a plurality of physically separated devices each including a processor.

Advantages of the Invention

[0031] According to the present invention, it is possible to provide an automatic navigation system and a ship control method capable of performing appropriate navigation according to the situation. [Brief explanation of the drawing]

[0032] [Figure 1] Figure 1 is a diagram illustrating the outline of a system for remotely monitoring a vessel equipped with an automatic ship handling system according to one embodiment of the present invention. [Figure 2] Figure 2 is a block diagram illustrating an example of a ship's configuration. [Figure 3] Figure 3 is a flowchart illustrating an example of automatic ship handling control. [Modes for carrying out the invention]

[0033] Hereinafter, embodiments of this invention will be described in detail with reference to the accompanying drawings.

[0034] [Remote monitoring system] Figure 1 is a diagram illustrating the outline of a system for remotely monitoring a vessel equipped with an automatic ship handling system according to one embodiment of the present invention.

[0035] Ship 1 is equipped with onboard system 2. Onboard system 2 constitutes an automated navigation system that enables the unmanned ship 1 to be autonomously operated from its departure point to its destination. However, manual operation by a user may be performed at the departure point and / or destination, and manual operation may also be performed as needed at locations other than the departure point and / or destination. Automated navigation systems are typically used in ships that transport goods and other cargo.

[0036] The onboard system 2 is equipped with a communication device 53. A remote monitoring base 100 (an example of a remote ship handling base) for monitoring the ship 1 is equipped with a remote monitoring system 101 (an example of a remote ship handling system). The remote monitoring system 101 can communicate with the onboard system 2 via the communication device 53. More specifically, the communication device 53 and the remote monitoring system 101 are communicated via a wireless data communication network 3, such as a mobile phone network or a satellite phone network, and are communicated via the wireless data communication network 3.

[0037] The communication device 53 transmits status information of the onboard system 2 (i.e., status information of the ship 1) to the remote monitoring system 101. The remote monitoring system 101 collects the status information received from the communication device 53. The remote monitoring system 101 includes a computer 102 for information processing, a display 103 for providing information to the monitor, and an input device 104 operated by the monitor. The computer 102 displays the status information received from the communication device 53 of the ship 1 on the display 103. The monitor understands the status of the ship 1 based on the display on the display 103. The computer 102 may display an alarm on the display 103 when a specific abnormal condition occurs in the ship 1. Further alarm devices, such as an alarm sound generator, may be provided in addition to the display 103 to generate alarms.

[0038] The input device 104 and display 103 of the remote monitoring system 101 may provide a remote operation interface for remotely operating the vessel 1. For example, when a monitor inputs a remote operation command, the computer 102 sends a command to switch to remote control mode to the communication device 53 of the vessel 1. As a result, when the onboard system 2 switches to remote control mode, the monitor can operate the remote operation interface to send a remote operation signal to the onboard system 2 and remotely operate the vessel 1.

[0039] For example, an image captured by a remote-controlled camera 45 (see Figure 2) mounted on the ship 1 is transmitted from the communication device 53 to the remote monitoring system 101, and the image is displayed on the display 103. The monitor (remote operator) operates the input device 104 while viewing the image. As a result, the computer 102 transmits a remote control signal to the communication device 53 on the ship 1. The onboard system 2 performs remote control in response to this remote control signal, thereby achieving remote operation of the ship 1.

[0040] [Ship composition] Figure 2 is a block diagram illustrating an example configuration of the vessel 1. The vessel 1 includes a hull 11 and an onboard system 2 (automatic steering system) mounted on the hull 11. The onboard system 2 includes various pieces of equipment (outfitting equipment) fitted to the hull 11. The outfitting equipment includes a main controller 41 for overall control of the equipment fitted to the vessel 1, a propulsion system that provides thrust to the hull 11, and a steering device for changing the direction of travel of the hull 11. In this embodiment, a communication device 53 is also one of the outfitting equipment. In this embodiment, input equipment (steering equipment) for manual steering is also provided as outfitting equipment. In this example, the input equipment includes a steering wheel 12 and a remote control 15.

[0041] In this example, the propulsion system includes outboard motors 20. Specifically, one or more outboard motors 20 are located at the stern. In this example, multiple outboard motors 20 (more specifically, two) are mounted side by side at the stern. In this example, the outboard motors 20 are engine-driven outboard motors that use an engine 21 (internal combustion engine) as a power source to drive a propeller 25. Of course, electric outboard motors powered by electric motors may also be used. Specifically, the two outboard motors 20 include a port outboard motor 20P and a starboard outboard motor 20S, which are mounted side by side at the stern.

[0042] In this example, the steering device is a steering 30 that steers the outboard motor 20 left or right. One steering 30 is provided for each outboard motor 20, and in this example, two steering 30s are provided. The two steering 30s are the port steering 30P and the starboard steering 30S, corresponding to the port outboard motor 20P and the starboard outboard motor 20S, respectively.

[0043] The steering wheel 12 is rotated by the user during manual operation. The angle of operation of the steering wheel 12 is detected by the angle of operation sensor 13 and input to the helm ECU (electronic control unit) 14. The remote control 15 is equipped with an accelerator lever 16 that is operated by the user to adjust the direction (forward or reverse) and magnitude of the thrust force generated by the outboard motor 20 during manual operation. The operating position of the accelerator lever 16 is detected by the accelerator position sensor 17 and input to the remote control ECU 18.

[0044] The outboard motor 20 includes an engine 21, a propeller 25 driven by the engine 21, a shift mechanism 26, and an engine ECU 23. The shift mechanism 26 has multiple shift positions, namely a forward position, a reverse position, and a neutral position. The forward position is the shift position in which the propeller 25 rotates forward by the driving force of the engine 21. The reverse position is the shift position in which the propeller 25 rotates backward by the driving force of the engine 21. The neutral position is the shift position in which power transmission between the engine 21 and the propeller 25 is interrupted. The engine ECU 23 controls the operation of a shift actuator 27 that operates the shift mechanism 26, thereby controlling the direction of the thrust. The engine ECU 23 also controls the operation of a throttle actuator 22 that drives the throttle valve of the engine 21, thereby controlling the magnitude of the thrust.

[0045] The steering system 30 includes a steering actuator 31 and a steering ECU 32 that controls it. The steering actuator 31 generates power to rotate the outboard motor 20 left and right around a steering axis (not shown). This changes the direction of the thrust force that the outboard motor 20 imparts to the hull 11, thereby changing the direction of travel of the vessel 1. The steering system 30 may be an integrated unit with the outboard motor 20 or a separate unit. Figure 2 shows an example where the steering system 30 is configured as an integrated unit with the outboard motor 20 (for example, built into the outboard motor 20).

[0046] The outfitting equipment further includes an automatic navigation camera 44, a remote navigation camera 45, a GPS (Global Positioning System) receiver 46, a compass sensor 47, a radar 48, a millimeter-wave radar 49, an electronic chart 50, a depth sensor 51, a remote control ECU 52, an anchoring device 55, and a display device 60. The GPS receiver 46 is an example of a GNNS (Global Navigation Satellite System) position detection device and an example of a position sensor that detects the position of the vessel 1.

[0047] The automatic navigation camera 44 includes at least one camera that images the area around the vessel 1 and is primarily used to detect obstacles around the vessel 1 during automatic navigation control. The remote navigation camera 45 includes at least one camera that images the area around the vessel 1 and is primarily used to provide images for remote navigation from the remote monitoring base 100. The heading sensor 47 detects the heading of the vessel 1 and outputs heading information. The radar 48 and millimeter-wave radar 49 are used to detect obstacles around the vessel 1. The radar 48 provides wide-area obstacle information, and the millimeter-wave radar 49 is used to detect obstacles at short range. The electronic chart 50 is a device that provides chart data. The remote control ECU 52 is a controller for remote control that generates propulsion commands and steering commands based on commands (remote control signals) from the remote monitoring base 100. The remote control ECU 52 typically includes a processor and memory and is configured to realize the necessary functions by having the processor execute programs stored in memory.

[0048] The anchoring device 55 includes, for example, an anchor, a rope connected to the anchor, a reel for winding / unwinding the rope, and an electric motor for driving the reel. By controlling the electric motor with the main controller 41, anchoring and release can be performed unattended.

[0049] The display device 60 is a device for visually informing other vessels of the status of vessel 1. The display device 60 may include a lighting device 61, a shape display device 62, a rotating light 63, etc. The lighting device 61 is mainly used for nighttime display, and the shape display device 62 is mainly used for daytime display. The rotating light 63 may be used for display both daytime and nighttime.

[0050] The display device 60 is operated under the control of the main controller 41 and is configured to display the status of the ship 1 (especially the operating status). The display device 60 is configured to distinguish and display multiple states, including the state in which the main controller 41 is performing automatic ship handling control and the state in which automatic ship handling control cannot be performed.

[0051] More specifically, the lighting device 61 and the shape display device 62 are configured to indicate a stationary state. The indication of a stationary state may also be an indication of being at anchor. The indication of being at anchor by the lighting device 61 is the illumination of one white 360-degree light that emits light in all directions. The indication of being at anchor by the shape display device 62 is the display of one spherical shape (for example, a black sphere).

[0052] The lighting device 61 and the shape display device 62 are further configured to indicate a state of operational impairment. For example, when the vessel is being remotely operated by the remote monitoring system 101, a state of operational impairment may be indicated. The lighting device 61 indicates a state of operational impairment by illuminating two red all-around lights arranged on a vertical line. The shape display device 62 indicates a state of operational impairment by displaying two spherical shapes (for example, black spheres) arranged on a vertical line.

[0053] The rotating light 63 is a light that emits light while rotating the direction in which it is emitted around the vessel 1. The rotating light 63 may be activated, for example, during the execution of automatic steering control and / or remote steering to visually notify other vessels that the vessel is in an automatic steering state and / or a remote steering state. For example, the automatic steering state and the remote steering state may be distinguished and indicated by varying the speed of rotation.

[0054] A network for data communication, namely the shipboard network 10, is established on board the vessel. The shipboard system 2 is composed of the shipboard network 10 and various outfitting equipment connected to the shipboard network 10.

[0055] The onboard network 10 is connected to the helm ECU 14, remote control ECU 18, engine ECU 23, and steering ECU 32. Therefore, thrust commands from the remote control ECU 18 are transmitted to the engine ECU 23 via the onboard network 10. A thrust command is a signal that commands the direction (forward or reverse) and magnitude of thrust for each outboard motor 20. In this embodiment, the thrust command includes a shift command that commands the shift position of the shift mechanism 26 and an output command that commands the output (e.g., rotational speed) of the engine 21. Also, steering commands from the helm ECU 14 are transmitted to the steering ECU 32 via the onboard network 10. A steering command is a command signal corresponding to the operating direction (rotation direction) and operating angle of the steering wheel 12, and is a signal that commands the steering direction and steering angle of the outboard motor 20.

[0056] The onboard network 10 is further connected to a main controller 41. The main controller 41 typically includes a processor and memory, and is configured to perform the necessary functions by having the processor execute programs stored in memory. The main controller 41 is programmed to perform automatic ship handling control. When performing automatic ship handling control, the main controller 41 sends thrust commands to the engine ECU 23 via the onboard network 10 and steering commands to the steering ECU 32 via the onboard network 10. As a result, the outboard motor 20 (propulsion system) and the steering system 30 (steering device) are controlled by the main controller 41.

[0057] The main controller 41 can also obtain various information from the remote control ECU 18, helm ECU 14, engine ECU 23, and steering ECU 32. Therefore, the main controller 41 can obtain information on steering commands received by the steering ECU 32 and information on the detection results of various sensors 33 provided in the steering 30. The sensors 33 include, for example, a steering angle sensor. The steering angle sensor detects the actual steering angle of the outboard motor 20. The steering angle sensor may also be a sensor that detects the amount of operation of the steering actuator 31. Furthermore, the main controller 41 can obtain various information from the engine ECU 23. For example, it can obtain information on thrust commands received by the engine ECU 23 and information on the detection results of various sensors 24 provided in the outboard motor 20. The sensors 24 include, for example, a throttle opening sensor, engine rotational speed sensor, engine temperature sensor, cooling water pressure sensor, oil pressure sensor, shift position sensor, fuel pressure sensor, and fuel level sensor.

[0058] The ship's network 10 is further connected to an automatic navigation camera 44, a remote navigation camera 45, a GPS receiver 46, a compass sensor 47, a radar 48, a millimeter-wave radar 49, an electronic chart 50, a depth sensor 51, a remote control ECU 52, an anchoring device 55, a lighting device 61, a shape display device 62, a rotating light 63, and the like. The ship's network 10 is also connected to a gauge 42 for displaying various information and a communication device 53. The communication device 53 may transmit information such as the status of the ship 1, more specifically, configuration information of the ship 1 (especially the ship's system 2), information on failures occurring in the ship's system 2, and detection values ​​of sensors 24 and 33 to the remote monitoring system 101 (see Figure 1). The communication device 53 also transmits information on the control status by the main controller 41 to the remote monitoring system 101 as status information of the ship 1. In particular, the communication device 53 transmits stop control information to the remote monitoring system 101 indicating whether the main controller 41 is performing stop control. The communication device 53 can also transmit images captured by the remote operation camera 45 to the remote monitoring system 101. Furthermore, the communication device 53 receives various commands from the remote monitoring system 101 and transmits them to the main controller 41, remote control ECU 52, etc., via the ship's network 10.

[0059] The gauge 42 has the function of a display device that informs the user of, for example, the remaining fuel level, the engine speed and shift position of each outboard motor 20, the remaining battery level, etc. The gauge 42 may also be equipped with an input device 43 such as input buttons or a touch panel, and the user may be able to input various commands by operating the input device 43. The input device 43 may be provided separately from the gauge 42.

[0060] The helmsman's seat, where the steering wheel 12 and remote control 15 are located, is equipped with a main switch 19 that is operated to turn the power to the outboard motor 20 on / off and to start / stop their engines 21.

[0061] [Overview of Automated Vessel Operation] The user can input a destination, for example, by operating the input device 43. Specifically, by operating the input device 43, the user can display a map read from the electronic chart 50 on the gauge 42 and specify and input a destination on that map. Of course, the destination may also be input by other methods such as coordinate input. The main controller 41 has an autopilot function and obtains the current location from the GPS receiver 46, uses the obtained current location as the departure point, and calculates the route to the input destination. The calculated route is displayed on the map on the gauge 42. The user may modify the route by operating the input device 43 as needed. The main controller 41 may store a history of previously set routes in memory. In this case, the user may read the history and set a route from the departure point to the destination.

[0062] Once the route is set, the user operates the main switch 19 to start the engine 21 of the outboard motor 20, inputs an automatic steering start command from the input device 43, and disembarks. As a result, the vessel 1 becomes unmanned. Upon receiving the automatic steering start command, the main controller 41 starts automatic steering control after a waiting period required for the user to disembark. Automatic steering control involves controlling the outboard motor 20 (propulsion system) and the steering 30 (steering device) for automatic steering from the departure point to the destination.

[0063] The main controller 41 issues thrust and steering commands to avoid obstacles based on the outputs of the automatic navigation camera 44, radar 48, and millimeter-wave radar 49, so that the current position detected by the GPS receiver 46 moves towards the destination according to the set course. The thrust command is given to the engine ECU 23, and the steering command is given to the steering ECU 32. The engine ECU 23 controls the shift actuator 27 and the throttle actuator 22 according to the thrust command. The steering ECU 32 controls the steering actuator 31 according to the steering command. As a result, thrust forces of magnitude and direction corresponding to the thrust and steering commands act on the hull 11.

[0064] Upon arrival at the destination, the user waiting at the destination performs the prescribed mooring procedure, boards the vessel 1, and operates the input device 43 to input a command to terminate automatic ship handling control. This causes the main controller 41 to terminate automatic ship handling control. The user may, if necessary, move the vessel 1 manually or stop the engine 21 by operating the main switch 19.

[0065] [Overview of remote ship operation] There may be situations where the vessel 1 is forced to stop on the water far from both its destination and departure point. For example, if a malfunction occurs in the automatic navigation camera 44 and obstacle detection becomes unreliable, it is preferable to interrupt or cancel the automatic navigation control. In such cases, the vessel may be operated remotely from the remote monitoring system 101 (remote navigation).

[0066] Remote operation can be initiated by a remote operation start command from the remote monitoring system 101. In the remote monitoring system 101, a remote operation start command is transmitted to the ship's communication device 53 when a monitor performs a predetermined input operation. Upon receiving the remote operation start command, the communication device 53 transmits the command to the main controller 41 and the remote control ECU 52. As a result, the main controller 41 stops automatic operation control and enters remote control mode, and the remote control ECU 52 starts control for remote operation (remote control). In remote control mode, the main controller 41 transmits images captured by the remote operation camera 45 to the remote monitoring system 101 via the communication device 53.

[0067] In the remote monitoring system 101, the monitor (remote operator) displays images received via the communication device 53 on the display 103 and performs input operations for remote operation, causing the computer 102 to issue a remote operation signal. This remote operation signal is transmitted to the communication device 53 and provided from the communication device 53 to the remote control ECU 52. The remote control ECU 52 converts the remote operation signal received via the communication device 53 into a propulsion command and steering command in a format compatible with the onboard system 2. The remote control ECU 52 then provides the propulsion command to the engine ECU 23 and the steering command to the steering ECU 32. The engine ECU 23 controls the shift actuator 27 and the throttle actuator 22 according to the propulsion command. The steering ECU 32 controls the steering actuator 31 according to the steering command. As a result, a propulsion force of magnitude and direction corresponding to the remote operation signal acts on the hull 11.

[0068] [Automatic ship handling control] Figure 3 is a flowchart illustrating an example of automatic ship handling control by the main controller 41.

[0069] In this embodiment, the automatic ship handling control (automatic ship handling step) includes camera ship handling control, which performs recognition processing on images acquired by the automatic ship handling camera 44 and controls the propulsion system (outboard motor 20 in this embodiment; the same applies hereinafter) and steering system (steering 30 in this embodiment; the same applies hereinafter) based on the results of the recognition processing. The automatic ship handling control in this embodiment also includes position sensor ship handling control, which controls the propulsion system and steering system based on position information acquired by a GPS receiver 46, which is an example of a position sensor.

[0070] In this embodiment, the mode of automatic ship handling control differs depending on the water area in which the ship 1 is located, and the response to a malfunction in the automatic ship handling control also differs. In this embodiment, the water area is divided from the perspective of automatic ship handling control into the departure port water area, which is the water area near the departure port; the destination port water area, which is the water area near the destination port (destination port); and the out-of-port water area, which is the water area through which the ship 1 passes between the departure port water area and the destination port water area. The departure port water area is typically within the harbor of the departure port, and may include the water area near it. The destination port water area is typically within the harbor of the destination port, and may include the water area near it. The out-of-port water area is typically the open ocean water area outside the harbor.

[0071] The departure port waters and destination port waters are designated waters where automatic navigation is possible by camera navigation control, rather than by position sensor navigation control. For each port registered in the electronic chart 50, such designated waters are predetermined and registered in the electronic chart 50. Therefore, the designated waters registered for the departure port are the departure port waters, and the designated waters registered for the destination port are the destination port waters. When there is one or more intermediate ports (intermediate ports) to be passed through between the departure port and the destination port, the intermediate port is considered the destination (intermediate destination) until arrival at the intermediate port, and when departing from the intermediate port, the intermediate port is considered the departure port.

[0072] For each port, waters outside the designated registered water area are considered off-port waters. In automatic ship handling control in off-port waters, position sensor-based ship handling control is primarily used, with camera-based ship handling control used as a supplement for obstacle avoidance. Specifically, obstacle avoidance control is performed by controlling the propulsion system and steering device to avoid obstacles based on recognition processing of images acquired by the automatic ship handling camera 44.

[0073] The automatic ship handling control preferably includes automatic de-docking control for leaving the pier at the departure point at the time of departure. Furthermore, the automatic ship handling control preferably includes automatic docking control for docking at the pier at the destination point at the time of arrival. Typically, the automatic ship handling control preferably includes automatic de-docking control, automatic departure port control, automatic open-sea control, automatic destination port control, and automatic docking control. Automatic departure port control is automatic ship handling control in the waters of the departure port. Automatic open-sea control is automatic ship handling control in the waters outside the port. Automatic destination port control is automatic ship handling control in the waters of the destination port.

[0074] As described above, the user uses the input device 43, etc., to input the destination and set the route, and then inputs the command to start automatic navigation and disembarks. The main controller 41 then uses the current location detected by the GPS receiver 46 and the electronic chart 50 to identify the departure port area. The main controller 41 also uses the input destination and the electronic chart 50 to identify the target port area. The main controller 41 then defines the area between the departure port area and the target port area, including the route, as the out-of-port area.

[0075] [Pre-departure procedures] Prior to departure, the main controller 41 determines whether or not to start automatic docking control, that is, whether or not departure is permitted. Specifically, the main controller 41 determines whether or not departure has already taken place (step S1), and if so, determines whether or not there are any obstacles to departure (step S2). If it is determined that there are no obstacles to departure (step S2: NO), automatic docking control (step S3) is started. Automatic docking control is a control system that automatically moves the ship 1 away from the pier and departs by camera-based ship handling control, which performs image recognition on images (video) of the ship's surroundings captured by the automatic ship handling camera 44, and issues propulsion and steering commands while automatically analyzing the recognition results.

[0076] If it is determined that there is a reason for an obstacle to departure that would hinder automatic docking control (Step S2: YES), automatic docking control is not performed, that is, the vessel 1 is not allowed to depart, and a departure hold process (Step S4) is executed. The departure hold process may include an obstacle reason display process that displays the reason for the obstacle to departure on the gauge 42. The departure hold process may also include an alarm process that generates an alarm indicating that an obstacle to departure has occurred. The alarm process may include an alarm sound generation process, a notification process to the remote monitoring system 101 via the communication device 53, or a notification process from the communication device 53 to a pre-registered mobile terminal (for example, a smartphone held by the person in charge at the departure port).

[0077] For example, the reasons for the inability to depart include reasons for the inability to use automatic navigation cameras 44, remote navigation cameras 45, remote control ECUs 52, communication devices 53, GPS receivers 46, radar 48, millimeter-wave radar 49, electronic charts 50, propulsion systems (outboard motors 20), steering systems (steering 30), etc., and may also include other reasons for the inability to depart.

[0078] Reasons for delays in departure related to the automatic navigation camera 44 include complete failure, water droplet adhesion, backlighting, wave reflection, and misalignment of mounting position. "Complete failure" means that the output image of the automatic navigation camera 44 cannot be acquired. "Water droplet adhesion" means that water droplets are reflected in the output image of the automatic navigation camera 44. "Backlighting" means that the output image of the automatic navigation camera 44 is distorted by backlighting. "Wave reflection" means that strong reflected light from the water surface is reflected in the output image of the automatic navigation camera 44, causing image distortion. "Misalignment of mounting position" means that the relative position between the imaging area of ​​the automatic navigation camera 44 and the hull 11 is shifted from the predetermined position. When these reasons for delays are reported and departure is put on hold, the person in charge at the departure port may be able to resolve the issue by taking the necessary measures, which may enable departure, i.e., the start of automatic docking control. For example, if backlighting or glare from waves is an obstacle, these obstacles can be resolved by changing the direction of vessel 1 through manual maneuvering by the person in charge at the departure port. In the case of water droplet adhesion, it can sometimes be resolved by activating water droplet removal devices such as wipers or air blowing devices. Therefore, when water droplet adhesion is detected, the main controller 41 may activate the water droplet removal device, and if the water droplet adhesion is not resolved even after that, it may detect the water droplet adhesion as an obstacle to departure.

[0079] The reasons for delays in departure related to the remote control camera 45 are the same as those for the automatic control camera 44, including complete failure, water droplet adhesion, backlighting, wave reflection, and misalignment of the mounting position. When there are any reasons for delays related to the remote control camera 45, it is preferable not to depart because it is not possible to switch from automatic control to remote control. The countermeasures for the reasons for delays are the same as for the automatic control camera 44. The same applies to water droplet adhesion; when water droplet adhesion is detected, the main controller 41 may activate the water droplet removal device, and if the water droplet adhesion is not resolved even after this, the water droplet adhesion may be detected as a reason for delays in departure.

[0080] A reason for a departure failure related to the remote control ECU 52 is a complete failure. For example, the main controller 41 may determine that the remote control ECU 52 is faulty when it cannot establish communication with the remote control ECU 52, or when the remote control ECU 52 sends an error code indicating equipment malfunction. When there is a reason for a failure related to the remote control ECU 52, it is preferable not to depart because it is not possible to switch from automatic operation to remote operation.

[0081] Reasons for departure failure related to the communication device 53 include complete failure and being out of range. "Complete failure" may refer to a state in which the main controller 41 cannot communicate with the communication device 53, or a state in which the communication device 53 is sending an error code indicating equipment malfunction. "Out of range" may refer to a state in which the communication device 53 cannot establish communication with the remote monitoring system 101, and may refer to a state in which the communication device 53 is sending an error code indicating being out of range. When communication with the remote monitoring system 101 cannot be established, the state of the ship 1 in automatic operation mode cannot be monitored, and switching from automatic operation to remote operation is not possible, so it is preferable not to depart.

[0082] Reasons for departure problems related to the GPS receiver 46 include complete failure, large error, momentary interruption, and satellite loss. "Complete failure" is a state in which the main controller 41 cannot acquire data generated by the GPS receiver 46. "Large error" is a state in which, when the GPS receiver 46 provides error data along with position data, the error data exceeds a predetermined threshold. "Momentary interruption" is a state in which radio waves from satellites are temporarily not received due to structures such as bridges. "Satellite loss" is a state in which radio waves from the required number of positioning satellites are not received. When these reasons for failure are notified and the departure is put on hold, the person in charge at the departure port may be able to resolve the reason for failure by taking the necessary action, thereby enabling departure, i.e., the start of automatic docking control. For example, "large error," "momentary interruption," and "satellite loss" can sometimes be resolved by waiting for a while or by changing the position of the vessel 1 through manual maneuvering.

[0083] Reasons for departure obstruction related to radar 48 and millimeter-wave radar 49 include complete failure and misalignment of mounting positions. The main controller 41 may determine that radar 48 and millimeter-wave radar 49 are in a state of "complete failure" if it cannot communicate with them. The main controller 41 may also determine that radar 48 and millimeter-wave radar 49 are "misaligned" if they do not properly detect known obstacles (such as land or buildings) around their current location (for example, if they do not match the electronic chart data). The person in charge at the departure port may be able to resolve the obstruction by checking the mounting status of radar 48 and millimeter-wave radar 49 and correcting their mounting positions as necessary.

[0084] The reasons for departure obstruction related to the electronic chart 50 may include a discrepancy in detected topography. A "discrepancy in detected topography" refers to a case where the difference between the water depth detected by the depth sensor 51 and the water depth shown in the electronic chart 50 is greater than a threshold. The cause of this obstruction is often that the position of the vessel 1 is unsuitable for departure, rather than a malfunction of the electronic chart 50. The obstruction may be resolved by manually maneuvering the vessel 1 to a position suitable for departure by the person in charge at the departure port.

[0085] Reasons for failure to depart related to the propulsion system include, for example, reasons for failure related to the shift mechanism 26, specifically, failure of the shift actuator 27, sticking of the shift mechanism 26, and disconnection of the shift position sensor. Failure of the shift actuator 27 means that the shift actuator 27 is not operating, and this includes not only cases where there is a problem with the shift actuator 27, but also cases where the power line to the shift actuator 27 is disconnected. Sticking of the shift mechanism 26 means that the shift position cannot be changed. Disconnection of the shift position sensor means that the output of the shift position sensor cannot be obtained, and the shift position cannot be detected. Typically, these failures are detected by the engine ECU 23 and notified to the main controller 41. Repairs are required for departure. Reasons for failure to depart related to the propulsion system also include, for example, reasons for failure related to the throttle, specifically, failure of the throttle actuator 22, sticking of the throttle valve, and disconnection of the throttle opening sensor. A failure of the throttle actuator 22 means that the throttle actuator 22 is not operating, and this includes not only cases where there is a malfunction in the throttle actuator 22, but also cases where the power supply wire to the throttle actuator 22 is broken. A sticking throttle valve means that the opening degree of the throttle valve cannot be changed, or there is a significant delay in changing the opening degree. A broken wire in the throttle opening sensor means that the throttle opening sensor cannot output, and the throttle opening degree cannot be detected. Typically, these failures are detected by the engine ECU 23 and notified to the main controller 41. Repairs are required before departure.

[0086] Reasons for departure problems related to the steering system include failure of the steering actuator 31, seizure of the steering mechanism, and disconnection of the steering angle sensor. Failure of the steering actuator 31 means that the steering actuator 31 is not operating, and this includes not only cases where there is a problem with the steering actuator 31, but also cases where the power supply wire to the steering actuator 31 is disconnected. Seizure of the steering mechanism means that the steering angle cannot be changed. Disconnection of the steering angle sensor means that the output of the steering angle sensor cannot be obtained, and the steering angle cannot be detected. Typically, these failures are detected by the steering ECU 32 and notified to the main controller 41. Repairs are required before departure.

[0087] Other possible reasons for departure problems include the pier's complex shape being unsuitable for automatic berthing control, heavy traffic from other vessels making automatic berthing control unsuitable, engine 21 not being in operation, or fuel levels being below a threshold. In these cases, the problem can be resolved by the person in charge at the departure port taking the necessary measures.

[0088] [Processing in the waters of the departure port] After the automatic docking control is initiated and the vessel departs (leaving the dock) (Step S1: NO), the main controller 41 determines whether the current position of the vessel 1 is within the departure port waters (Step S5). Specifically, the main controller 41 acquires position data detected by the GPS receiver 46 and determines whether the current position indicated by that position data is within the departure port waters. If it is within the departure port waters (Step S5: YES), the main controller 41 determines whether to execute (start or continue) the automatic departure port control.

[0089] Whether or not automatic departure port control can be performed is determined by the presence or absence of obstacles that would hinder automatic departure port control (departure port obstacles) (step S6). If it is determined that there are no departure port obstacles and that automatic departure port control can be performed (started or continued) (step S6: NO), automatic departure port control is performed (step S7). Automatic departure port control is a control that automatically navigates the ship 1 according to the planned route by camera operation control, which is performed by recognizing images of the area around the ship captured by the automatic ship operation camera 44 and automatically analyzing the recognition results. If there are departure port obstacles (step S6: YES) and it is determined that automatic departure port control cannot be performed, the main controller 41 performs departure port obstacle processing (step S8).

[0090] The departure port fault handling may include an alarm process that generates an alarm indicating that a fault (departure port fault) has occurred that makes it impossible to execute (start or continue) automatic departure port control. The alarm process may include an alarm sound generation process, a notification process to the remote monitoring system 101 via the communication device 53, or a notification process from the communication device 53 to a pre-registered mobile terminal (for example, a smartphone held by the person in charge at the departure port).

[0091] The obstacle handling within the departure port may include automatic stop control. Automatic stop control is a control to stop the vessel 1 and keep it in that position. Specifically, the automatic stop control may include anchoring control, which involves activating the anchoring device 55 to drop anchor and keep the vessel at berth. Furthermore, if possible, the automatic stop control may include fixed-point holding control, which involves controlling the propulsion system and steering system to maintain the current position of the vessel 1. In addition, depending on the specific cause of the obstacle, the obstacle handling within the departure port may include camera-operated return handling, which involves returning to the departure point (an example of a predetermined target position) using camera-operated control.

[0092] For example, obstacles within the departure port include obstacles related to the automatic navigation camera 44, remote navigation camera 45, remote control ECU 52, communication device 53, GPS receiver 46, radar 48, millimeter-wave radar 49, electronic chart 50, propulsion system (outboard motor 20), steering system (steering 30), etc., and may also include other obstacles.

[0093] The causes of malfunctions related to the automatic navigation camera 44 within the departure port are the same as those for the causes of malfunctions upon departure. When a malfunction occurs related to the automatic navigation camera 44, the main controller 41 performs a stop control as a malfunction handling procedure within the departure port. In this case, if possible, the system can switch to remote navigation from the remote monitoring system 101 to take necessary action (remote control step). If a complete failure is the cause of the malfunction, the malfunction handling procedure within the departure port preferably includes an SOS transmission process. When backlighting is the cause of the malfunction, it may be possible to resolve these malfunctions by changing the orientation of the vessel 1 via remote navigation. If the malfunction related to the automatic navigation camera 44 cannot be resolved, it is preferable to return the vessel 1 to the departure port via remote navigation whenever possible.

[0094] The causes of malfunctions within the departure port concerning the remote-controlled camera 45 are the same as those for the causes of malfunctions upon departure. When a malfunction occurs with respect to the remote-controlled camera 45, the main controller 41 executes a stop control. In this case, if possible, the controller can switch to remote control from the remote monitoring system 101 to take necessary action (remote control step). If the cause of the malfunction is a complete failure, the malfunction processing within the departure port preferably includes an SOS transmission process. For example, when backlighting is the cause of the malfunction, these malfunctions can sometimes be resolved by changing the direction of the vessel 1 via remote control. If the malfunction cannot be resolved, it is preferable to return the vessel 1 to the departure port via remote control whenever possible. Furthermore, camera operation control is possible even if there is a malfunction with the remote-controlled camera 45. Therefore, when there is a malfunction within the departure port concerning the remote-controlled camera 45, the malfunction processing within the departure port preferably includes a camera operation return process in which the vessel returns to the departure port via camera operation control (camera operation step).

[0095] The causes of failures within the departure port related to the remote control ECU 52 and the causes of failures within the departure port related to the communication device 53 are the same as those for the causes of failures upon departure. When there is a cause of failure within the departure port related to the remote control ECU 52 or a cause of failure within the departure port related to the communication device 53, the failure handling within the departure port preferably includes camera-operated return handling, which involves returning to the departure port by camera-operated control.

[0096] The causes of a malfunction within the departure port related to the GPS receiver 46 are the same as those for a malfunction at departure. When a "complete failure" of the GPS receiver 46 is detected, the malfunction handling within the departure port preferably includes stopping control. After that, if possible, it is preferable to switch to remote operation by the remote monitoring system 101 and return the vessel 1 to the departure port (remote control step). When there is a "large error", the malfunction handling within the departure port preferably includes stopping control. In this case, it is preferable to wait until the error recovers to a state below a threshold. As for "momentary interruptions", since they occur when passing under a bridge, for example, if momentary interruptions are normal when compared with the electronic chart 50, it may not be judged as a malfunction within the departure port. For unexpected momentary interruptions, for example, if more than a predetermined number of momentary interruptions occur within a predetermined time, it may be judged as a malfunction and stopping control may be performed as a malfunction handling within the departure port. After that, if possible, it is preferable to switch to remote operation and return the vessel 1 to the departure port. In the case of "satellite loss," as with the case of "large error," it is preferable that the fault handling within the departure port includes stopping control. In this case, it is preferable to wait until the necessary number of satellite signals are reacquired and the error recovers to a state below the threshold. If a long time passes without recovery, it is preferable to switch to remote control and return vessel 1 to the departure point.

[0097] Since automatic navigation within the departure port waters is controlled by camera-based navigation, there is no need to use position data detected by the GPS receiver 46. Therefore, the handling of malfunctions related to the GPS receiver 46 within the departure port may be a camera-based navigation return process that docks the ship at the departure pier using automatic navigation controlled by camera-based navigation.

[0098] The causes of obstacles within the departure port related to the radar 48, millimeter-wave radar 49, and electronic chart 50 are the same as those for obstacles upon departure. When there are obstacles related to the radar 48, millimeter-wave radar 49, or electronic chart 50, the handling of obstacles within the departure port preferably includes camera-operated return-to-port processing, which involves returning to the departure port by camera-operated control.

[0099] The causes of obstacles within the departure port related to the propulsion system include, as with the causes of obstacles within the departure port, obstacles related to the shift mechanism 26 and obstacles related to the throttle. Similarly, obstacles related to the steering system are the same as those related to obstacles within the departure port. When an obstacle occurs within the departure port related to the propulsion system or steering system, it is preferable to include stop control. Furthermore, it is preferable that the obstacle handling within the departure port in this case includes SOS transmission processing. Generally, remote operation is difficult when an obstacle occurs in the propulsion system or steering system.

[0100] Other potential obstacles within the departure port include collisions with obstacles such as breakwaters or other vessels, and running aground on shallow waters. These can be detected by equipping the vessel with appropriate sensors and recognized by the main controller 41. When these obstacles occur, the main controller 41 preferably performs stop control and SOS transmission processing as part of the departure port obstacle handling. There may also be cases where the main controller 41 gives up camera-based ship handling control based on images acquired by the automatic ship handling camera 44. Specifically, this may occur when the gap in the breakwater cannot be recognized from the images acquired by the automatic ship handling camera 44, when the vessel cannot return to the planned route due to the effects of strong currents, or when the recognition processing of the images from the automatic ship handling camera 44 is poor due to insufficient brightness. When these obstacles occur, the main controller 41 preferably performs stop control as part of the departure port obstacle handling. Then, as far as possible, it is preferable to switch to remote ship handling from the remote monitoring system 101 and attempt to return to the route toward the destination (remote control step).

[0101] [Processing in waters outside the harbor] When the ship leaves the departure port area through automatic maneuvering using automatic departure port control (Step S5: NO, Step S9: NO), it enters the open sea area between the departure port area and the destination port area. Based on this, the main controller 41 determines whether to execute (start or continue) automatic ocean control. Specifically, it determines whether there are any obstacles that would hinder automatic ocean control (ocean obstacles) (Step S13). If there are no ocean obstacles (Step S13: NO), automatic ocean control is executed (Step S14). Automatic ocean control includes position sensor maneuvering control, which automatically navigates the ship 1 according to a planned route based on current position data acquired by a GPS receiver 46, which is an example of a position sensor. Automatic ocean control also further includes obstacle avoidance control, which controls the propulsion and steering devices to avoid obstacles based on recognition processing of images acquired by an automatic maneuvering camera 44.

[0102] If an offshore obstruction is detected (Step S13: YES) and automatic offshore control is deemed impossible, the main controller 41 executes offshore obstruction processing.

[0103] The offshore obstacle handling may include alarm processing that generates an alarm indicating that an offshore obstacle has occurred. The alarm processing may include alarm sound generation processing, notification processing to the remote monitoring system 101 via the communication device 53, and notification processing from the communication device 53 to a pre-registered mobile terminal (for example, a smartphone held by a person in charge at the departure port or destination port). The offshore obstacle handling may include automatic stop control. Automatic stop control is control to stop the vessel 1 and keep it in that position. Specifically, automatic stop control may include anchoring control that activates the anchoring device 55 to drop anchor and keep the vessel at berth. Also, if possible, automatic stop control may include fixed-point holding control that maintains the current position of the vessel 1 by controlling the propulsion system and steering device. Furthermore, depending on the specific obstacle, the offshore obstacle handling may include direction sensor steering control that detects the direction using the direction sensor 47 and moves towards the waters of the departure port or destination port.

[0104] For example, offshore obstruction causes include obstruction causes related to automatic navigation cameras 44, remote navigation cameras 45, remote control ECUs 52, communication devices 53, GPS receivers 46, radar 48, millimeter-wave radar 49, electronic charts 50, propulsion systems (outboard motors 20), steering systems (steering 30), etc., and may also include other obstruction causes.

[0105] The offshore obstacles related to the automatic navigation camera 44 are the same as those related to departure obstacles. When an offshore obstacle related to the automatic navigation camera 44 occurs, the main controller 41 performs a stop control as offshore obstacle processing. In this case, if possible, it can switch to remote navigation from the remote monitoring system 101 to take necessary action (remote control step). If the obstacle is a complete failure, the offshore obstacle processing preferably includes SOS transmission processing. When backlighting is the obstacle, these obstacles can sometimes be resolved by changing the direction of the vessel 1 via remote navigation. Even if the obstacle related to the automatic navigation camera 44 cannot be resolved, it is preferable to wait for the obstacle to be resolved while guiding the vessel 1 toward its destination via remote navigation, as much as possible (especially in cases other than complete failure).

[0106] The offshore obstruction causes related to the remote-controlled camera 45 are the same as those for departure obstruction causes. When an obstruction cause related to the remote-controlled camera 45 occurs, the main controller 41 performs a stop control. In this case, if possible, the system can switch to remote control from the remote monitoring system 101 to take necessary action (remote control step). If the obstruction cause is a complete failure, the offshore obstruction processing preferably includes SOS transmission processing. For example, when backlighting is the obstruction cause, these obstruction causes can sometimes be resolved by changing the direction of the vessel 1 via remote control. Even if the obstruction cause related to the remote-controlled camera 45 cannot be resolved, it is preferable to wait for the obstruction cause to be resolved while moving the vessel 1 toward its destination via remote control, as much as possible (especially in cases other than complete failure).

[0107] The offshore obstacles related to the remote control ECU 52 and the offshore obstacles related to the communication device 53 are the same as those related to departure obstacles. When there is an offshore obstacle related to the remote control ECU 52 or the offshore obstacle related to the communication device 53, the offshore obstacle handling preferably includes a process of heading towards the destination by automatic ship handling control, including position sensor ship handling control and obstacle avoidance control. For example, the main controller 41 may enter the waters of the destination port, and after the ship 1 has reached the vicinity of the destination by automatic ship handling control, it may perform a stop control. The main controller 41 may also perform SOS transmission processing.

[0108] The offshore obstacles related to the GPS receiver 46 are the same as those related to departure obstacles. When a "complete failure" of the GPS receiver 46 is detected, the offshore obstacle processing preferably includes stopping control (stopping step). Subsequently, if possible, it is preferable to switch to remote operation by the remote monitoring system 101 to direct the vessel 1 toward its destination (remote control step). When the GPS receiver 46 completely fails, position detection is not possible. Therefore, the main controller 41 makes the decisions in steps S5 and S9 based on the last position information acquired immediately before the GPS receiver 46 failed. If the last position information is in waters outside the harbor, it is determined whether or not there is an offshore obstacle (step S13). If a "complete failure" of the GPS receiver 46 is detected, stopping control is executed. When the fault related to the GPS receiver 46 is "large error", the offshore obstacle processing preferably includes continuing automatic operation control, including position sensor operation control and obstacle avoidance control, toward the destination. For example, the main controller 41 may enter the port waters of the destination, and after automatically controlling the ship 1 to reach the vicinity of the destination, it may perform a stopping control. After that, if possible, it is preferable to switch to remote control by the remote monitoring system 101 to dock the ship 1 at the destination pier. In the case of "momentary interruption" and "satellite loss," the same processing as for in-port obstruction processing may be performed as offshore obstruction processing. However, if position data cannot be acquired, it is preferable to avoid remote control in waters outside the port.

[0109] If a "complete failure" of the GPS receiver 46 is detected in waters outside the harbor, the main controller 41 may perform automatic maneuvering control toward the departure port waters or destination port waters by controlling the propulsion system and steering system while detecting the heading using the heading sensor 47 (heading sensor maneuvering step). More specifically, the main controller 41 determines whether the last position information detected by the GPS receiver 46 immediately before the failure indicates a position within a predetermined distance (for example, within 100 kilometers) from the departure port waters or destination port waters. If the last position information indicates a position within a predetermined distance of the departure port waters, the main controller 41 directs the vessel 1 toward the departure port waters using heading sensor maneuvering control. If the last position information indicates a position within a predetermined distance of the destination port waters, the main controller 41 directs the vessel 1 toward the destination port waters using heading sensor maneuvering control. When the vessel 1 reaches the waters of the departure port or destination port and the automatic navigation camera 44 is able to recognize land, buildings, etc. from the image, the system may switch to camera navigation control and return to the departure port or destination port. Alternatively, the main controller 41 may perform a stop control and wait for remote navigation from the remote monitoring base 100. If the last position information indicates a position in waters outside the port, it is preferable for the main controller 41 to perform a stop control.

[0110] The offshore obstruction events related to radar 48 and millimeter-wave radar 49 are the same as those for departure obstruction events. When there is an obstruction event related to radar 48 or millimeter-wave radar 49, offshore obstruction handling preferably includes stopping control. Where possible, it is preferable to switch to remote control and remotely steer the vessel toward the destination port (remote control step).

[0111] The offshore obstacles related to the electronic chart 50 are the same as those related to departure obstacles. When an offshore obstacle related to the electronic chart 50 occurs in waters outside the harbor, it is preferable to return to the departure port using automatic ship handling control (position sensor ship handling control and obstacle avoidance control).

[0112] Offshore obstacles related to the propulsion system include, similar to departure obstacles, obstacles related to the shift mechanism 26 and obstacles related to the throttle. When an obstacle related to the propulsion system occurs, the offshore obstacle handling preferably includes continuing automatic maneuvering and heading towards the vicinity of the destination. For example, the main controller 41 may enter the waters of the destination port, and after the vessel 1 has reached the vicinity of the destination by automatic maneuvering control, it may perform stopping control. After that, if possible, it is preferable to switch to remote control by the remote monitoring system 101 to dock the vessel 1 at the destination pier. However, if the shift mechanism 26 is stuck in the neutral position, the propulsion system cannot generate thrust. In this case, the offshore obstacle handling includes stopping control at the current location without heading towards the vicinity of the destination. The offshore obstacle handling may also include SOS transmission processing.

[0113] The causes of offshore obstacles related to the steering gear are the same as those for obstacles to departure. When an offshore obstacle occurs related to the steering gear, it is preferable to include stop control. Furthermore, it is preferable to include SOS transmission processing in this case. If the steering gear becomes stuck in the straight-ahead position, the course of the vessel 1 can be changed by adjusting the relative magnitudes of the thrust of the left and right propulsion engines (outboard motors 20P, 20S). Therefore, it is also possible to switch to remote control and navigate the vessel 1 to the vicinity of the destination before performing stop control and SOS transmission processing.

[0114] Other offshore obstacles include collisions with other vessels or other obstacles, and running aground on shallow waters. These can be detected by the main controller 41 using appropriate sensors and recognized by the main controller 41. When these obstacles occur, the main controller 41 preferably performs stop control and SOS transmission processing as offshore obstacle handling. There may also be cases where the main controller 41 gives up on position sensor-based maneuvering control. Specifically, this may occur when it is not possible to return to the planned course due to the effects of strong currents, or when insufficient brightness causes poor recognition processing of the image from the automatic maneuvering camera 44, making obstacle avoidance control difficult. When these obstacles occur, the main controller 41 preferably performs stop control as offshore obstacle handling. Then, as far as possible, it is preferable to switch to remote maneuvering from the remote monitoring system 101 and attempt to return to the course toward the destination (remote control step).

[0115] [Processing within the designated port area] By navigating vessel 1 in the waters outside the harbor according to the navigation route, vessel 1 enters the waters of the destination port (Step S9: YES). Based on this, the main controller 41 determines whether or not to execute (start or continue) automatic destination port control. Specifically, it is determined whether or not there are any obstacles within the destination port that would prevent the execution (start or continue) of automatic destination port control (Step S10). If it is determined that there are no obstacles within the destination port (Step S10: NO), automatic destination port control is executed (Step S11). Automatic destination port control is a control that automatically navigates vessel 1 according to the planned route by camera operation control, which is performed by automatically analyzing images (video) of the area around the vessel captured by, for example, an automatic navigation camera 44. If there are obstacles within the destination port (Step S10: YES) and automatic departure port control cannot be executed, the main controller 41 executes destination port obstacle processing (Step S12).

[0116] The port-based obstacle handling may include alarm processing that generates an alarm indicating that an obstacle has occurred within the port. The alarm processing may include alarm sound generation processing, notification processing to the remote monitoring system 101 via the communication device 53, and notification processing from the communication device 53 to a pre-registered mobile terminal (for example, a smartphone held by the person in charge at the port). The port-based obstacle handling may include automatic docking control. Automatic docking control is control to stop the vessel 1 and keep it in that position. Specifically, automatic docking control may include anchoring control that activates the anchoring device 55 to drop anchor and keep the vessel at berth. Also, if possible, automatic docking control may include fixed-point holding control that maintains the current position of the vessel 1 by controlling the propulsion system and steering device. Furthermore, depending on the specific obstacle, the port-based obstacle handling may include camera-operated docking processing to dock at the destination using camera-operated control.

[0117] For example, the causes of obstruction within the target port include causes of obstruction related to the automatic navigation camera 44, the remote navigation camera 45, the remote control ECU 52, the communication device 53, the GPS receiver 46, the radar 48, the millimeter-wave radar 49, the electronic chart 50, the propulsion system (outboard motor 20), the steering system (steering 30), etc., and may also include other causes of obstruction.

[0118] The causes of obstacles within the destination port related to the automatic navigation camera 44 are the same as those for obstacles to departure. When an obstacle occurs with respect to the automatic navigation camera 44, the main controller 41 performs a stop control as part of the obstacle handling within the destination port. In this case, if possible, the controller can switch to remote navigation from the remote monitoring system 101 to take the necessary action (remote control step). If the cause of the obstacle is a complete failure, it is preferable that the obstacle handling within the destination port includes an SOS transmission process. When backlighting is the cause of the obstacle, it may be possible to resolve these obstacles by changing the orientation of the vessel 1 via remote navigation. If the obstacles with respect to the automatic navigation camera 44 cannot be resolved, it is preferable to dock the vessel 1 at the destination pier via remote navigation whenever possible.

[0119] The causes of obstruction within the target port concerning the remote-controlled camera 45 are the same as those for obstruction causes when departing. When an obstruction occurs concerning the remote-controlled camera 45, the main controller 41 performs a stop control. In this case, if possible, it can switch to remote control from the remote monitoring system 101 and take the necessary action (remote control step). If a complete failure is the cause of the obstruction, it is preferable that the obstruction handling within the target port includes SOS transmission processing. For example, when backlighting is the cause of the obstruction, it may be possible to resolve these obstructions by changing the orientation of the vessel 1 via remote control. If the obstruction cannot be resolved, it is preferable to dock the vessel 1 at the destination pier via remote control whenever possible.

[0120] The reasons for obstacles within the target port related to the remote control ECU 52 are the same as those for obstacles when departing. When there are obstacles related to the remote control ECU 52, the handling of obstacles within the target port preferably includes an automatic maneuvering process to dock at the destination pier using camera-based ship handling control.

[0121] The causes of interference within the target port concerning the communication device 53 are the same as those for causes of interference upon departure. When there is an interference cause concerning the communication device 53, the handling of interference within the target port preferably includes stopping control and SOS transmission processing.

[0122] The obstacles within the destination port related to the GPS receiver 46 are the same as those for obstacles upon departure. Since automatic navigation within the destination port waters is controlled by camera navigation, it is not necessary to use the position data detected by the GPS receiver 46. Therefore, the obstacle handling within the destination port for obstacles related to the GPS receiver 46 is preferably a process of docking at the pier of the destination (an example of a predetermined target position) by automatic navigation controlled by camera navigation (camera navigation step).

[0123] The obstacles within the destination port related to the radar 48, millimeter-wave radar 49, and electronic chart 50 are the same as those for obstacles when departing. Since automatic navigation within the destination port waters is controlled by camera navigation, it can continue even if an obstacle related to the radar 48, millimeter-wave radar 49, or electronic chart 50 occurs. Therefore, when there is an obstacle related to the radar 48, millimeter-wave radar 49, or electronic chart 50, the handling of the obstacle within the destination port is preferably an automatic navigation process that docks at the destination pier using camera navigation control.

[0124] The causes of obstruction within the destination port related to the propulsion system include, as in the case of obstruction causes for departure, obstruction causes related to the shift mechanism 26 and obstruction causes related to the throttle. Similarly, obstruction causes related to the steering system are the same as in the case of obstruction causes for departure. When an obstruction cause related to the propulsion system or steering system occurs, the obstruction handling within the destination port preferably includes stopping control. Furthermore, in this case, the obstruction handling within the destination port preferably includes SOS transmission processing. When an obstruction cause occurs in the propulsion system or steering system, it is generally difficult to dock the vessel 1 at the destination pier by remote operation.

[0125] Other potential hazards within the target port include collisions with obstacles such as breakwaters and other vessels, and running aground on shallow waters. These can be detected by appropriate sensors and recognized by the main controller 41. When these hazards occur, the main controller 41 preferably performs stop control and SOS transmission processing as part of the target port hazard handling. There may also be cases where the main controller 41 gives up camera-based ship handling control based on images acquired by the automatic ship handling camera 44. Specifically, this may occur when the gap in the breakwater cannot be recognized from the images acquired by the automatic ship handling camera 44, when the ship cannot recover to the planned route due to the effects of strong currents, or when the recognition processing of the images from the automatic ship handling camera 44 is poor due to insufficient brightness. When these hazards occur, the main controller 41 preferably performs stop control as part of the target port hazard handling, and also preferably performs SOS transmission processing. If possible, the controller may switch to remote ship handling to dock the vessel 1 at the destination pier (remote control step). However, it is preferable that a person in charge at the destination board vessel 1 and manually maneuver it to dock at the destination pier.

[0126] [summary] As described above, according to this embodiment, the main controller 41 can appropriately perform automatic ship handling control by making autonomous decisions according to the situation. More specifically, the main controller 41 determines whether it is before departure, while navigating in the departure port waters, while navigating in the destination port waters, or while navigating in waters outside the port, and performs appropriate fault handling according to the cause of the fault. As a result, for example, even if the GPS receiver 46 (position sensor) fails in the departure port waters or destination port waters where navigation is possible with camera ship handling control alone, automatic ship handling by camera ship handling control can be continued. On the other hand, if the GPS receiver 46 fails in waters outside the port where position sensor ship handling control is used, the ship 1 can be stopped. Furthermore, even if the GPS receiver 46 fails in waters outside the port, if the last position information indicates a position within a predetermined distance from the departure port waters or destination port waters, the main controller 41 can perform direction sensor ship handling control to guide the ship 1 to the departure port waters or destination port waters.

[0127] [Differentiation] Although one embodiment of the present invention has been described above, the present invention can be implemented in other forms.

[0128] For example, the aforementioned failure causes are all examples, and the main controller 41 may make judgments regarding fewer or more failure causes. Furthermore, the main controller 41 may perform appropriate processing for each failure cause that differs from the explanation above.

[0129] Furthermore, while the above-described embodiment uses an outboard motor as an example of a propulsion system, the configuration of a propulsion system installed on a ship can take various forms, such as an inboard motor, an inboard / outboard motor, or a water jet. Also, it is sufficient to have at least one propulsion system, and three or more propulsion systems may be installed.

[0130] Furthermore, various design modifications can be made within the scope of the matters described in the patent claims. [Explanation of symbols]

[0131] 1: Ship, 2: Onboard system (automatic navigation system), 11: Hull, 20: Outboard motor (propulsion system), 30: Steering (rudder device), 41: Main controller (controller), 44: Camera for automatic navigation, 45: Camera for remote navigation, 46: GPS receiver (position sensor), 47: Direction sensor, 52: Remote control ECU (controller), 53: Communication device, 100: Remote monitoring base, 101: Remote monitoring system

Claims

1. A propulsion system for propelling a ship, A steering device for changing the course of the aforementioned vessel, A position sensor for acquiring the position information of the aforementioned vessel, A camera that images the area around the aforementioned vessel, Includes a controller that performs automatic navigation control for controlling the propulsion system and the steering device for automatic navigation from the departure point to the destination, The automatic ship handling control includes camera-based ship handling control, which performs recognition processing on images acquired by the camera and controls the propulsion system and steering system based on the recognition processing, and position sensor-based ship handling control, which controls the propulsion system and steering system based on position information acquired by the position sensor. The controller is an automatic ship handling system that, when the position sensor fails within a predetermined water area where automatic ship handling is possible solely by the camera handling control, directs the ship towards a predetermined target position within the predetermined water area where the failure occurred, using the camera handling control.

2. The automatic ship handling system according to claim 1, wherein the predetermined target position is the departure point or the destination.

3. The automatic ship handling system according to claim 1, wherein the controller performs the position sensor ship handling control outside the predetermined water area.

4. The automatic ship handling system according to claim 3, wherein the controller further performs obstacle avoidance control, which controls the propulsion system and the steering device to avoid obstacles based on recognition processing of images acquired by the camera outside the predetermined water area.

5. The automatic ship handling system according to claim 3, wherein the controller performs a stop control to stop the ship if the last position information acquired immediately before the position sensor fails represents a position outside the predetermined water area.

6. The system further includes a compass sensor that acquires the compass information of the aforementioned vessel, The automatic ship handling system according to claim 3, wherein the controller, when the position sensor fails outside the predetermined water area, performs direction sensor steering control, which controls the propulsion system and the steering device to approach the predetermined water area using direction information acquired by the direction sensor.

7. The automatic ship handling system according to claim 6, wherein the controller executes the direction sensor ship handling control when the last position information acquired immediately before the position sensor fails represents a position within a predetermined distance from the predetermined water area, and when the last position information represents a position outside the predetermined distance from the predetermined water area, the controller does not execute the direction sensor ship handling control and instead executes a stop control to stop the ship.

8. The system further includes a communication device for communicating with a remote ship handling base in order to remotely operate the propulsion system and the steering device, The automatic ship handling system according to claim 1, wherein the controller notifies the remote ship handling base of status information of the automatic ship handling system via the communication device, and performs remote control of the propulsion system and the steering device based on remote control signals received from the remote ship handling base via the communication device.

9. The automatic ship handling system according to claim 8, wherein the status information of the automatic ship handling system includes stop control information indicating whether the controller is performing stop control to stop the ship.

10. The hull and, A ship comprising the automatic ship handling system according to any one of claims 1 to 9, which is installed on the hull.

11. A ship control method that controls a propulsion system for propelling a ship and a steering device for changing the ship's course using a controller, The automatic maneuvering step includes, based on the output signal of a position sensor for acquiring the position information of the vessel and the output signal of a camera imaging the area around the vessel, the controller performs automatic maneuvering control to control the propulsion system and the steering device for automatic maneuvering from the departure point to the destination, The automatic ship handling control includes camera-based ship handling control, which performs recognition processing on images acquired by the camera and controls the propulsion system and steering system based on the recognition processing, and position sensor-based ship handling control, which controls the propulsion system and steering system based on position information acquired by the position sensor. The ship control method further includes a camera maneuvering step in which, if the position sensor fails within a predetermined water area where the ship can be automatically maneuvered by camera maneuvering alone, the camera maneuvering control directs the ship towards a predetermined target position within the predetermined water area where the failure occurred.

12. The ship control method according to claim 11, wherein the predetermined target position is the departure point or the destination.

13. The ship control method according to claim 11, wherein in the automatic ship handling step, the controller performs the position sensor ship handling control outside the predetermined water area.

14. The ship control method according to claim 13, wherein in the automatic ship handling step, the controller further performs obstacle avoidance control, which controls the propulsion system and the steering device to avoid obstacles outside the predetermined water area based on recognition processing of images acquired by the camera.

15. The ship control method according to claim 13, further comprising a stop step in which the controller performs a stop control to stop the ship if the last position information acquired immediately before the position sensor fails represents a position outside the predetermined water area.

16. The ship control method according to claim 13, further comprising a direction sensor steering step, in which, if the position sensor fails outside the predetermined water area, the controller controls the propulsion system and steering device using the direction information of the ship acquired by the direction sensor to approach the predetermined water area.

17. The compass sensor maneuvering step is executed when the last position information acquired immediately before the position sensor fails represents a position within a predetermined distance from the predetermined water area. The ship control method according to claim 16, further comprising a stopping step in which, if the last position information represents a position outside the predetermined distance range from the predetermined water area, the direction sensor maneuvering step is not performed, and the controller performs stopping control to stop the ship.

18. The controller further controls a communication device for communicating with a remote ship handling base in order to remotely operate the propulsion system and the steering device. The ship control method according to claim 11, further comprising a remote control step in which the controller notifies the remote ship control base of the status information of the ship via the communication device, and performs remote control to control the propulsion system and the steering device based on remote control signals received from the remote ship control base via the communication device.

19. The ship control method according to claim 18, wherein the ship status information includes stop control information indicating whether the controller is performing stop control to stop the ship.

Images