Obstacle estimation system

Abstract

[Problem] To provide an obstacle estimation system for improving safety of a ship by improving obstacle estimation accuracy. [Solution] An obstacle estimation system according to the present disclosure includes a plurality of data acquisition means and a control unit executing information processing and estimates information on an obstacle when a ship navigates. The control unit executes: ship information estimation processing for estimating ship position information and ship orientation information on the basis of first data acquired from at least one of the data acquisition means; ship information correction processing for correcting the ship position information and the ship orientation information on the basis of second data acquired from at least one of the data acquisition means; obstacle information estimation processing for estimating the position and size of an obstacle on the basis of the ship position information and the ship orientation information corrected in the ship information correction processing and third data acquired from at least one of the data acquisition means; and output information generation processing for generating output information based on information on the position and size of the obstacle estimated in the obstacle information estimation processing.

Description

Obstacle Estimation System

[0001] The present disclosure relates to an obstacle estimation system.

[0002] Conventionally, when a ship navigates, it is necessary to avoid obstacles such as floating objects on the water surface and other ships, and technology for doing so is needed. For example, Patent Document 1 discloses a system that integrates sensor data from multiple sensors to detect foreign objects present on the ship's path of travel and assists the ship in avoiding collisions.

[0003] Japanese Patent Application Laid-Open No. 2019-036010

[0004] However, even while a ship is moving, it may experience rocking and swaying due to the effects of waves and wind, and may be affected by tidal currents, making the ship's position and attitude unstable and making it difficult to ensure accurate estimation of obstacles.

[0005] Therefore, the present disclosure has been made in consideration of the above problems, and its purpose is to provide an obstacle estimation system that improves the accuracy of obstacle estimation and enhances the safety of ships.

[0006] According to the present disclosure, there is provided an obstacle estimation system comprising a plurality of data acquisition means and a control unit that performs information processing, and that estimates information regarding obstacles when a ship is navigating, wherein the control unit performs: a ship information estimation process that estimates ship position information and ship attitude information based on first data acquired from at least one data acquisition means; a ship information correction process that corrects the ship position information and ship attitude information based on second data acquired from at least one data acquisition means; an obstacle information estimation process that estimates the position and size of an obstacle based on the ship position information and ship attitude information corrected by the ship information correction process and third data acquired from at least one data acquisition means; and an output information generation process that generates output information based on the obstacle position and size information estimated by the obstacle information estimation process.

[0007] According to the present disclosure, it is possible to provide an obstacle estimation system that improves the accuracy of obstacle estimation and enhances the safety of ships.

[0008] 1 is a diagram illustrating an obstacle estimation system according to an embodiment of the present disclosure; FIG. 2 is a diagram illustrating a ship to which an example of the obstacle estimation system according to the embodiment is applied; and FIG. 3 is a flowchart illustrating a series of controls in the obstacle estimation system according to the embodiment.

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] FIG. 1 shows an example of an obstacle estimation system (hereinafter also simply referred to as the "system") according to this embodiment. The system of this embodiment can be installed on any type of vessel, and is particularly suitable for relatively small vessels. Specifically, the system of this embodiment can be employed in fishing boats, water taxis, small dual-purpose vessels, pleasure fishing boats, passenger ships, transportation boats, work boats, fireboats / patrol boats, pleasure yachts, pleasure motorboats, special work vessels, etc., and can also be employed in large passenger ships, tankers, cargo ships, etc.

[0011] The system of this embodiment is an obstacle estimation system that comprises multiple data acquisition means and a control unit that performs information processing, and estimates information about obstacles when a ship is navigating.The control unit performs a ship information estimation process that estimates ship position information and ship attitude information based on first data acquired from at least one data acquisition means, a ship information correction process that corrects the ship position information and ship attitude information based on second data acquired from at least one data acquisition means, an obstacle information estimation process that estimates the position and size of an obstacle based on the ship position information and ship attitude information corrected by the ship information correction process and third data acquired from at least one data acquisition means, and an output information generation process that generates output information based on the obstacle position and size information estimated by the obstacle information estimation process.

[0012] The system of this embodiment is an obstacle estimation system 1 for estimating obstacles when a ship is navigating. The system 1 includes, for example, a data acquisition unit 10, a control unit 20, a memory unit 30, an input unit 40, an output unit 50, a communication unit 60, and a ship operation control unit 70. Each component is connected via wire or wirelessly and can communicate with each other. In this example, the control unit 20, the memory unit 30, the input unit 40, the output unit 50, and the communication unit 60 are implemented in an information processing device (computer), but this is not limited to this, and each component may be independent.

[0013] The data acquisition unit 10 includes multiple data acquisition means, such as various sensors and cameras, and acquires various sensor data and image data. The data acquisition unit may include, for example, a camera, radar devices such as Lidar (Light Detection and Ranging), marine radar, and millimeter-wave radar, ultrasonic ranging sensors, GNSS modules, and QZSS modules, as well as positioning devices such as AIS, inertial measurement units (IMUs), inertial sensors, wind direction sensors, wind speed sensors, speed sensors (ground speed sensors, water speed sensors), direction sensors, acceleration sensors, gyro sensors, magnetic compasses, satellite compasses, temperature sensors, humidity sensors, barometric pressure sensors, altitude sensors, and infrared sensors. The data acquisition unit 10 can acquire information regarding the ship's current position (coordinates), attitude (heading, such as bow direction), movement speed, movement direction, bow direction, and turning amount, information about the surrounding environment, and information about obstacles such as other ships and people. AIS is a system for transmitting and receiving various information between ships and between ships and onshore facilities, and can wirelessly exchange ship information, such as each ship's position, attitude (course), movement speed, and destination. The direction sensor may be a magnetic direction sensor that calculates the heading direction using geomagnetism, a gyrocompass, a GPS compass, or the like.

[0014] The control unit 20 includes a processor (computing device) such as a CPU and is configured to execute various information processes based on programs stored in the storage unit 30. The control unit 20 can execute a ship information estimation process to estimate ship position information and ship attitude information based on data (first data) from the data acquisition unit 10. The ship position information can be estimated using coordinate information as positioning data acquired by the GNSS module as the current position. The ship position information may also be estimated by a ship position estimation process based on a comparison of map data (three-dimensional point cloud data) acquired from Lidar with map information stored in the storage unit. The ship position information may be expressed as two-dimensional or three-dimensional coordinate information in a specific coordinate system, and may be expressed in latitude and longitude. The ship attitude is expressed as orientation (angle) information in a specific two-dimensional or three-dimensional coordinate system. The attitude information may also be expressed in east-west, north-south, or north-south directions.

[0015] The control unit 20 can execute a ship information correction process to correct the ship position information and ship attitude information based on the data (second data) from the data acquisition unit 10. The control unit 20 can correct the ship position information and ship attitude information based on speed data acquired from a lidar or a camera, for example.

[0016] The control unit 20 can execute a speed estimation process to estimate the ship's speed based on data from the data acquisition unit 10. For example, the control unit 20 can calculate the distance traveled in a given time period (0.01 seconds, 0.1 seconds, 1 second, etc.) from position information from a given time period ago and current position information, and estimate the ship's speed from the information on the distance traveled in the given time period.

[0017] The control unit 20 can execute an obstacle information estimation process to estimate the position and size of an obstacle based on the ship's position information and ship's attitude information corrected by the ship information correction process, and the data (third data) from the data acquisition unit 10. The control unit 20 can also estimate the type of obstacle based on the ship's position information and ship's attitude information corrected by the ship information correction process, and the data (third data) from the data acquisition unit 10.

[0018] The control unit 20 can estimate the type of obstacle by, for example, selecting one of candidate obstacle types stored in advance in the storage unit 30. Information on the type of obstacle can include type information such as ships, specific ship classifications (fishing boats, water taxis, multipurpose boats, recreational fishing boats, passenger boats, transportation boats, work boats, fireboats / patrol boats, pleasure yachts, pleasure motorboats, special work boats, etc.), marine structures, people (including swimmers, surfers, yachts, paddleboards, etc.), rocks, shores, driftwood, seaweed, marine life, etc. Information on the type of obstacle is stored in association with size information (area, height).

[0019] The control unit 20 can estimate the position, posture, type, and size of an obstacle by, for example, image analysis of image data acquired by a camera. In the obstacle information estimation process, the control unit 20 may estimate the position, posture, type, and size of an obstacle using machine learning technology that uses a pre-trained learning model. When the control unit 20 estimates the position of an obstacle based on image data acquired by a camera, the control unit 20 estimates the three-dimensional position (coordinates) of the obstacle in the camera coordinate system, and then converts the camera coordinate system into a world coordinate system by coordinate transformation processing based on the camera position (coordinates) and shooting orientation information, thereby calculating the position (coordinates) of the obstacle in the world coordinate system. The control unit 20 can estimate the bow direction of a ship, the direction of a person's face (front direction), and the direction (direction of travel) of an object used, such as a surfboard, as obstacles by image analysis of image data from the camera or analysis of three-dimensional data (point cloud data, model data, etc.). The control unit 20 can estimate the three-dimensional position and posture of obstacles on or underwater based on sensor data from radar devices, Lidar, etc., or a combination of multiple devices, not limited to image data from cameras such as those described above.

[0020] The storage unit 30 may store candidate obstacle types, corresponding size values ​​(information on area and height in a planar view), and size ranges (upper and lower limits) in advance in association with each other. For example, the area range of a ship may be 3 m^2 or more and 9200 m^2 or less, and the height range of a ship (height above the water surface) may be 0.5 m or more and 100 m or less. The area range of a person may be 0.2 m^2 or more and 5 m^2 or less, and the height range of a person (height above the water surface) may be 0.1 m or more and 3 m or less. The storage unit 30 stores appropriate avoidance distance information associated with obstacle information such as the position, type, and size of the obstacle. The storage unit 30 may also store information related to a formula for calculating the appropriate avoidance distance. For example, the appropriate avoidance distance may be calculated according to the size (area in a planar view) of a ship as an obstacle. Furthermore, the parameters used in the calculation may include information such as the type of obstacle, movement speed, movement direction, etc. The memory unit 30 may also include information on a program that simulates movement of the ship along its navigation route (planned movement route) and whether or not it will come into contact with an obstacle. The memory unit 30 stores information on each data acquisition means. The information on the data acquisition means may be, for example, the position of the data acquisition means on the ship (e.g., relative position with respect to a specific point such as the center or center of gravity of the ship), relative attitude (e.g., direction relative to the bow direction of the ship), etc., but is not limited to these.

[0021] The control unit 20 may estimate the moving speed, moving direction, and attitude of an obstacle based on the ship's position information and ship's attitude information corrected in the ship information correction process, the ship's moving speed information and moving direction information, and data (third data) from the data acquisition unit 10. The control unit 20 can acquire moving speed information and moving direction information of obstacles such as other ships relative to the ship based on data from Lidar. The control unit 20 can also acquire moving speed information, moving direction information, etc. of obstacles using data from a radar device.

[0022] In the obstacle information estimation process, the control unit 20 can analyze image data acquired by the camera to estimate information such as the type, size, position, shape, and color of the obstacle. The control unit 20 may also estimate the obstacle information based on any information about the obstacle estimated from the image data and obstacle-related information pre-stored in the storage unit. For example, the control unit 20 may estimate the size of the obstacle from the image data, refer to the storage unit, and determine (estimate) an obstacle type from a candidate obstacle associated with a size range that matches the estimated size. The control unit 20 may also analyze image data acquired by the camera to estimate the type of the obstacle, and refer to the storage unit to determine (estimate) a candidate size associated with the type of the obstacle. Furthermore, the type, size, and position of the obstacle may be estimated based on detection data acquired from a radar device, detection data from a lidar, or a combination of these, rather than on image data acquired by the camera. For example, 3D data of an obstacle can be generated from distance data to multiple points on the obstacle using Lidar, and the 3D shape, type, size, position, etc. of the obstacle can be estimated based on the 3D data. In this way, the control unit 20 can select from the obstacle candidates by comparing the information on the obstacle candidate stored in the storage unit 30 with the data on the obstacle acquired from the data acquisition unit 10, and estimate the type, size, etc. of the detected obstacle. Note that the control unit 20 can also estimate the position and size of the obstacle based only on the data from the data acquisition unit 10.

[0023] The control unit 20 can store various information such as generated information in the storage unit 30, update the information in the storage unit 30, output the information from the output unit 50, and transmit the information to external devices, other ships, etc. via the communication unit 60. The control unit 20 can also accept information acquired by the data acquisition unit 10, user operation information input via the input unit 40, and information received by the communication unit 60, and perform information processing based on the information.

[0024] The storage unit 30 can store various types of information stored in advance, information acquired from the data acquisition unit 10, information received from external devices or other ships via the communication unit 60, information generated by the control unit 20, information input via the input unit 40, etc., and is composed of a non-volatile memory, a hard disk, etc. The storage unit 30 can store information on various parameters indicating the shape of the ship (e.g., three-dimensional model data), characteristics (characteristics related to power, speed, and turning amount), information on obstacles, two-dimensional or three-dimensional map information, navigation history information, navigation plan information, tidal information, etc.

[0025] The input unit 40 receives input information based on operations by the user, etc. The input unit 40 is composed of mechanical buttons, switches, operation levers, touch panels, etc. The input unit 40 may also include a microphone that allows voice input.

[0026] The output unit 50 outputs various types of information as images (video), sounds, etc. The output unit 50 may include, for example, a display unit such as a liquid crystal monitor or a touch panel that displays images, an audio output unit such as a speaker that displays sounds, a vibration generating unit (vibration device) that generates vibrations, etc.

[0027] The communication unit 60 is connected to a network such as the Internet or wireless communication, and can transmit data to external devices (servers, control devices, etc.) or other ships, and can receive data from external devices.

[0028] The vessel operation control unit 70 may include an auto-rudder that controls steering, an auto-throttle that controls a power unit (engine, motor, etc.) for rotating a propeller or thruster, etc. The vessel operation control unit 70 controls the operation of the vessel 70 (movements such as moving forward, backward, moving left and right, turning, etc.) based on instruction information from at least one of the control unit 20, the memory unit 30, the input unit 40, and the communication unit 60, and can navigate the vessel along a predetermined route.

[0029] Fig. 2 shows an example of the present system installed on a ship 100. The example in Fig. 2 includes a lidar, a camera, a GNSS antenna, an AIS antenna, and a wind direction and speed meter as a data acquisition unit 10, a control unit as a control unit 20, a monitor as an output unit 50, and an auto-rudder and auto-throttle as a ship operation control unit 70. In the example in Fig. 2, the lidar and the camera are installed on the front (bow side) and rear (stern side) of the ship 100, respectively, so that data about the surroundings of the ship 100, particularly the front and rear sides, can be detected with high accuracy. The configuration of the present system installed on a ship is not limited to the example shown in the figure, and other sensors, etc. may also be included.

[0030] FIG. 3 shows an example of a processing flow relating to a method for estimating obstacle information in the system of this embodiment.

[0031] The control unit 20 executes a ship information estimation process (S101) that estimates ship position information and ship attitude information based on first data acquired from at least one data acquisition means, a ship information correction process (S102) that corrects ship position information and ship attitude information based on second data acquired from at least one data acquisition means, an obstacle information estimation process (S103) that estimates the position and size of an obstacle based on the ship position information and ship attitude information corrected in the ship information correction process and third data acquired from at least one data acquisition means, and an output information generation process (S104) that generates output information based on the obstacle position and size information estimated in the obstacle information estimation process. The first data, second data, and third data may each include data acquired from a common data acquisition means. Furthermore, the first data, second data, and third data may each include data acquired from a single data acquisition means or from multiple data acquisition means.

[0032] In the ship information estimation process (S101), for example, the ship's three-dimensional position is estimated based on coordinate information from a GNSS antenna as a data acquisition means, and the ship's three-dimensional attitude information is estimated based on orientation data acquired from an inertial measurement unit. The ship information estimation process (S101) may be performed based on data acquired from only one data acquisition means, but from the perspective of improving estimation accuracy, it is preferable to perform the estimation process based on data acquired from multiple data acquisition means.

[0033] In the ship information correction process (S102), the ship position information and ship attitude information estimated in S101 are corrected based on, for example, speed information acquired from at least one of the Lidar and the camera. The control unit 20 estimates the current position and attitude (a predetermined time after the time) using, for example, the position (coordinates), movement direction information, and movement speed information from a predetermined time ago (0.01 seconds, 0.1 seconds, 1 second, etc.). The estimated position and attitude information is then compared with the ship position information and ship attitude information estimated in the ship information estimation process (S101). If they do not match, the ship position information and ship attitude information can be corrected by adopting the intermediate values ​​as accurate current position and current attitude information. Note that the ship information correction process may be performed using other methods. For example, the ship position information and ship attitude information may be corrected using position information and attitude information calculated by a ship position and attitude estimation process based on image data from a camera, or a combination of these methods. Furthermore, from the viewpoint of increasing the accuracy of the correction process, it is preferable to execute the correction process based on data acquired from a plurality of data acquisition means.

[0034] Next, in the obstacle information estimation process (S103), the position and attitude of the camera are estimated based on the ship's position information and ship's attitude information corrected in the ship information correction process (the camera's position and attitude in the ship's coordinate system), and the relative position data of the obstacle in the camera coordinate system is used to perform coordinate transformation to estimate the position of the obstacle in the world coordinate system. Furthermore, the camera image data is analyzed to estimate the type and size of the obstacle. The position, type, and size of the obstacle may be estimated using detection data acquired from the radar device alone or in combination.

[0035] Then, in the output information generation process (S104), for example, image data or audio data including information on the position and size of the obstacle can be generated as output information, and the position of the obstacle can be displayed on a map on a monitor, the type and size of the obstacle can be displayed in text or an image, or audio can be output from a speaker to warn the driver.

[0036] Here, the content of the output information is not particularly limited, and for example, navigation route information for navigating while avoiding obstacles may be generated and displayed on a monitor, or applied to an autopilot system for automatic navigation.

[0037] The output information may include information on the distance (appropriate avoidance distance) to be maintained from an obstacle. For example, if the size of the obstacle is small (e.g., 1 m^2 or less), the appropriate avoidance distance may be set small (e.g., 5 m, 10 m), and as the size of the obstacle increases (e.g., 10 m^2 or more), the appropriate avoidance distance may be set large (e.g., 20 m, 50 m). For example, if the obstacle is a person, the appropriate avoidance distance may be set large (e.g., 20 m, 50 m), and if the obstacle is a floating object such as a buoy, driftwood, or seaweed, the appropriate avoidance distance may be set small (e.g., 5 m, 10 m). The appropriate avoidance distance may be determined based on appropriate avoidance distance information pre-stored in the storage unit according to at least one of the type and size of the obstacle. In this case, information on the type and size of the obstacle and information on the appropriate avoidance distance are pre-stored in association with each other. Furthermore, if the obstacle is another ship, the appropriate avoidance distance may be calculated based on the moving speed and moving direction of the other ship.

[0038] As described above, in the system of this embodiment, the control unit 20 executes the following processes: a ship information estimation process (S101) that estimates ship position information and ship attitude information based on first data acquired from at least one data acquisition means; a ship information correction process (S102) that corrects ship position information and ship attitude information based on second data acquired from at least one data acquisition means; an obstacle information estimation process (S103) that estimates the position and size of an obstacle based on the ship position information and ship attitude information corrected in the ship information correction process and third data acquired from at least one data acquisition means; and an output information generation process (S104) that generates output information based on the information on the obstacle position and size estimated in the obstacle information estimation process. In this way, by correcting the ship information estimated in the ship information correction process before executing the obstacle information estimation process, the accuracy of obstacle estimation can be improved, which makes it easier to avoid contact with obstacles, thereby improving the safety of the ship. Furthermore, according to this embodiment, by repeatedly executing the above process at predetermined intervals such as 0.01 seconds, 0.1 seconds, etc., it is possible to estimate obstacle information with high accuracy substantially in real time. Note that the obstacle type may be estimated by the obstacle information estimation process, and in that case, information on the obstacle type may be output as image, text, audio, etc.

[0039] In the system of this embodiment, the obstacle information estimation process may include a data correction process that corrects the third data based on the ship's position information and ship's attitude information corrected in the ship information correction process. That is, for example, the third data, such as radar detection data, camera image data, and Lidar data, may be corrected based on the corrected ship's position information and ship's attitude information, thereby further improving the accuracy of obstacle estimation. Note that the correction process may include a so-called clustering process.

[0040] In the system of this embodiment, the output information may include information on the distance that the vessel should keep away from an obstacle based on information on the position and size of the obstacle. With this configuration, the vessel can navigate while safely and efficiently avoiding obstacles.

[0041] In the system of this embodiment, the information about the size of an obstacle may include information about the height from the water surface to the top of the obstacle. With this configuration, for example, it is possible to determine the distance to avoid the obstacle depending on its height, and to improve the accuracy of estimating the type of obstacle.

[0042] In the system of this embodiment, the obstacle information estimation process may include a process for determining whether the obstacle is another ship or a person. With this configuration, for example, the user can be notified that the obstacle is another ship or a person that requires special attention, and be urged to take caution or safely avoid the obstacle.

[0043] In the system of this embodiment, the first data may include positioning data received from a positioning satellite and data acquired from an inertial measurement unit. With this configuration, the own-ship information estimation process can be performed quickly and efficiently.

[0044] In the system of this embodiment, the second data may include speed data acquired from at least one of the Lidar and the camera. With this configuration, even if positioning data received from a positioning satellite cannot be acquired, the own ship information correction process can be performed with high accuracy.

[0045] In the system of this embodiment, the third data may include data acquired from Lidar. With this configuration, for example, it is possible to estimate information such as the position, size, and shape of an obstacle with high accuracy using the three-dimensional point cloud data acquired from Lidar.

[0046] In the system of this embodiment, the third data may include image data acquired from a camera. With this configuration, for example, it is possible to estimate information such as the relative position, size, shape, etc. of an obstacle with high accuracy by image analysis of the image data.

[0047] In the system of this embodiment, in environments where signals from positioning satellites cannot be received or signal interference occurs, such as under bridges, in other words, where GNSS and the like do not function, self-position estimation processing can be performed based on data from at least one of the Lidar and the camera.

[0048] When the control unit 20 determines based on the signal from the data acquisition unit 10 that a signal from a positioning satellite has not been received, the control unit 20 may notify the ship by image or sound from the output unit 50 that a signal from a positioning satellite has not been received, or may automatically switch the ship's position estimation method. The ship's position estimation method switching process may involve, for example, switching from a ship's position estimation process based on data such as GNSS to a ship's position estimation process based on data from Lidar and a camera. Then, when it determines again that a signal from a positioning satellite has been received, the ship's position estimation method may be returned to the original ship's position estimation method based on data from the positioning satellite.

[0049] In the system of this embodiment, the control unit 20 may determine the type of data to be used in each process based on priority information previously stored in the storage unit 30. For example, in the obstacle information estimation process, the priority may be set in advance so that the highest priority is given to a combination of camera image data and Lidar data, followed by camera image data only, Lidar data only, and radar device data only. The priority may be set taking into consideration, for example, high estimation accuracy and a low information processing load.

[0050] In the system of this embodiment, the control unit 20 may repeatedly determine whether each data acquisition means is functioning properly based on the presence or absence of a signal from the data acquisition means, and may determine the type of data to use for each process based on the determination result. For example, if it is determined that GNSS data cannot be acquired, the control unit 20 may execute a ship information estimation process using data acquired by Lidar to estimate the ship's position information and attitude information.

[0051] Information on the determination result as to whether each data acquisition means is functioning properly may be output from the output unit 50 as an image, sound, signal, or the like.

[0052] The system of this embodiment can be installed when a new ship is manufactured, but it can also be implemented by installing various components such as various sensors, cameras, drive units, power supplies, and information processing devices on an existing ship.

[0053] The system of this embodiment can be installed on a newly constructed ship, but can also be installed (retrofitted) on an existing ship. That is, the data acquisition unit 10, control unit 20, memory unit 30, input unit 40, output unit 50, communication unit 60, and ship operation control unit 70 may all or partly be formed as a system unit, and the system unit may be installed on an existing ship later.

[0054] In the system of this embodiment, for example, weather information and ocean current information may be acquired via the data acquisition unit 10 or the communication unit 60. In this case, for example, by using the weather information and ocean current information in at least one of the above-mentioned own ship information estimation process, own ship information correction process, and obstacle information estimation process, the accuracy of estimating the own ship or obstacles can be further improved.

[0055] In the system of this embodiment, output information about obstacles generated by the control unit, and information such as weather information, ocean current information, and tide information acquired via the data acquisition unit 10 or the communication unit 60 may be transmitted to other ships. In this case, for example, other ships can be notified of emergencies such as natural disasters and accidents. Information may also be shared with rescue teams and the coast guard to respond to marine distress and emergencies.

[0056] In the system of this embodiment, the current position and attitude of the ship may be predicted based on speed data of the ship at a predetermined point in time in the past, as described above, or the accuracy of the ship's position information and attitude estimation may be improved by predicting the ship's movements using machine learning or the like from information such as parameters indicating the ship's characteristics and navigation history.

[0057] In the system of this embodiment, the control unit may delete unnecessary obstacle information if the obstacle information acquired from multiple data acquisition means is duplicated. That is, the control unit performs a duplication determination process to determine whether the multiple obstacle information estimated (detected) by the multiple data acquisition means is the same obstacle. If it determines that there are two or more obstacle information pieces for one obstacle, it may delete the remaining obstacle information pieces and leave only one obstacle information piece in the storage unit. The duplication determination process may determine that the position information (coordinates) of another ship acquired by a first data acquisition means (e.g., AIS) and the position information of another ship (obstacle) estimated from data from a second data acquisition means (e.g., camera, lidar, radar device) are the same, or if the value (distance) indicating the difference between these two points (coordinates) is within a predetermined threshold. On the other hand, if it determines that the value (distance) indicating the difference between the two points (coordinates) is outside the predetermined threshold (exceeding the threshold), it may determine that the two obstacles are different, and the two obstacle information pieces may be maintained without being deleted. The first data acquisition means and the second data acquisition means may be a single data acquisition means or a combination of multiple data acquisition means. Although the first data acquisition means and the second data acquisition means are assumed to be different data acquisition means, some of them may be the same data acquisition means.

[0058] In the system of this embodiment, the control unit 20 can also determine the appropriate avoidance distance based on the attitude (direction) of the obstacle. For example, if the attitude (bow direction) of a ship acting as an obstacle is pointing toward the host ship, the parameters can be set so that the appropriate avoidance distance is greater than when the bow direction is the same as that of the host ship. If the obstacle is a person using a surfboard, paddleboard, or the like, the parameters can be similarly determined based on the person's orientation (the orientation of an object used by a person, such as a surfboard, canoe, yacht, or paddleboard) and the direction of the person's face. Such information about the obstacle's attitude and information about the parameters may be associated with each other and stored in advance in the storage unit.

[0059] In the system of this embodiment, the control unit 20 can execute a navigation route generation process that generates navigation route information for avoiding obstacles and navigating to a destination based on an appropriate avoidance distance (parameter). The navigation route generation process generates navigation route information that is the appropriate avoidance distance away from obstacles based on position information (coordinates) of a start position (e.g., the ship's current position), position information (coordinates) of a destination, 3D map information including the seabed topography, and obstacle information including parameters. In this process, determining the appropriate avoidance distance based on not only the position information of the obstacle but also the type and size information of the obstacle enables safe and efficient navigation without taking unnecessary detours.

[0060] In the system of this embodiment, the control unit 20 may generate a navigation route using a route search algorithm stored in the storage unit. For example, a navigation route may be generated by assigning a cost (numerical value) to each node, which is the smallest unit of area obtained by dividing a map (nautical chart), and searching to minimize the cost from the start node to the destination node. A parameter based on an appropriate avoidance distance may be used to set the cost. For example, the closer a node is to an obstacle, the higher the cost value, and nodes that are further away than the appropriate avoidance distance have a cost set to a predetermined value or less. In this way, the cost value may be used as a parameter related to the appropriate avoidance distance. In this case, the predetermined value may be 0 or any predetermined value. A search is performed to minimize the cost from the departure point to the destination using an estimated value of the cost (distance) from each node to the destination. The cost of each node used in the route search may be determined using any information other than obstacle information and map information (including water depth information and unnavigable area information). For example, the cost may be determined based on meteorological information (including wind direction, wind speed, etc.), ocean current information, wave size information, etc. For example, the higher the wind speed, the higher the cost; nodes where winds are blowing in the opposite direction to the direction of travel have a higher cost; nodes where ocean currents are occurring in the opposite direction to the direction of travel have a higher cost; and the larger the waves, the higher the cost.

[0061] The control unit 20 may generate a navigation route using a simulation program stored in the storage unit. For example, when the ship moves along a planned navigation route (planned travel route) at a predetermined speed (constant speed or a predetermined variable speed), the control unit 20 may determine the timing at which the ship approaches each obstacle, whether the ship will come into contact with the obstacle, the distance to the obstacle at the closest approach, and whether the ship will approach the obstacle by more than the appropriate avoidance distance (whether appropriate avoidance is possible). By repeatedly executing the simulation program while changing the planned travel route and the ship's speed, the control unit 20 can generate (search) a navigation route (including speed information) that allows the ship to navigate at an appropriate distance from obstacles and reach the destination in the shortest time. Machine learning based on a predetermined learning model may be used to generate such a navigation route.

[0062] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0063] The devices described in this specification may be realized as a single device, or may be realized by multiple devices (e.g., cloud servers) some or all of which are connected via a network. For example, the control unit and the storage unit may be realized by different servers connected to each other via a network.

[0064] The series of processes performed by the device described herein may be implemented using software, hardware, or a combination of software and hardware. A computer program for implementing each function of the control unit according to this embodiment may be created and installed on a PC or the like. A computer-readable recording medium storing such a computer program may also be provided. Examples of the recording medium include a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory. The computer program may also be distributed, for example, via a network, without using a recording medium.

[0065] Furthermore, the processes described herein using flowchart diagrams do not necessarily have to be performed in the order shown. Some process steps may be performed in parallel. Additional process steps may be employed, and some process steps may be omitted.

[0066] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0067] Note that the following configurations also fall within the technical scope of the present disclosure: (Item 1) An obstacle estimation system comprising a plurality of data acquisition means and a control unit that executes information processing, and that estimates information about an obstacle when a ship is navigating, wherein the control unit executes: an own ship information estimation process that estimates own ship position information and own ship attitude information based on first data acquired from at least one data acquisition means; an own ship information correction process that corrects the own ship position information and own ship attitude information based on second data acquired from at least one data acquisition means; an obstacle information estimation process that estimates the position and size of an obstacle based on the own ship position information and own ship attitude information corrected in the own ship information correction process and third data acquired from at least one data acquisition means; and an output information generation process that generates output information based on the information about the position and size of the obstacle estimated in the obstacle information estimation process. (Item 2) The obstacle estimation system according to claim 1, wherein the obstacle information estimation process estimates the type of obstacle based on the ship's position information and ship's attitude information corrected in the ship information correction process, and third data acquired from at least one data acquisition means. (Item 3) The obstacle estimation system according to item 1, wherein the obstacle information estimation process includes a data correction process that corrects the third data based on the ship's position information and ship's attitude information corrected in the ship information correction process. (Item 4) The obstacle estimation system according to item 2, wherein the output information includes information on the distance the ship should keep from the obstacle based on information on the position, type, and size of the obstacle. (Item 5) The obstacle estimation system according to item 1 or 2, wherein the information on the size of the obstacle includes information on the height from the water surface to the apex of the obstacle. (Item 6) The obstacle estimation system according to item 2, wherein the obstacle information estimation process includes a process of determining whether the type of obstacle is another ship or a person. (Item 7) The obstacle estimation system according to item 1 or 2, wherein the first data includes positioning data received from a positioning satellite and data acquired from an inertial measurement unit. (Item 8) The obstacle estimation system according to item 1 or 2, wherein the second data includes velocity data acquired from at least one of a lidar and a camera.(Item 9) The obstacle estimation system according to item 1 or 2, wherein the third data includes data acquired from a lidar. (Item 10) The obstacle estimation system according to item 1 or 2, wherein the third data includes image data acquired from a camera. (Item 11) The obstacle estimation system according to item 1 or 2, wherein the third data includes data acquired from a radar device.

[0068] REFERENCE SIGNS LIST 1 Obstacle estimation system 10 Data acquisition unit 20 Control unit 30 Storage unit 40 Input unit 50 Output unit 60 Communication unit 70 Ship operation control unit

Claims

1. An obstacle estimation system comprising multiple data acquisition means and a control unit that performs information processing, and which estimates information regarding obstacles when a ship navigates, wherein the control unit executes: an own ship information estimation process that estimates own ship position information and own ship attitude information based on first data acquired from at least one data acquisition means; an own ship information correction process that corrects the own ship position information and own ship attitude information based on second data acquired from at least one data acquisition means; an obstacle information estimation process that estimates the position and size of an obstacle based on the own ship position information and own ship attitude information corrected by the own ship information correction process and third data acquired from at least one data acquisition means; and an output information generation process that generates output information based on the obstacle position and size information estimated by the obstacle information estimation process.

2. An obstacle estimation system as described in claim 1, wherein the obstacle information estimation process estimates the type of obstacle based on the ship's position information and ship's attitude information corrected by the ship's information correction process, and third data acquired from at least one data acquisition means.

3. An obstacle estimation system as described in claim 1, wherein the obstacle information estimation process includes a data correction process that corrects the third data based on the ship's position information and ship's attitude information corrected in the ship's information correction process.

4. An obstacle estimation system as described in claim 2, wherein the output information includes information on the distance that the vessel should keep away from the obstacle based on information on the position, type, and size of the obstacle.

5. An obstacle estimation system according to claim 1 or 2, wherein the information on the size of the obstacle includes information on the height from the water surface to the top of the obstacle.

6. An obstacle estimation system as described in claim 2, wherein the obstacle information estimation process includes a process for determining whether the type of the obstacle is another ship or a person.

7. The obstacle estimation system according to claim 1 or 2, wherein the first data includes positioning data received from a positioning satellite and data obtained from an inertial measurement unit.

8. The obstacle estimation system according to claim 1 or 2, wherein the second data includes speed data obtained from at least one of a Lidar and a camera.

9. The obstacle estimation system according to claim 1 or 2, wherein the third data includes data acquired from a Lidar.

10. The obstacle estimation system according to claim 1 or 2, wherein the third data includes image data acquired from a camera.

11. The obstacle estimation system according to claim 1 or 2, wherein the third data includes data acquired from a radar device.

Images