Information processing device, control method, program, and storage medium

Abstract

To communicate the presence of objects around the vessel to the operator in an intuitively easy-to-understand manner. [Solution] In the information processing device, the object detection means detects an object based on point cloud data generated by a measuring device installed on the ship. The position relationship acquisition means acquires the relative position relationship between the object and the ship. The display control means displays information regarding the position relationship on the display means in a display mode corresponding to the position relationship.

Description

[Technical Field]

[0001] This disclosure relates to the processing of data measured on a vessel. [Background technology]

[0002] Conventionally, techniques have been known for estimating the self-position of a moving object by comparing (matching) shape data of surrounding objects measured using measuring devices such as laser scanners with map information in which the shapes of surrounding objects are pre-stored. For example, Patent Document 1 discloses an autonomous mobile system that determines whether a detected object in a voxel, which is divided into a space according to a predetermined rule, is a stationary or moving object, and performs matching between map information and measurement data for voxels in which stationary objects exist. Patent Document 2 also discloses a scan matching method that estimates the self-position by comparing voxel data, which includes the mean vector and covariance matrix of stationary objects for each voxel, with point cloud data output by a lidar.

[0003] Furthermore, Patent Document 3 discloses a docking support device that detects obstacles around a vessel when it is docking and outputs a determination result on whether or not docking is possible based on the detection results of the obstacles. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] International release WO2013 / 076829 [Patent Document 2] International release WO2018 / 221453 [Patent Document 3] Japanese Patent Publication No. 2020-19372 [Overview of the project] [Problems that the invention aims to solve]

[0005] In ship operation, it is important to grasp the surrounding situation not only when landing. For example, when there are obstacles around the ship, it is necessary to navigate around the obstacles. Also, when there are following waves around the ship, by navigating at an appropriate angle with respect to the following waves, the impact and sway on the ship can be reduced. Therefore, it is required to detect obstacles, following waves, etc. around the ship and convey them to the operator in an intuitive and easy-to-understand manner.

[0006] The present disclosure has been made to solve the above problems, and the main object is to provide an information processing device capable of conveying the presence of an object around a ship to an operator in an intuitive and easy-to-understand manner.

Means for Solving the Problems

[0007] The invention according to the claim is an information processing device, comprising: an object detection means for detecting an object based on point cloud data generated by a measuring device provided on a ship; a position relationship acquisition means for acquiring a relative positional relationship between the object and the ship; and a display control means for displaying information regarding the position relationship on a display means in a display mode corresponding to the position relationship.

[0008] Also, the invention according to the claim is a control method executed by a computer, which detects an object based on point cloud data generated by a measuring device provided on a ship, acquires a relative positional relationship between the object and the ship, and displays information regarding the position relationship on a display means in a display mode corresponding to the position relationship.

[0009] Also, the invention according to the claim is a program that causes a computer to execute a process of detecting an object based on point cloud data generated by a measuring device provided on a ship, acquiring a relative positional relationship between the object and the ship, and displaying information regarding the position relationship on a display means in a display mode corresponding to the position relationship.

Brief Description of the Drawings

[0010] [Figure 1] This is a schematic diagram of the driver assistance system. [Figure 2] This is a block diagram showing the configuration of an information processing device. [Figure 3] This figure shows the self-position that the self-position estimation unit should estimate, represented in three-dimensional Cartesian coordinates. [Figure 4] An example of a general data structure for voxel data is shown. [Figure 5] This is a diagram illustrating the water level as seen from the rider's perspective. [Figure 6] This diagram illustrates the reflection of light emitted from a lidar onto the water's surface. [Figure 7] This diagram illustrates the point cloud data used to estimate water level. [Figure 8] This is a diagram illustrating a method for detecting obstacles. [Figure 9] This is a diagram illustrating the method for detecting undertow waves. [Figure 10] This is a block diagram showing the functional configuration of the obstacle / wake detection unit. [Figure 11] This diagram explains how the search range is determined. [Figure 12] The results of a simulation to detect straight lines using the Hough transform are shown. [Figure 13] An example of Euclidean clustering is shown. [Figure 14] The simulation results of Euclidean clustering are shown. [Figure 15] This figure shows the relationship between the distance of an object and the interval of point cloud data. [Figure 16] The simulation results are shown for cases where the grouping threshold and score threshold are fixed, and for cases where they are set adaptively. [Figure 17] This shows water surface reflection data obtained around the vessel. [Figure 18] This document describes a method for removing wakes and obstacles from water surface reflection data. [Figure 19] Examples of obstacle and wake markings are shown. [Figure 20] This is a flowchart for obstacle / undertow detection processing. [Figure 21] This is a flowchart of the wave detection process. [Figure 22] This is a diagram illustrating a method for detecting straight lines. [Figure 23] This is a flowchart of the obstacle detection process. [Figure 24] This is a flowchart of the water surface position estimation process. [Figure 25] This is a flowchart and explanatory diagram of the wave information calculation process. [Figure 26] This is a flowchart for the obstacle information calculation process. [Figure 27] This is a flowchart for the screen display process of undertow information. [Figure 28] This is a diagram illustrating the emphasis parameters. [Figure 29] This is a flowchart for the screen display process of obstacle information. [Figure 30] This is an explanatory diagram of the water surface position estimation method according to Modification Example 1. [Figure 31] An example of wake detection using variation 2 is shown. [Modes for carrying out the invention]

[0011] In one preferred embodiment of the present invention, the information processing device includes object detection means for detecting an object based on point cloud data generated by a measuring device installed on a ship, position relationship acquisition means for acquiring the relative position relationship between the object and the ship, and display control means for displaying information relating to the position relationship on a display means in a display manner corresponding to the position relationship.

[0012] In the above-described information processing device, the object detection means detects an object based on point cloud data generated by a measuring device installed on the ship. The position relationship acquisition means acquires the relative position relationship between the object and the ship. The display control means displays the position relationship information on the display means in a display manner appropriate to the position relationship. This makes it possible to display information regarding the position relationship between the ship and the object in an appropriate display manner.

[0013] In one embodiment of the above-described information processing device, the display control means changes the display mode of the positional relationship information based on the degree of danger the object poses to the vessel, which is determined based on the positional relationship. In this embodiment, the display mode is changed according to the degree of danger. In a preferred example, the display control means emphasizes the display of the positional relationship information as the degree of danger increases.

[0014] In another embodiment of the information processing device described above, the positional relationship information includes the position of the vessel, the position of the object, the direction of movement of the object, the speed of movement of the object, the height of the object, and the distance between the vessel and the object. This allows the operator to easily understand the positional relationship with the object.

[0015] In another embodiment of the information processing apparatus described above, the object includes at least one of an obstacle and a wake, and the positional relationship information includes information indicating whether the object is an obstacle or a wake. In a preferred example in this case, if the object is a wake, the display control means displays at least one of the height of the wake and the angle in the direction from which the wake extends as positional relationship information. This enables the operator to navigate appropriately with respect to the wake.

[0016] In another preferred embodiment of the present invention, a control method performed by a computer detects an object based on point cloud data generated by a measuring device installed on the ship, obtains the relative positional relationship between the object and the ship, and displays information regarding the positional relationship on a display means in a display manner appropriate to the positional relationship. This makes it possible to display information regarding the positional relationship between the ship and the object in an appropriate display manner.

[0017] In another preferred embodiment of the present invention, the program causes a computer to perform the following processes: detect an object based on point cloud data generated by a measuring device installed on a ship, acquire the relative positional relationship between the object and the ship, and display information regarding the positional relationship on a display means in a display manner corresponding to the positional relationship. By executing this program on a computer, the above-described information processing device can be realized. This program can be stored and handled on a storage medium. [Examples]

[0018] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. Note that any character with "^" or "-" above it will be referred to as "A^" or "A" in this specification for convenience. - This is represented as (where "A" is any letter).

[0019] (1) Overview of the driver assistance system Figure 1 shows a schematic configuration of the driver assistance system according to this embodiment. The driver assistance system comprises an information processing device 1 that moves together with the ship, which is a moving object, and a group of sensors 2 mounted on the ship. Hereafter, the ship that moves together with the information processing device 1 will also be referred to as the "target ship".

[0020] The information processing device 1 is electrically connected to the sensor group 2 and estimates the position of the target vessel (also called "self-position") based on the outputs of the various sensors included in the sensor group 2. Based on the self-position estimation result, the information processing device 1 provides driving support such as automatic driving control of the target vessel. Driving support includes docking support such as automatic docking. Here, "docking" includes not only docking the target vessel at a quay but also docking the target vessel at a structure such as a pier. The information processing device 1 may be a navigation device installed on the target vessel or an electronic control device built into the vessel.

[0021] Furthermore, the information processing device 1 stores a map database (DB:DataBase) 10 containing voxel data "VD". Voxel data VD is data that records position information of stationary structures for each voxel, which represents a cube (normal grid) that is the smallest unit of three-dimensional space. Voxel data VD includes data that represents the measured point cloud data of stationary structures within each voxel using a normal distribution, and is used for scan matching using NDT (Normal Distributions Transform), as described later. The information processing device 1 uses NDT scan matching to estimate, for example, the position on the plane, height position, yaw angle, pitch angle, and roll angle of the target vessel. Unless otherwise specified, the self-position is assumed to include attitude angles such as the yaw angle of the target vessel.

[0022] Sensor group 2 includes various external and internal sensors installed on the target vessel. In this embodiment, sensor group 2 includes a Lidar (Light Detection and Ranging, or Laser Illuminated Detection and Ranging) 3, a speed sensor 4 for detecting the speed of the target vessel, a GPS (Global Positioning Satellite) receiver 5, and an inertial measurement unit (IMU) 6 for measuring the acceleration and angular velocity of the target vessel in three axes.

[0023] The lidar 3 discretely measures the distance to an object in the external environment by emitting a pulsed laser within a predetermined angular range in the horizontal and vertical directions, and generates three-dimensional point cloud data indicating the position of the object. In this case, the lidar 3 has an irradiation unit that irradiates laser light while changing the irradiation direction, a light receiving unit that receives reflected light (scattered light) of the irradiated laser light, and an output unit that outputs scan data (points that constitute the point cloud data, hereafter referred to as "measurement points") based on the light receiving signal output by the light receiving unit. The measurement points are generated based on the irradiation direction corresponding to the laser light received by the light receiving unit and the response delay time of the laser light specified based on the light receiving signal described above. Generally, the accuracy of the lidar's distance measurement is higher the closer the distance to the object, and lower the accuracy the farther the distance. The lidar 3 is an example of a "measurement device" in the present invention. The speed sensor 4 may be, for example, a speedometer using Doppler or a speedometer using GNSS.

[0024] Furthermore, the sensor group 2 may have a receiver that generates positioning results from a GNSS other than GPS, instead of the GPS receiver 5.

[0025] (2) Configuration of an information processing device Figure 2 is a block diagram showing an example of the hardware configuration of the information processing device 1. The information processing device 1 mainly consists of an interface 11, a memory 12, and a controller 13. Each of these elements is interconnected via a bus line.

[0026] Interface 11 performs interface operations related to the exchange of data between the information processing device 1 and external devices. In this embodiment, Interface 11 acquires output data from each sensor in the sensor group 2, such as the lidar 3, speed sensor 4, GPS receiver 5, and IMU 6, and supplies it to the controller 13. Interface 11 also supplies signals related to the control of the target vessel, generated by the controller 13, to each component of the target vessel that controls the operation of the target vessel. For example, the target vessel includes a drive source such as an engine or electric motor, a propeller that generates thrust in the direction of travel based on the driving force of the drive source, a thruster that generates thrust in the lateral direction based on the driving force of the drive source, and a rudder, etc., which is a mechanism for freely determining the direction of travel of the vessel. During automatic operation such as automatic docking, Interface 11 supplies control signals generated by the controller 13 to each of these components. If the target vessel is equipped with an electronic control device, Interface 11 supplies control signals generated by the controller 13 to the electronic control device. Interface 11 may be a wireless interface such as a network adapter for wireless communication, or it may be a hardware interface for connecting to external devices via cables, etc. Interface 11 may also perform interface operations with various peripheral devices such as input devices, display devices, and sound output devices.

[0027] Memory 12 is composed of various volatile and non-volatile memories such as RAM (Random Access Memory), ROM (Read Only Memory), hard disk drive, and flash memory. Memory 12 stores programs for the controller 13 to execute predetermined processes. Note that the programs executed by the controller 13 may be stored in storage media other than memory 12.

[0028] Memory 12 also stores map DB 10, which includes voxel data VD. In addition to voxel data VD, map DB 10 includes, for example, information about berthing locations (including shores and piers) and information about waterways to which ships can move. Map DB 10 may also be stored in an external storage device of the information processing device 1, such as a hard disk connected to the information processing device 1 via interface 11. The storage device may be a server device that communicates with the information processing device 1. The storage device may also consist of multiple devices. Map DB 10 may also be updated periodically. In this case, for example, controller 13 receives partial map information about the area to which its own position belongs from the server device that manages map information via interface 11 and reflects it in map DB 10.

[0029] In addition to the map DB 10, the memory 12 also stores information necessary for the processing performed by the information processing device 1 in this embodiment. For example, the memory 12 stores information used to set the downsampling size when downsampling is performed on the point cloud data obtained when the lidar 3 performs one cycle of scanning.

[0030] The controller 13 includes one or more processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a TPU (Tensor Processing Unit), and controls the entire information processing device 1. In this case, the controller 13 performs processing related to self-localization and driving assistance by executing programs stored in memory 12, etc.

[0031] Furthermore, the controller 13 functionally includes a self-position estimation unit 15 and an obstacle / wake detection unit 16. The controller 13 also functions as a "point cloud data acquisition means," a "water surface reflection data extraction means," a "water surface height calculation means," a "detection means," and a computer that executes programs.

[0032] The self-position estimation unit 15 estimates its own position by performing NDT-based scan matching (NDT scan matching) based on point cloud data based on the output of the LIDA 3 and voxel data VD corresponding to the voxel to which the point cloud data belongs. Here, the point cloud data to be processed by the self-position estimation unit 15 may be point cloud data generated by the LIDA 3, or it may be point cloud data after downsampling the point cloud data.

[0033] The obstacle / wake detection unit 16 uses the point cloud data output by the lidar 3 to detect obstacles and wakes around the ship.

[0034] The display device 17 displays information about obstacles and wakes around the detected vessel on a monitor or other device.

[0035] (3) Self-location estimation Next, we will explain the self-position estimation based on NDT scan matching performed by the self-position estimation unit 15.

[0036] Figure 3 shows the self-position to be estimated by the self-position estimation unit 15, represented in three-dimensional Cartesian coordinates. As shown in Figure 3, the self-position on a plane defined on the three-dimensional Cartesian coordinate system (x, y, z) is represented by the coordinates "(x, y, z)", the roll angle "φ", the pitch angle "θ", and the yaw angle (direction) "ψ". Here, the roll angle φ is defined as the rotation angle around the direction of travel of the target vessel, the pitch angle θ is the elevation angle of the direction of travel of the target vessel relative to the xy plane, and the yaw angle ψ is defined as the angle between the direction of travel of the target vessel and the x-axis. The coordinates (x, y, z) are, for example, absolute positions corresponding to a combination of latitude, longitude, and altitude, or world coordinates indicating a position with a predetermined point as the origin. The self-position estimation unit 15 then performs self-position estimation using these x, y, z, φ, θ, and ψ as estimation parameters.

[0037] Next, we will explain the voxel data VD used for NDT scan matching. The voxel data VD includes data representing the measured point cloud data of stationary structures within each voxel according to a normal distribution.

[0038] Figure 4 shows an example of the schematic data structure of voxel data VD. The voxel data VD includes information on parameters when expressing the point cloud in a voxel by a normal distribution. In this embodiment, as shown in FIG. 4, it includes a voxel ID, voxel coordinates, a mean vector, and a covariance matrix.

[0039] The "voxel coordinates" indicate the absolute three-dimensional coordinates of a reference position such as the center position of each voxel. Each voxel is a cube obtained by dividing space into a grid pattern, and since its shape and size are determined in advance, it is possible to specify the space of each voxel by the voxel coordinates. The voxel coordinates may be used as a voxel ID.

[0040] The "mean vector" and "covariance matrix" indicate the mean vector and covariance matrix corresponding to the parameters when expressing the point cloud in the target voxel by a normal distribution. The coordinates of an arbitrary point "i" in an arbitrary voxel "n" are X n (i) = [x n (i), y n (i), z n (i)] T is defined as, and when the number of points in voxel n is "N n ", the mean vector "μ n " and covariance matrix "V n " in voxel n are represented by the following formulas (1) and (2), respectively.

[0041]

Number

[0042]

Number

[0043] Next, the outline of NDT scan matching using the voxel data VD will be described.

[0044] Scan matching using NDT for ships involves estimation parameters that include the amount of movement in the horizontal plane (here, x and y coordinates) and the orientation of the ship. P=[t x t y t z t φ t θ t ψ ] T This will lead to the estimation of "t x " is the amount of movement in the x direction, "t y " is the amount of movement in the y direction, "t z " is the amount of movement in the z direction, "t φ " is the roll angle, "t θ " is the pitch angle, "t ψ The symbol " indicates the yaw angle.

[0045] Also, the coordinates of the point cloud data output by the LIDA3 are X L (j) = [x n (j), y n (j), z n (j)] T Therefore, X L (j) Average value "L' n This can be expressed by the following equation (3).

[0046]

number

[0047] Then, the self-position estimation unit 15 searches for voxel data VD that corresponds to the point cloud data converted to an absolute coordinate system (also called the "world coordinate system") which is the same coordinate system as the map DB10, and the mean vector μ contained in that voxel data VD. n and the covariance matrix V n Using this, the evaluation function value of voxel n (also called the "individual evaluation function value") "E nThe self-position estimation unit 15 calculates the individual evaluation function value E of voxel n based on the following equation (4). n Calculate.

[0048]

number

[0049] The self-localization unit 15 then calculates an overall evaluation function value (also called a "score value") "E(k)" for all voxels subject to matching, as shown by the following equation (5). The score value E serves as an indicator of the goodness of fit of the matching.

[0050]

number

[0051]

number

[0052] (4) Obstacle / wake detection Next, the detection of obstacles and wakes by the obstacle / wake detection unit 16 will be explained. The obstacle / wake detection unit 16 uses the water level calculated in the processing up to one time point in advance to detect obstacles and wakes. If there is an obstacle near the vessel, it is necessary to navigate while avoiding collision or contact with the obstacle. Obstacles include, for example, other vessels, pilings, bridge piers, buoys, nets, and debris. Also, if there is a wake from another vessel nearby, care must be taken in navigating the vessel to avoid being affected by it and becoming excessively turbulent. Therefore, the obstacle / wake detection unit 16 uses the water level to detect the presence of obstacles and wakes near the vessel.

[0053] (4-1) Estimation of water level Figure 5 illustrates the water level as seen from lidar 3. The draft of a vessel changes depending on the number of passengers and the amount of cargo. That is, the height to the water level as seen from lidar 3 changes. As shown in Figure 5(A), when the draft of the vessel is low, the water level as seen from lidar 3 will be lower. On the other hand, as shown in Figure 5(B), when the draft of the vessel is high, the water level as seen from lidar 3 will be higher. Therefore, as shown in Figure 5(C), by setting a search range with a predetermined width based on the water level, it becomes possible to correctly detect obstacles and wakes.

[0054] Figure 6 illustrates the reflection of light emitted from Lida 3 onto the water surface. A portion of the light emitted from Lida 3, which is directed downwards, may be reflected from the water surface and return to Lida 3. Now, let's assume that Lida 3 on a ship is emitting laser light, as shown in Figure 6(A). Figure 6(B) shows the light received by Lida 3 on a ship near a quay. In Figure 6(B), beam 101 is light that has been directly irradiated onto an object without being reflected from the water surface, and a portion of the scattered light has returned to Lida 3 and been received. Beam 102 is light emitted from Lida 3, reflected from the water surface, and returned directly to Lida 3 and is one of the water surface reflected lights (hereinafter also called "direct water surface reflected light"). Beam 103 is emitted from LiDAR 3, and when the light reflects off the water surface and hits the quay or other object, some of the scattered light is reflected back off the water surface and received by LiDAR 3. It is one type of water surface reflected light (hereinafter also referred to as "indirect water surface reflected light"). Since LiDAR 3 cannot recognize that the light is reflected from the water surface, when it receives beam 102, LiDAR 3 perceives that there is an object at that water surface position. Similarly, when it receives beam 103, LiDAR 3 perceives that there is an object below the water surface. Therefore, LiDAR 3 that receives beam 103 will output incorrect point cloud data indicating a position inside the quay, as shown in the figure.

[0055] Figure 7 illustrates the point cloud data used to estimate the water surface height (hereinafter also referred to as "water surface position"). Figure 7(A) is a view of the ship from the rear, and Figure 7(B) is a view of the ship from above. Around the ship, due to fluctuations in the water surface, the beam from lidar 3 is sometimes almost perpendicular to the water surface, generating direct water surface reflected light such as beam 102 mentioned above. On the other hand, when the ship is close to the shore, the beam from lidar 3 is reflected by the shore or other surfaces, generating indirect water surface reflected light such as beam 103 mentioned above. Therefore, the obstacle / wake detection unit 16 acquires multiple point cloud data of direct water surface reflected light near the ship and estimates the water surface position by averaging their z-coordinate values. Since the ship floats on the water, the amount it sinks into the water changes depending on the number of passengers and the amount of cargo, and the height from lidar 3 to the water surface changes. Therefore, it is possible to always calculate the distance from lidar 3 to the water surface using the above method.

[0056] Specifically, the obstacle / wake detection unit 16 extracts point cloud data measured at a location far from the shore but close to the vessel from the point cloud data output by the lidar 3. Here, a location far from the shore refers to a location that is more than a predetermined distance from the shore. The shore location can be a docking location (including shores and piers) stored in the map DB 10. The shore may also refer to a land location or structure other than a docking location. By using point cloud data measured at a location far from the shore, point cloud data of indirect water surface reflections can be excluded.

[0057] Furthermore, a position close to the vessel refers to a position within a predetermined range from the vessel's own position. By using point cloud data measured at a position close to the vessel, it becomes possible to estimate the water surface position with high accuracy using point cloud data obtained by directly measuring the reflected light from the water surface (hereinafter also referred to as "direct water surface reflection data").

[0058] (4-2) Obstacle detection Next, the obstacle detection method will be explained. Figure 8 is a diagram illustrating the obstacle detection method. After the water surface position is estimated as described above, the obstacle / wake detection unit 16 performs Euclidean clustering on the point cloud data at heights near the water surface position. As shown in Figure 8(A), if a "cluster" (hereinafter also called a "group") is detected by the Euclidean clustering process, the obstacle / wake detection unit 16 tentatively determines that the cluster to be an obstacle candidate. The obstacle / wake detection unit 16 similarly detects clusters in frames at multiple time points, and if it detects clusters of similar size at each time point, it determines that the cluster is some kind of obstacle.

[0059] Furthermore, when detecting small obstacles on the water, such as buoys, the water surface reflection component can also be valuable information from a detection perspective. In Figure 8(B), beam 111 is emitted from lidar 3, reflected by the buoy, and returned to lidar 3. On the other hand, beam 112 is emitted from lidar 3, reflected from the water surface, hits the buoy, and a portion of the scattered light is reflected again from the water surface and returned to lidar 3 for reception. In the case of small obstacles such as buoys, the amount of data that is directly reflected back from the buoy as beam 111 is small, so by including data of the component reflected from the water surface, such as beam 112, the amount of data used for analysis is increased and utilized for clustering. This increases the amount of data subject to clustering processing, thus improving the performance of clustering.

[0060] If the obstacle / wake detection unit 16 determines that a detected cluster is an obstacle, it subtracts the water surface position from the z-coordinate of the highest point of the obstacle to calculate the height Ho of the obstacle that is above the water surface, as shown in Figure 8(B).

[0061] (4-3) Method for detecting undertow Next, the method for detecting wakes will be explained. Figure 9 is a diagram illustrating the wake detection method. After the water surface position is calculated as described above, the obstacle / wake detection unit 16 performs a Hough transform on the point cloud data at heights near the water surface position, ignoring the z-coordinate and treating it as a point cloud in a two-dimensional plane. As shown in Figure 9(A), if a "straight line" is detected by the Hough transform process, the obstacle / wake detection unit 16 tentatively determines that the straight line is a wake candidate. The obstacle / wake detection unit 16 similarly detects straight lines in frames at multiple time points, and if a straight line with similar coefficients is detected at each time point, it determines that the straight line is a wake.

[0062] Furthermore, when detecting a wake, the water surface reflection component can also be valuable information from a detection perspective. In Figure 9(B), beam 113 is emitted from lidar 3, reflected by the wake, and returns to lidar 3. On the other hand, beam 114 is emitted from lidar 3, reflected from the water surface, hits the wake, and a portion of the scattered light is reflected again from the water surface and returns to lidar 3 for reception. In the case of a wake, the amount of data that is directly reflected back from the wake as beam 113 is small, so by including data of the component reflected from the water surface, as in beam 114, the amount of data used for analysis is increased and utilized in the Hough transform. This increases the amount of data subject to Hough transform processing, thus improving the performance of the Hough transform.

[0063] The obstacle / wake detection unit 16, after determining the presence of a wake using the two-dimensional data as described above, re-evaluates the z-coordinate for the points determined to be part of the wake. Specifically, the obstacle / wake detection unit 16 calculates the average z-coordinate using only the points whose z-coordinate value is higher than the water surface height, and then subtracts the water surface position from this average value to calculate the wake height Hw from the water surface.

[0064] (4-4) Example of an Obstacle / Wave Detection Unit Next, an embodiment of the obstacle / wake detection unit 16 will be described. In the following embodiment, the obstacle / wake detection unit 16 processes in the order of wake detection → obstacle detection → water surface position estimation, thereby facilitating subsequent processing. Specifically, the water surface position estimated by the water surface position estimation block 132 is used to determine the height of the wake and obstacle from the water surface, and is also used to set the search range for the point cloud data at the next time point.

[0065] Figure 10 is a block diagram showing the functional configuration of the obstacle / wake detection unit 16. The obstacle / wake detection unit 16 receives point cloud data measured by the lidar 3 and outputs wake information and obstacle information. The obstacle / wake detection unit 16 comprises a search range setting block 121, a line extraction block 122, a wake detection block 123, a wake information calculation block 124, a wake data removal block 125, a Euclidean clustering block 126, an obstacle detection block 127, an obstacle information calculation block 128, an obstacle data removal block 129, a mean / variance calculation block 130, a time filter block 131, a water surface position estimation block 132, and a display control unit 133.

[0066] The search range setting block 121 extracts point cloud data of water surface reflections directly from the input point cloud data and sets the search range in the height direction for obstacles and wakes. The obstacle / wake detection unit 16 detects obstacles and wakes by extracting and analyzing point cloud data belonging to the search range set with the water surface position as the center, as shown in Figure 5(C). However, if the ship is rocking or the waves are large, obstacles and wakes floating on the water surface may fall outside the search range and become undetectable. On the other hand, if the search range is increased to avoid this, irrelevant data may be included when the waves are small, reducing the detection accuracy.

[0067] Therefore, the search range setting block 121 calculates the standard deviation of the z-coordinate values ​​of the direct water surface reflection data obtained around the vessel, as described above, and sets the search range using that standard deviation value. Specifically, the search range setting block 121 estimates the wave height using the standard deviation of the z-coordinate values ​​of the direct water surface reflection data, and sets the search range according to the wave height. If the standard deviation of the z-coordinate values ​​of the direct water surface reflection data is small, the wave height is presumed to be small, as shown in Figure 11(A). In this case, the search range setting block 121 narrows the search range. For example, the search range setting block 121 sets the search range to around the average value of the z-coordinate values ​​of the direct water surface reflection data. This reduces the inclusion of noise, thereby improving the accuracy of detecting obstacles and wakes.

[0068] On the other hand, if the standard deviation of the z-coordinate values ​​of direct water surface reflection is large, it is presumed that the wave height is large, as shown in Figure 11(B). Therefore, the search range setting block 121 widens the search range. That is, the search range setting block 121 sets a wider search range than when the wave height is small, centered on the average value of the z-coordinate values ​​of the direct water surface reflection data.

[0069] For example, as shown in Figure 11(C), the search range setting block 121 may set the search range to a range of ±3σ centered on the average value of the z-coordinate of the direct water surface reflection data, using the standard deviation σ of the z-coordinate values ​​of the direct water surface reflection data. This allows for a wider search range even when waves are high, preventing the detection of obstacles and wakes from being missed. The search range setting block 121 outputs the set search range to the line extraction block 122.

[0070] The line extraction block 122 extracts straight lines from direct water surface reflection data (hereinafter also referred to as "search data") measured within the search range around the vessel using the Hough transform. The line extraction block 122 outputs the extracted straight lines to the wake detection block 123. Since a discretized 2D array is used when detecting straight lines with the Hough transform, the resulting straight lines are approximate. Therefore, the line extraction block 122 and the wake detection block 123 obtain more accurate straight lines using the following procedure. (Step 1) Calculate the approximate straight line using the Hough transform. (Step 2) Extract data where the distance to the approximate straight line is within a predetermined threshold (straight line distance threshold). (Step 3) Perform principal component analysis using the extracted data to recalculate the straight line, which will be used as the straight line for the undertow wave.

[0071] Figure 12 shows the results of a simulation performed to detect a straight line using the procedure described above. As shown in the figure, the straight line 141 obtained by the Hough transform is an approximate straight line, and therefore there is a slight discrepancy with the data. By extracting data within the straight-line distance threshold from the straight line 141 (marked with squares in Figure 12) and recalculating the straight line using principal component analysis with the extracted data, an accurate wake line 142 can be obtained.

[0072] The wake detection block 123 determines that the recalculated straight line is a wake and outputs wake data indicating the wake to the wake information calculation block 124 and the wake data removal block 125. The wake information calculation block 124 calculates the position, distance, angle, and height of the wake based on the equation of the straight line indicating the wake and the ship's own position, and outputs it as wake information.

[0073] The wake data removal block 125 removes wake data from the search data measured within the search range around the vessel and outputs it to the Euclidean clustering block 126. The Euclidean clustering block 126 performs Euclidean clustering on the input search data, detects clusters in the search data, and outputs them to the obstacle detection block 127.

[0074] In Euclidean clustering, the distance between all target points and all other points is calculated, and points whose inter-point distance is shorter than a predetermined value (hereinafter referred to as the "grouping threshold") are placed in the same group. Next, within each group, those containing a predetermined number of points or more (hereinafter referred to as the "point threshold") are considered clusters. Groups with a small number of points are likely to be noise and are therefore not considered clusters.

[0075] Figure 13 shows an example of Euclidean clustering. Figure 13(A) shows multiple points to be clustered. Grouping was performed by calculating the distance between each point shown in Figure 13(A) and comparing it with a grouping threshold. Since the distances indicated by each arrow in Figure 13(B) are greater than the grouping threshold, five groups A to E are obtained, shown by dashed lines in Figure 13(B). Next, the number of points belonging to each group was compared with a point threshold (here, "6"), and as shown in Figure 13(C), only groups A and C, which contain more points than the point threshold, were ultimately determined to be clusters.

[0076] Figure 14 shows the simulation results of Euclidean clustering. Figure 14(A) shows the simulation results when undertow data remains during Euclidean clustering. In Euclidean clustering, when grouping is performed using a grouping threshold, if undertow data remains, there is a risk of misidentifying the obstacle and the undertow as being in the same cluster. In the example in Figure 14(A), because the undertow and the obstacle were close together, the data for the obstacle and the undertow were grouped together, and since the score of that group was higher than the score threshold, they were detected as being in the same cluster.

[0077] Figure 14(B) shows the simulation results when Euclidean clustering is performed after removing wake wave data. To distinguish obstacles from wake waves, wake wave detection is performed first, and then Euclidean clustering is performed after removing the data identified as wake wave data. In this case, obstacles are correctly detected as clusters without being affected by the wake wave data.

[0078] Generally, since the lidar's light beam is emitted radially, the distance between data points increases with distance. Therefore, as shown in Figure 15, the further away the data point, the greater the distance between adjacent data points. Also, even for objects of the same size, the number of detected points will be higher when they are close and lower when they are far away. Therefore, in Euclidean clustering, by setting the grouping threshold and the number of points threshold according to the data distance value, clustering can be performed under conditions as similar as possible for both objects close to and far from the lidar.

[0079] Figure 16 shows the results of a simulation in which the grouping threshold was increased as the distance to the data points increased, and the score threshold was decreased as the distance to the centroid of the group increased.

[0080] Figure 16(A) shows Grouping threshold = 2.0m Score threshold = 6 points The simulation results for this scenario are shown below.

[0081] Figure 16(B) shows Grouping threshold = a × (data distance) Score threshold = b / (distance to the centroid of the group) The simulation results for this scenario are shown. Note that in this simulation, a=0.2 and b=80 were used, but in reality, these values ​​should be set to suit the characteristics of the LIDA3.

[0082] As can be seen by comparing Figure 16(A) and Figure 16(B), in Figure 16(B), in addition to cluster 1, which is located near the vessel, cluster 2, which is located farther away from the vessel, is also detected. In cluster 2, the distance between each data point is approximately 3m, but the grouping threshold calculated using the distance from the vessel to the data is approximately 4.5m. Since the distance is closer than this threshold, it is classified as the same group. Also, although there are 4 data points, the score threshold calculated using the distance to the centroid of the group is approximately 3.2. Since the number of points is higher than this threshold, it is determined to be a cluster. Furthermore, using the above formula, the grouping threshold for cluster 1 calculated using the distance from the vessel to the data is approximately 2.5m, and the score threshold is approximately 7.1, which is not significantly different from the fixed values ​​in Figure 16(A). In this way, adaptive threshold setting can minimize the failure to detect clusters and false detections, thereby improving the performance of obstacle detection.

[0083] The obstacle detection block 127 outputs point cloud data (hereinafter referred to as "obstacle data") indicating obstacles detected by Euclidean clustering to the obstacle information calculation block 128 and the obstacle data removal block 129. Based on the obstacle data, the obstacle information calculation block 128 calculates the position, distance, angle, size, and height of the obstacles relative to the ship's own position and outputs them as obstacle information.

[0084] The obstacle data removal block 129 removes obstacle data from the search data measured within the search range around the vessel and outputs it to the mean / variance calculation block 130. This is because when estimating the water surface position from direct water surface reflection data around the vessel, the presence of wakes or obstacles makes it impossible to accurately estimate the water surface position.

[0085] Figure 17(A) shows direct surface reflection data obtained when there are wakes and obstacles around the vessel. In this case, data from positions higher than the water surface, and indirect surface reflections due to obstacles and wakes (such as beam 112 in Figure 8(B) and beam 114 in Figure 9(B)) become sources of error in surface position estimation. Therefore, surface position estimation is performed using search data after removing wakes and obstacles using the wake data removal block 125 and obstacle data removal block 129, as shown in Figure 17(B). Specifically, as shown in Figure 18, from state 1, where there are wakes and obstacles around the vessel, the wakes are detected and removed as shown in state 2, resulting in state 3. Next, obstacles are detected and removed as shown in state 4, and direct surface reflection data that does not include wakes or obstacles is obtained as shown in state 5.

[0086] Specifically, the mean / variance calculation block 130 calculates the mean and variance of the z-coordinate values ​​of the direct water surface reflection data obtained around the vessel and outputs them to the time filter block 131. The time filter block 131 averages or filters the mean of the z-coordinate values ​​of the input direct water surface reflection data with past water surface positions. The water surface position estimation block 132 estimates the water surface position using the mean of the z-coordinate values ​​after averaging or filtering and the variance of the z-coordinate values ​​of the search data.

[0087] When estimating the water surface position, if the variance of the direct water surface reflection data around the vessel is large, it can be expected that the waves are high due to the passage of other vessels, or that there are floating objects that were not detected as obstacles. Therefore, if the variance is smaller than a predetermined value, the water surface position estimation block 132 estimates and updates the water surface position using the average value of the direct water surface reflection data. On the other hand, if the variance is greater than or equal to the predetermined value, the water surface position estimation block 132 does not update the water surface position and maintains the previous value. Here, the "predetermined value" may be a fixed value, or it may be a value set based on the average value of past variances, for example, twice the average value of the variance. The water surface position estimation block 132 then outputs the estimated water surface position to the search range setting block 121, the wake information calculation block 124, and the obstacle information calculation block 128. In this way, the water surface position is updated based on the newly obtained direct water surface reflection data, and wakes and obstacles are detected.

[0088] The display control unit 133 is configured, for example, as a liquid crystal display device, and displays information about the ship's surroundings on the display device 17 based on the wake information calculated by the wake information calculation block 124 and the obstacle information calculated by the obstacle information calculation block 128.

[0089] Figure 19(A) shows an example of displaying surrounding information when there are obstacles around a vessel. The surrounding information is displayed on the display screen of the display control unit 133. The surrounding information basically schematically shows the state of the area within a predetermined distance from the vessel as seen from above. First, the display control unit 133 displays the vessel 80 near the center of the display screen of the display device 17. The display control unit 133 also identifies the positional relationship between the vessel and the obstacle based on the position, speed, and direction of movement of the obstacle detected by the obstacle / wake detection unit 16, and displays the obstacle 82 on the display screen to indicate the identified positional relationship. In the example of Figure 19(A), the display control unit 133 displays the point cloud (measurement points) 81 that constitute the obstacle, and also displays the obstacle 82 as a figure surrounding these point clouds 81. In this case, the figure representing the obstacle 82 may be highlighted by using a conspicuous color or by making the display blink to emphasize the presence of the obstacle 82. Alternatively, the point cloud 81 may not be displayed, and only the detected obstacle 82 may be displayed.

[0090] Furthermore, the display control unit 133 displays information indicating the relative positional relationship between the ship and the obstacle (hereinafter also referred to as "positional relationship information") as surrounding information. Specifically, an arrow 84 indicating the direction of travel of the obstacle 82 is displayed, and the speed of movement of the obstacle 82 (v=0.13 [m / s]) is displayed near the arrow 84. In addition, a straight line 85 indicating the direction of the obstacle 82 relative to the ship 80 is displayed, and the distance between the ship 80 and the obstacle 82 (d=2.12 [m]) is displayed near the straight line 85. Furthermore, the width of the obstacle 82 (w=0.21 [m]) and the height of the obstacle 82 (h=0.15 [m]) are displayed near the obstacle 82.

[0091] Here, the display control unit 133 changes the display manner of the positional relationship information according to the degree of danger posed by the obstacle to the vessel. Basically, the display control unit 133 displays the positional relationship information in a more emphasized manner, that is, a manner that attracts the attention of the operator, the higher the degree of danger. Specifically, the display control unit 133 emphasizes the arrow 84 and the numbers indicating the speed of movement as the obstacle 82 gets closer and the speed of movement of the obstacle 82 increases. For example, the arrow 84 is made thicker and the size of the numbers indicating the speed of movement is increased. In addition, the color of the arrow 84 and the numbers indicating the speed of movement may be made to be a conspicuous color or to blink. In this case, considering the direction of movement of the vessel 80 and the obstacle 82, the arrow 84 and the numbers indicating the speed of movement may be emphasized as described above when the obstacle 82 is moving towards the vessel 80, and the arrow 84 and the numbers indicating the speed of movement may not be emphasized when the obstacle 82 is moving away from the vessel 80. Furthermore, the display control unit 133 emphasizes the display of the straight line 85 and the numbers indicating the distance to the obstacle 82 as the distance between the ship 80 and the obstacle 82 decreases. For example, the straight line 85 is made thicker, and the size of the numbers indicating the distance to the obstacle is increased. Alternatively, the color of the straight line 85 and the numbers indicating the distance to the obstacle 82 may be made more conspicuous, or they may be made to flash. This allows the operator to intuitively understand the degree of danger posed by the obstacle 82.

[0092] In the example above, the display control unit 133 displays the positional relationship information in a display mode with higher emphasis as the risk level increases. However, instead, the risk level may be classified into multiple stages using thresholds. For example, the display control unit 133 may classify the risk level into two stages using one threshold. In that case, the display control unit 133 displays the positional relationship information in two display modes with different emphasis levels. Alternatively, the display control unit 133 may classify the risk level into three or more stages and display the positional relationship information in a display mode with emphasis levels corresponding to each stage.

[0093] Figure 19(B) shows an example of displaying surrounding information when there is a wake around a vessel. In the example in Figure 19(B), the display control unit 133 displays the point cloud (measurement points) 81 that constitute the obstacle, and also highlights the wake 86 as a shape surrounding the point cloud 81. Alternatively, the point cloud 81 may not be displayed, and only the detected wake 86 may be displayed.

[0094] In the example in Figure 19(B), surrounding information includes an arrow 84 indicating the direction of travel of the wake 86, and the speed of the wake 86 (v=0.41 [m / s]) is displayed near the arrow 84. Furthermore, a straight line 85 indicating the direction of the obstacle 82 relative to the vessel 80 is displayed, and the distance between the vessel 80 and the obstacle 82 (d=4.45 [m]) is displayed near the straight line 85. In the example in Figure 19(B), the display control unit 133 displays the positional relationship information with a higher degree of emphasis the higher the level of danger. Here, since the direction of travel of the wake 86 is towards the vessel 80, the arrow 84 is made thicker and the size of the number indicating the speed is increased.

[0095] Furthermore, in the case of a wake, the display control unit 133 displays the angle of the wake 86 as seen from the ship (θ = 42.5 [deg]). The angle of the wake 86 is the angle between the direction of travel of the ship 80 and the direction in which the wake 86 extends. Generally, it is said that entering the wake at an angle of about 45 degrees reduces the impact and rolling of the ship. Therefore, the angle of the wake 86 may be displayed to guide the operator to cross the wake at an angle that minimizes impact and rolling. Alternatively, instead of displaying the angle of the wake 86 relative to the ship 80, a sector or similar shape indicating a range around 45 degrees relative to the wake may be displayed to guide the operator to enter the wake within that angle range.

[0096] Furthermore, in the case of a wake, the display control unit 133 displays the height of the wake 86 (h=0.23 [m]) near the wake 86. In this case, the larger the wake, the larger the size of the number indicating the wake height. Alternatively, the color of the displayed wake 86 (i.e., the figure representing the wake) may be changed to a color close to red to indicate the wake height.

[0097] (4-5) Obstacle / Sweep detection process Next, the obstacle / wave detection process performed by the obstacle / wave detection unit 16 will be described. Figure 20 is a flowchart of the obstacle / wave detection process. This process is realized when the controller shown in Figure 2 executes a pre-prepared program and operates as each element shown in Figure 10.

[0098] First, the obstacle / wake detection unit 16 acquires point cloud data measured by the lidar 3 (step S11). Next, the search range setting block 121 determines the search range from the estimated water surface position up to one time step prior and the standard deviation σ of the z-coordinate values ​​of the direct water surface reflection data obtained around the vessel (step S12). For example, when the standard deviation σ is, Search range = Estimated water surface position ±3σ Let's assume that. Then, the search range setting block 121 extracts point cloud data within the determined search range and uses it as search data for wake detection (step S13).

[0099] Next, the obstacle / wake detection unit 16 performs wake detection processing (step S14). Figure 21 is a flowchart of the wake detection processing. First, the line extraction block 122 takes each point of the search data obtained from the search range and temporarily ignores the z value to obtain two-dimensional data of x and y (step S101). Next, the line extraction block 122 calculates (θ,ρ) for all search points by changing θ in the range of 0 to 180 degrees using the following equation (7) (step S102). In order to create a discretized two-dimensional array with (θ,ρ) as the element, "θ" and "ρ" are expressed as integers.

[0100]

number

[0101] Next, the linear extraction block 122 examines the number of each (θ,ρ) and extracts those that are greater than a predetermined value and represent a local maximum (step S103). When n (θ,ρ) are extracted, (θ1,ρ1)~(θ n ρ n Next, the line extraction block 122 extracts the extracted (θ1,ρ1)~(θ n ρ n Substitute ) into equation (7) and obtain n lines L1~L n Generate (step S104).

[0102] Next, the wake detection block 123 again checks all the search points using the generated n lines L1~L n The distance to the target is calculated, and those below a predetermined distance are designated as wake wave data (step S105). Next, the wake wave detection block 123 takes the above wake wave data and converts it into 3D data including the z-value (step S106). Then, the wake wave detection block 123 uses the extracted wake wave data to recalculate the equations of the n lines using the least squares method or principal component analysis (step S107). Finally, the process returns to the main routine shown in Figure 20.

[0103] Next, the wake data removal block 125 removes the wake data from the search data to obtain search data for obstacle detection (step S15).

[0104] Next, the obstacle / wake detection unit 16 performs obstacle detection processing (step S16). Figure 23 is a flowchart of the obstacle detection processing. First, the Euclidean clustering block 126 calculates the distance between all search data points and all other search data points for all search data points (step S111). If there are n search data points, then n(n-1) distances between points will be calculated. Next, the Euclidean clustering block 126 selects the first target data point (step S112), calculates the distance r1 from the ship to the target data point, and calculates the grouping threshold T1 using a predetermined coefficient a (step S113). For example, T1 = a·r1. That is, the grouping threshold T1 will be a different value for each target data point.

[0105] Next, the Euclidean clustering block 126 groups data where the distance between the target data points is less than the grouping threshold T1 (step S114). Then, the Euclidean clustering block 126 determines whether all the search data has been included (step S115). If not all the search data has been included (step S115: No), the Euclidean clustering block 126 selects the next target data (step S116) and returns to step S113.

[0106] On the other hand, if all the search data is included (step S115: Yes), the Euclidean clustering block 126 determines the centroid position for each extracted group and calculates the distance r2 to that centroid position. Then, it sets the score threshold T2 using a predetermined coefficient b (step S117). For example, T2 = b / r2. That is, the score threshold T2 will be a different value for each group.

[0107] Next, the Euclidean clustering block 126 determines that a cluster is one that contains a number of data points equal to or greater than the score threshold T2 for each group, and the obstacle detection block 127 identifies that cluster as an obstacle (step S118). Then, the process returns to the main routine shown in Figure 20.

[0108] Next, the obstacle data removal block 129 removes data identified as obstacles from the search data to obtain data for water surface position estimation (step S17).

[0109] Next, the obstacle / wake detection unit 16 performs water surface position estimation processing (step S18). Figure 24 is a flowchart of the water surface position estimation processing. First, the mean / variance calculation block 130 determines that data that is far from the shore, close to the ship's position, and near the water surface position is water surface reflection data (step S121). Next, the mean / variance calculation block 130 obtains water surface reflection data from multiple scan frames, and once a predetermined number of data points is reached, it calculates the average value and variance of those data points in the z direction (step S122).

[0110] Next, the mean / variance calculation block 130 determines whether the variance value is less than a predetermined value (step S123). If the variance value is not less than the predetermined value (step S123: No), the process proceeds to step S125. On the other hand, if the variance value is less than the predetermined value (step S123: Yes), the time filter block 131 filters the calculated z-values ​​against past estimated water surface positions and updates the water surface position (step S124). Next, the water surface position estimation block 132 outputs the calculated water surface position and variance value (step S125). Then, the process returns to the main routine shown in Figure 20.

[0111] Next, the obstacle / wake detection unit 16 performs wake information calculation processing (step S19). Figure 25(A) is a flowchart of the wake information calculation processing. First, the wake information calculation block 124 calculates the shortest distance to the straight line detected by the wake detection block 123, using the ship's own position as a reference, and sets this as the distance to the wake. It also finds the position at that distance and sets this as the position of the wake. Furthermore, it calculates the slope from the coefficient of that straight line and sets this as the angle of the wake (step S131).

[0112] As shown in Figure 25(B), the shortest distance from the ship's own position to the straight line is the distance to the foot of the perpendicular drawn to the straight line. However, since the straight line detected as a wake is a line segment, as shown in Figure 25(C), the endpoint of the data detected as a wake may also be the shortest distance. Therefore, the wake information calculation block 124 checks whether the coordinates to the foot of the perpendicular lie within the line segment, and if not, it uses the distance to the endpoint of the line segment as the shortest distance.

[0113] Next, the wake information calculation block 124 calculates the average of the z-coordinate values ​​using only the points whose z-values ​​are higher than the estimated water surface position, and uses the estimated water surface position to calculate the height of the wake from the water surface (step S132). Alternatively, the maximum value of the z-coordinate values ​​may be used as the height of the wake. Then, the process returns to the main routine shown in Figure 20.

[0114] Next, the obstacle / wake detection unit 16 performs obstacle information calculation processing (step S20). Figure 26 is a flowchart of the obstacle information calculation processing. First, the obstacle information calculation block 128 uses the ship's own position as a reference and extracts the cluster data with the shortest distance from the detected obstacles, and sets that as the location of the obstacle. It also calculates the distance to that data and sets that as the distance to the obstacle. Furthermore, it calculates the angle of the obstacle from the coordinates of that data (step S141).

[0115] Next, the obstacle information calculation block 128 extracts the two points furthest apart in the x and y two-dimensional plane from the cluster data, and uses that distance as the horizontal size of the obstacle. It also subtracts the water surface position estimated from the z coordinate of the highest point in the cluster data to obtain the height of the obstacle from the water surface (step S142). Then, the process returns to the main routine shown in Figure 20.

[0116] Next, the obstacle / wake detection unit 16 determines whether it has detected similar wakes in multiple frames (step S21). Although they will not be exactly the same if the ship itself or the wake is moving, if the values ​​calculated in step S19 are only slightly different, it is determined that they are similar wakes. If no similar wakes are detected (step S21: No), the process proceeds to step S24. On the other hand, if similar wakes are detected (step S21: Yes), the wake information calculation block 124 determines that the data is a wake and outputs the wake information to the hull system (step S22).

[0117] Next, the display control unit 133 performs screen display processing of the undertow information (step S23). Figure 27 is a flowchart of the screen display processing of the undertow information. The display control unit 133 performs this processing each time undertow information is acquired.

[0118] First, the display control unit 133 obtains wake information from the wake information calculation block 124 and obtains the position p, distance d, angle θ, and height h. The display control unit 133 also calculates the difference between the position of the wake obtained previously and the current position, and calculates the relative velocity v and its vector (step S151).

[0119] Next, the display control unit 133 determines whether the velocity vector is in the direction of the ship (step S152). If the velocity vector is not in the direction of the ship (step S152: No), the display control unit 133 sets all font sizes and the line widths of lines and borders to the normal size s min Then, the positional relationship information is displayed on the display screen of the display device 17 (step S156). The screen display processing of the undertow information is then completed.

[0120] On the other hand, if the velocity vector is in the direction of the ship (step S152: Yes), the display control unit 133 increases the emphasis parameters s1 to s4 and S for each value of the positional relationship information. Specifically, s1 is increased as the relative velocity v increases, s2 is increased as the distance d decreases, s3 is increased as the height h increases, and s4 is increased as the angle θ' (=|θ-45°|) increases. Also, S = s1 + s2 + s3 + s4 (step S153).

[0121] Figure 28(A) illustrates the emphasis parameter s. The emphasis parameter s is calculated according to the values ​​of the variables (v, d, h, θ') on the horizontal axis, within the range of a predetermined lower limit (normal size) and upper limit (maximum size). Note that a and b are set for the variables v, d, h, and θ', respectively.

[0122] Next, the display control unit 133 displays the numerical values ​​of the variables v, d, h, and θ' on the display screen, using the values ​​of the emphasis parameters s1 to s4 as the font size. The display control unit 133 also draws an arrow 84 representing the relative velocity v on the screen, using the emphasis parameter s1 as the line width. At this time, the length of the arrow 84 is set according to the value of the relative velocity v. The display control unit 133 also draws a straight line 85 from the ship to the position of the wake, using the emphasis parameter s2 as the line width. The display control unit 133 also draws a frame 86 surrounding the wake data, using the emphasis parameter S as the line width (step S154).

[0123] Next, the display control unit 133 further flashes the font, lines, and borders if the values ​​of the emphasis parameters s1 to s4 each exceed predetermined thresholds (step S155). Then, the screen display processing of the undertow information is completed, and the program returns to the main routine in Figure 20.

[0124] Next, the obstacle / wake detection unit 16 determines whether it has detected similar obstacles in multiple frames (step S24). Although they will not be exactly the same if the ship itself or the obstacle is moving, if the values ​​calculated in step S20 are only slightly different, it will be determined that they are similar obstacles. If no similar obstacles are detected (step S24: No), the process ends. On the other hand, if similar obstacles are detected (step S24: Yes), the obstacle information calculation block 128 determines that the data is an obstacle and outputs the obstacle information to the hull system (step S25).

[0125] Next, the display control unit 133 performs the screen display processing of obstacle information (step S26). Figure 29 is a flowchart of the screen display processing of obstacle information. The display control unit 133 performs this processing each time obstacle information is acquired.

[0126] First, the display control unit 133 obtains obstacle information from the obstacle information calculation block 128 and acquires the position p, distance d, size w, and height h. The display control unit 133 also calculates the difference between the previously acquired position of the obstacle and the current position, and calculates the relative velocity v and its vector (step S161).

[0127] Next, the display control unit 133 determines whether the velocity vector is in the direction of the ship (step S162). If the velocity vector is not in the direction of the ship (step S162: No), the display control unit 133 sets all font sizes and the line widths of lines and borders to the normal size s min Then, the positional relationship information is displayed on the screen (step S166). The screen display process for obstacle information is then completed.

[0128] On the other hand, if the velocity vector is in the direction of the ship (step S162: Yes), the display control unit 133 increases the emphasis parameters s1 to s4 and S for each value of the positional relationship information. Specifically, the larger the relative velocity v, the larger s1 is increased; the smaller the distance d, the larger s2 is increased; the larger the height h, the larger s3 is increased; and the larger the size w, the larger s4 is increased. Also, S = s1 + s2 + s3 + s4 (step S163).

[0129] Figure 28(B) illustrates the emphasis parameter s. The emphasis parameter s is calculated according to the values ​​of the variables (v, d, h, w) on the horizontal axis, within the range of a predetermined lower limit (normal size) and upper limit (maximum size). Note that a and b are set for the variables v, d, h, and w, respectively.

[0130] Next, the display control unit 133 displays the numerical values ​​of variables v, d, h, and w on the display screen, using the values ​​of emphasis parameters s1 to s4 as the font size. The display control unit 133 also draws an arrow 84 representing the relative velocity v on the screen, using emphasis parameter s1 as the line width. At this time, the length of the arrow 84 is determined according to the value of the relative velocity v. The display control unit 133 also draws a straight line 85 from the ship to the position of the obstacle, using emphasis parameter s2 as the line width. The display control unit 133 also draws a frame 82 surrounding the obstacle data, using emphasis parameter S as the line width (step S164).

[0131] Next, the display control unit 133 further flashes the font, lines, and borders if the values ​​of the emphasis parameters s1 to s4 each exceed predetermined thresholds (step S165). Then, the screen display processing of the obstacle information is completed, and the obstacle / wake detection processing in Figure 20 is also completed.

[0132] (4-6) Modifications (Variation 1) The above water surface position estimation uses the variance of water surface reflection data. However, as illustrated in Figure 30, if the hull is statically tilted in the roll direction due to cargo imbalance or other reasons, the variance of the water surface reflection data will also increase. When estimating the water surface position in such a situation, the water surface position estimation block 132 may process the water surface reflection data on the starboard side and the water surface reflection data on the port side separately, and determine the water surface position on the starboard side and the water surface position on the port side separately. Alternatively, the water surface position estimation block 132 can estimate the water surface position without separating the starboard and port sides by applying a coordinate transformation to the water surface reflection data that rotates the roll angle so that the difference between the average values ​​of the starboard side and the average values ​​of the port side of the water surface reflection data becomes small.

[0133] (Modification 2) In the above embodiment, the straight line extraction block 122 extracts straight lines of the undertow wave according to the following steps 1 to 3. (Step 1) Calculate the approximate straight line using the Hough transform. (Step 2) Extract data where the distance to the approximate straight line is within a predetermined threshold (straight line distance threshold). (Step 3) Perform principal component analysis using the extracted data to recalculate the straight line, which will be used as the straight line for the undertow wave.

[0134] Alternatively, step 4 may be added, and steps 2 and 3 may be repeatedly executed based on the result of step 4. (Step 4) If the extracted data changes and the equation of the line changes, return to Step 2. Once the equation of the line no longer changes, consider it the line representing the wake wave.

[0135] The graph on the left in Figure 31 shows an example where a straight line is obtained without performing step 4 above, while the graph on the right shows an example where the straight line generation is converged after performing up to step 4. By adding step 4, the extraction of underswing data is eliminated, and as a result, the accuracy of the straight line can be improved.

[0136] The present invention has been described above with reference to the examples, but the present invention is not limited to the above examples. Various modifications to the structure and details of the present invention can be made that can be understood by a person skilled in the art within the scope of the present invention. That is, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that a person skilled in the art could make in accordance with the technical idea. Furthermore, each disclosure of the above-mentioned patent documents, etc., that has been cited is incorporated herein by reference. [Explanation of Symbols]

[0137] 1. Information Processing Device 2 Sensor Groups 3 Riders 4 Speed ​​Sensor 5 GPS receivers 6 IMU 10 Map Database 13 Controllers 15 Self-position estimation part 16 Obstacle / Surface Wave Detection Unit 121 Search range setting block 122 Line Extraction Block 123 Undertow detection block 124 Undertow Information Calculation Block 125 Undertow data removal block 126 Euclidean clustering blocks 127 Obstacle detection block 128 Obstacle Information Calculation Block 129 Obstacle data removal block 130 Mean and Variance Calculation Block 131-hour filter block 132 Water surface position estimation block 133 Display Control Unit

Claims

[Claim 1] An object detection means for detecting an object based on point cloud data generated by a measuring device installed on a ship, A position relationship acquisition means for acquiring the relative positional relationship between the object and the vessel, A display control means that displays the positional relationship information on a display means in a display manner corresponding to the positional relationship, An information processing device equipped with the following features.

Images