Information processing device, control method, program, and storage medium

Abstract

This invention provides an information processing device that can calculate parameters used in docking assistance with high accuracy. [Solution] The information processing device comprises an acquisition means, a nearest neighbor extraction means, and a reference point setting means. The acquisition means acquires measurement data, which is a set of data identified by a pair of indices representing lateral position and vertical position generated by a measuring device installed on the ship. The nearest neighbor extraction means extracts data representing the nearest neighbor point closest to the ship's reference point from the data representing the measurement point at the berthing location, for each index representing the lateral position. The reference point setting means sets the reference point at the next processing time based on the nearest neighbor point.

Description

[Technical Field]

[0001] This disclosure relates to the procedures for when a vessel is docked. [Background technology]

[0002] Technologies for assisting with ship docking (berthing) have been known for some time. For example, Patent Document 1 describes a method for controlling the attitude of a ship in an automatic docking device that performs automatic ship docking, such that light emitted from a lidar is reflected by objects around the docking position and received by the lidar. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-59403 [Overview of the project] [Problems that the invention aims to solve]

[0004] For ships, safe and smooth docking at a designated berthing location is crucial, and the realization of docking support systems for ship handling assistance and automated navigation is particularly desired. Therefore, docking support systems require high-precision calculation of parameters such as distance, speed, and angle relative to the intended docking location.

[0005] This disclosure is made to solve the above-mentioned problems, and its main purpose is to provide an information processing device that can calculate parameters used in docking assistance with high accuracy. [Means for solving the problem]

[0006] The invention described in the claim is an information processing device comprising: acquisition means for acquiring measurement data which is a set of data identified by a pair of indexes representing lateral position and indexes representing vertical position generated by a measuring device installed on a ship; nearest neighbor extraction means for extracting data representing the nearest neighbor point closest to the reference point of the ship from the data representing the measurement point at the berthing location for each index representing the lateral position; and reference point setting means for setting the reference point at the next processing time based on the nearest neighbor point.

[0007] Furthermore, the invention described in the claim is a control method executed by a computer, which acquires measurement data, which is a set of data identified by a pair of indices representing lateral position and longitudinal position generated by a measuring device installed on a ship, extracts data representing the nearest neighbor point closest to the ship's reference point from the data representing the measurement point at the berthing location for each index representing the lateral position, and sets the reference point at the next processing time based on the nearest neighbor point.

[0008] Furthermore, the invention described in the claim is a program that acquires measurement data which is a set of data identified by a pair of indices representing lateral position and longitudinal position generated by a measuring device installed on a ship, extracts data representing the nearest neighbor point closest to the ship's reference point from the data representing the measurement point at the berthing location for each index representing the lateral position, and causes a computer to execute a process to set the reference point at the next processing time based on the nearest neighbor point. [Brief explanation of the drawing]

[0009] [Figure 1A] Block diagram of the flight support system. [Figure 1B] A top view illustrating the field of view of the ship and lidar included in the navigation support system. [Figure 1C] A diagram showing the field of view of a ship and a rider from the rear. [Figure 2]A block diagram showing an example of the hardware configuration of an information processing device. [Figure 3] Functional block diagram related to berthing support processing. [Figure 4A] A diagram showing a rider capturing the quay as it docks. [Figure 4B] A perspective view of a quay wall, clearly showing the straight line of the side where the quay docks. [Figure 5A] A diagram showing an example of a data structure for confidence level information. [Figure 5B] A diagram showing an example of the indicators and confidence levels included in confidence information. [Figure 5C] An overhead view of the target vessel and berthing location, clearly indicating the indicators shown in Figure 5B. [Figure 6A] A diagram showing an example of a hull coordinate system based on the hull of the target vessel. [Figure 6B] A perspective view of a structure with its normal vectors clearly indicated. [Figure 7A] This figure shows an example of the nearest neighbor points obtained as edge point cloud SPs for docking locations. [Figure 7B] A diagram that explicitly shows the unit vector uL used for calculating distance dq. [Figure 8] A diagram showing an overview of the process for setting the reference point on the ship side in the first embodiment. [Figure 9A] This figure illustrates an example of incorrectly selecting an object on the water surface as the nearest neighbor. [Figure 9B] This diagram illustrates an example of a case where an object with height relative to the upper surface of a docking location is mistakenly selected as the nearest neighbor. [Figure 9C] This figure illustrates an example of how to extract the appropriate nearest neighbor when the distance from the vessel to the berthing location is large. [Figure 9D] This figure illustrates an example of how to extract the appropriate nearest neighbor when the distance from the ship to the berthing location is very short. [Figure 9E] This figure illustrates an example of incorrectly selecting an object on the water surface as the nearest neighbor. [Figure 9F] This diagram shows an example of extracting the nearest neighbor point by setting the location where the lid is installed as the reference point R. [Figure 10A] A flowchart illustrating the overview of the docking support process in the first embodiment. [Figure 10B] A flowchart illustrating an example of the process involved in setting a reference point. [Figure 11A] A diagram illustrating an example of a situation in which the second embodiment can be applied. [Figure 11B] A top view illustrating the field of view of the ship and lidar included in the navigation support system. [Figure 12] A diagram showing an overview of the process for extracting the most recent MPA (Maximum Pacing Marks). [Figure 13A] Figure 12 shows an example of the nearest neighbor point (MPA) extracted by the process shown. [Figure 13B] A top view showing the quaying side straight line L and the extraction window EWA. [Figure 14] A diagram showing an overview of the process for extracting the most recent MPB (Multiple Key Bounds). [Figure 15A] Figure 14 shows an example of the nearest neighbor MPB extracted by the process described. [Figure 15B] A top view showing the extraction window EWB. [Figure 16A] A top view showing the origin O, the docking side line L, the nearest neighbor point KP, and the distance df. [Figure 16B] A recent diagram explicitly showing the unit vector u used in the search for emphasis point KP. [Figure 17] A flowchart illustrating the overview of the docking support process in the second embodiment. [Figure 18] A top view showing the marker line Lm. [Figure 19A] This diagram illustrates an example where multiple fenders of different sizes are installed outside the field of view of the radar. [Figure 19B] A top view showing the difference value Δdrf. [Figure 19C] This figure shows the temporal change in the difference value Δdrf and the difference value Δdrf_max. [Figure 19D] A flowchart illustrating an example of the process for calculating the shortest distance from a ship to its fenders. [Figure 19E] A top view showing the distance df_min. [Modes for carrying out the invention]

[0010] In one preferred embodiment of the present invention, the information processing device includes: acquisition means for acquiring measurement data which is a set of data identified by a pair of indices representing lateral position and longitudinal position generated by a measuring device installed on a ship; nearest neighbor extraction means for extracting data representing the nearest neighbor point closest to the reference point of the ship from the data representing the measurement point at the berthing location, for each index representing the lateral position; and reference point setting means for setting the reference point at the next processing time based on the nearest neighbor point.

[0011] The above-described information processing device comprises an acquisition means, a nearest neighbor extraction means, and a reference point setting means. The acquisition means acquires measurement data, which is a set of data identified by a pair of indices representing lateral position and vertical position generated by a measuring device installed on the vessel. The nearest neighbor extraction means extracts data representing the nearest neighbor point closest to the reference point of the vessel from the data representing the measurement point at the berthing location, for each index representing the lateral position. The reference point setting means sets the reference point at the next processing time based on the nearest neighbor point. This makes it possible to calculate parameters used in berthing assistance with high accuracy.

[0012] In one embodiment of the information processing device described above, the reference point setting means sets the reference point based on the reliability of the nearest neighbor point.

[0013] In one embodiment of the information processing device described above, the reference point setting means sets the position where the measuring device is installed on the ship as the reference point when the reliability of the nearest neighbor point is less than or equal to a predetermined value.

[0014] In one embodiment of the information processing device described above, the reference point setting means sets the reference point at a position higher or lower than the position where the measuring device is installed, depending on the distance between the vessel and the berthing location, when the reliability of the nearest point is higher than a predetermined value.

[0015] In one embodiment of the above-described information processing device, the reference point setting means sets, as reference points when the reliability of the nearest neighbor point is higher than a predetermined value, a first reference point used for extracting the data representing the nearest neighbor point, and a second reference point used for extracting the data representing the measurement point of the fender installed at the berthing location.

[0016] In one embodiment of the information processing device described above, the reference point setting means sets the first reference point to have a large elevation angle when viewed from the quay, and sets the second reference point to have a small elevation angle when viewed from the quay.

[0017] In one embodiment of the above-described information processing device, the device further includes a means for calculating the distance to the opposite shore, which corresponds to the distance from the vessel to the quay of the docking location, based on the search result for the nearest neighbor using the first reference point.

[0018] In one embodiment of the above-described information processing device, the device further includes a fender distance calculation means that calculates a fender distance corresponding to the distance from the ship to the fender, based on the search result for the nearest neighbor point using the second reference point.

[0019] In one embodiment of the above-described information processing device, the fender distance calculation unit calculates the overhang length, which is the maximum value of the difference between the distance to the opposite shore and the fender distance calculated within a predetermined time, and calculates the fender distance by subtracting the overhang length from the distance to the opposite shore.

[0020] In another embodiment of the present invention, a control method executed by a computer acquires measurement data, which is a set of data identified by a pair of indices representing lateral position and longitudinal position generated by a measuring device installed on the ship. From the data representing the measurement point at the berthing location, data representing the nearest neighbor point closest to the ship's reference point is extracted for each index representing the lateral position, and the reference point at the next processing time is set based on the nearest neighbor point. This makes it possible to calculate parameters used in berthing assistance with high accuracy.

[0021] In yet another embodiment of the present invention, the program acquires measurement data, which is a set of data identified by a pair of indices representing lateral position and longitudinal position generated by a measuring device installed on the ship. The program extracts data representing the nearest neighbor point closest to the ship's reference point from the data representing the measurement point at the berthing location, for each index representing the lateral position, and causes the computer to perform a process to set the reference point at the next processing time based on the nearest neighbor point. By executing this program on a computer, the above information processing device can be realized. This program can be stored and used on a storage medium. [Examples]

[0022] Preferred embodiments of the present invention will be described below with reference to the drawings.

[0023] <First Example> First, the first embodiment will be described below.

[0024] [Overview of the driver assistance system] Figures 1A to 1C show the schematic configuration of the navigation support system according to this embodiment. Specifically, Figure 1A shows a block diagram of the navigation support system, Figure 1B is a top view illustrating the field of view (also called the "measurement range" or "distance-measurable range") 90 of the vessel and the lidar 3 described later, which are included in the navigation support system, and Figure 1C is a rear view showing the field of view 90 of the vessel and the lidar 3. The navigation support system comprises an information processing device 1 that moves together with the vessel, which is a moving object, and a group of sensors 2 mounted on the vessel. Hereafter, the vessel on which the navigation support system is installed will also be called the "target vessel".

[0025] The information processing device 1 is electrically connected to the sensor group 2 and provides operational support for the target vessel based on the outputs of the various sensors included in the sensor group 2. Operational support includes berthing support such as automatic docking. Here, "berthing" includes not only docking the target vessel at a quay but also docking it at a structure such as a pier. Furthermore, hereafter, "berthing location" refers to the general term for structures such as quays and piers that are the target of berthing. The information processing device 1 may be a navigation device installed on the vessel or an electronic control device built into the vessel.

[0026] Sensor group 2 includes various external and internal sensors installed on the ship. In this embodiment, sensor group 2 includes, for example, a Lidar (Light Detection and Ranging, or Laser Illuminated Detection and Ranging) 3.

[0027] LIDA 3 is an external sensor that discretely measures the distance to an object in the outside world by emitting a pulsed laser within a predetermined angular range in the horizontal direction (see Figure 1B) and a predetermined angular range in the vertical direction (see Figure 1C), and generates three-dimensional point cloud data indicating the position of the object. In the examples in Figures 1B and 1C, LIDA 3 is provided on the ship, with one LIDA pointed towards the port side and another towards the starboard side. Note that the arrangement of LIDA 3 is not limited to the examples in Figures 1B and 1C. For example, the target ship may have multiple LIDA 3 (for example, LIDAs provided at the front and rear of the target ship) that measure in the same lateral direction so that multiple measurement data of the berthing location can be obtained simultaneously when docking. Also, the number of LIDA 3 installed on the target ship is not limited to two; it may be one or three or more.

[0028] LIDA 3 comprises an irradiation unit that irradiates laser light while changing the irradiation direction, a light receiving unit that receives reflected light (scattered light) of the irradiated laser light, and an output unit that outputs scan data based on the received signal output by the light receiving unit. The data measured for each direction of laser light irradiation (scanning position) is generated based on the irradiation direction corresponding to the laser light received by the light receiving unit and the response delay time of the laser light specified based on the received signal described above. Hereafter, a point measured by irradiation with laser light within the measurement range of LIDA 3, or the measurement data thereof, will also be referred to as the "measured point".

[0029] Here, point cloud data can be considered as an image (frame) where each measurement direction is represented by a pixel, and the measured distance and reflectance value for each measurement direction are represented by the pixel value. In this case, the direction of laser beam emission (i.e., measurement direction) differs in elevation and depression angles in the vertical arrangement of pixels, and the direction of laser beam emission differs in horizontal angles in the horizontal arrangement of pixels. Hereafter, when the point cloud data is considered as an image, the measured points corresponding to the rows of pixels (i.e., vertical columns) whose horizontal index positions coincide will also be called "vertical lines." Furthermore, when the point cloud data is considered as an image, the horizontal index will be called the "horizontal number," and the vertical index will be called the "vertical number."

[0030] Furthermore, LIDA 3 is not limited to the scanning type LIDA described above, but may also be a flash type LIDA that generates 3D data by diffusing laser light into the field of view of a 2D array sensor. LIDA 3 is an example of a "measuring device" in the present invention.

[0031] [Configuration of the information processing device] Figure 2 is a block diagram showing an example of the hardware configuration of an information processing device. The information processing device 1 mainly consists of an interface 11, a memory 12, and a controller 13. Each of these elements is interconnected via a bus line.

[0032] Interface 11 performs interface operations related to the exchange of data between the information processing device 1 and external devices. In this embodiment, interface 11 acquires output data from each sensor in the sensor group 2 and supplies it to the controller 13. Interface 11 also supplies signals related to the control of the target vessel, generated by the controller 13, to each component of the target vessel that controls the operation of the target vessel. For example, the target vessel includes a drive source such as an engine or electric motor, a propeller that generates thrust in the direction of travel based on the driving force of the drive source, a thruster that generates thrust in the lateral direction based on the driving force of the drive source, and a rudder, etc., which is a mechanism for freely determining the direction of travel of the vessel. During automatic operation such as automatic docking, interface 11 supplies control signals generated by the controller 13 to each of these components. If the target vessel is equipped with an electronic control device, interface 11 supplies control signals generated by the controller 13 to the electronic control device. Interface 11 may be a wireless interface such as a network adapter for wireless communication, or it may be a hardware interface for connecting to external devices by cables, etc. Furthermore, interface 11 may perform interface operations with various peripheral devices such as input devices, display devices, and sound output devices.

[0033] Memory 12 is composed of various volatile and non-volatile memories such as RAM (Random Access Memory), ROM (Read Only Memory), hard disk drive, and flash memory. Memory 12 stores programs for the controller 13 to execute predetermined processes. Note that the programs executed by the controller 13 may be stored in storage media other than memory 12.

[0034] Furthermore, memory 12 stores information necessary for the processing performed by the information processing device 1 in this embodiment. For example, memory 12 may store map data including information about the location of the docking place. In another example, memory 12 stores information about the downsampling size when downsampling is performed on the point cloud data obtained when the lidar 3 performs one cycle of scanning.

[0035] The controller 13 includes one or more processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a TPU (Tensor Processing Unit), and controls the entire information processing device 1. In this case, the controller 13 performs processing related to the operation support of the target vessel by executing programs stored in memory 12, etc.

[0036] Furthermore, the controller 13 functionally includes a docking location detection unit 15 and a docking parameter calculation unit 16. The docking location detection unit 15 performs processing related to the detection of docking locations based on the point cloud data output by the lidar 3. The docking parameter calculation unit 16 calculates the parameters necessary for docking at the docking location (also called "docking parameters"). Here, the docking parameters include the distance between the target vessel and the docking location (distance to the opposite shore), the approach angle of the target vessel to the docking location, and the speed at which the target vessel approaches the docking location (docking speed). The docking parameter calculation unit 16 also calculates information representing the reliability of docking at the docking location (also called "reliability information") based on the processing results of the docking location detection unit 15 and the docking parameters. The controller 13 functions as an "acquisition means," a "nearest neighbor extraction means," a "reference point setting means," and a computer that executes programs.

[0037] Furthermore, the processing performed by the controller 13 is not limited to being implemented by software through a program, but may also be implemented by a combination of hardware, firmware, and software. Additionally, the processing performed by the controller 13 may be implemented using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the program executed by the controller 13 in this embodiment may be implemented using this integrated circuit.

[0038] [Overview of docking support procedures] Next, an overview of the docking support process performed by the information processing device 1 will be described. Based on the point cloud data of the lidar 3 measured in the direction in which the docking location exists, the information processing device 1 generates a straight line along the side of the docking location (also called the "docking side line L"). In other words, the docking side line L is a straight line along the side of the quay wall of the docking location. Then, based on the docking side line L, the information processing device 1 calculates docking parameters such as the distance to the opposite shore.

[0039] Figure 3 is a functional block diagram of the docking location detection unit 15 and the docking parameter calculation unit 16 related to docking support processing. Functionally, the docking location detection unit 15 includes a normal vector calculation block 20, a field of view / detection surface identification block 21, a normal number identification block 22, a mean / variance calculation block 23, and a docking status determination block 24. Functionally, the docking parameter calculation unit 16 includes a nearest point search block 26, a straight line generation block 27, a distance calculation block 28, an entry angle calculation block 29, a docking speed calculation block 30, a reliability information generation block 40, a nearest neighbor extraction block 41, and a reference point setting block 42.

[0040] The normal vector calculation block 20 calculates the normal vector of the surface formed by the berthing location (also called the "berthing surface") based on the point cloud data generated by the lidar 3 in the direction in which the berthing location exists. In this case, the normal vector calculation block 20 calculates the normal vector based on, for example, the point cloud data generated by the lidar 3, which includes the berthing side of the target vessel in its measurement range. Information regarding the measurement range of the lidar 3 and the direction of the berthing location may be pre-registered in, for example, memory 12.

[0041] In this case, the normal vector calculation block 20 preferably performs downsampling of the point cloud data and removal of data obtained by the reflection of laser light from the water surface (also called "water surface reflection data").

[0042] In this case, first, the normal vector calculation block 20 removes data located below the water surface position from the point cloud data generated by the lidar 3, treating it as water surface reflection data (i.e., false detection data). The normal vector calculation block 20 estimates the water surface position based, for example, on the average value in the height direction of the point cloud data generated by the lidar 3 when there are no objects other than the water surface in the surrounding area. Then, the normal vector calculation block 20 performs downsampling on the point cloud data after the water surface reflection data has been removed, which is a process that integrates the measured points for each grid space of a predetermined size. Finally, the normal vector calculation block 20 calculates a normal vector for each measured point indicated by the downsampled point cloud data using multiple surrounding measured points. Note that downsampling may be performed before removing the data reflected from the water surface.

[0043] The field of view / detection surface identification block 21 identifies the surface of the docking location that is within the field of view angle of the Lidar 3 (also called the "inner surface of the field of view") and the surface of the docking location detected based on the normal vector calculated by the normal vector calculation block 20 (also called the "detection surface"). In this case, the field of view / detection surface identification block 21 determines whether the upper surface and / or the side surface of the docking location are included in the inner surface of the field of view and the detection surface.

[0044] The normal vector number identification block 22 extracts the vertical normal vector and the normal vector perpendicular to it (i.e., the horizontal direction) from the normal vector calculated by the normal vector calculation block 20, and calculates the number of vertical normal vectors and the number of horizontal normal vectors. Here, the normal vector number identification block 22 considers the vertical normal vector to represent the normal to the measured point on the upper surface of the docking location, and the horizontal normal vector to represent the normal to the measured point on the side of the docking location, and calculates the number of each as an indicator of the reliability of the docking location.

[0045] The mean and variance calculation block 23 extracts the vertical normal vector and the normal vector perpendicular to it (i.e., horizontal) from the normal vectors calculated by the normal vector calculation block 20, and calculates the mean and variance of the vertical normal vector and the mean and variance of the horizontal normal vector.

[0046] The docking status determination block 24 obtains the processing results of the field of view / detection surface identification block 21, the normal number identification block 22, and the mean / variance calculation block 23, which are identified or calculated based on the same point cloud data, as determination results representing the detection status of the docking location at the time the point cloud data was generated. The docking status determination block 24 then supplies the processing results of the field of view / detection surface identification block 21, the normal number identification block 22, and the mean / variance calculation block 23 as determination results of the detection status of the docking location to the docking parameter calculation unit 16.

[0047] The nearest neighbor search block 26 searches for the nearest neighbor point for each vertical line, based on the measured points that make up the point cloud data and the reference points set by the reference point setting block 42. For example, as shown in Figure 4A, when the lidar 3 is detecting a quay where the ship is docked, the nearest neighbor point, which is the point closest to the ship, will be the edge portion between the top and side surfaces of the quay.

[0048] The line generation block 27 generates a shoreline line L, which is a straight line along the side of the shoreing location, based on the nearest neighbor determined by the nearest neighbor search block 26. Through this process, the line generation block 27 can generate a shoreline line L, for example, as shown in Figure 4B. Figure 4B is a perspective view of a quay wall with the shoreline line clearly indicated.

[0049] The nearest neighbor point extraction block 41 extracts one nearest neighbor point from among the nearest neighbor points, based on the nearest neighbor points of each vertical line searched by the nearest neighbor point search block 26, the shoreline side line L generated by the line generation block 27, and the reference point set by the reference point setting block 42, which is the point that is closest to the reference point and has the shortest distance from the shoreline side line L.

[0050] The opposite shore distance calculation block 28 calculates the opposite shore distance, which corresponds to the shortest distance between the target vessel and the berthing location, based on the berthing side line L generated by the line generation block 27. Here, if there are multiple lidars 3 that can measure the berthing location, the opposite shore distance calculation block 28 generates a berthing side line L by combining the point cloud data of multiple lidars 3 and calculates the shortest distance to each lidar 3 as the opposite shore distance. Alternatively, the opposite shore distance calculation block 28 may generate a berthing side line L for each point cloud data of lidar 3 and calculate the opposite shore distance as the shortest distance between each berthing side line L and each lidar 3. Furthermore, the opposite shore distance calculation block 28 may calculate the opposite shore distance as the shortest distance from a reference point such as the vessel's center position to the berthing side line L. Note that instead of considering the shortest distance for each lidar 3 as the opposite shore distance, the opposite shore distance calculation block 28 may define the opposite shore distance as the shorter of the shortest distances for each lidar 3, or define the opposite shore distance as the average of these shortest distances. Furthermore, if the opposite shore distance calculation block 28 can calculate the distance to a fender installed on the quay, for example, the distance to the fender may be set as the opposite shore distance.

[0051] The approach angle calculation block 29 calculates the approach angle of the target vessel to the berthing location based on the berthing side line L generated by the line generation block 27. Specifically, the approach angle calculation block 29 calculates the approach angle using the function "atan2", which finds the arctangent from two arguments that define the tangent. More specifically, the approach angle calculation block 29 calculates the approach angle from the direction vector of the berthing side line L by calculating the function atan2.

[0052] The berthing speed calculation block 30 calculates the berthing speed, which is the speed at which the target vessel approaches the berthing location, based on the distance to the opposite shore calculated by the distance to the opposite shore calculation block 28. For example, the berthing speed calculation block 30 calculates the berthing speed as the change in the distance to the opposite shore (shortest distance) over time.

[0053] The reliability information generation block 40 generates reliability information based on the processing results of the docking status determination block 24, the nearest point search block 26, the opposite bank distance calculation block 28, and the approach angle calculation block 29.

[0054] Here, we will explain the first specific example of generating confidence information. The confidence information generation block 40 generates flags for each element such as the field of view at the time of detection of the docking location, the surface detection of the docking location, the number of normal vectors, and the variance, and generates confidence information from the vector of the generated flags. Hereafter, a flag of "1" indicates that the confidence level of the corresponding element is higher than a predetermined value, and a flag of "0" indicates that the confidence level of the corresponding element is less than or equal to the predetermined value.

[0055] Figure 5A shows an example of the data structure of confidence information generated by the confidence information generation block 40. As shown in Figure 5A, the confidence information has the following items: "Top view", "Side view", "Linear view", "Distance", and "Angle". The item "Top view" has the sub-items "Field of view angle", "Detection", "Normal number", and "Variance", and the item "Side view" has the sub-items "Field of view angle", "Detection", "Normal number", and "Variance". The item "Nearest neighbor" has the sub-item "Variance", the item "Distance" has the sub-items "Amount of change" and "Rate of change", and the item "Angle" has the sub-item "Amount of change".

[0056] Here, the reliability information generation block 40 registers a flag in the "Field of View Angle" sub-item of the "Top Surface" item, which is set to "1" if the top surface of the docking location is within the field of view, and to "0" if the top surface is outside the field of view. The reliability information generation block 40 also registers a flag in the "Detection" sub-item of the "Top Surface" item, which is set to "1" if the top surface of the docking location is a detection surface, and to "0" if the top surface is not a detection surface. The reliability information generation block 40 also registers a flag in the "Number of Normals" sub-item of the "Top Surface" item, which is set to "1" if the number of normal vectors to the top surface of the docking location is equal to or greater than a predetermined threshold (e.g., 10), and to "0" if the number is less than the threshold. Furthermore, the reliability information generation block 40 registers a flag in the "Variance" sub-item of the "Upper Surface" item, where "1" is set if the variance of the x, y, and z components of the normal vector to the upper surface of the docking location is all less than a predetermined threshold (e.g., 1.0), and "0" is set if any of the variances is equal to or greater than the threshold. The reliability information generation block 40 also registers flags in each sub-item of the "Side" item, which are determined by the same rules as for each sub-item of the "Upper Surface" item.

[0057] Furthermore, the confidence information generation block 40 registers a flag in the "Variance" sub-item of the "Nearest Neighbor" item, which is set to "1" if the variance of the nearest neighbors searched for each vertical line by the nearest neighbor search block 26 is less than a predetermined threshold (e.g., 0.3), and to "0" if the variance is equal to or greater than the threshold.

[0058] Furthermore, the reliability information generation block 40 registers a flag in the "Change Amount" sub-item of the "Distance" item, which is set to "1" if the change amount of the distance to the opposite bank calculated by the opposite bank distance calculation block 28 from one time point in the previous period is less than a predetermined threshold (e.g., 1.0 m), and to "0" if the change amount is equal to or greater than the threshold. Furthermore, the reliability information generation block 40 registers a flag in the "Rate of Change" sub-item of the "Distance" item, which is set to "1" if the rate of change of the distance to the opposite bank calculated by the opposite bank distance calculation block 28 from one time point in the previous period is less than a predetermined threshold (e.g., ±10%), and to "0" if the rate of change is equal to or greater than the threshold. Furthermore, the reliability information generation block 40 registers a flag in the "Change Amount" sub-item of the "Angle" item, which is set to "1" if the change amount of the approach angle calculated by the approach angle calculation block 29 from one time point in the previous period is less than a predetermined threshold (e.g., 1.0 degrees), and to "0" if the change amount is equal to or greater than the threshold.

[0059] The thresholds mentioned above are set to conforming values ​​pre-stored in, for example, memory 12. Furthermore, confidence information may be generated for each Writer 3.

[0060] Next, a second specific example related to the generation of confidence information will be explained. Figure 5B is a diagram showing an example of the indicators and confidence levels included in the confidence information. Figure 5C is an overhead view of the target vessel and berthing location with the indicators shown in Figure 5B clearly indicated. Here, markers M0 and M1, which are reference points for berthing, are provided on the quay where the berthing takes place. The information processing device 1 detects these markers M0 and M1 based on the point cloud data of the lidar 3 and performs various processes such as generating the berthing side line L using the measured points (neighborhood point set) near the edge of the berthing location that exist between markers M0 and M1. Hereafter, the measured points near the edge of the berthing location that exist between markers M0 and M1 will also be called "target points". If markers M0 and M1 are not detected, for example, all of the neighboring point set will be considered as target points.

[0061] The index "c3" is an index based on the score of the target point, and here it is expressed as a linear function with the variable x as the score of the target point as an example. The index "c2" is an index based on the standard deviation of the target point, and here it is expressed as a linear function with the variable x as the standard deviation of the target point as an example. The index "c1" is an index indicating whether both the front and rear shore walls can be measured by two lidars 3. Here, as an example, when both can be measured, it is set to "1.0", and when only one can be measured, it is set to "0.0". The index "c0" is an index based on the interval between both ends of the target point (both ends in the direction along the shore contact side straight line L), and it is expressed as a linear function with the variable x as the interval between the above-mentioned both ends. Also, the indices c0 to c3 are calculated so as to be restricted to the range of 0 to 1.

[0062] The calculation reliability "c" is the reliability based on the above-mentioned indices c0 to c3. Here, it is the weighted average value of the indices c0 to c3 using the weight coefficients "w0" to "w3" according to the importance of the indices c0 to c3. Also, an example of the set values of the weight coefficients w0 to w3 is shown. Since each of the indices c0 to c3 is in the range of 0 to 1, the calculation reliability c, which is their weighted average value, is also calculated as a numerical value in the range of 0 to 1.

[0063] The overall reliability "r" is based on the side detection reliability "q s ", which is the reliability regarding the detection of the side surface of the shore wall, the top surface detection reliability "q u ", which is the reliability regarding the detection of the top surface of the shore wall, the marker detection reliability "m0", which is the reliability regarding the detection of the marker M0, the marker detection reliability "m1", which is the reliability regarding the detection of the marker M1, and the calculation reliability c. Here, for each reliability q s 、q u 、m0, m1, c, the weighted average value of the reliability q qs 、q qu 、m0, m1, c is obtained using the weight coefficients "w m0 ", "w m1 ", "w c ", "w s ", "w u " according to the importance. Also, the weight coefficients w qs 、wqu , w m0 , w m1 , w c An example of the setting value is shown in the diagram. Note that the information processing device 1, for example, the side detection confidence level q s This is calculated based on the "Side" item of the reliability information shown in Figure 5A, and the top surface detection reliability q u The confidence level may be calculated based on the "top view" item in the confidence level information of the same figure. Alternatively, the information processing device 1 may calculate the marker detection confidence level m0 and marker detection confidence level m1 based, for example, on the number of points measured in the point cloud data from the respective lidar 3 for marker M0 and marker M1. Then, the side detection confidence level q s Top surface detection confidence q u The marker detection confidence levels m0 and m1 are both calculated to be in the range of 0 to 1. Furthermore, if both the forward and aft quays are detected, the side detection confidence level q of the forward quay is calculated. s0 , the confidence level of detection on the side of the rear quay q s1 , detection confidence q on the upper surface of the forward quay u0 , detection confidence q on the upper surface of the rear quay u1 It is calculated as follows: Side detection confidence q s Top surface detection confidence q u Since the marker detection confidence m0, marker detection confidence m1, and calculation confidence c are all in the range of 0 to 1, the overall confidence score r, which is their weighted average, is also calculated as a value in the range of 0 to 1. Therefore, the closer the overall confidence score r is to 1, the higher the reliability of the calculated docking parameters, and the closer the overall confidence score r is to 0, the lower the reliability of the calculated docking parameters.

[0064] In other words, the confidence information generated by the second specific example includes each indicator and confidence level calculated using the method described above.

[0065] The reference point setting block 42 performs a process to set a reference point corresponding to the ship's reference position used in the processing of the nearest neighbor search block 26 and the nearest neighbor extraction block 41, based on the confidence information generated by the confidence information generation block 40 and a nearest neighbor extracted by the nearest neighbor extraction block 41.

[0066] Next, specific examples of the processing of the normal vector calculation block 20, the normal number determination block 22, and the mean / variance calculation block 23 will be explained with reference to Figures 6A and 6B.

[0067] Figure 6A shows an example of a hull coordinate system based on the hull of the target vessel. As shown in Figure 6A, the forward direction of the target vessel is the "x" coordinate, the side direction of the target vessel is the "y" coordinate, and the height direction of the target vessel is the "z" coordinate. The measurement data from the coordinate system based on LIDA 3, measured by LIDA 3, is converted to the hull coordinate system shown in Figure 6A. The process of converting point cloud data from a coordinate system based on a LIDA installed on a moving object to the coordinate system of the moving object is disclosed, for example, in International Publication WO2019 / 188745.

[0068] Figure 6B is a perspective view of the quay where the vessel docks, clearly showing the measurement points (represented by the LIDA 3) and the normal vectors calculated based on those measurement points. In Figure 6B, the measurement points are indicated by circles, and the normal vectors are indicated by arrows. This example shows that both the top and side surfaces of the quay were measured by the LIDA 3.

[0069] As shown in Figure 6B, the normal vector calculation block 20 calculates normal vectors for the measurement points on the side and top surfaces of the quay wall. Since normal vectors are vectors perpendicular to the target plane or curved surface, they are calculated using multiple measurement points that can form a surface. Therefore, a grid with predetermined lengths for length and width, or a circle with a predetermined radius, is set up, and the calculation is performed using the measurement points located within it. In this case, the normal vector calculation block 20 may calculate a normal vector for each measurement point, or it may calculate normal vectors at predetermined intervals. The normal number identification block 22 then determines that normal vectors whose z component is greater than a predetermined threshold are normal vectors oriented in the vertical direction. It is assumed that the normal vectors are unit vectors. Furthermore, normal vectors whose z component is less than a predetermined threshold are determined to be normal vectors oriented in the horizontal direction. The normal number identification block 22 then identifies the number of vertical normal vectors (5 in this case) and the number of horizontal normal vectors (4 in this case). Furthermore, the mean / variance calculation block 23 calculates the mean and variance of the vertical normal vector and the mean and variance of the horizontal normal vector. Note that for edge areas, the measurement direction will be diagonal because the surrounding measurement points are on the top or side.

[0070] [How to set and use reference points] Next, the method for setting and using the reference point in this embodiment will be described. The docking parameter calculation unit 16 sets the position where the rider 3 is installed as the reference point R at processing time t. t The settings are configured as follows. The docking parameter calculation unit 16 also performs a process to search for the nearest neighbor point closest to the reference point R for each vertical line, based on each measured point that constitutes the point cloud data generated by the lidar 3 at processing time t and the reference point R. Through this process, the docking parameter calculation unit 16 acquires point cloud data (hereinafter also referred to as the "docking location edge point cloud") at processing time t, as shown in Figure 4B, which includes multiple measured points (nearest neighbor points) that constitute the edge of the docking location (more specifically, the boundary of the top surface and side surface).

[0071] As shown in Figure 7A, the docking parameter calculation unit 16 generates a docking side line L at processing time t based on the docking location edge point cloud SP acquired at processing time t. For example, at processing time t, the docking parameter calculation unit 16 acquires a docking location edge point cloud SP including the nearest neighbor point as shown in Figure 7A by searching for the nearest neighbor point closest to the reference point for each vertical line, based on the measured point that constitutes the point cloud data obtained from the lidar 3 and a pre-set reference point. Then, the docking parameter calculation unit 16 generates a docking side line L by applying principal component analysis or the least squares method to the docking location edge point cloud SP acquired as described above.

[0072] The docking parameter calculation unit 16 extracts a nearest neighbor point HP that offers the shortest distance from the reference point R to the docking side line L in the horizontal plane (xy plane). Then, it calculates the distance in the horizontal plane (xy plane) from the reference point R to the nearest neighbor point HP.

[0073] Here, if we let G be the centroid of the point being measured, as indicated by the docking location edge point cloud SP, then as can be seen from Figure 7B, the vector HP-G and the unit vector u of the docking side line L are shown. L Since the two points are parallel, the relationship HP-G=au (where a is a constant) holds. Therefore, the docking parameter calculation unit 16 can calculate the coordinate position of the nearest point HP using the following formula (1).

number

[0074] Furthermore, the docking parameter calculation unit 16 uses the following formula (2) to calculate the distance d on the horizontal plane (xy plane) between the reference point R and the nearest neighbor point HP. q Calculate (see Figure 7B). Note that in formula (2) below, HP x represents the x-coordinate of the nearest point HP, HP y represents the y-coordinate of the nearest point HP, R x represents the x-coordinate of the reference point R, y This represents the y-coordinate of the reference point R. That is, the distance d qThis value is calculated as the distance between the vessel and the berthing location. The berthing parameter calculation unit 16 also calculates the z-coordinate value R of the reference point R. z The z-coordinate value of the nearest point HP is HP z Subtracting this value, the height h, which is the vertical (z-direction) distance between the reference point R and the nearest neighbor point HP, is calculated. Note that the docking parameter calculation unit 16 calculates the opposite bank distance d r This is calculated as the distance from the origin O, such as the center position of the ship, to the straight line L on the side of the shore. In this case, the coordinates of the origin O are used instead of the reference point R, and the calculation can be performed using the above formula (1) and the following formula (2). The shore parameter calculation unit 16 also calculates the distance d across the shore with the reference point R as the origin O. r When calculating the above distance d, q distance d across the river r That is also acceptable.

number

[0075] The docking parameter calculation unit 16 determines the level of confidence of each measured point (nearest neighbor) included in the docking location edge point cloud SP by referring to the flag registered in the sub-item "Variance" of the item "Nearest Neighbor" among the confidence information generated by the confidence information generation block 40.

[0076] The docking parameter calculation unit 16 determines that the confidence level of each measured point (nearest neighbor) included in the docking location edge point cloud SP is less than or equal to a predetermined value if the flag registered in the sub-item "Variance" of the "Nearest Neighbor" item of the confidence information generated according to the first specific example described above is "0". Alternatively, the docking parameter calculation unit 16 determines that the confidence level of each measured point (nearest neighbor) included in the docking location edge point cloud SP is less than or equal to a predetermined value if, for example, the overall confidence level r included in the confidence information generated according to the second specific example described above is 0.3 or less. Then, if the docking parameter calculation unit 16 makes such a determination, it sets the coordinate position of the reference point R at the processing time t+1 following processing time t to the position where the lidar 3 is installed [L x L y L z ]T Set to this.

[0077] The docking parameter calculation unit 16 determines that the confidence level of each measured point (nearest neighbor) included in the docking location edge point cloud SP is higher than a predetermined value if the flag registered in the sub-item "Variance" of the confidence information item "Nearest Neighbor" generated according to the first specific example described above is "1". Alternatively, the docking parameter calculation unit 16 determines that the confidence level of each measured point (nearest neighbor) included in the docking location edge point cloud SP is higher than a predetermined value if, for example, the overall confidence level r included in the confidence information generated according to the second specific example described above is greater than 0.3. Then, if the docking parameter calculation unit 16 makes such a determination, it sets the coordinate position of the reference point R at the next processing time t+1 after processing time t to [L x L y L z -h+d q tanθ] T Set to [L]. Note that in this embodiment, the coordinate position of the reference point R is [L] x L y L z -h+d q tanθ] T The value of θ in this context is set to a predetermined value (for example, 30°). The angle θ is set as the elevation angle of the reference point R with respect to the nearest point HP. In other words, the angle θ is set as the elevation angle when the reference point R is viewed from the quay wall.

[0078] Subsequently, the docking parameter calculation unit 16 repeatedly performs processes such as setting reference points, acquiring edge point clouds of docking locations, and generating straight lines on the docking side from processing time t+1 onward.

[0079] The outline of the process for setting the reference point on the ship in this embodiment can be represented, for example, as shown in Figure 8. Figure 8 is a diagram showing the outline of the process for setting the reference point on the ship in the first embodiment.

[0080] Incidentally, with conventionally known techniques, for example, as shown in Figure 9A, when the distance from the vessel to the berthing location is large, objects on the water surface such as water surface reflections and splashes may be mistakenly extracted as the nearest neighbor. Also, with conventionally known techniques, for example, as shown in Figure 9B, when the distance from the vessel to the berthing location is very small, objects that have height relative to the upper surface of the berthing location may be mistakenly extracted as the nearest neighbor. In contrast, with the processing described above, for example, when the confidence level of each measured point (nearest neighbor) included in the berthing location edge point cloud is higher than a predetermined value, and the distance from the vessel to the berthing location is large, the reference point used for searching (extracting) the berthing location edge point cloud can be set to a position higher than the position where the lidar 3 is installed. This prevents water surface reflection data near the water surface from being captured in the nearest neighbor search. Furthermore, according to the processing described above, for example, when the confidence level of each measured point (nearest neighbor) included in the berthing location edge point cloud is higher than a predetermined value, and the distance from the vessel to the berthing location is short, the reference point used for searching (extracting) the berthing location edge point cloud can be set to a position lower than the location where LIDA 3 is installed. This prevents objects on the quay from being detected in the nearest neighbor search. Therefore, according to the processing described above, whether the distance from the vessel to the berthing location is far or near, the angle θ corresponding to the elevation angle when viewing the reference point R from the quay becomes constant, and nearest neighbor search can be performed under the same conditions that do not depend on the distance to the quay (see Figures 9C and 9D), thus helping to maintain a constant performance of the navigation support system. Furthermore, according to the processing described above, for example, when the confidence level of each measured point (nearest neighbor) included in the berthing location edge point cloud is below a predetermined value, the reference point used for searching (extracting) the berthing location edge point cloud can be set to the location where LIDA 3 is installed.Specifically, if the docking parameter calculation unit 16 mistakenly extracts an object on the water surface as the nearest neighbor point included in the docking location edge point cloud at processing time t (see Figure 9E), the confidence level of each measured point (nearest neighbor point) included in the docking location edge point cloud will be below a predetermined value. Therefore, the unit sets the position where the lidar 3 is installed as the next reference point R (at processing time t+1), and uses this next reference point R to search for and extract the nearest neighbor point (see Figure 9F). Thus, according to the processing described above, a docking location edge point cloud corresponding to the position of the reference point can be extracted, and an appropriate docking side line can be generated based on each nearest neighbor point included in the docking location edge point cloud.

[0081] [Processing flow] Figure 10A is a flowchart illustrating the overview of the docking support process in this embodiment. The information processing device 1 repeatedly executes the processes shown in the flowchart of Figure 10A.

[0082] First, the information processing device 1 acquires point cloud data in the direction of the berthing location (step S11). In this case, the information processing device 1 acquires, for example, point cloud data generated by a lidar 3 that includes the berthing side of the target vessel within its measurement range. The information processing device 1 may also further downsample the acquired point cloud data and remove data reflected from the water surface.

[0083] Next, the docking location detection unit 15 of the information processing device 1 calculates normal vectors based on the point cloud data acquired in step S11 (step S12). Furthermore, in step S12, the docking location detection unit 15 calculates the number of normal vectors and the variance of the normal vectors. Also, based on the processing results of step S12, the docking location detection unit 15 identifies the inner surface of the field of view and the detection surface (step S13).

[0084] Next, the docking parameter calculation unit 16 obtains the current reference point R (for processing time t) from the memory 12 (step S14).

[0085] Next, the docking parameter calculation unit 16 performs a process to search for the nearest neighbor point closest to the reference point R, based on the point cloud data acquired in step S11 and the reference point R acquired in step S14, for each vertical line (step S15).

[0086] Next, the docking parameter calculation unit 16 generates a docking side line L using the nearest neighbor points for each vertical line obtained in step S15 (step S16).

[0087] Next, the docking parameter calculation unit 16 uses the docking side line L calculated in step S16 to calculate the docking parameters, which are the distance to the opposite shore, the approach angle, and the docking speed (step S17).

[0088] Next, the docking parameter calculation unit 16 generates confidence information (step S18) based on the results of identifying the inner surface of the field of view and the detection surface in step S13 and the calculation results of the docking parameters in step S16.

[0089] The berthing parameter calculation unit 16 extracts one nearest neighbor point (hereinafter also referred to as the quay nearest neighbor point) from among the nearest neighbor points obtained in step S15, the point where the distance between the reference point R obtained in step S14 and the berthing side line L generated in step S16 is the shortest. Then, based on the quay nearest neighbor point extracted as described above and the confidence information generated in step S17, the berthing parameter calculation unit 16 performs the next processing related to setting the reference point R (at processing time t+1) (step S19).

[0090] Here, a specific example of the process performed in step S19 will be explained with reference to Figure 10B. Figure 10B is a flowchart showing an example of the process related to setting the reference point.

[0091] After setting the angle θ to a predetermined value (step S31), the docking parameter calculation unit 16 determines whether the confidence level included in the confidence level information generated in step S17 is greater than the predetermined value (step S32).

[0092] If the confidence level is below a predetermined value (step S32: No), the docking parameter calculation unit 16 sets the installation position of the rider 3 as the next reference point R (step S33), stores the set next reference point R in the memory 12 (step S35), and then continues the process of step S20. If the confidence level is greater than a predetermined value (step S32: Yes), the docking parameter calculation unit 16 calculates the distance d q Then, the next reference point R is set using the height h and angle θ (step S34), and the set next reference point R is stored in memory 12 (step S35), after which the process of step S20 is continued. The nearest point to the quay is at a distance d q It is used when calculating [something].

[0093] After performing the processing in step S19, the information processing device 1 controls the vessel based on the reliability information (step S20). As a result, the information processing device 1 can accurately control the vessel regarding berthing based on a reliability level that accurately reflects the berthing status.

[0094] The information processing device 1 then determines whether or not the target vessel has docked (step S21). In this case, the information processing device 1 determines whether or not the target vessel has docked based on, for example, the output signal of the sensor group 2 or user input via the interface 11. If the information processing device 1 determines that the target vessel has docked (step S21; Yes), it terminates the flowchart process. On the other hand, if the information processing device 1 determines that the target vessel has not docked (step S21; No), it returns to step S11.

[0095] According to the process described above, the acquisition means acquires measurement data, which is a set of data identified by a pair of indices representing lateral position and vertical position generated by a measuring device installed on the vessel. According to the process described above, the nearest neighbor extraction means extracts data representing the nearest neighbor point closest to the vessel's reference point from the data representing the measurement point at the berthing location, for each index representing lateral position. According to the process described above, the reference point setting means sets the reference point at the next processing time based on the nearest neighbor point. According to the process described above, the reference point setting means sets the reference point based on the confidence level of the nearest neighbor point. According to the process described above, if the confidence level of the nearest neighbor point is below a predetermined value, the reference point setting means sets the location where the measuring device on the vessel is installed as the reference point. According to the process described above, if the confidence level of the nearest neighbor point is higher than a predetermined value, the reference point setting means sets the height of the reference point according to the distance between the vessel and the berthing location.

[0096] As described above, according to this embodiment, the reference point on the vessel side can be changed (in the z direction) according to the conditions of the berthing location or the vicinity of the berthing location, a berthing location edge point cloud can be extracted according to the position of the reference point, and an appropriate berthing side line can be generated based on each nearest neighbor point included in the berthing location edge point cloud. Therefore, according to this embodiment, parameters used in berthing assistance, such as distance, speed, and angle, can be calculated with high accuracy based on the berthing side line generated as described above.

[0097] <Second Example> Next, a second embodiment will be described below. In this embodiment, explanations of parts to which the same configuration as in the first embodiment can be applied will be omitted as appropriate, and the explanation will focus on parts that differ from the first embodiment. Specifically, in this embodiment, the system configuration, hardware configuration, and functional configuration are substantially the same as in the first embodiment, but the content of the processing performed by the berthing parameter calculation unit 16 differs from that of the first embodiment. Therefore, the explanation below will mainly focus on the processing performed by the berthing parameter calculation unit 16.

[0098] [How to set and use reference points] Here, the method for setting and using the reference point in this embodiment will be described. In this embodiment, for example, as shown in Figure 11A, the case in which a vessel is docked in a berthing location where it is difficult to generate an appropriate berthing side straight line due to the presence of fenders along the wall surface will be described. In this embodiment, for example, as shown in Figure 11A, it will be described that one marker having retroreflective material is placed on the forward side and one on the aft side of the vessel at the berthing location. In this embodiment, for example, as shown in Figure 11B, it will be described that the vessel is equipped with two lidars 3 having a field of view range 91 on the port forward and starboard forward sides of the vessel, and two lidars having a field of view range 92 on the port aft and starboard aft sides of the vessel. Furthermore, in this embodiment, it is explained that a marker placed on the forward side of the vessel at the berthing location (hereinafter also referred to as the forward marker) may be included in the field of view 91, and a marker placed on the rear side of the vessel at the berthing location (hereinafter also referred to as the rear marker) may be included in the field of view 92. Furthermore, in this embodiment, it is explained that the reliability of each measured point (nearest neighbor) included in the berthing location edge point cloud SP is determined to be higher than a predetermined value by the same processing as in the first embodiment. Figure 11A is a diagram showing an example of a situation to which the second embodiment can be applied. Figure 11B is a top view illustrating the field of view of the vessel and lidar included in the navigation support system.

[0099] The docking parameter calculation unit 16, for example, by setting θ = 60°, determines the coordinate position of the reference point R as [L x L y L z -h+d q [tan60°] T The reference point R is set to a position with a large elevation angle when viewed from the quay. In other words, the nearest neighbor search is performed from relatively above. The berthing parameter calculation unit 16 also performs a process to search for the nearest neighbor point MPA closest to the reference point R for each vertical line, based on each measured point that constitutes the point cloud data generated by the lidar 3 at processing time t+1 and the reference point R set as described above. With this process, for example, as shown in Figure 12, it is possible to extract measured points at the edge of the berthing location as the nearest neighbor point MPA in the vertical line while avoiding the extraction of measured points on the fenders as the nearest neighbor point MPA in the vertical line. Furthermore, with the above process, for example, the nearest neighbor point MPA shown in Figure 13A is extracted. Figure 12 is a diagram showing an overview of the process related to the extraction of nearest neighbor point MPA. Figure 13A is a diagram showing an example of the nearest neighbor point MPA extracted by the process shown in Figure 12.

[0100] The docking parameter calculation unit 16 sets an extraction window EWA, which is a rectangular area on the xy plane, based on the measurement point indicating the position of the forward marker and the measurement point indicating the position of the rear marker. The docking parameter calculation unit 16 also obtains a docking location edge point cloud SP at processing time t+1 by extracting those points that belong within the range of the extraction window EWA from among the nearest neighbor points MPA extracted for each vertical line. The docking parameter calculation unit 16 also generates a docking side line L at processing time t+1 based on the docking location edge point cloud SP. Through this process, for example, a rectangular extraction window EWA as shown in Figure 13B is set on the xy plane. Furthermore, through the above process, for example, as shown in Figure 13B, multiple nearest neighbor points MPA belonging within the range of the extraction window EWA are obtained as a docking location edge point cloud SP, and a docking side line L is generated based on the docking location edge point cloud SP. Figure 13B is a top view that clearly shows the extraction window EWA and the docking side line L.

[0101] After performing the above-described processing, the docking parameter calculation unit 16 resets the coordinate position of the reference point R to [L], for example, by resetting θ=15°. x L y L z -h+d q tan15°] T The reference point R is reset. This process allows the reference point R to be set at a position with a small elevation angle when viewed from the quay. In other words, the nearest neighbor search is performed from relatively below. The berthing parameter calculation unit 16 also performs a process to search for the nearest nearest MPB closest to the reference point R for each vertical line, based on each measured point obtained by removing noise from the point cloud data generated by the lidar 3 at processing time t+1, and the reference point R that was reset as described above. This process allows, for example, the measured points on the fenders to be extracted as nearest nearest MPBs in the vertical line, as shown in Figure 14. Furthermore, the above process extracts nearest nearest MPBs such as those shown in Figure 15A. Figure 14 is a diagram showing an overview of the process related to the extraction of nearest nearest MPBs. Figure 15A is a diagram showing an example of nearest nearest MPBs extracted by the process shown in Figure 14.

[0102] The berthing parameter calculation unit 16 sets an extraction window EWB, which is a rectangular area on the xy plane, based on the measurement point indicating the position of the forward marker and the measurement point indicating the position of the rear marker. It is preferable to set the extraction window EWB with a slightly different rectangular area than the extraction window EWA so that detection points of fenders installed on the side of the quay wall can be extracted. For example, the extraction window EWB is set as a rectangular area on the xy plane having an x-boundary closer to the side of the vessel than the extraction window EWA, and a y-boundary further from the rear of the vessel than the extraction window EWA. The berthing parameter calculation unit 16 also obtains a fender point group BP indicating the position of the fenders corresponding to the berthing side line L by extracting those points that belong within the range of the extraction window EWB from among the nearest neighbor points MPB extracted for each vertical line. Through this process, for example, an extraction window EWB larger than the extraction window EWA is set on the xy plane, as shown in Figure 15B. Furthermore, according to the aforementioned process, for example, as shown in Figure 15B, multiple nearest neighbor points (MPB) belonging to the range of the extraction window (EWB) are acquired as a fender point group (BP). Figure 15B is a top view clearly showing the extraction window (EWB).

[0103] The berthing parameter calculation unit 16 extracts one nearest neighbor point KP from among the nearest neighbor points included in the fender point group BP, which is the point with the shortest distance from the origin O of the hull coordinate system. The origin O can be set, for example, to the center position or center of gravity position of the target vessel.

[0104] Here, we will explain a specific example of a process that can be used to extract one nearest neighbor point KP from among the nearest neighbor points included in the fender point group BP, with reference to Figures 16A and 16B. Figure 16A shows the origin O, the berthing side line L, the nearest neighbor point KP, and the distance d. f Figure 16B is a top view that clearly shows the unit vector "u" used to search for the nearest neighbor KP.

[0105] First, the docking parameter calculation unit 16 applies principal component analysis and least squares method to the docking location edge point cloud SP to generate the docking side line L shown in the following formula (3).

[0106]

number

[0107] Here, "[x0y0z0] T '' indicates the centroid of the measured point shown by the docking location edge point cloud SP, and "[abc] T " indicates the direction vector, and "k" indicates the parameter. For example, when performing principal component analysis, the eigenvector corresponding to the largest eigenvalue becomes the direction vector of L.

[0108] The docking parameter calculation unit 16 then calculates a unit vector u that is perpendicular to the docking side line L in a two-dimensional plane, as shown in the following equation (4). The unit vector u is represented as a vector originating from the origin O, as shown in Figure 16B.

[0109]

number

[0110] The dot product of the (x,y) component of data p representing the measured point in the fender point group BP and the unit vector u corresponds to the projection of data p onto vector u. Therefore, the following equation (5) for the distance "d" shown in Figure 16B holds true.

[0111]

number

[0112] Therefore, the docking parameter calculation unit 16 calculates the distance d based on the following formula (6).

[0113]

number

[0114] The berthing parameter calculation unit 16 then calculates the distance for each measurement point represented by the fender point group BP based on the above formula (6), and extracts the data of the measurement point showing the shortest distance as the nearest neighbor point KP. Furthermore, by applying the data representing the nearest neighbor point KP to the above formula (6), the berthing parameter calculation unit 16 calculates the distance d on the horizontal plane (xy plane) between the origin O and the fender. f Calculate.

[0115] Generally, fenders are difficult to detect with a lidar because they are made of low-reflectivity materials such as black rubber, and their rounded shape makes it difficult to identify edges such as those on a quay. Therefore, it is difficult to generate a straight line from the fender point cloud BP. For this reason, as described above, the method of searching for the nearest neighbor KP using the quay side straight line L is employed. If the fenders were made of a highly reflective material and had a shape that made it easy to identify edges, it would be possible to generate a straight line from the fender point cloud BP and use that line to search for the nearest neighbor KP.

[0116] According to the process described above, even if fenders are present at the berthing location, a point cloud of berthing location edges corresponding to the position of the reference point can be extracted, and an appropriate berthing side line can be generated based on each nearest neighbor point included in the berthing location edge point cloud. Furthermore, according to the process described above, if fenders are present at the berthing location, a point cloud of fenders corresponding to the position of the reference point can be extracted, and the distance from the vessel to the fenders can be calculated with high accuracy based on each nearest neighbor point included in the fender point cloud.

[0117] [Processing flow] Figure 17 is a flowchart illustrating the overview of the docking support process in this embodiment. The information processing device 1 repeatedly executes the processes shown in the flowchart of Figure 17.

[0118] In steps S51 to S53, the docking parameter calculation unit 16 performs the same processing as in steps S11 to S13. After that, the docking parameter calculation unit 16 obtains the current first reference point R1 from the memory 12 as a reference point to be used for detecting the edge portion of the docking location (step S54).

[0119] Next, in steps S55 to S57, the berthing parameter calculation unit 16 performs the same processing as in steps S15 to S17, using the first reference point R1 obtained in step S54 as needed. After that, the berthing parameter calculation unit 16 obtains the current second reference point R2 from the memory 12 as a reference point to be used for detecting the fenders installed at the berthing location (step S58).

[0120] Next, the docking parameter calculation unit 16 performs a process to search for the nearest neighbor point closest to the second reference point R2 obtained in step S58, for each vertical line (step S59).

[0121] Next, the berthing parameter calculation unit 16 obtains a fender point group from among the nearest neighbor points obtained in step S59, which indicates the position of the fenders corresponding to the berthing side line L. The berthing parameter calculation unit 16 also extracts one nearest neighbor point from among the nearest neighbor points included in the fender point group that is the shortest distance from the origin O of the hull coordinate system (hereinafter also referred to as the fender nearest neighbor point). Then, based on the fender nearest neighbor point extracted as described above and the berthing side line L, the berthing parameter calculation unit 16 calculates the distance d from the ship (origin O) to the fender. f Calculate (step S60).

[0122] Next, in step S61, the docking parameter calculation unit 16 performs the same processing as in step S18. After that, the docking parameter calculation unit 16 sets the next first reference point R1 by performing the same processing as shown in Figure 10B (step S62). The docking parameter calculation unit 16 also calculates the distance d q Instead, distance d fThe next second reference point R2 is set by performing the same process as shown in Figure 10B using the method (step S63). Subsequently, the docking parameter calculation unit 16 performs the same process as in steps S20 and S21 in steps S64 and S65.

[0123] According to the process described above, the reference point setting means sets a first reference point used for extracting data representing the nearest neighbor point, and a second reference point used for extracting data representing the measurement point of the fender installed at the berthing location, as reference points when the reliability of the nearest neighbor point is higher than a predetermined value. Furthermore, according to the process described above, the reference point setting means sets the first reference point to have a large elevation angle when viewed from the quay, and sets the second reference point to have a small elevation angle when viewed from the quay.

[0124] As described above, according to this embodiment, even if fenders are present at the berthing location, the distance from the vessel to the fenders can be calculated with high accuracy as the distance to the opposite shore. In other words, according to this embodiment, the parameters used in berthing assistance can be calculated with high accuracy.

[0125] [Differentiation] The following describes a suitable modification of the second embodiment described above.

[0126] (Variation 1) The berthing parameter calculation unit 16 may, if it detects a situation that makes it difficult to acquire the berthing location edge point cloud, such as when fenders larger than a predetermined size are densely installed at the berthing location, generate a marker line Lm, which is a straight line passing through the forward marker and the aft marker, instead of a berthing side line. In such a case, the berthing parameter calculation unit 16 may use the marker line Lm to calculate the distance d f The following calculation is required. Furthermore, the marker line Lm should be generated as a straight line, for example, as shown in Figure 18. Figure 18 is a top view clearly showing the marker line Lm.

[0127] (Modification 2) If the lidar 3 has a field of view of less than 180° in the lateral direction of the target vessel, it will not be able to detect multiple fenders of different sizes even if they are installed in an area outside that field of view in the lateral direction, as shown in Figure 19A. Furthermore, even if the lidar 3 has a field of view of 180° or more in the lateral direction of the target vessel, it may not be able to detect the point to be measured that is the shortest distance from the target vessel to the fender, for example, due to the surface of the fender being made of black rubber. In this modified example, a method for calculating the shortest distance from the vessel to the fender will be described, taking into consideration the occurrence of the above situations.

[0128] The docking parameter calculation unit 16 calculates, for example, the distance d from the origin O to the docking location. r And the distance d across the river r Distance d from f The difference value Δd is calculated by subtracting rf (See Figure 19B) Based on this, the distance d corresponds to the shortest distance from the ship to the fender. f_min Calculate.

[0129] Specifically, the docking parameter calculation unit 16 calculates, for example, the difference value Δd from the time docking assistance is initiated onward. rf By continuously calculating the difference value Δd rf It detects the temporal change of the docking parameter 16, for example, the difference value Δd rf At the point when a predetermined time has elapsed since the start of the calculation, the difference value Δd within that predetermined time is calculated. rf The difference value Δd corresponds to the maximum value of Δd. rf_max Obtain the difference value Δd (see Figure 19C). rf_max This is updated to the latest value as time passes. The docking parameter calculation unit 16 calculates the opposite shore distance d r The latest difference value Δd rf_max By subtracting d, the distance d f_min Calculate.

[0130] Here, the processing flow related to the processing of this modified example will be described. FIG. 19D is a flowchart showing an example of the processing related to the calculation of the shortest distance from a ship to a fender.

[0131] The shore approach parameter calculation unit 16 determines whether at least a part of the hull of the target ship has entered the shore approach area (step S81). Note that the shore approach area is the area where the target ship should be located when approaching the shore.

[0132] If the hull of the target ship has not entered the shore approach area (step S81: No), the shore approach parameter calculation unit 16 repeats the process of step S81. Also, if the hull of the target ship has entered the shore approach area (step S81: Yes), the shore approach parameter calculation unit 16 sets the difference value Δd rf_max to 0 (step S82), and then calculates the opposite shore distance d r and the distance d f (step S83).

[0133] Subsequently, the shore approach parameter calculation unit 16 calculates the difference value Δd r by subtracting the distance d f from the opposite shore distance d rf (step S84).

[0134] Subsequently, the shore approach parameter calculation unit 16 determines whether the difference value Δd rf is greater than the difference value Δd rf_max (step S85).

[0135] If the difference value Δd rf is less than or equal to the difference value Δd rf_max (step S85: No), in the state where the difference value Δd rf_max is maintained (step S86), the distance d r corresponding to the shortest distance from the ship to the fender is calculated by subtracting the difference value Δd rf_max from the opposite shore distance d f_min (step S88). Also, if the difference value Δd rf is the difference value Δdrf_max If it is greater than (Step S85: Yes), then the difference value Δd rf The difference value Δd rf_max In the updated state (step S87), the distance to the opposite bank d r From the difference value Δd rf_max By reducing this, the distance d corresponds to the shortest distance from the ship to the fender. f_min Calculate (Step S88).

[0136] Subsequently, the information processing device 1 determines whether or not the target vessel has docked (step S89). In this case, the information processing device 1 determines whether or not the target vessel has docked based, for example, on the output signals of the sensor group 2 or user input via the interface 11. If the information processing device 1 determines that the target vessel has docked (step S89: Yes), it terminates the flowchart process. On the other hand, if the information processing device 1 determines that the target vessel has not docked (step S89: No), it returns to step S83.

[0137] By the way, the difference value Δd rf This corresponds to the overhang length, which is the length that the fender closest to the ship extends from the wall of the berthing location. And, according to the process described above, the maximum value of the aforementioned overhang length is the difference value Δd rf_max We are continuously calculating this difference value Δd rf_max Using distance d f_min The system is designed to calculate the distance d (see Figure 19E). f_min This can be used as the distance to the opposite shore, including the fenders. Therefore, according to the process described above, for example, even if the lidar has a field of view of less than 180° in the lateral direction of the target vessel and various sizes of fenders are installed at the berthing location, the shortest distance from the target vessel to the largest fender can be calculated as the distance to the opposite shore. Furthermore, according to the process described above, for example, even if there are many instances where the measurement points of the fenders are not detected, the shortest distance to the fenders can be calculated.

[0138] In each of the embodiments described above, the program can be stored using various types of non-transitory computer-readable medium and supplied to a control unit, etc., which is a computer. Non-transitory computer-readable medium includes various types of tangible storage medium. Examples of non-transitory computer-readable medium include magnetic storage medium (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage medium (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)).

[0139] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made that are understandable to those skilled in the art within the scope of the present invention. That is, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that those skilled in the art could make in accordance with the technical idea. Furthermore, each disclosure of the above-mentioned patent documents and other references is incorporated herein by reference. [Explanation of Symbols]

[0140] 1. Information Processing Device 2 Sensor Groups 3 Riders

Claims

[Claim 1] An acquisition means for acquiring measurement data, which is a set of data identified by a pair of indices representing lateral position and vertical position generated by a measuring device installed on a ship, A nearest neighbor extraction means extracts data representing the nearest neighbor point closest to the reference point of the vessel from the data representing the measurement point at the berthing location, for each index representing the lateral position, A reference point setting means for setting the reference point at the next processing time based on the nearest neighbor point, An information processing device having

Images