Information processing device, control method, program, and storage medium

Abstract

The present invention provides an information processing device capable of calculating parameters used in docking assistance with high accuracy. [Solution] The information processing device comprises an acquisition means, a marker position acquisition means, and a search range setting means. The acquisition means acquires measurement data generated by a measuring device installed on the ship. The marker position acquisition means acquires the position of at least one marker installed at the berthing location based on the measurement data. The search range setting means sets the search range of the point cloud of the edge portion of the berthing location based on the position of at least one marker.

Description

[Technical Field]

[0001] This disclosure relates to the procedures for when a vessel is docked. [Background technology]

[0002] Technologies for assisting with ship docking (berthing) have been known for some time. For example, Patent Document 1 describes a method for controlling the attitude of a ship in an automatic docking device that performs automatic ship docking, such that light emitted from a lidar is reflected by objects around the docking position and received by the lidar. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-59403 [Overview of the project] [Problems that the invention aims to solve]

[0004] For ships, safe and smooth docking at a designated berthing location is crucial, and the realization of docking support systems for ship handling assistance and automated navigation is particularly desired. Therefore, docking support systems require high-precision calculation of parameters such as distance, speed, and angle relative to the intended docking location.

[0005] This disclosure is made to solve the above-mentioned problems, and its main purpose is to provide an information processing device that can calculate parameters used in docking assistance with high accuracy. [Means for solving the problem]

[0006] The invention described in the claim is an information processing device comprising: acquisition means for acquiring measurement data generated by a measuring device installed on a ship; marker position acquisition means for acquiring the position of at least one marker installed at a berthing location based on the measurement data; and search range setting means for setting a search range for a point cloud of the edge portion of the berthing location based on the position of the at least one marker.

[0007] Furthermore, the invention described in the claim is a control method performed by a computer, comprising: acquiring measurement data generated by a measuring device installed on a ship; acquiring the position of at least one marker installed at a berthing location based on the measurement data; and setting a search range for the point cloud of the edge portion of the berthing location based on the position of the at least one marker.

[0008] Furthermore, the invention described in the claim is a program that causes a computer to perform the following processes: acquire measurement data generated by a measuring device installed on a ship; acquire the position of at least one marker installed at a berthing location based on the measurement data; and set the search range of the point cloud of the edge portion of the berthing location based on the position of the at least one marker. [Brief explanation of the drawing]

[0009] [Figure 1A] Block diagram of the flight support system. [Figure 1B] A top view illustrating the field of view of the ship and lidar included in the navigation support system. [Figure 1C] A diagram showing the field of view of a ship and a rider from the rear. [Figure 2] A block diagram showing an example of the hardware configuration of an information processing device. [Figure 3] Functional block diagram related to berthing support processing. [Figure 4A] A diagram showing a rider capturing the quay as it docks. [Figure 4B] A perspective view of a quay wall, clearly showing the straight line of the side where the quay docks. [Figure 5A] A diagram showing an example of a data structure for confidence level information. [Figure 5B] A diagram showing an example of the indicators and confidence levels included in confidence information. [Figure 5C] An overhead view of the target vessel and berthing location, clearly indicating the indicators shown in Figure 5B. [Figure 6A] A diagram showing an example of a hull coordinate system based on the hull of the target vessel. [Figure 6B] A perspective view of a structure with its normal vectors clearly indicated. [Figure 7] A diagram showing an example of a docking area with forward and aft markers. [Figure 8] A top view clearly showing the extraction window EWA. [Figure 9A] A diagram showing the nearest neighbor points for each vertical line at the SBP (Side Bypass) berthing location. [Figure 9B] This figure shows an example of each nearest neighbor point obtained as a docking location edge point cloud by the extraction window EWA. [Figure 10A] A diagram showing heights hmf and hmr. [Figure 10B] A top view showing the marker interval rm. [Figure 10C] A top view showing distances wmf and wmr. [Figure 11] This figure shows examples of the nearest neighbor points obtained when marker CM is determined to be a forward marker. [Figure 12] This figure shows an example of a straight line generated when marker CM is assumed to be a forward marker. [Figure 13] This figure shows an example of a straight line generated when marker CM is assumed to be a rear marker. [Figure 14A] A flowchart illustrating the overview of the docking support process. [Figure 14B] A flowchart illustrating an example of the process for acquiring the edge point cloud of the docking location. [Figure 15A] This figure shows an example of when incorrect edge lines are generated. [Figure 15B] This figure shows an example of when the correct edge line is generated. [Figure 15C] This figure shows an example of when the correct edge line is generated. [Figure 16] A diagram illustrating an example of the process for acquiring the edge point cloud of the shoreing location. [Figure 17A] A top view clearly showing the origin O, distances df and dr, and the front end FE and rear end RE. [Figure 17B] A diagram showing the unit vector u used to calculate distances df and dr. [Figure 18] A diagram illustrating an example of the process for calculating distance df and distance dr. [Modes for carrying out the invention]

[0010] In one preferred embodiment of the present invention, the information processing device includes: acquisition means for acquiring measurement data generated by a measuring device installed on a ship; marker position acquisition means for acquiring the position of at least one marker installed at a berthing location based on the measurement data; and search range setting means for setting a search range for a point cloud of the edge portion of the berthing location based on the position of the at least one marker.

[0011] The above-described information processing device includes an acquisition means, a marker position acquisition means, and a search range setting means. The acquisition means acquires measurement data generated by a measuring device installed on the ship. The marker position acquisition means acquires the position of at least one marker installed at the berthing location based on the measurement data. The search range setting means sets the search range of the point cloud of the edge portion of the berthing location based on the position of the at least one marker. This makes it possible to calculate parameters used in berthing assistance with high accuracy.

[0012] In one embodiment of the above-described information processing device, the marker position acquisition means acquires the position of a first marker provided on the front side of the vessel and the position of a second marker provided on the rear side of the vessel, and the search range setting means sets a substantially rectangular or substantially cuboidal region corresponding to the positions of the first and second markers as the search range.

[0013] In one embodiment of the information processing device described above, the search range setting means sets the search range using the distance from a straight line along the edge portion of the docking location to the first marker and the distance from the straight line to the second marker.

[0014] In one embodiment of the above-described information processing device, the marker position acquisition means acquires the position of one of two markers provided at the berthing location, which is provided on the forward or aft side of the vessel, and the search range setting means sets the search range based on the distance between the two markers and the position of the one marker.

[0015] In one embodiment of the above-described information processing device, the search range setting means sets the search range using the distance from a straight line along the edge portion of the docking location to one marker.

[0016] In one embodiment of the information processing device described above, the search range setting means sets the search range using the entire length of the vessel.

[0017] In one embodiment of the above-described information processing device, the marker position acquisition means acquires the position of a first marker provided on the front side of the vessel and the position of a second marker provided on the rear side of the vessel, and further includes distance calculation means that calculates the distance from a reference point of the vessel to the edge of a berthing area, which is the area in which the vessel should be located when berthing at the berthing area, based on the position of the first marker, the position of the second marker, and a straight line along the edge portion of the berthing area.

[0018] In one embodiment of the above-described information processing device, two markers are provided at the docking location such that they are at different heights from the upper surface of the docking location.

[0019] In another embodiment of the present invention, a control method executed by a computer acquires measurement data generated by a measuring device installed on the ship, acquires the position of at least one marker installed at the berthing location based on the measurement data, and sets the search range of the point cloud of the edge portion of the berthing location based on the position of the at least one marker. This makes it possible to calculate parameters used in berthing assistance with high accuracy.

[0020] In yet another embodiment of the present invention, the program causes a computer to perform the following processes: acquire measurement data generated by a measuring device installed on a ship; acquire the position of at least one marker installed at the berthing location based on the measurement data; and set the search range for the point cloud of the edge portion of the berthing location based on the position of the at least one marker. By executing this program on a computer, the above-described information processing device can be realized. This program can be stored and used on a storage medium. [Examples]

[0021] Preferred embodiments of the present invention will be described below with reference to the drawings.

[0022] <First Example> First, the first embodiment will be described below.

[0023] [Overview of the driver assistance system] Figures 1A to 1C show the schematic configuration of the navigation support system according to this embodiment. Specifically, Figure 1A shows a block diagram of the navigation support system, Figure 1B is a top view illustrating the field of view (also called the "measurement range" or "distance-measurable range") 90 of the vessel and the lidar 3 described later, which are included in the navigation support system, and Figure 1C is a rear view showing the field of view 90 of the vessel and the lidar 3. The navigation support system comprises an information processing device 1 that moves together with the vessel, which is a moving object, and a group of sensors 2 mounted on the vessel. Hereafter, the vessel on which the navigation support system is installed will also be called the "target vessel".

[0024] The information processing device 1 is electrically connected to the sensor group 2 and provides operational support for the target vessel based on the outputs of the various sensors included in the sensor group 2. Operational support includes berthing support such as automatic docking. Here, "berthing" includes not only docking the target vessel at a quay but also docking it at a structure such as a pier. Furthermore, hereafter, "berthing location" refers to the general term for structures such as quays and piers that are the target of berthing. The information processing device 1 may be a navigation device installed on the vessel or an electronic control device built into the vessel.

[0025] Sensor group 2 includes various external and internal sensors installed on the ship. In this embodiment, sensor group 2 includes, for example, a Lidar (Light Detection and Ranging, or Laser Illuminated Detection and Ranging) 3.

[0026] The LIDA3 is an external sensor that discretely measures the distance to an object in the external environment by emitting a pulsed laser within a predetermined angular range in the horizontal direction (see Figure 1B) and a predetermined angular range in the vertical direction (see Figure 1C), and generates three-dimensional point cloud data indicating the position of the object.

[0027] LIDA 3 comprises an irradiation unit that irradiates laser light while changing the irradiation direction, a light receiving unit that receives reflected light (scattered light) of the irradiated laser light, and an output unit that outputs scan data based on the received signal output by the light receiving unit. The data measured for each direction of laser light irradiation (scanning position) is generated based on the irradiation direction corresponding to the laser light received by the light receiving unit and the response delay time of the laser light specified based on the received signal described above. Hereafter, the point measured by the irradiation of laser light within the measurement range of LIDA 3, or the data thereof, will also be referred to as the "measured point".

[0028] In the examples shown in Figures 1B and 1C, the target vessel is equipped with four lidars 3: one pointed forward on the port side, one pointed aft on the port side, one pointed forward on the starboard side, and one pointed aft on the starboard side. When the target vessel approaches the berthing location, the forward and aft lidars 3, located on the side of the vessel that is docking alongside the berthing location, generate point cloud data measuring the berthing location. Hereafter, the forward lidar 3 that measures the berthing location will be referred to as the "forward lidar," and the aft lidar 3 that measures the berthing location will be referred to as the "backward lidar." The forward and aft measuring lidars are examples of the "first measuring device" and "second measuring device." Note that the arrangement of lidars 3 is not limited to the examples shown in Figures 1B and 1C.

[0029] Here, point cloud data can be considered as an image (frame) where each measurement direction is represented by a pixel, and the measured distance and reflectance value for each measurement direction are represented by the pixel value. In this case, the direction of laser beam emission (i.e., measurement direction) differs in elevation and depression angles in the vertical arrangement of pixels, and the direction of laser beam emission differs in horizontal angles in the horizontal arrangement of pixels. Hereafter, when the point cloud data is considered as an image, the measured points corresponding to the rows of pixels (i.e., vertical columns) whose horizontal index positions coincide will also be called "vertical lines." Furthermore, when the point cloud data is considered as an image, the horizontal index will be called the "horizontal number," and the vertical index will be called the "vertical number."

[0030] LiDAR 3 is not limited to the scanning type LiDAR described above, but may also be a flash type LiDAR that generates 3D data by diffusing laser light into the field of view of a 2D array sensor. LiDAR 3 is an example of a "measuring device" in the present invention.

[0031] [Configuration of Information Processing Device] Figure 2 is a block diagram showing an example of the hardware configuration of the information processing device 1. The information processing device 1 mainly consists of an interface 11, a memory 12, and a controller 13. Each of these elements is interconnected via a bus line.

[0032] Interface 11 performs interface operations related to the exchange of data between the information processing device 1 and external devices. In this embodiment, interface 11 acquires output data from each sensor in the sensor group 2 and supplies it to the controller 13. Interface 11 also supplies signals related to the control of the target vessel, generated by the controller 13, to each component of the target vessel that controls the operation of the target vessel. For example, the target vessel includes a drive source such as an engine or electric motor, a propeller that generates thrust in the direction of travel based on the driving force of the drive source, a thruster that generates thrust in the lateral direction based on the driving force of the drive source, and a rudder, etc., which is a mechanism for freely determining the direction of travel of the vessel. During automatic operation such as automatic docking, interface 11 supplies control signals generated by the controller 13 to each of these components. If the target vessel is equipped with an electronic control device, interface 11 supplies control signals generated by the controller 13 to the electronic control device. Interface 11 may be a wireless interface such as a network adapter for wireless communication, or it may be a hardware interface for connecting to external devices by cables, etc. Furthermore, interface 11 may perform interface operations with various peripheral devices such as input devices, display devices, and sound output devices.

[0033] Memory 12 is composed of various volatile and non-volatile memories such as RAM (Random Access Memory), ROM (Read Only Memory), hard disk drive, and flash memory. Memory 12 stores programs for the controller 13 to execute predetermined processes. Note that the programs executed by the controller 13 may be stored in storage media other than memory 12.

[0034] Furthermore, memory 12 stores information necessary for the processing performed by the information processing device 1 in this embodiment. For example, memory 12 may store map data including information about the location of the docking place. In another example, memory 12 stores information about the downsampling size when downsampling is performed on the point cloud data obtained when the lidar 3 performs one cycle of scanning.

[0035] The controller 13 includes one or more processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a TPU (Tensor Processing Unit), and controls the entire information processing device 1. In this case, the controller 13 performs processing related to the operation support of the target vessel by executing programs stored in memory 12, etc.

[0036] Furthermore, the controller 13 functionally includes a docking location detection unit 15 and a docking parameter calculation unit 16. The docking location detection unit 15 performs processing related to the detection of docking locations based on the point cloud data output by the lidar 3. The docking parameter calculation unit 16 calculates the parameters necessary for docking at the docking location (also called "docking parameters"). Here, the docking parameters include the distance between the target vessel and the docking location (distance to the opposite shore), the approach angle of the target vessel to the docking location, and the speed at which the target vessel approaches the docking location (docking speed). The docking parameter calculation unit 16 also calculates information representing the reliability of docking at the docking location (also called "reliability information") based on the processing results of the docking location detection unit 15 and the docking parameters. The controller 13 functions as an "acquisition means," a "marker position acquisition means," a "search range setting means," a "distance calculation means," and a computer that executes programs.

[0037] Furthermore, the processing performed by the controller 13 is not limited to being implemented by software through a program, but may also be implemented by a combination of hardware, firmware, and software. Additionally, the processing performed by the controller 13 may be implemented using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the program executed by the controller 13 in this embodiment may be implemented using this integrated circuit.

[0038] [Overview of docking support procedures] Next, an overview of the docking support process performed by the information processing device 1 will be described. Based on the point cloud data of the lidar 3 measured in the direction in which the docking location exists, the information processing device 1 generates a straight line along the side of the docking location (also called the "docking side line L"). In other words, the docking side line L is a straight line along the side of the quay wall of the docking location. Then, based on the docking side line L, the information processing device 1 calculates docking parameters such as the distance to the opposite shore.

[0039] Figure 3 is a functional block diagram of the docking location detection unit 15 and the docking parameter calculation unit 16 related to docking support processing. Functionally, the docking location detection unit 15 includes a normal vector calculation block 20, a field of view / detection surface identification block 21, a normal number identification block 22, a mean / variance calculation block 23, and a docking status determination block 24. Functionally, the docking parameter calculation unit 16 includes a neighboring point search block 26, a straight line generation block 27, a distance calculation block 28, an entry angle calculation block 29, a docking speed calculation block 30, a reliability information generation block 40, a marker detection block 41, and a docking location edge point cloud extraction block 42.

[0040] The normal vector calculation block 20 calculates the normal vector of the surface formed by the berthing location (also called the "berthing surface") based on the point cloud data generated by the lidar 3 in the direction in which the berthing location exists. In this case, the normal vector calculation block 20 calculates the normal vector based on, for example, the point cloud data generated by the lidar 3, which includes the berthing side of the target vessel in its measurement range. Information regarding the measurement range of the lidar 3 and the direction of the berthing location may be pre-registered in, for example, memory 12.

[0041] In this case, the normal vector calculation block 20 preferably performs downsampling of the point cloud data and removal of data obtained by the reflection of laser light from the water surface (also called "water surface reflection data").

[0042] In this case, first, the normal vector calculation block 20 removes data located below the water surface position from the point cloud data generated by the lidar 3, treating it as water surface reflection data (i.e., false detection data). The normal vector calculation block 20 estimates the water surface position based, for example, on the average value in the height direction of the point cloud data generated by the lidar 3 when there are no objects other than the water surface in the surrounding area. Then, the normal vector calculation block 20 performs downsampling on the point cloud data after the water surface reflection data has been removed, which is a process that integrates the measured points for each grid space of a predetermined size. Finally, the normal vector calculation block 20 calculates a normal vector for each measured point indicated by the downsampled point cloud data using multiple surrounding measured points. Note that downsampling may be performed before removing the data reflected from the water surface.

[0043] The field of view / detection surface identification block 21 identifies the surface of the docking location that is within the field of view angle of the Lidar 3 (also called the "inner surface of the field of view") and the surface of the docking location detected based on the normal vector calculated by the normal vector calculation block 20 (also called the "detection surface"). In this case, the field of view / detection surface identification block 21 determines whether the upper surface and / or the side surface of the docking location are included in the inner surface of the field of view and the detection surface.

[0044] The normal vector number identification block 22 extracts the vertical normal vector and the normal vector perpendicular to it (i.e., the horizontal direction) from the normal vector calculated by the normal vector calculation block 20, and calculates the number of vertical normal vectors and the number of horizontal normal vectors. Here, the normal vector number identification block 22 considers the vertical normal vector to represent the normal to the measured point on the upper surface of the docking location, and the horizontal normal vector to represent the normal to the measured point on the side of the docking location, and calculates the number of each as an indicator of the reliability of the docking location.

[0045] The mean and variance calculation block 23 extracts the vertical normal vector and the normal vector perpendicular to it (i.e., horizontal) from the normal vectors calculated by the normal vector calculation block 20, and calculates the mean and variance of the vertical normal vector and the mean and variance of the horizontal normal vector.

[0046] The docking status determination block 24 obtains the processing results of the field of view / detection surface identification block 21, the normal number identification block 22, and the mean / variance calculation block 23, which are identified or calculated based on the same point cloud data, as determination results representing the detection status of the docking location at the time the point cloud data was generated. The docking status determination block 24 then supplies the processing results of the field of view / detection surface identification block 21, the normal number identification block 22, and the mean / variance calculation block 23 as determination results of the detection status of the docking location to the docking parameter calculation unit 16.

[0047] The marker detection block 41 detects at least one marker placed at the docking location based on the measurement points that constitute the point cloud data, and obtains the coordinate position of the detected at least one marker.

[0048] The nearest neighbor search block 26 searches for the nearest neighbor point closest to the target vessel, line by line, from the measured points that make up the point cloud data. For example, as shown in Figure 4A, when the lidar 3 is detecting a quay where the vessel is docked, the nearest neighbor point, which is the point closest to the vessel, will be the edge between the top and side surfaces of the quay. The set of points obtained by the nearest neighbor search for each vertical line will be a collection of points near the edge of the quay.

[0049] The docking location edge point cloud extraction block 42 extracts a point cloud of the edge portion of the docking location within the search range, based on the coordinate position of at least one marker obtained by the marker detection block 41 and the nearest neighbor point found by the nearest neighbor point search block 26.

[0050] The line generation block 27 generates a shoreline line L, which is a straight line along the side of the shoreing location, using principal component analysis and least squares method based on the nearest neighbor determined by the nearest neighbor search block 26. Through this process, the line generation block 27 can generate a shoreline line L, for example, as shown in Figure 4B. Figure 4B is a perspective view of a quay wall with the shoreline line clearly indicated.

[0051] The opposite shore distance calculation block 28 calculates the opposite shore distance, which corresponds to the shortest distance between the target vessel and the berthing location, based on the berthing side line L generated by the line generation block 27. Here, if there are multiple lidars 3 that can measure the berthing location, the opposite shore distance calculation block 28 generates a berthing side line L by combining the point cloud data of multiple lidars 3 and calculates the shortest distance to each lidar 3 as the opposite shore distance. Alternatively, the opposite shore distance calculation block 28 may generate a berthing side line L for each point cloud data of lidar 3 and calculate the opposite shore distance as the shortest distance between each berthing side line L and each lidar 3. Furthermore, the opposite shore distance calculation block 28 may calculate the opposite shore distance as the shortest distance from a reference point such as the vessel's center position to the berthing side line L. Note that instead of considering the shortest distance for each lidar 3 as the opposite shore distance, the opposite shore distance calculation block 28 may define the opposite shore distance as the shorter of the shortest distances for each lidar 3, or define the opposite shore distance as the average of these shortest distances. The shortest distance to the shoreline line L can be determined by calculating the length of the perpendicular to the shoreline line L in the xy-plane.

[0052] The approach angle calculation block 29 calculates the approach angle of the target vessel to the berthing location based on the berthing side line L generated by the line generation block 27. Specifically, the approach angle calculation block 29 calculates the approach angle using the function "atan2", which finds the arctangent from two arguments that define the tangent. More specifically, the approach angle calculation block 29 calculates the approach angle from the direction vector of the berthing side line L by calculating the function atan2.

[0053] The berthing speed calculation block 30 calculates the berthing speed, which is the speed at which the target vessel approaches the berthing location, based on the distance to the opposite shore calculated by the distance to the opposite shore calculation block 28. For example, the berthing speed calculation block 30 calculates the berthing speed as the change in the distance to the opposite shore (shortest distance) over time.

[0054] The reliability information generation block 40 generates reliability information based on the processing results of the docking status determination block 24, the straight line generation block 27, the opposite bank distance calculation block 28, and the approach angle calculation block 29.

[0055] Here, we will explain the first specific example of generating confidence information. The confidence information generation block 40 generates flags for each element such as the field of view at the time of detection of the docking location, the surface detection of the docking location, the number of normal vectors, and the variance, and generates confidence information from the vector of the generated flags. Hereafter, a flag of "1" will indicate that the confidence level of the corresponding element is high, and a flag of "0" will indicate that the confidence level of the corresponding element is low.

[0056] Figure 5A shows an example of the data structure of confidence information generated by the confidence information generation block 40. As shown in Figure 5A, the confidence information has the following items: "Top view", "Side view", "Linear view", "Distance", and "Angle". The item "Top view" has the sub-items "Field of view angle", "Detection", "Normal number", and "Variance", and the item "Side view" has the sub-items "Field of view angle", "Detection", "Normal number", and "Variance". The item "Linear view" has the sub-item "Absolute value", the item "Distance" has the sub-items "Amount of change" and "Rate of change", and the item "Angle" has the sub-item "Amount of change".

[0057] Here, the reliability information generation block 40 registers a flag in the "Field of View Angle" sub-item of the "Top Surface" item, which is set to "1" if the top surface of the docking location is within the field of view, and to "0" if the top surface is outside the field of view. The reliability information generation block 40 also registers a flag in the "Detection" sub-item of the "Top Surface" item, which is set to "1" if the top surface of the docking location is a detection surface, and to "0" if the top surface is not a detection surface. The reliability information generation block 40 also registers a flag in the "Number of Normals" sub-item of the "Top Surface" item, which is set to "1" if the number of normal vectors to the top surface of the docking location is equal to or greater than a predetermined threshold (e.g., 10), and to "0" if the number is less than the threshold. Furthermore, the reliability information generation block 40 registers a flag in the "Variance" sub-item of the "Upper Surface" item, where "1" is set if the variance of the x, y, and z components of the normal vector to the upper surface of the docking location is all less than a predetermined threshold (e.g., 1.0), and "0" is set if any of the variances is equal to or greater than the threshold. The reliability information generation block 40 also registers flags in each sub-item of the "Side" item, which are determined by the same rules as for each sub-item of the "Upper Surface" item.

[0058] Furthermore, the reliability information generation block 40 registers a flag in the "Absolute Value" sub-item of the "Line" item that represents the reliability of the berthing side line L generated by the generation method described later. For example, if there are riders 3 at the front and rear of the target vessel, the reliability information generation block 40 registers a flag that is "1" if the difference for each component between the direction vector of the berthing side line L generated by the generation method described later and the direction vector of the line connecting the nearest points of the riders 3 at the front and rear of the target vessel is all less than a threshold, and "0" if any of the differences are greater than or equal to the threshold.

[0059] Furthermore, the reliability information generation block 40 registers a flag in the "Change Amount" sub-item of the "Distance" item, which is set to "1" if the change amount of the distance to the opposite bank calculated by the opposite bank distance calculation block 28 from one time point in the previous period is less than a predetermined threshold (e.g., 1.0 m), and to "0" if the change amount is equal to or greater than the threshold. Furthermore, the reliability information generation block 40 registers a flag in the "Rate of Change" sub-item of the "Distance" item, which is set to "1" if the rate of change of the distance to the opposite bank calculated by the opposite bank distance calculation block 28 from one time point in the previous period is less than a predetermined threshold (e.g., ±10%), and to "0" if the rate of change is equal to or greater than the threshold. Furthermore, the reliability information generation block 40 registers a flag in the "Change Amount" sub-item of the "Angle" item, which is set to "1" if the change amount of the approach angle calculated by the approach angle calculation block 29 from one time point in the previous period is less than a predetermined threshold (e.g., 1.0 degrees), and to "0" if the change amount is equal to or greater than the threshold.

[0060] The thresholds mentioned above are set to conforming values ​​pre-stored in, for example, memory 12. Furthermore, confidence information may be generated for each Writer 3.

[0061] Next, a second specific example related to the generation of confidence information will be explained. Figure 5B is a diagram showing an example of the indicators and confidence levels included in the confidence information. Figure 5C is an overhead view of the target vessel and berthing location with the indicators shown in Figure 5B clearly indicated. Here, markers M0 and M1, which are reference points for berthing, are provided on the quay where the berthing takes place. The information processing device 1 detects these markers M0 and M1 based on the point cloud data of the lidar 3 and performs various processes such as generating the berthing side line L using the measured points (neighborhood point set) near the edge of the berthing location that exist between markers M0 and M1. Hereafter, the measured points near the edge of the berthing location that exist between markers M0 and M1 will also be called "target points". If markers M0 and M1 are not detected, for example, all of the neighboring point set will be considered as target points.

[0062] The index "c3" is an index based on the number of target points, and is expressed here as an example as a linear function with the variable x being the number of target points. The index "c2" is an index based on the standard deviation of the target points, and is expressed here as an example as a linear function with the variable x being the standard deviation of the target points. The index "c1" is an index indicating whether both the front and rear quays can be measured by the two lidars 3, and as an example here, the value is set to "1.0" when both can be measured, and "0.0" when only one can be measured. The index "c0" is an index based on the distance between both ends of the target points (both ends in the direction along the berthing side straight line L), and is expressed as a linear function with the variable x being the distance between the aforementioned both ends. Further, the indices c0 to c3 are calculated so as to be limited to the range of 0 to 1.

[0063] The calculation reliability "c" is a reliability based on each of the aforementioned indices c0 to c3, and here it is a weighted average value of the indices c0 to c3 using weighting coefficients "w0" to "w3" corresponding to the importance of the indices c0 to c3. Further, an example of set values for the weighting coefficients w0 to w3 is illustrated. Since each of the indices c0 to c3 is in the range of 0 to 1, the calculation reliability c, which is the weighted average value thereof, is also calculated as a numerical value in the range of 0 to 1.

[0064] The overall reliability "r" is a reliability based on the side detection reliability "q s ", which is the reliability related to detection of the side surface of a quay, the top surface detection reliability "q u ", which is the reliability related to detection of the top surface of a quay, the marker detection reliability "m0" which is the reliability related to detection of the marker M0, the marker detection reliability "m1" which is the reliability related to detection of the marker M1, and the calculation reliability c. Here, each reliability q s , q u , m0, m1, c uses weighting coefficients "w qs ", "w qu ", "w m0 ", "w m1 ", "w c " corresponding to their respective importance, and is a weighted average value of the reliabilities q s , q u , m0, m1, c. Further, the weighting coefficients w qs , wqu , w m0 , w m1 , w c An example of the setting value is shown in the diagram. Note that the information processing device 1, for example, the side detection confidence level q s This is calculated based on the "Side" item of the reliability information shown in Figure 5A, and the top surface detection reliability q u The confidence level may be calculated based on the "top view" item in the confidence level information of the same figure. Alternatively, the information processing device 1 may calculate the marker detection confidence level m0 and marker detection confidence level m1 based, for example, on the number of points measured in the point cloud data from the respective lidar 3 for marker M0 and marker M1. Then, the side detection confidence level q s Top surface detection confidence q u The marker detection confidence levels m0 and m1 are both calculated to be in the range of 0 to 1. Furthermore, if both the forward and aft quays are detected, the side detection confidence level q of the forward quay is calculated. s0 , the confidence level of detection on the side of the rear quay q s1 , detection confidence q on the upper surface of the forward quay u0 , detection confidence q on the upper surface of the rear quay u1 It is calculated as follows: Side detection confidence q s Top surface detection confidence q u Since the marker detection confidence m0, marker detection confidence m1, and calculation confidence c are all in the range of 0 to 1, the overall confidence score r, which is their weighted average, is also calculated as a value in the range of 0 to 1. Therefore, the closer the overall confidence score r is to 1, the higher the reliability of the calculated docking parameters, and the closer the overall confidence score r is to 0, the lower the reliability of the calculated docking parameters.

[0065] In other words, the confidence information generated by the second specific example includes each indicator and confidence level calculated using the method described above.

[0066] Next, specific examples of the processing of the normal vector calculation block 20, the normal number determination block 22, and the mean / variance calculation block 23 will be explained with reference to Figures 6A and 6B.

[0067] Figure 6A shows an example of a hull coordinate system based on the hull of the target vessel. As shown in Figure 6A, the forward direction of the target vessel is the "x" coordinate, the side direction of the target vessel is the "y" coordinate, and the height direction of the target vessel is the "z" coordinate. The measurement data from the coordinate system based on LIDA 3, measured by LIDA 3, is converted to the hull coordinate system shown in Figure 6A. The process of converting point cloud data from a coordinate system based on a LIDA installed on a moving object to the coordinate system of the moving object is disclosed, for example, in International Publication WO2019 / 188745.

[0068] Figure 6B is a perspective view of the quay where the vessel docks, clearly showing the measurement points (represented by the LIDA 3) and the normal vectors calculated based on those measurement points. In Figure 6B, the measurement points are indicated by circles, and the normal vectors are indicated by arrows. This example shows that both the top and side surfaces of the quay were measured by the LIDA 3.

[0069] As shown in Figure 6B, the normal vector calculation block 20 calculates normal vectors for the measurement points on the side and top surfaces of the quay wall. Since normal vectors are vectors perpendicular to the target plane or curved surface, they are calculated using multiple measurement points that can form a surface. Therefore, a grid with predetermined lengths for length and width, or a circle with a predetermined radius, is set up, and the calculation is performed using the measurement points located within it. In this case, the normal vector calculation block 20 may calculate a normal vector for each measurement point, or it may calculate normal vectors at predetermined intervals. The normal number identification block 22 then determines that normal vectors whose z component is greater than a predetermined threshold are normal vectors oriented in the vertical direction. It is assumed that the normal vectors are unit vectors. Furthermore, normal vectors whose z component is less than a predetermined threshold are determined to be normal vectors oriented in the horizontal direction. The normal number identification block 22 then identifies the number of vertical normal vectors (5 in this case) and the number of horizontal normal vectors (4 in this case). Furthermore, the mean / variance calculation block 23 calculates the mean and variance of the vertical normal vector and the mean and variance of the horizontal normal vector. Note that for edge areas, the measurement direction will be diagonal because the surrounding measurement points are on the top or side.

[0070] [Details on how to generate the straight line along the shoreline] Next, we will explain how to generate the straight line L on the side of the shore.

[0071] (1st generation method) Figure 7 shows an example of a docking area where forward and rearward markers are provided.

[0072] This generation method describes the case where a vessel is docked at a berthing area (SBP) having an L-shape, as shown in Figure 7. Furthermore, this generation method is described assuming that two markers with high reflectivity characteristics, such as retroreflective plates, are provided at the berthing area (SBP). In the following descriptions of this generation method, unless otherwise specified, the marker provided on the forward side of the vessel docked at the berthing area (SBP) will be referred to as the forward marker FM, and the marker provided on the aft side will be referred to as the aft marker RM. In this generation method, it is also described assuming that the forward marker FM and the aft marker RM are positioned at different heights from the top surface of the berthing area (SBP). Note that the forward marker FM and the aft marker RM may have different shapes or sizes, for example, as long as it is possible to distinguish whether they are located on the forward or aft side of the vessel.

[0073] The berthing parameter calculation unit 16 obtains the point cloud data FMG constituting the forward marker FM and the point cloud data RMG constituting the rear marker RM by, for example, extracting points to be measured with a predetermined intensity or higher from the point cloud data generated by the lidar 3. Furthermore, the berthing parameter calculation unit 16 calculates the coordinate position M of the forward marker FM in the ship coordinate system based on the centroid position of each point to be measured included in the point cloud data FMG. f [m fx m fy m fz ] T The docking parameter calculation unit 16 obtains the coordinate position M of the rear marker RM in the ship coordinate system, based on the centroid position of each measured point included in the point cloud data RMG. r [m rx m ry m rz ] T Obtain it.

[0074] Subsequently, the docking parameter calculation unit 16 calculates the coordinate position M of the forward marker FM. f And the coordinate position M of the rear marker RM. r Based on a predetermined margin Δw, a rectangular extraction window EWA is set on the xy plane.

[0075] Specifically, the extraction window EWA is, for example, the coordinate position FWA[m] as shown in Figure 8. fx +Δw m ry +Δw] T And, coordinate position FWB[m fx +Δw m fy ―Δw] T And, coordinate position RWA[m rx ―Δw m ry +Δw] T (and coordinate position RWB[m rx ―Δw m fy ―Δw] T It is set as a rectangular region with vertices and . Figure 8 is a top view that clearly shows the extraction window EWA.

[0076] Note that the extraction window EWA is not limited to being set as a rectangular (or roughly rectangular) region on the xy-plane, but may also be set as, for example, a cuboid (or roughly cuboid) region in xyz space. In such cases, the coordinate value m fz and m rz The relatively small coordinate value among them should be set as the upper limit of the z-direction of the extraction window EWA. That is, the docking parameter calculation unit 16 calculates the coordinate position M f and M r A rectangular or cuboidal region can be set as the search range for the edges of the docking location SBP.

[0077] As described above, the berthing parameter calculation unit 16 searches for the nearest neighbor point closest to the target vessel for each vertical line from the measurement points that make up the point cloud data of the berthing location SBP, and obtains the set of nearest neighbor points SG of the berthing location SBP.

[0078] The docking parameter calculation unit 16 extracts the measured points that belong within the range of the extraction window EWA for the neighboring point set SG, thereby obtaining point cloud data (hereinafter also referred to as the "docking location edge point cloud") that includes multiple measured points (nearest neighbors) that constitute the edge (more specifically, the boundary of the top surface and side surface) of the docking location SBP. Through this process, for example, it is possible to extract the neighboring point set SG that includes each nearest neighbor point as shown in Figure 9A, and to obtain the docking location edge point cloud SP that includes each nearest neighbor point as shown in Figure 9B. Figure 9A is a diagram showing the nearest neighbor points for each vertical line in the docking location SBP. Figure 9B is a diagram showing an example of each nearest neighbor point obtained as the docking location edge point cloud by the extraction window EWA.

[0079] The docking parameter calculation unit 16 then generates a docking side line L by applying principal component analysis or the least squares method to each point to be measured (nearest neighbor) included in the docking location edge point cloud SP.

[0080] Furthermore, according to this generation method, the docking parameter calculation unit 16 calculates, for example, the average value of the z-coordinates of multiple points to be measured that correspond to the upper surface of the docking location SBP, and the coordinate position M f The z-coordinate value m fz Based on the above, the height h of the installation position of the forward marker FM when the upper surface of the docking area SBP is used as the reference point. mf The following may be calculated. Furthermore, according to this generation method, the docking parameter calculation unit 16 calculates, for example, the average value of the z coordinates of multiple points to be measured corresponding to the upper surface of the docking location SBP, and the coordinate position M r The z-coordinate value m rz Based on the above, the height h of the installation position of the rear marker RM when the upper surface of the docking area SBP is used as the reference point. mr The following may be calculated (see Figure 10A). Furthermore, according to this generation method, the docking parameter calculation unit 16 calculates, for example, the coordinate position M f and M r Based on this, the marker interval r corresponds to the distance between the front marker FM and the rear marker RM. mmay be calculated (see FIG. 10B). In addition, according to the present generation method, the berthing parameter calculation unit 16 calculates, for example, the distance w from the berthing side straight line L to the front marker FM mf and the distance w from the berthing side straight line L to the rear marker RM mr may be calculated (see FIG. 10C). In addition, according to the present generation method, the berthing parameter calculation unit 16 calculates the height h mf , the height h mr , the marker interval r m , the distance w mf , the distance w mr may be stored in a storage medium such as the memory 12. FIG. 10A is a diagram clearly showing the height h of the marker installation position mf and the height h mr . FIG. 10B is a top view clearly showing the marker interval r m . FIG. 10C is a top view clearly showing the distance w mf and the distance w mr .

[0081] In addition, according to the present generation method, instead of the predetermined margin Δw, the extraction window EWA may be generated using a margin w corresponding to (the actually measured value of) the distance from the berthing side straight line L to the front marker FM mf and a margin w corresponding to (the actually measured value of) the distance from the berthing side straight line L to the rear marker RM mr . In such a case, the extraction window EWA is defined by the coordinate position FWA[m fx +w mf m ry +w mf T , the coordinate position FWB[m fx +w mf m fy -w mf T , the coordinate position RWA[m rx -w mr m ry +w mr T and the coordinate position RWB[m rx -w mr m fy -w mr T ​​​​This is set as a roughly rectangular area with vertices and . In other words, the docking parameter calculation unit 16 can set the search range for the edge of the docking location SBP using the distance from the docking side line L to the forward marker FM and the distance from the docking side line L to the rear marker RM.

[0082] (Second generation method) This generation method describes an example where data corresponding to one of the two markers placed at the berthing location SBP is obtained, but data corresponding to the other marker is not obtained.

[0083] The berthing parameter calculation unit 16 obtains point cloud data CMG corresponding to one of the two markers CM provided at the berthing location SBP by, for example, extracting points to be measured with a predetermined intensity or higher from the point cloud data generated by the lidar 3. Furthermore, based on the centroid position of each point to be measured included in the point cloud data CMG, the berthing parameter calculation unit 16 calculates the coordinate position CP[m] of marker CM in the ship coordinate system. cx m cy m cz ] T The docking parameter calculation unit 16 obtains the average value of the z coordinates of multiple points to be measured corresponding to the upper surface of the docking location SBP, and the z coordinate value m of the coordinate position CP. cz Based on the above, the height h of the installation position of the marker CM when the upper surface of the docking location SBP is used as the reference point. mc Calculate.

[0084] The docking parameter calculation unit 16 calculates the height h mc And the height h calculated in the past mf and h mr By comparing and , it is determined whether marker CM is the front marker FM or the rear marker RM. Note that height h mf and h mr For example, this information should be stored in memory 12 as information related to assisting the docking of the target vessel.

[0085] The docking parameter calculation unit 16 calculates, for example, height h mc is height h mr Height h mf If it is close, it is determined that marker CM is the forward marker FM. Also, when the docking parameter calculation unit 16 determines that marker CM is the forward marker FM, it calculates the x-coordinate value m of coordinate position CP from the set of neighboring points SG. cx Each measurement point having an x-coordinate value less than or equal to the value obtained by adding a predetermined margin Δw to the value is extracted. Furthermore, the docking parameter calculation unit 16 extracts each measurement point detected as described above from the marker interval r whose distance to marker CM has been calculated in the past. m The docking location edge point cloud SP is obtained by extracting the measurement points that meet the following criteria. That is, if marker CM is determined to be the forward marker FM, the x-coordinate value x that satisfies both of the following equations (1) and (2) is obtained from the neighboring point set SG. i and y coordinate value y i Data containing these elements is acquired as a docking location edge point cloud SP. For example, if marker CM is determined to be a forward marker FM, a docking location edge point cloud SP including each nearest neighbor point, as shown in Figure 11, is acquired. Figure 11 shows an example of each nearest neighbor point acquired when marker CM is determined to be a forward marker.

[0086]

number

number

[0087] On the other hand, the docking parameter calculation unit 16 calculates, for example, height h mc is height h mf Height h mr If it is close, it is determined that marker CM is the rear marker RM. Also, if the docking parameter calculation unit 16 determines that marker CM is the rear marker RM, it calculates the x-coordinate value m of coordinate position CP from the set of neighboring points SG. cxEach measurement point having an x-coordinate value less than or equal to the value obtained by subtracting a predetermined margin Δw from the value is extracted. Furthermore, the docking parameter calculation unit 16 extracts from each measurement point as described above the distance to marker CM from the marker interval r that was previously calculated. m The docking location edge point cloud SP is obtained by extracting the measurement points that meet the following criteria. That is, if marker CM is determined to be the rear marker RM, the x coordinate value x that satisfies both equations (3) and (4) below is obtained from the neighboring point set SG. i and y coordinate value y i Data containing this information is acquired as a docking location edge point cloud SP.

[0088]

number

number

[0089] The docking parameter calculation unit 16 obtains the docking location edge point cloud SP by performing extraction processing using formulas (1) and (2) on the set of neighboring points SG obtained as described above, if marker CM is determined to be a forward marker FM, and by performing extraction processing using formulas (3) and (4) if marker CM is determined to be a rear marker RM.

[0090] The docking parameter calculation unit 16 then generates a docking side line L by applying principal component analysis or the least squares method to each point to be measured (nearest neighbor) included in the docking location edge point cloud SP.

[0091] According to the process described above, the docking parameter calculation unit 16 calculates the marker interval r m Based on the coordinate position CP of the marker CM, the search range for the edge point cloud of the docking location SBP can be set.

[0092] Furthermore, according to this generation method, for example, the marker interval r m If information indicating the marker interval r is not stored in memory 12, mAlternatively, the total length of the target vessel (length in the x-direction) may be applied to the above formulas (2) and (4) to obtain the berthing location edge point cloud SP. For example, the quay map DB is referenced to obtain information on the upper limit of the total length of vessels that can berth. If the total length of the target vessel is shorter than the obtained upper limit, it can be determined that the target vessel can berth. Therefore, the marker spacing of the quay is considered to be at least longer than the total length of the target vessel. Thus, by using the total length of the target vessel for the search, the search range is narrowed, but it is possible to prevent the extraction of unnecessary point clouds. That is, the berthing parameter calculation unit 16 uses the marker spacing r m Instead, the overall length of the target vessel can be used to set the search range for the edge of the berthing location SBP.

[0093] Furthermore, according to this generation method, in the above formula (1), instead of the predetermined margin Δw, a margin w corresponding to the distance (measured value) from the straight line L on the shore to the forward marker FM is used. mf The following may also be applied. Furthermore, according to this generation method, in the above formula (3), instead of the predetermined margin Δw, a margin w corresponding to the distance (measured value) from the docking side line L to the rear marker RM is used. mr The following may also be applied: The docking parameter calculation unit 16 can use the distance from the docking side line L to the marker CM to set the search range for the edge of the docking location SBP.

[0094] Furthermore, according to this generation method, for example, if marker information MJ, which is information relating to a marker installed at a docking location, is stored in a storage medium such as memory 12, the docking parameter calculation unit 16 may perform processing using the information contained in marker information MJ.

[0095] The marker information MJ includes information about the forward marker FM, such as the installation location (latitude, longitude, and elevation) and the height from the top of the quay (height h). mf (equivalent to), distance from the quay (distance w mfIt is sufficient if the marker information MJ includes information about the rear marker RM, such as the installation location (latitude, longitude, and elevation) and the height from the top of the quay (height h). mr (equivalent to), distance from the quay (distance w mr It is sufficient if the marker information MJ includes information related to the distance between markers (equivalent to), shape, and size. m It is sufficient if the information includes information related to (equivalent to) the above.

[0096] (Third generation method) This generation method describes an example where data corresponding to one of two markers placed at the berthing location SBP is obtained, but data corresponding to the other marker is not obtained, and it is not possible to determine whether the one marker is the forward marker FM or the rear marker RM.

[0097] The shoreing parameter calculation unit 16, assuming that marker CM is forward marker FM, performs the same processing as described in the second generation method to obtain x coordinate values ​​x that satisfy both equations (1) and (2) above. i and y coordinate value y i The data of each measured point having the marker CM is acquired as a docking location edge point cloud SPD. The docking parameter calculation unit 16 then applies principal component analysis or least squares method to each measured point (nearest neighbor) included in the docking location edge point cloud SPD to generate a straight line LD, which is a straight line representing the edge of the docking location SBP, assuming that marker CM is a forward marker FM. Through this process, for example, a docking location edge point cloud SPD with each nearest neighbor point as shown in Figure 12 is acquired. Furthermore, through the above process, for example, a straight line LD as shown in Figure 12 is generated. Figure 12 is a diagram showing an example of a straight line generated when marker CM is assumed to be a forward marker.

[0098] The shoreing parameter calculation unit 16, assuming that marker CM is the aft marker RM, performs the same processing as described in the second generation method to obtain an x-coordinate value x that satisfies both equations (3) and (4) above. i and y coordinate value y i The data of each measured point having the marker CM is acquired as a docking location edge point cloud SPE. The docking parameter calculation unit 16 then applies principal component analysis or least squares method to each measured point (nearest neighbor) included in the docking location edge point cloud SPE to generate a straight line LE, which is a straight line representing the edge of the docking location SBP assuming that marker CM is a rear marker RM. Through this process, for example, a docking location edge point cloud SPE with each nearest neighbor point as shown in Figure 13 is acquired. Furthermore, through the above process, for example, a straight line LE as shown in Figure 13 is generated. Figure 13 is a diagram showing an example of a straight line generated when marker CM is assumed to be a rear marker.

[0099] The docking parameter calculation unit 16 calculates the variance VD for each measured point included in the docking location edge point cloud SPD with respect to the straight line LD, and the variance VE for each measured point included in the docking location edge point cloud SPE with respect to the straight line LE. Then, by comparing the variances VD and VE, the docking parameter calculation unit 16 selects the straight line with the relatively smaller calculated variance as the docking side straight line L along the edge of the docking location SBP. In the example shown in Figures 12 and 13, the variance VE is smaller than the variance VD, so the straight line LE is selected as the docking side straight line L.

[0100] Furthermore, even in the case of a single marker, where only one marker is placed in the area where docking is to take place, the map data in memory 12 contains information indicating that it is a single marker, as well as the forward distance d from the marker. mf and rear distance d mr If the information is stored as information relating to the docking area, the docking location edge point cloud SP can be obtained. Note that the forward distance d mf and rear distance d mrThis will be explained later. As shown in Figure 16, the docking parameter calculation unit 16 reads marker information of the docking target area from the map data, recognizes that it is a single marker, and, for the set of neighboring points SG obtained as described above, it calculates the coordinates of the marker CM and the forward distance d from the marker. mf and rear distance d mr Using this method, extraction processes are performed according to the following formulas (5) and (6), and extraction processes according to (7) and (8). By combining the results of these extraction processes, the shore location edge point cloud SP is obtained. Figure 16 shows an example of the process for obtaining the shore location edge point cloud.

[0101]

number

number

number

number

[0102] [Processing flow] Figure 14A is a flowchart illustrating the overview of the docking support process in this embodiment. The information processing device 1 repeatedly executes the processes shown in the flowchart of Figure 14A.

[0103] First, the information processing device 1 acquires point cloud data in the direction of the berthing location (step S11). In this case, the information processing device 1 acquires, for example, point cloud data generated by a lidar 3 that includes the berthing side of the target vessel within its measurement range. The information processing device 1 may also further downsample the acquired point cloud data and remove data reflected from the water surface.

[0104] Next, the docking location detection unit 15 of the information processing device 1 calculates normal vectors based on the point cloud data acquired in step S11 (step S12). Furthermore, in step S12, the docking location detection unit 15 calculates the number of normal vectors and the variance of the normal vectors. Also, based on the processing results of step S12, the docking location detection unit 15 identifies the inner surface of the field of view and the detection surface (step S13).

[0105] Next, the docking parameter calculation unit 16 performs a nearest neighbor search for each vertical line based on the point cloud data acquired in step S11 to determine the nearest neighbor point for each vertical line (step S14).

[0106] Next, the docking parameter calculation unit 16 acquires a docking location edge point cloud including the nearest neighbor point for each vertical line, based on one of the first to third generation methods described above (step S15).

[0107] Here, an example of the process performed in step S15 will be explained with reference to Figure 14B. Figure 14B is a flowchart showing an example of the process related to acquiring the shore location edge point cloud.

[0108] The docking parameter calculation unit 16 detects a marker placed at the docking location (step S31), and then obtains map data or previously calculated marker information (marker height, distance from the edge of the quay, distance between markers, etc.) from the memory 12 (step S32).

[0109] The shoreing parameter calculation unit 16 determines the number of markers detected in step S31 and the positions of the markers detected in step S31 based on the map data or marker information acquired in step S32 (step S33).

[0110] Based on the identification result of step S33, the docking parameter calculation unit 16 determines whether there is one or two markers detected in step S31 (step S34).

[0111] If the docking parameter calculation unit 16 detects one marker in step S31, it performs the process described in step S36 below. If the docking parameter calculation unit 16 detects two markers in step S31, it sets the search range for the edge of the docking location using the extraction window (step S35), and then performs the process described in step S41 below.

[0112] Based on the identification result in step S33, the docking parameter calculation unit 16 determines whether the marker installed at the docking location is a single marker or not (step S36).

[0113] The berthing parameter calculation unit 16 performs the process described in step S38 below if the marker installed at the berthing location is not a single marker (step S36: No). Also, if the berthing parameter calculation unit 16 is a single marker, it calculates the forward distance d included in the map data acquired in step S32. mf and rear distance d mr After setting the search range using (step S37), the process of step S41 described below is performed. The docking parameter calculation unit 16 calculates, for example, the forward distance d in the map data obtained in step S32. mf and rear distance d mr If the data is not included, or if map data has not been obtained by step S32, the process described in step S38 below is performed.

[0114] Based on the identification result of step S33, the shore-docking parameter calculation unit 16 determines whether the marker detected in step S31 can be identified as a forward marker or a rear marker (step S38).

[0115] The shoreing parameter calculation unit 16 determines the marker interval r if the marker detected in step S31 can be identified as a forward marker or a rear marker (step S38: Yes). mAfter setting the search range using the x-coordinate of the marker position (step S39), the process of step S41 described below is performed. Furthermore, if the marker detected in step S31 cannot be identified as a forward marker or a rear marker (step S38: No), the docking parameter calculation unit 16 sets two types of search ranges corresponding to the cases where the marker is a forward marker and where it is a rear marker, respectively (step S40), and then performs the process of step S41 described below.

[0116] The docking parameter calculation unit 16 performs a process to acquire a docking location edge point cloud using the search range set by the processing up to step S40 (step S41). Then, using the processing results up to step S41, the docking parameter calculation unit 16 updates the marker information (marker height, distance from the edge of the quay, distance between markers, etc.) stored in the memory 12 (step S42), and then continues the processing from step S16 onward.

[0117] The docking parameter calculation unit 16 generates a docking side line L using the docking location edge point cloud including the nearest neighbor point for each vertical line obtained in step S15 (step S16). At this time, if one marker is detected whose position is unknown (either forward or backward), there is a quay edge point cloud assuming it is forward and a quay edge point cloud assuming it is backward. Therefore, a line is generated using both sets of markers, the variance of the quay edge point clouds for each line is calculated, and the line with the smaller variance value is selected as the docking side line L.

[0118] Next, the docking parameter calculation unit 16 uses the docking side line L calculated in step S16 to calculate the docking parameters, which are the distance to the opposite shore, the approach angle, and the docking speed (step S17).

[0119] Then, the berthing parameter calculation unit 16 generates confidence information based on the results of identifying the inner surface of the field of view and the detection surface in step S13 and the calculation results of the berthing parameters in step S17 (step S18). Subsequently, the information processing device 1 controls the vessel based on the confidence information (step S19). As a result, the information processing device 1 can accurately perform vessel control related to berthing based on a confidence level that accurately reflects the berthing situation.

[0120] The information processing device 1 then determines whether or not the target vessel has docked (step S20). In this case, the information processing device 1 determines whether or not the target vessel has docked based, for example, on the output signals of the sensor group 2 or user input via the interface 11. If the information processing device 1 determines that the target vessel has docked (step S20; Yes), it terminates the flowchart process. On the other hand, if the information processing device 1 determines that the target vessel has not docked (step S20; No), it returns to step S11.

[0121] According to the process described above, the acquisition means acquires measurement data generated by a measuring device installed on the ship. Also according to the process described above, the marker position acquisition means acquires the position of at least one of the two markers installed at the berthing location based on the measurement data. Also according to the process described above, the search range setting means sets the search range of the edge of the berthing location based on the position of at least one marker.

[0122] In the descriptions of the first to third generation methods and the flowchart above, a set of nearest neighbor points was generated by collecting the nearest neighbor search points for each vertical line, and then the docking location edge point cloud was extracted using markers. However, the order can be changed, and the target point cloud can be extracted from the point cloud data of LIDA 3 using markers, and then the docking location edge point cloud can be extracted by performing nearest neighbor search for each vertical line on the extracted point cloud data. The same result can be obtained with that order as well.

[0123] Here, for example, if the lidar captures a quay surface that is not the target, the point cloud of berthing location edges will not be aligned in a single straight line. As a result, even if a straight line is found using principal component analysis or least squares method, it will not be possible to generate the correct edge line of the target quay (see Figure 15A). Furthermore, if an incorrect edge line is generated, there is a risk that an incorrect distance will be calculated. If an incorrect edge line is used as the berthing side line L, calculating the distance by finding the perpendicular to that line will not result in the correct distance to the opposite shore. In contrast, according to this embodiment, even if there are other quay surfaces near the berthing quay, only the point cloud located between the markers is used, so a correct straight line can be generated (see Figure 15B). Therefore, by calculating the distance of the perpendicular to the correctly generated berthing side line L, the correct distance to the opposite shore can be determined. Also, according to this embodiment, for example, even if the lidar captures a ship at anchor and mistakenly identifies its side as a wall, those point clouds are excluded by using markers, so a correct straight line can be generated (see Figure 15C). Therefore, even in such cases, the correct distance to the opposite shore can be determined by calculating the distance of the perpendicular to the correctly generated shoreline L. Figure 15A shows an example where an incorrect edge line is generated. Figures 15B and 15C show examples where a correct edge line is generated. Accordingly, according to this embodiment, based on the shoreline generated as described above, parameters used in shore landing assistance, such as distance, speed, and angle, can be calculated with high accuracy.

[0124] If there are three or more markers in the area where the vessel should dock, then only the markers at both the front and rear ends of the docking area need to be targeted.

[0125] Furthermore, to distinguish whether a marker is in front of or behind a marker, we varied the marker height relative to the top of the quay, but other methods are also possible. For example, the size, shape, and orientation of the front and rear markers could be different. By determining the size, shape, and orientation of a marker from the marker point cloud data detected by LIDA3, it is possible to distinguish whether a marker is in front of or behind a marker. In that case, information such as the size, shape, and orientation of the markers may be stored in the map database.

[0126] <Second Example> Next, a second embodiment will be described below. In this embodiment, explanations of parts to which the same configuration as in the first embodiment can be applied will be omitted as appropriate, and the explanation will focus on parts that differ from the first embodiment. Specifically, in this embodiment, the system configuration, hardware configuration, and functional configuration are substantially the same as in the first embodiment, but the content of the processing performed by the berthing parameter calculation unit 16 differs from that of the first embodiment. Therefore, the explanation below will mainly focus on the processing performed by the berthing parameter calculation unit 16.

[0127] [Processing related to the detection of the docking area] The berthing parameter calculation unit 16 performs the same processing as described above in the first embodiment, thereby determining the coordinate position M of the forward marker FM in the ship coordinate system. f [m fx m fy m fz ] T And the coordinate position M of the rear marker RM in the ship's coordinate system. r [m rx m ry m rz ] T The and are obtained respectively. The docking parameter calculation unit 16 generates the docking side line L by performing the same processing as described above in the first embodiment. The docking parameter calculation unit 16 also obtains the coordinate position M f and M rBased on this, the rectangular area between the front end FE and the rear end RE (see Figure 17A) is identified as the docking area. The docking parameter calculation unit 16 also determines the coordinate position M f and M r Based on the docking side line L and the distance d from the origin O of the hull coordinate system to the forward end of the docking area, f and the distance d from the origin O to the rear end of the shore-docking area. r The following is calculated. The origin O corresponds to the reference point of the target vessel, and can be set, for example, at the center position or center of gravity position of the target vessel. The berthing area is the area where the target vessel should be located when berthing at the berthing location. Figure 17A shows the origin O and the distance d f and distance d r This is a top view clearly showing the front end FE and the rear end RE.

[0128] Here, the distance d f and distance d r A specific example of the process for calculating the distance d will be explained with reference to Figures 17A and 17B. Figure 17B shows the distance d f and distance d r This figure clearly shows the unit vector "u" used in the calculation.

[0129] The shoreing parameter calculation unit 16 generates the shoreing side line L shown in the following formula (9) by performing the processing described above in the first embodiment.

[0130]

number

[0131] Here, "[x0y0z0] T '' indicates the centroid of the measured point shown by the docking location edge point cloud, and "[abc] T " indicates the direction vector, and "t" indicates the parameter. For example, when performing principal component analysis, the eigenvector corresponding to the largest eigenvalue becomes the direction vector of the shoreline line L.

[0132] The docking parameter calculation unit 16 then calculates a unit vector u that is perpendicular to the docking side line L in a two-dimensional plane, as shown in the following formula (10). The unit vector u is represented as a vector originating from the origin O, as shown in Figure 17B.

[0133]

number

[0134] Coordinate position M f The dot product of the (x,y) component and the unit vector u is given by the coordinate position M f This corresponds to the projection onto the vector u. Therefore, the distance "d" shown in Figure 17B f The following equation (11) holds true for ".

[0135]

number

[0136] Therefore, the docking parameter calculation unit 16 calculates the distance d f This is calculated based on the following formula (12).

[0137]

number

[0138] Furthermore, the docking parameter calculation unit 16 uses the same method as described above to calculate the distance d r This is calculated based on the following formula (13). According to formulas (12) and (13), when the origin O is between FE and RE as shown in Figure 17A, the distance d f distance d r Both values ​​are positive, and when the origin O is located ahead of FE, the distance d f The distance d is a negative value. r The value is positive, and when the origin O is located behind RE, the distance d f d is a positive value for distance d r This will result in a negative value.

[0139]

number

[0140] [Processing flow] The processing flow in this embodiment is substantially the same as the processing flow (Figure 14A) described in the first embodiment. On the other hand, in this embodiment, for example, in step S17, the distance d is calculated using the straight line L of the shore side generated in step S16. f and distance d r The process should be performed to calculate the distance d. f and distance d r This can be used to calculate the docking parameters in step S17.

[0141] As described above, according to this embodiment, the distance from the current position of the target vessel to the edge of the berthing area can be calculated with high accuracy as a parameter used in berthing assistance.

[0142] If there are three or more markers in the area where the vessel should dock, then only the markers at both the front and rear ends of the docking area need to be targeted.

[0143] Furthermore, even if only one marker is placed in the area where docking should occur, the map data in memory 112 will include the forward distance d from the marker. mf and rear distance d mr If it is stored as the docking area, then distance d f and distance d r The berthing parameter calculation unit 16 can calculate the forward distance d from the marker when a ship is about to dock. As shown in Figure 18, when a ship is about to dock, the berthing parameter calculation unit 16 refers to marker information from map data and calculates the forward distance d from the marker. mf and rear distance d mr Obtain the distance d of the marker along the line L. m The distance d is calculated and obtained. mf distance d mr From, distance d f distance d rThis is calculated using the following formulas (14) and (15). Figure 18 shows the distance d f and distance d r This figure shows an example of the process involved in the calculation of [the value].

[0144]

number

number

[0145] In each of the embodiments described above, the program can be stored using various types of non-transitory computer-readable medium and supplied to a control unit, etc., which is a computer. Non-transitory computer-readable medium includes various types of tangible storage medium. Examples of non-transitory computer-readable medium include magnetic storage medium (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage medium (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)).

[0146] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made that are understandable to those skilled in the art within the scope of the present invention. That is, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that those skilled in the art could make in accordance with the technical idea. Furthermore, each disclosure of the above-mentioned patent documents and other references is incorporated herein by reference. [Explanation of Symbols]

[0147] 1. Information Processing Device 2 Sensor Groups 3 Riders

Claims

[Claim 1] An acquisition means for acquiring measurement data generated by a measuring device installed on a ship, A marker position acquisition means that acquires the position of at least one of the markers installed at the docking location based on the aforementioned measurement data, A search range setting means that sets the search range of the point cloud of the edge portion of the docking location based on the position of at least one marker, An information processing device having

Images