Method and apparatus for controlling autonomous berthing of ship

Abstract

The present disclosure relates to a method and an apparatus for controlling autonomous berthing of a ship. The method for controlling autonomous berthing of a ship according to an embodiment of the present disclosure may comprise: receiving images and posture data collected through a camera; deriving obstacle information around the ship on the basis of the images and posture data; and deriving a berthing area on the basis of the obstacle information.

Description

Method and device for autonomous berthing control of a vessel

[0001] The present invention relates to a method and device for autonomous berthing control of a ship.

[0002] Typically, users of small vessels operate the steering wheel and throttle to navigate, berth, or unberth their vessels. However, due to the inexperience of users of recreational vessels and the influence of environmental disturbances such as currents and winds, vessels often experience difficulties navigating, especially when berthing or unberthing in confined spaces. These difficulties hinder the general public's access to operating or manoeuvring recreational vessels.

[0003] Accordingly, various technologies to assist ship operation are being actively developed.

[0004] The background technology described above is technical information that the inventor possessed for the purpose of deriving the present invention or acquired in the process of deriving the present invention, and cannot necessarily be considered as publicly known technology disclosed to the general public prior to the application for the present invention.

[0005] The purpose of the present disclosure is to provide a method and device for autonomous ship berthing control. The problems addressed by the present disclosure are not limited to those mentioned above. Other problems and advantages of the present disclosure not mentioned above can be understood through the following description and will be more clearly understood through the embodiments of the present disclosure. Furthermore, it will be appreciated that the problems and advantages addressed by the present disclosure can be realized by the means and combinations thereof set forth in the claims.

[0006] A first aspect of the present disclosure may provide an autonomous berthing control method, including: receiving images and attitude data collected through a camera; calculating obstacle information around a vessel based on the images and attitude data; calculating a berthing area based on the obstacle information; and displaying the berthing area.

[0007] A second aspect of the present disclosure provides an autonomous docking control device comprising: a memory having at least one program stored therein; and a processor that operates by executing the at least one program; wherein the processor receives images and attitude data collected through a camera, calculates obstacle information around the vessel based on the images and attitude data, calculates a docking area based on the obstacle information, and displays the docking area.

[0008] A third aspect of the present disclosure can provide a computer-readable recording medium having recorded thereon a program for executing the method according to the first aspect on a computer.

[0009] According to various embodiments of the present disclosure, autonomous eye contact can be provided to a user using only data collected through a camera, without expensive equipment such as a lidar sensor.

[0010] Figure 1 is a conceptual diagram for explaining the autonomous navigation system of the present disclosure.

[0011] FIG. 2 is a flowchart illustrating the overall eye control process according to one embodiment of the present disclosure.

[0012] FIG. 3 is a drawing specifically illustrating a process for calculating a viewing area according to one embodiment of the present disclosure.

[0013] FIG. 4 is a diagram for explaining a process of processing an acquired image according to one embodiment of the present disclosure.

[0014] FIG. 5 illustrates an example for explaining a process of processing an acquired image according to one embodiment of the present disclosure.

[0015] FIG. 6 is a diagram for explaining a process of generating obstacle information according to one embodiment of the present disclosure.

[0016] FIG. 7 illustrates an example for explaining a process for generating obstacle information according to one embodiment of the present disclosure.

[0017] FIG. 8 is a flowchart illustrating a process for estimating a berth point according to one embodiment of the present disclosure.

[0018] FIGS. 9A to 9D illustrate examples for explaining a process of estimating a berth point according to one embodiment of the present disclosure.

[0019] FIG. 10 is a flowchart illustrating a process for calculating a viewing area according to one embodiment of the present disclosure.

[0020] FIGS. 11A to 11D illustrate examples for explaining a process for calculating an eyepiece area according to one embodiment of the present disclosure.

[0021] FIG. 12 is a flowchart of a method for controlling autonomous berthing of a ship according to one embodiment of the present disclosure.

[0022] FIG. 13 is a flowchart of a method for generating information for autonomous docking of a vessel according to one embodiment of the present disclosure.

[0023] FIG. 14 is a flowchart of a method for calculating a berthing area of ​​a vessel according to one embodiment of the present disclosure.

[0024] FIG. 15 is a block diagram of a device according to one embodiment of the present disclosure.

[0025] A method according to one embodiment of the present invention for solving the above technical problem is an autonomous berthing control method, comprising: a step of receiving images and attitude data collected through a camera; a step of calculating obstacle information around a vessel based on the images and attitude data; a step of calculating a berthing area based on the obstacle information; and a step of displaying the berthing area.

[0026] In the above method, the step of receiving a contact point and a contact method is further included; and the contact area can be calculated based on the contact point and the contact method.

[0027] In the above method, the method may further include a step of calculating control logic based on the contact area; and a step of generating a control command based on the control logic.

[0028] In the above method, the contact area can be updated according to a preset cycle.

[0029] In the above method, the image may include one or more images collected by one or more cameras having different viewpoints centered on the vessel.

[0030] In the above method, the step of generating obstacle information may include a step of generating probability data for an obstacle corresponding to each of the one or more images; and a step of generating the obstacle information by matching the probability data.

[0031] In the above method, the step of calculating the anchorage area may include: a step of determining a berth area based on the obstacle information; and a step of generating a boundary area including two obstacle points corresponding to the berth entrance based on the berth area.

[0032] In the above method, the step of calculating the anchorage area may further include the step of calculating an outer boundary area, which is a safety area secured in the opposite direction to the boundary area and includes two obstacle points corresponding to the berth entrance.

[0033] In the above method, the step of displaying the berthing area may further include a step of displaying the berthing area using a square reflecting the size of the vessel so that the center of the square corresponds to the berthing point.

[0034] In the above method, the method may further include a step of determining whether an obstacle exists between the vessel and the berthing point; and a step of stopping autonomous berthing based on determining that an obstacle exists between the vessel and the berthing point.

[0035] In the above method, the step of receiving the berthing point and berthing method may include the step of storing information on a berthing target point including the latitude and longitude of the berthing point when the vessel is berthed; the step of displaying information on the berthing target point as a candidate berthing point to a user; and the step of receiving a selection input regarding one of the candidate berthing points from the user.

[0036] In the above method, the step of receiving the berthing point and berthing method may include the step of displaying the berthing area as a candidate berthing point to the user while the vessel is in operation; the step of receiving a selection input regarding one of the candidate berthing points from the user; and the step of storing the latitude and longitude of the selected candidate berthing point as information regarding the berthing target point.

[0037] In the above method, if information about the point where the docking is completed is not stored based on the completion of docking, the method may further include a step of storing at least some of the latitude and longitude of the point where the docking is completed, the type of berth, the entrance direction of the berth, the docking direction, and the docking method.

[0038] According to another embodiment of the present invention for solving the above technical problem, the autonomous berthing control device comprises: a memory in which at least one program is stored; and a processor that operates by executing the at least one program; wherein the processor receives one or more images collected through one or more cameras having different viewpoints centered on the ship and attitude data of the ship, calculates obstacle information around the ship by matching the one or more images based on the attitude data, and calculates a berthing area including two obstacle points corresponding to a berth entrance based on the obstacle information, and displays the berthing area.

[0039] One embodiment of the present invention can provide a computer-readable recording medium having recorded thereon a program for executing the above method on a computer.

[0040] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments presented below, but can be implemented in various different forms, and it should be understood that it includes all transformations, equivalents, and substitutes included in the spirit and technical scope of the present invention. The embodiments presented below are provided to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the invention of the scope of the invention. In describing the present invention, if a detailed description of a related known technology is judged to obscure the gist of the present invention, the detailed description thereof will be omitted.

[0041] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0042] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a given function. Furthermore, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented by algorithms that execute on one or more processors. Furthermore, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and / or data processing. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations.

[0043] Additionally, the connecting lines or connecting members between components depicted in the drawings are merely exemplary representations of functional connections and / or physical or circuit connections. In an actual device, connections between components may be represented by various functional connections, physical connections, or circuit connections that may be replaced or added.

[0044] Additionally, terms including ordinal numbers, such as "first" or "second," used in the specification may be used to describe various components, but the components should not be limited by the terms. The terms may be used to distinguish one component from another.

[0045] Figure 1 is a conceptual diagram for explaining an autonomous navigation system according to the present disclosure.

[0046] Referring to FIG. 1, the autonomous navigation system (100) may include a ship's control device (110), an EIU (Engine Interface Unit 120), an Autonomous Navigation Processor (130, hereinafter, an autonomous navigation processing unit), and an engine (140).

[0047] The steering device (110) may include at least a portion of a throttle lever, a steering wheel, and a joystick. However, the present invention is not limited thereto, and the steering device (110) may include other ship devices. The steering device (110) may be referred to as a helm station in some embodiments.

[0048] The EIU (120) may refer to a device that acquires a signal (S1) from a steering device (110) included in a ship through a communication network within the ship and transmits it to the engine (140). The signal (S1) may include a message or protocol of several steering devices (110) included in the ship. The EIU (120) may transmit the signal (S1) of the steering device (110) as is to the engine (140), or may convert the signal (S1) of the steering device (110) and then transmit the converted signal (S2) to the engine (140). The present invention is not limited thereto.

[0049] The EIU (120) may be a device that enables switching between the autonomous navigation mode (or automatic navigation mode) and the manual navigation mode of the ship by transmitting or injecting a signal (S1) of the steering device (110). The EIU (120) may determine whether the current navigation mode of the ship is the autonomous navigation mode or the manual navigation mode based on a control command received from the autonomous navigation processing device (130). The EIU (120) may also receive the navigation status of the ship determined from the autonomous navigation processing device (130). In this case, the EIU (120) may operate based on the determined navigation status. According to an embodiment, the EIU (120) may determine the autonomous navigation mode by detecting a control value output from the steering device (110) even when the user does not control the steering devices (110).

[0050] The EIU (120) can be connected to the ship's steering device (110) and engine (140) via an internal communication network. The internal communication network can include a CAN (Controller Area Network). Here, CAN can refer to an internal communication network of a ship, automobile, etc. that can perform data transmission between ECUs (Engine Control Units), control of various steering devices (110), control of a system, etc., and is not limited thereto. The internal communication network can refer to any communication network that can transmit data between the ship's steering device (110) and the engine (140). According to an embodiment, a user can check the status of the linkage between the steering devices (110), the autonomous navigation processing unit (130), and the engine (140) by using the EIU (120).

[0051] The autonomous navigation processing unit (130) may be a device that processes control commands for controlling the ship's steering device (110) when the ship is in autonomous navigation mode. In other words, even when the user does not operate the steering device (110), the autonomous navigation processing unit (130) may generate commands for controlling the ship's steering device (110). In addition, the autonomous navigation processing unit (130) may transmit the ship's navigation status to the EIU (120) so that the EIU (120) may operate based on the ship's navigation status.

[0052] The engine (140) may be a device that operates based on control commands from the ship's steering devices (110). For example, when the ship is in manual navigation mode, the engine (140) may operate based on user input operating the steering device (110). As another example, when the ship is in autonomous navigation mode, the engine (140) may operate based on control commands received from the autonomous navigation processing device (130).

[0053] Meanwhile, various embodiments of the present disclosure may be understood to be performed by the autonomous navigation processing device (130) of FIG. 1 or by a part of the autonomous navigation processing device (130) of FIG. 1.

[0054] FIG. 2 is a flowchart illustrating the overall eye control process according to one embodiment of the present disclosure.

[0055] The autonomous docking control process illustrated in Fig. 2 can be performed by the autonomous navigation processing device (130) described above.

[0056] At step 210, in one embodiment, the autonomous navigation processing device (130) may receive surrounding images and position data of the vessel.

[0057] In one embodiment, the autonomous navigation processing unit (130) may receive images of the vessel's surroundings captured by cameras mounted on the vessel. The vessel may be equipped with one or more cameras. The one or more cameras may capture images from different viewpoints centered around the vessel. In one embodiment, the vessel may be equipped with at least four cameras, each having a field of view for the front, rear, left, and right sides centered around the vessel. In one embodiment, the one or more cameras mounted on the vessel may be cameras attached to the outer plating of the vessel's hull. In one embodiment, images captured by the one or more cameras mounted on the vessel may include a curved water surface boundary.

[0058] In one embodiment, the autonomous navigation processing unit (130) may receive location data collected by sensors mounted on the vessel. For example, the location data may be GPS data.

[0059] In one embodiment, the autonomous navigation processing unit (130) may receive additional ship attitude data, or may receive attitude data instead of position data. The attitude data may be IMU data.

[0060] At step 220, in one embodiment, the autonomous navigation processing device (130) may receive a docking point and a docking method.

[0061] In one embodiment, the berthing point and / or berthing method may be input by the user. For example, the autonomous navigation processing unit (130) may display the vessel's surroundings or one or more berthing points and / or berthing methods to the user via a map or other means through the vessel's input / output device, and the user may select the berthing point and / or berthing method through the input / output device. The input / output device may transmit the user's input for selecting the berthing point and / or berthing method to the autonomous navigation processing unit (130).

[0062] In other embodiments, the eye contact points and / or eye contact methods may be preset. In one embodiment, the eye contact method may be automatically set to match a specific eye contact point.

[0063] In one embodiment, the eyepiece arrangement may include a front eyepiece, a rear eyepiece, and a side eyepiece.

[0064] At step 230, in one embodiment, the autonomous navigation processing device (130) may calculate a landing area.

[0065] In one embodiment, the autonomous navigation processing device (130) can calculate a berthing area based on the received berthing point and berthing method. In the present disclosure, the berthing area refers to the area surrounding the berthing point, and may refer to an area associated with the berthing of a vessel. The berthing area of ​​the present disclosure will be described in detail later.

[0066] At step 240, in one embodiment, the autonomous navigation processing unit (130) may produce control logic.

[0067] In one embodiment, the autonomous navigation processing device (130) may calculate control logic based on the berthing point, berthing method, and berthing area. The control logic may refer to logic for reducing the difference between the current vessel position and attitude and the target vessel position and attitude. Based on the berthing point, berthing method, and berthing area, the target vessel position and attitude may be calculated. The target vessel position and attitude may be calculated stepwise or sequentially.

[0068] At step 250, in one embodiment, the autonomous navigation processing unit (130) may generate and transmit a control command.

[0069] In one embodiment, the autonomous navigation processing unit (130) may generate control commands for controlling the vessel based on the calculated control logic. The control commands may refer to commands for controlling the vessel's engine output and / or steering wheel angle, etc., necessary for controlling the vessel according to the calculated control logic.

[0070] FIG. 3 is a drawing specifically illustrating a process for calculating a viewing area according to one embodiment of the present disclosure.

[0071] The process of calculating the eye contact area illustrated in FIG. 3 may correspond to the process of calculating the eye contact area (220) of FIG. 2, and may be understood to be performed by the autonomous navigation processing device (130).

[0072] At step 310, in one embodiment, the autonomous navigation processing device (130) may estimate an obstacle.

[0073] In one embodiment, the autonomous navigation processing unit (130) can estimate obstacles based on image and sensor data.

[0074] In one embodiment, the images may be collected by one or more cameras mounted on the vessel.

[0075] In one embodiment, one or more sensors may be mounted on the vessel, and one or more sensors may transmit sensor data to the autonomous navigation processing unit (130). For example, the sensor data may include attitude data. The attitude data may be data from an inertial measurement unit (IMU). Since the vessel is above the water surface, the images collected through the camera may need to be corrected depending on the vessel's attitude. In this case, the images collected through the camera may be corrected based on sensor data from the inertial measurement unit.

[0076] A specific embodiment of how the autonomous navigation processing device (130) estimates obstacles will be described in detail later.

[0077] At step 320, in one embodiment, the autonomous navigation processing device (130) may estimate the landing area.

[0078] In one embodiment, the autonomous navigation processing device (130) may generate obstacle information as a result of obstacle estimation. In one embodiment, the autonomous navigation processing device (130) may estimate the contact area based on the obstacle information.

[0079] In one embodiment, the autonomous navigation processing unit (130) can estimate the docking area based on location data. For example, sensor data may include location data. The location data may be GPS data. For example, the location data may include location data of the vessel and location data of the docking point.

[0080] In one embodiment, the autonomous navigation processing device (130) may generate tethering area data as a result of tethering area estimation.

[0081] A specific embodiment of how the autonomous navigation processing device (130) estimates the docking area will be described in detail later.

[0082] FIG. 4 is a diagram for explaining a process of processing an acquired image according to one embodiment of the present disclosure.

[0083] It can be understood that the process of processing the acquired image illustrated in Fig. 4 is performed by the autonomous navigation processing device (130).

[0084] The process of processing the acquired image illustrated in FIG. 4 can be understood as part of the aforementioned obstacle estimation process. That is, the autonomous navigation processing device (130) estimating the obstacle may include processing the acquired image.

[0085] Referring to FIG. 4, in one embodiment, the autonomous navigation processing device (130) may include a segmentation unit (410), a boundary detection unit (420), and a synthesis unit (430). Each component included in the autonomous navigation processing device (130) illustrated in FIG. 4 may be understood as functional blocks corresponding to functions performed by the processor of the autonomous navigation processing device (130).

[0086] In one embodiment, the autonomous navigation processing device (130) can perform segmentation on the image (401).

[0087] Referring to FIG. 4, an image (401) is input to a segmentation unit (410), and the segmentation unit (410) generates and outputs segment data (402) as a result of segmentation.

[0088] Segmentation may involve analyzing an image pixel by pixel to segment objects or regions. In one embodiment, the autonomous navigation processing unit (130) or segmentation unit (410) may include a segmentation model for performing segmentation. For example, the segmentation model may be U-net, mask R-CNN, etc.

[0089] In one embodiment, the segment data (402) may include data about the class to which each pixel included in the image belongs. That is, the segment data (402) may be a pixel-by-pixel class map.

[0090] In one embodiment, the segment data (402) may further include data regarding pixels representing boundaries between regions where classes are classified. In other words, based on data regarding the class to which each pixel included in the image belongs, the image may include regions where classes are classified, and in this case, the segment data (402) may further include data regarding whether a pixel corresponds to a boundary between regions where classes are classified. In other words, the segment data (402) may be a pixel-by-pixel inter-class boundary probability map. Whether a pixel corresponds to a boundary may be determined based on whether the probability for the first class and the probability for the second class of the pixel are similar. For example, if the difference between the probability for the first class and the probability for the second class of the pixel is less than or equal to a threshold value, the probability for the first class and the probability for the second class may be determined to be similar, and the pixel may be determined to correspond to a boundary.

[0091] In one embodiment, the autonomous navigation processing device (130) can detect a boundary from an image (401).

[0092] Referring to FIG. 4, an image (401) is input to a boundary detection unit (420), and the boundary detection unit (420) generates and outputs boundary map data (403) as a result of boundary detection.

[0093] Edge detection is a technique for finding the outline of an object in an image, and can be performed by analyzing intensity changes between pixels. For example, the edge detection unit (420) can detect the boundary of the sea surface or the boundary between water and an obstacle. In one embodiment, the autonomous navigation processing unit (130) or the edge detection unit (420) may include a edge detection model for edge detection. The edge detection model may detect the boundary based on a filter, or may include an artificial intelligence learning and inference model trained to detect the boundary. For example, the edge detection model may be holistically nested edge detection (HED), richer convolutional features (RCF), etc.

[0094] In one embodiment, the boundary map data (403) may include information about boundaries contained in the image. That is, the boundary map data (403) may be a boundary probability map.

[0095] In one embodiment, the autonomous navigation processing device (130) can synthesize segment data (402) which is a result of segmentation and boundary map data (403) which is a result of boundary detection.

[0096] Referring to FIG. 4, segment data (402) and boundary map data (403) are input to a synthesis unit (430), and the synthesis unit (430) generates and outputs probability data (404) as a result of synthesis.

[0097] In one embodiment, the probability data (404) may include probability information per pixel of the image. Specifically, the probability data (404) may include a value indicating the probability that each pixel in the image belongs to a specific class.

[0098] In one embodiment, the synthesis unit (430) may combine the class probability map of the segment data (402) and the boundary probability map of the boundary map data (403). Specifically, the synthesis unit (430) may correct the probability for pixels near the boundary or weaken the probability for pixels inside an object that is outside the boundary.

[0099] In one embodiment, generating the probability data (404) as a result of the synthesis may include correcting the detected boundaries. In one embodiment, generating the probability data (404) may include correcting false-negative boundaries and / or false-positive boundaries. In one embodiment, generating the probability data (404) may include refining false-detected boundaries based on the results of correcting the false-detected boundaries.

[0100] In one embodiment, generating the probability data (404) as a result of the synthesis may include correcting undetected boundaries. Specifically, the synthesis unit (430) may correct undetected boundaries by performing an element-wise sum on the segment data (402) and the boundary map data (403) to compensate for the undetected portions of the boundaries.

[0101] In one embodiment, generating the probability data (404) as a result of the synthesis may include correcting misdetected boundaries. Specifically, the synthesis unit (430) may correct misdetected boundaries by performing element-wise multiplication on the segment data (402) and the boundary map data (403) to compensate for the misdetected portions of the boundaries.

[0102] In one embodiment, the synthesis unit (430) can refine the misdetected boundaries by multiplying the corrected boundary result detected according to the aforementioned process by segment data (402) (element-wise multiplication). The refined result of the misdetected boundaries can be probability data (404).

[0103] In the present disclosure, segment data (402) and boundary map data (403) are combined and used together to derive more accurate segmentation results or improved boundary information, thereby providing highly reliable autonomous navigation.

[0104] In one embodiment, the operations of collecting images from multiple cameras, generating probability data using the images, and generating obstacle information based on the probability data may be performed repeatedly at preset intervals. For example, the operations may be performed at preset intervals or in real time, and the obstacle information may be updated accordingly.

[0105] FIG. 5 illustrates an example for explaining a process of processing an acquired image according to one embodiment of the present disclosure.

[0106] Referring to FIG. 5, an acquired image (501) is illustrated.

[0107] As described above, segmentation is performed on the image (501), and a boundary can be detected from the image (501).

[0108] Referring to FIG. 5, segment data (502) and boundary map data (503) are illustrated.

[0109] As described above, segment data (502) and boundary map data (503) can be synthesized.

[0110] Referring to FIG. 5, probability data (504) is shown as a result of the synthesis of segment data (502) and boundary map data (503).

[0111] For ease of understanding, the segment data (502), boundary map data (503), and probability data (504) are depicted as a type of image, but the segment data (502), boundary map data (503), and probability data (504) are composed of data including values ​​corresponding to each pixel, and are not necessarily expressed as images as depicted in FIG. 5.

[0112] FIG. 6 is a diagram for explaining a process of generating obstacle information according to one embodiment of the present disclosure.

[0113] It can be understood that the process of generating obstacle information illustrated in Fig. 6 is performed by the autonomous navigation processing device (130).

[0114] The process of generating obstacle information illustrated in FIG. 6 can be understood as part of the aforementioned obstacle estimation process. That is, the autonomous navigation processing device (130) estimating obstacles may include a process of generating obstacle information.

[0115] Referring to FIG. 6, in one embodiment, the autonomous navigation processing device (130) may include a projection unit (600). The projection unit (600) illustrated in FIG. 6 may be understood as a functional block corresponding to a function performed by the processor of the autonomous navigation processing device (130).

[0116] In one embodiment, one or more probability data (601) may include probability data each calculated in response to images captured by one or more cameras. The images captured by one or more cameras may be images captured at the same time. Here, the probability data may be data calculated by the process described above with reference to FIGS. 4 and 5 .

[0117] For example, if a ship is equipped with four cameras, one or more probability data (601) may include first camera probability data calculated in response to an image collected by a first camera, second camera probability data calculated in response to an image collected by a second camera, third camera probability data calculated in response to an image collected by a third camera, and fourth camera probability data calculated in response to an image collected by a fourth camera.

[0118] In one embodiment, the autonomous navigation processing device (130) may generate obstacle information (603) based on one or more probability data (601). In one embodiment, the autonomous navigation processing device (130) may generate obstacle information (603) by performing a series of processes including projection.

[0119] In the present disclosure, obstacle information (603) may include information about the surrounding environment of the vessel, and specifically, may include information about the location, distance, and / or type of obstacle. As described above, the autonomous navigation processing device (130) may estimate the berthing area based on the obstacle information.

[0120] Referring to FIG. 6, one or more probability data (601) is input to a projection unit (600), and the projection unit (600) generates and outputs obstacle information (603) as a result of the projection. Here, the obstacle information (603) may include information regarding the direction and distance of obstacles existing around the ship.

[0121] In one embodiment, the projection unit (600) can perform projection to transform the point of view of each of one or more cameras into a common coordinate system.

[0122] In one embodiment, the projection unit (600) can perform camera calibration. In one embodiment, the projection unit (600) can perform camera calibration based on intrinsic and extrinsic parameters of one or more cameras mounted on the vessel.

[0123] In one embodiment, the projection unit (600) can perform image alignment. In one embodiment, the projection unit (600) can align multiple images or multiple probability data into a single data based on camera calibration. Image alignment may mean detecting and aligning overlapping areas between images collected from different cameras. In one embodiment, the projection unit (600) can align one or more probability data (601) to generate obstacle information (603).

[0124] In one embodiment, the projection unit (600) can perform filtering on one or more probability data (601) to generate one or more filtered probability data (601) and generate obstacle information (603) based on the one or more filtered probability data (601). For example, the projection unit (600) can generate the one or more filtered probability data (601) by excluding probability values ​​below a threshold value from the probability data (601). For example, the projection unit (600) can exclude pixels from the boundary in the probability data (601) that have a probability below a threshold value. For example, the threshold value can be 50%.

[0125] In one embodiment, obstacle information (603) may include information regarding the obstacle closest to the vessel. The projection unit (600) may generate obstacle information (603) so that it includes only information regarding the obstacle closest to the vessel in the first direction. The projection unit (600) may generate obstacle information (603) by filtering only pixel information closest to the vessel in the first direction centered on the vessel. That is, when multiple obstacles exist, including obstacles close to one direction and obstacles farther away, obstacle information (603) may be generated based on the closest obstacle, and information regarding obstacles further away may be deleted. By leaving only the information necessary for autonomous berthing of the vessel, computational efficiency and algorithmic safety may be improved.

[0126] In one embodiment, the projection unit (600) can receive sensor data (602). The projection unit (600) can generate obstacle information (603) based on the sensor data (602). For example, the sensor data (602) can be data from an inertial measurement device.

[0127] FIG. 7 illustrates an example for explaining a process for generating obstacle information according to one embodiment of the present disclosure.

[0128] The example shown in Fig. 7 is an example in which four cameras are installed on a ship.

[0129] Referring to FIG. 7, one or more probability data (701) are illustrated.

[0130] As described above, a series of processes including projections can be performed on one or more probability data (701).

[0131] Referring to FIG. 7, obstacle information (702) is illustrated.

[0132] As described above, obstacle information (702) can be generated by matching images collected from different cameras. As illustrated in FIG. 7, obstacle information (702) can be generated based on a viewpoint looking at the vessel from the air.

[0133] In one embodiment, obstacle information (702) may be mapped to a map centered on the vessel. Obstacle information (720) may be information that maps obstacle information around the vessel on a pixel-by-pixel basis onto the map. For example, obstacle information (702) may be a set of pixels where obstacles exist, and may be generated based on a viewpoint of the vessel viewed from the air. As described above, obstacle information (702) may be a set of pixels that are closest to the vessel in a first direction centered on the vessel among pixels where obstacles exist. Pixels where obstacles exist included in obstacle information (702) may be described as detected points in the following drawings.

[0134] In one embodiment, a map with obstacle information (702) mapped on it may be provided to a user of the vessel.

[0135] In one embodiment, the autonomous navigation processing device (130) may extract pixel information corresponding to the berth entrance from the obstacle information (702). The autonomous navigation processing device (130) may estimate a berthing area including pixel information corresponding to the berth entrance. The autonomous navigation processing device (130) may display the estimated berthing area.

[0136] Figure 8 is a flowchart illustrating a process for estimating a berth point according to one embodiment of the present disclosure. A berth point may refer to obstacle information adjacent to a berth. A berth point may refer to obstacle information corresponding to a berth occupying at least one side of a berth. A berth point may be obstacle information defining the berth closest to a berth point among multiple obstacle information.

[0137] FIGS. 9A to 9D illustrate examples for explaining a process of estimating a berth point according to one embodiment of the present disclosure.

[0138] The process of estimating the berth point illustrated in Fig. 8 can be performed by the autonomous navigation processing device (130) described above.

[0139] As described above, the autonomous navigation processing device (130) can estimate the berthing area based on obstacle information. The process of estimating the berthing point may be included in the process of estimating the berthing area.

[0140] In step 801, in one embodiment, the autonomous navigation processing device (130) may calculate a vector between the berthing point and the vessel.

[0141] As described above, the autonomous navigation processing device (130) can receive a docking point. For example, the docking point may be input by a user. The docking point may include latitude and longitude information.

[0142] The autonomous navigation processing device (130) can calculate a vector between the berthing point and the ship based on the received berthing point and position data.

[0143] Referring to FIG. 9A, a vessel (901) and a docking point (902) are illustrated. The position of the vessel (901) can be determined based on position data collected by sensors, and the docking point (902) can be determined based on received data. However, when based on a relative coordinate system, a vector between the docking point and the vessel can be calculated even without position data.

[0144] Referring to FIG. 9b, a vector (903) generated by the autonomous navigation processing device (130) is illustrated. The vector (903) can be generated based on the position of the ship (901) and the position of the berthing point (902).

[0145] Meanwhile, referring to FIG. 9B, one or more detected points are illustrated. In the present disclosure, the detected points represent the locations of obstacles included in the obstacle information generated and calculated according to the aforementioned process, and may be obstacle points. The detected points may include obstacle points within the vessel's area of ​​interest. In one embodiment, the detected points may be comprised of obstacle points for the obstacle closest to the vessel.

[0146] At step 802, in one embodiment, the autonomous navigation processing device (130) may determine whether an obstacle exists between the vessel and the berthing point.

[0147] In one embodiment, the autonomous navigation processing device (130) can determine whether an obstacle exists based on a vector between the position of the berthing point and the position of the vessel.

[0148] In step 803, in one embodiment, the autonomous navigation processing device (130) may determine a near point among the detected points.

[0149] In one embodiment, the autonomous navigation processing device (130) may determine a near point among the detected points based on determining that no obstacle exists between the vessel and the berthing point.

[0150] As described above, the detected points include all obstacle points within the vessel's area of ​​interest, while the near point may refer to an obstacle point among the detected points that is presumed to be associated with the berthing point. Therefore, the near point may be determined based on the berthing point. In one embodiment, the near point may be determined as a point located within a threshold distance from the berthing point among the detected points.

[0151] Referring to FIG. 9C, one or more detected points and one or more near points are depicted. As depicted in FIG. 9C, no obstacles (or obstacle points) exist between the vessel (901) and the berthing point (902). Furthermore, as depicted in FIG. 9C, one or more near points are depicted as being closer to the berthing point than detected points that are not near points.

[0152] At step 804, in one embodiment, the autonomous navigation processing device (130) may determine two entry points among the near points.

[0153] In one embodiment, the autonomous navigation processing device (130) may determine at least one entrance point based on the positional relationship between one or more near points and the vessel. For example, there may be two entrance points.

[0154] In one embodiment, the autonomous navigation processing device (130) may generate temporary vectors based on each of one or more near points and the location of the vessel (901). The autonomous navigation processing device (130) may determine a first entry point based on vectors having a greater slope than a vector (903) based on the location of the vessel (901) and the location of the berthing point (902) among the temporary vectors, and may determine a second entry point based on vectors having a less slope than a vector (903) based on the location of the vessel (901) and the location of the berthing point (902) among the temporary vectors. The first entry point may be determined as the nearest near point to the vessel (901) among the near points forming a vector having a greater slope than the vector (903), and the second entry point may be determined as the nearest near point to the vessel (901) among the near points forming a vector having a less slope than the vector (903).

[0155] Referring to Fig. 9d, two entrance points are illustrated. As illustrated in Fig. 9d, the two entrance points can be determined from one or more near points.

[0156] At step 805, in one embodiment, the autonomous navigation processing device (130) may determine whether the width of the inlet is greater than a threshold value.

[0157] In one embodiment, the width of the inlet can be calculated based on the distance between two inlet points.

[0158] In one embodiment, the threshold value may be preset to a value greater than the vessel's width. In another embodiment, the threshold value may be determined based on the vessel's width and the calculated disturbance. For example, the threshold value may be higher when the calculated disturbance is large than when the calculated disturbance is small.

[0159] In step 806, in one embodiment, the autonomous navigation processing device (130) may determine a near point between two entrance points as a berth point.

[0160] In one embodiment, the autonomous navigation processing device (130) may determine a berth point based on determining that the width of the inlet is greater than a threshold value.

[0161] In one embodiment, one or more near points between two entrance points are determined as berth points, and two near points located outside the two entrance points can be excluded from the berth points.

[0162] Referring to FIG. 9D, near points determined as berth points (grouped by dotted lines) among one or more near points are illustrated. While the berth points grouped by dotted lines in FIG. 9D are illustrated as not including an entry point, the present invention is not limited thereto, and the berth points may include an entry point.

[0163] At step 807, in one embodiment, the autonomous navigation processing device (130) may stop autonomous docking.

[0164] In one embodiment, the autonomous navigation processing device (130) may abort autonomous berthing based on determining that an obstacle exists between the vessel and the berthing point or that the width of the entrance is not greater than a threshold value.

[0165] FIG. 10 is a flowchart illustrating a process for calculating a viewing area according to one embodiment of the present disclosure.

[0166] FIGS. 11A to 11D illustrate examples for explaining a process for calculating an eyepiece area according to one embodiment of the present disclosure.

[0167] The process of calculating the contact area illustrated in Fig. 10 can be performed by the autonomous navigation processing device (130) described above.

[0168] As described above, the autonomous navigation processing device (130) can estimate the landing area based on obstacle information. The process of calculating the landing area may be included in the process of estimating the landing area.

[0169] In step 1010, in one embodiment, the autonomous navigation processing device (130) may determine a berth area that includes all berth points.

[0170] Here, the berth point may be a berth point detected and determined according to the process described above with reference to FIGS. 8 to 9d. The berth area may be an area of ​​the berth estimated based on the berth point. The berth area may be an area including all berth points surrounding the berthing point. The berth area may be an area including obstacle information corresponding to at least one side of an obstacle (e.g., a berth) adjacent to the berthing point.

[0171] In one embodiment, a berth area may be defined as the smallest area encompassing all of one or more berth points. For example, the berth area may be determined as the smallest rectangle encompassing all of one or more berth points. The autonomous navigation processing unit (130) may determine the berth area based on an algorithm that generates a rectangle based on one or more points.

[0172] Referring to FIG. 11A, one or more berth points are illustrated, determined based on one or more near points. Also, referring to FIG. 11A, a berth area (1110) including all of the one or more berth points is illustrated. The berth area (1110) of FIG. 11A may be the smallest rectangle that includes all of the one or more berth points.

[0173] In step 1020, in one embodiment, the autonomous navigation processing device (130) may generate a boundary area based on the berth area.

[0174] In this disclosure, the "limited area" may refer to the area a vessel can occupy when docking, taking into account vessel safety. Due to limitations in the viewpoint from which a vessel views a berth, the berth point or berth area may not reflect the actual area occupied by the berth. Therefore, the autonomous navigation processing device (130) may correct the berth area to create a limiting area. Accordingly, the limiting area may be an area corrected to resemble the shape of an actual berth.

[0175] In one embodiment, the boundary area may be determined as a rectangle generated based on the berth area. In one embodiment, the boundary area may be generated based on two entry points. Here, the two entry points may be two entry points determined according to the process described above with reference to FIGS. 8 to 9d.

[0176] Specifically, in one embodiment, the position and length of one side of the rectangular boundary region may be determined by a line segment connecting two entry points. In one embodiment, the length of the remaining side of the rectangular boundary region may be determined by the length of the longest side among the sides of the rectangular boundary region.

[0177] Referring to Fig. 11b, one or more berth points and two entrance points are illustrated. Also, referring to Fig. 11b, a boundary area (1120) generated based on the two entrance points and the berth area (1110) is illustrated. The boundary area (1120) of Fig. 11b has a rectangular shape, and the length of one side of the boundary area (1120) may be the length of a line segment connecting the two entrance points, and the length of the remaining side of the boundary area (1120) may be the length of the longest side among the sides of the square of the berth area (1110). The boundary area (1120) may include two entrance points.

[0178] In step 1030, in one embodiment, the autonomous navigation processing device (130) may produce a recommended area based on the boundary area.

[0179] In this disclosure, the recommended area may refer to an area recommended for a vessel to occupy when docking, taking into account vessel safety. That is, for vessel safety, it may be desirable to control the vessel so that it does not exceed the recommended area.

[0180] In one embodiment, the recommended area may be determined as a rectangle generated based on the boundary area. For example, the recommended area may be determined as a trapezoid generated based on the boundary area, but is not limited thereto. In one embodiment, the recommended area may be generated based on the anchor point.

[0181] Specifically, one side of the recommended area can be a line segment connecting two entry points, while the remaining three sides can be created so as not to include a berth point. Accordingly, both the recommended area and the limit area are configured so that one side includes two entry points, but the recommended area, unlike the limit area, can be configured so as not to include a berth point within its area. The size of the recommended area can be smaller than that of the limit area.

[0182] Referring to Fig. 11c, a recommended area (1130) generated based on a limit area (1120) is illustrated. The recommended area (1130) of Fig. 11c may have a trapezoidal shape, but is not limited thereto, and may also be defined in various rectangular shapes.

[0183] One side of the recommended area (1130) is a line segment connecting two entry points, and a line segment opposite the line segment connecting the two entry points may be the same length as the line segment connecting the two entry points or may be shorter than the line segment connecting the two entry points. The size of the recommended area (1130) may be smaller than the limit area (1120), and the recommended area (1130) may not include a berth point within it.

[0184] According to an embodiment, among the line segments forming the limit area (1120), the line segment corresponding to the length of the longest side of the side of the square of the pick area (1110) may have its length adjusted according to the distance between the line segment connecting the two entrance points of the recommendation area (1130) and the line segment facing the line segment connecting the two entrance points.

[0185] In step 1040, in one embodiment, the autonomous navigation processing device (130) may calculate an outer boundary area based on the boundary area. In one embodiment, the autonomous navigation processing device (130) may also calculate an outer boundary area based on the recommended area.

[0186] In this disclosure, the outer boundary area may refer to a safety area that can be secured in the direction in which the berth is open when a vessel berths at the berth. For example, when a vessel berths from the rear, the outer boundary area must be secured in the forward boundary area of ​​the vessel's bow.

[0187] In one embodiment, the outer boundary area may be determined as a rectangle generated in the opposite direction from the direction in which the obstacle is located, based on two entry points. Specifically, the outer boundary area may be generated based on detected points located opposite the direction in which the berth is open, i.e., the direction in which the berth is blocked. For example, when a vessel is berthing from behind, the outer boundary area may be generated based on detected points located along the bow of the vessel when berthing is completed.

[0188] Specifically, the position and length of one side of the rectangular outer boundary region can be determined as a line segment connecting two entry points. In one embodiment, the lengths of the remaining sides of the rectangular outer boundary region can be determined so that the rectangular outer boundary region becomes the largest rectangle that does not contain any other detected points.

[0189] Referring to Fig. 11d, a detected point and two entry points are illustrated. The detected point may be an obstacle point located in the direction of the vessel's bow when berthing is completed, as determined based on obstacle information. In addition, referring to Fig. 11d, an outer boundary area (1140) generated based on the boundary area (1120) is illustrated. The outer boundary area (1140) of Fig. 11d has a rectangular shape, and the length of one side of the outer boundary area (1140) is the length of the line segment connecting the two entry points, and the length of the remaining side of the outer boundary area (1140) may be a length determined so that the rectangular outer boundary area (1140) becomes the largest rectangle that does not include the detected point (1101). The outer boundary area (1140) includes two entry points and may be a safety area secured in the opposite direction to the boundary area (1120) or the recommended area (1130). The outer boundary area (1140) may be an area that includes two entry points and extends in the opposite direction to the boundary area (1120) based on the two entry points, but does not include a detected point in the opposite direction to the boundary area.

[0190] The aforementioned process of calculating the contact area with reference to FIGS. 10, 11a, and 11d can be performed repeatedly and periodically. That is, the contact area can be updated according to a preset cycle. Accordingly, after autonomous contact is initiated, the process of calculating the contact area can be continuously performed. For example, the preset cycle can be any suitable time, such as 0.1 seconds, 0.2 seconds, or 1 second.

[0191] Likewise, the process of generating obstacle information by processing the images described above with reference to FIGS. 4 to 7, which serves as the basis for calculating the eye contact area, can also be performed repeatedly and periodically. That is, the obstacle information can be updated according to a preset cycle. Accordingly, after autonomous eye contact is initiated, the process of calculating obstacle information can be continuously performed. For example, the preset cycle can be any suitable time, such as 0.1 seconds, 0.2 seconds, or 1 second.

[0192] In one embodiment, the autonomous navigation processing device (130) may determine and display a berthing area based on a limit area (1120) or a recommended area (1130). In one embodiment, the berthing area may reflect specifications such as the size of the vessel. In one embodiment, the berthing area may have a rectangular shape. For example, the berthing area may be determined and displayed so as to be included in the limit area (1120). For example, the berthing area may be determined and displayed so that a square corresponding to the berthing area includes at least one of two entry points, and the center of the square corresponding to the berthing area corresponds to the berthing point.

[0193] As described above, in the present disclosure, the autonomous navigation processing device (130) can calculate a berthing area, which can include a berth area, a limit area, a recommended area, and / or an outer limit area. The autonomous navigation processing device (130) of the present disclosure can safely achieve autonomous berthing by calculating control logic based on the calculated berthing area.

[0194] Below, a process for providing a user with an autonomous eye contact function or an autonomous eye contact control function according to the above-described embodiments is described.

[0195] In one embodiment, information regarding a docking target point may be stored. In the present disclosure, a docking target point may refer to a point that can be a candidate for a docking point, and a docking point may be set based on the docking target point. In one embodiment, information regarding a docking target point may include the docking direction, docking method, berth type, latitude and longitude of the berth, and the direction of the berth entrance.

[0196] In one embodiment, the autonomous navigation processing device (130) may store information regarding the docking target point in a database. Depending on the embodiment, the autonomous navigation processing device (130) may transmit information regarding the docking target point to a control server. In one embodiment, the autonomous navigation processing device (130) may receive information regarding the docking target point from the control server.

[0197] In one embodiment, information regarding the berthing target point may be collected while the vessel is at berthing. In one embodiment, if information regarding the berthing target point is not stored, information regarding the point at which berthing is completed may be automatically stored as information regarding the berthing target point based on the completion of autonomous berthing.

[0198] In one embodiment, information about a berthing target point can be collected in real time while the vessel is in operation. In one embodiment, information about a berthing target point can be stored by a user's input. For example, information about an estimated berth point or a calculated berthing area according to the above-described embodiments can be provided to the user, and the user can transmit an input to store information about a berthing target point based on the information about the estimated berth point or the calculated berthing area. For example, the user can transmit an input to store information about a berthing target point according to the berthing area by interacting with an object representing a calculated specific berthing area displayed through a display. For example, information about an estimated berth point or a calculated berthing area according to the above-described embodiments can be provided to the user in real time as information about a berthing target point (or information about a candidate berthing point), and information about a selected berthing target point (or information about a candidate berthing point) can be stored based on a selection input of any one of the provided berthing target point information (or information about a candidate berthing point).

[0199] In one embodiment, a docking point may be set based on a docking target point. As described above, the autonomous navigation processing device (130) may receive a docking point and a docking method. In one embodiment, receiving a docking point and a docking method may include receiving input from a user regarding a docking point and retrieving or receiving information regarding a target point corresponding to the received docking point. The user may select or set any one of one or more docking target points as the docking point. The docking method may be included in the information regarding the target point.

[0200] For example, a user may transmit an input for setting an eye contact point by interacting with any of the objects representing one or more eye contact target points displayed on the touch display. For example, the display of the eye contact target point may include an indication of an eye contact area corresponding to the eye contact target point.

[0201] For example, information about a target docking point (or information about a candidate docking point) may be displayed to the user, and the docking point may be set by the user requesting autonomous docking at the target docking point from the autonomous navigation processing device (130). In other words, the user may request autonomous docking at the target docking point by selecting information about the target docking point (or information about a candidate docking point).

[0202] In one embodiment, the autonomous navigation processing device (130) may calculate a recommended docking path based on receiving an input for setting a docking point. In one embodiment, the autonomous navigation processing device (130) may provide the calculated recommended docking path to the user. In one embodiment, the user may initiate autonomous docking through additional interaction. For example, the autonomous navigation processing device (130) may receive an input for selecting one of the previously stored docking target points or one of the docking target points detected in real time as the docking point, and may provide a recommended docking path accordingly.

[0203] Meanwhile, as previously described, obstacle information and docking areas can be updated at preset intervals. In one embodiment, the autonomous navigation processing unit (130) can terminate autonomous docking after autonomous docking has commenced based on the detection of an unsuitable element. Terminating autonomous docking may include providing a notification to the user.

[0204] In one embodiment, unsuitable elements may include an obstacle between the berthing area and the vessel, a berthing area in an area unsuitable for the vessel's specifications, an unsecured outer boundary area, and / or a non-existence of a berthing point or berthing area. That is, the autonomous navigation processing device (130) may stop autonomous berthing based on detecting an obstacle between the berthing area and the vessel, a berthing area in an area unsuitable for the vessel's specifications, an unsecured outer boundary area, and / or a non-existence of a berthing point or berthing area.

[0205] In one embodiment, the autonomous navigation processing device (130) may, after autonomous berthing has been initiated, determine that restoration control is necessary, provide a notification to the user, and then perform restoration control. Here, restoration control may refer to control that moves the vessel to a more suitable location for berthing when berthing along the berthing path is difficult.

[0206] FIG. 12 is a flowchart of a method for controlling autonomous berthing of a ship according to one embodiment of the present disclosure.

[0207] Each step of the method illustrated in FIG. 12 may be performed by an autonomous docking control device, specifically a processor of the autonomous docking control device, and the autonomous docking control device may be the autonomous navigation processing device (130) described above or may be included in the autonomous navigation processing device (130) described above.

[0208] At step 1210, the processor may receive image and posture data collected via the camera.

[0209] In one embodiment, the processor may further perform the step of receiving a contact point and a contact method.

[0210] In one embodiment, the step of receiving a contact point and a contact method may include the step of receiving input from a user regarding a contact point and the step of receiving information regarding a target point corresponding to the contact point.

[0211] In one embodiment, the eyepiece may be included in the information about the target point.

[0212] In one embodiment, the image may include one or more images collected by one or more cameras having different viewpoints centered on the vessel.

[0213] At step 1220, the processor can derive obstacle information around the vessel based on the image and attitude data.

[0214] In one embodiment, step 1220 may include generating probability data for an obstacle, corresponding to each of the one or more images.

[0215] At step 1230, the processor can calculate the eye contact area based on the obstacle information.

[0216] In one embodiment, the eye contact area can be calculated based on the eye contact point and eye contact method.

[0217] In one embodiment, step 1230 may include determining a berth area based on one or more detected points and generating a limit area that the vessel may occupy when berthing based on the berth area.

[0218] In one embodiment, step 1230 may further include calculating a recommended area that the vessel is recommended to occupy when docking, based on the limit area.

[0219] In one embodiment, step 1230 may further include calculating an outer limit area, which is a safe area within which the berth can be secured in an open direction during the docking process, based on the limit area.

[0220] In one embodiment, the processor may further perform the steps of: generating control logic based on the eyepiece area; and generating a control command based on the control logic.

[0221] In one embodiment, the processor may further perform the step of determining whether an obstacle exists between the vessel and the berthing point and the step of aborting the autonomous berthing based on determining that an obstacle exists between the vessel and the berthing point.

[0222] In one embodiment, the processor may further perform the step of storing information about the point at which the contact was completed as information about the contact target point based on the contact being completed.

[0223] FIG. 13 is a flowchart of a method for generating information for autonomous docking of a vessel according to one embodiment of the present disclosure.

[0224] Each step of the method illustrated in FIG. 13 may be performed by a device generating information for autonomous docking, specifically, a processor of the device generating information for autonomous docking, and the device generating information for autonomous docking may be the autonomous navigation processing device (130) described above or may be included in the autonomous navigation processing device (130) described above.

[0225] At step 1310, the processor may generate probability data about surrounding obstacles, corresponding to each of the one or more images, based on one or more images collected through each of the one or more cameras.

[0226] In one embodiment, one or more cameras may collect images from different viewpoints centered around the vessel.

[0227] In one embodiment, one or more of the images may be images collected at the same time.

[0228] In one embodiment, step 1310 may include performing segmentation on the first image to generate segment data, and detecting a boundary from the first image to generate boundary map data.

[0229] In one embodiment, step 1310 may further include the step of synthesizing segment data and boundary map data.

[0230] In one embodiment, the probability data may include probability information at the pixel level of each of one or more images.

[0231] In step 1320, the processor can generate obstacle information around the vessel based on probability data corresponding to each of one or more images.

[0232] In one embodiment, step 1320 may include transforming the viewpoints of each of the one or more cameras into a common coordinate system.

[0233] In one embodiment, step 1320 may include performing camera calibration.

[0234] In one embodiment, step 1320 may include detecting and aligning overlapping regions between one or more images.

[0235] In one embodiment, step 1320 may include filtering probability data corresponding to each of the one or more images.

[0236] In one embodiment, the processor may further perform the step of estimating the eye area based on obstacle information.

[0237] FIG. 14 is a flowchart of a method for calculating a berthing area of ​​a vessel according to one embodiment of the present disclosure.

[0238] Each step of the method illustrated in FIG. 14 can be performed by a device for calculating a landing area, specifically, a processor of the device for calculating a landing area, and the device for calculating a landing area can be the autonomous navigation processing device (130) described above or can be included in the autonomous navigation processing device (130) described above.

[0239] At step 1410, the processor may determine one or more berth points from among the one or more detected points.

[0240] In one embodiment, step 1410 may include determining one or more near points among the one or more detected points, determining two entry points among the one or more near points, and determining a near point between the two entry points as a berth point.

[0241] At step 1420, the processor may determine a berth area based on one or more berth points.

[0242] In one embodiment, the berth area may be determined to include all of one or more berth points.

[0243] At step 1430, the processor may generate a limit area that the vessel may occupy when berthing, based on the berth area.

[0244] In one embodiment, the processor may further perform the step of determining two entry points from among the one or more detected points.

[0245] In one embodiment, the boundary region can be generated based on two entry points.

[0246] In one embodiment, the boundary region may be a rectangle with one side being a line segment connecting two entry points.

[0247] In one embodiment, the processor may further perform the step of calculating a recommended area that the vessel is recommended to occupy when docking, based on the limit area.

[0248] In one embodiment, the recommended area may be a trapezoid with one side being the line segment connecting the two entry points.

[0249] In one embodiment, the processor can calculate an outer boundary area, which is a safe area within which the berth can be secured in an open direction during the docking process, based on the boundary area.

[0250] In one embodiment, the outer boundary region may be a rectangle with one side being a line segment connecting two entry points.

[0251] FIG. 15 is a block diagram of a device providing an interface for operating a ship according to one embodiment of the present disclosure.

[0252] The device (1500) illustrated in FIG. 15 may be the autonomous navigation processing device (130) described above.

[0253] Referring to FIG. 15, the device (1500) may include a communication unit (1510), a processor (1520), and a database (1530). Only components related to the embodiment are illustrated in the device (1500) of FIG. 15. Therefore, those skilled in the art will understand that other general components may be included in addition to the components illustrated in FIG. 15.

[0254] The communication unit (1510) may include one or more components that enable wired / wireless communication with an external server or external device. For example, the communication unit (1510) may include at least one of a short-range communication unit (not shown), a mobile communication unit (not shown), and a broadcast receiving unit (not shown).

[0255] DB (1530) is hardware that stores various data processed within the device (1500) and can store programs for processing and controlling the processor (1520). DB (1530) can store payment information, user information, etc.

[0256] DB (1530) may include random access memory (RAM) such as dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory.

[0257] The processor (1520) controls the overall operation of the device (1500). For example, the processor (1520) can control the input unit (not shown), the display (not shown), the communication unit (1510), the DB (1530), etc., by executing programs stored in the DB (1530). The processor (1520) can control the operation of the device (1500) by executing programs stored in the DB (1530).

[0258] The processor (1520) can control at least some of the operations of the device (1500) described above in FIGS. 1 to 14.

[0259] The processor (1520) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0260] In one embodiment, the device (1500) may be a mobile electronic device. For example, the device (1500) may be implemented as a smartphone, tablet PC, PC, smart TV, personal digital assistant (PDA), laptop, media player, navigation device, camera-equipped device, or other mobile electronic device. Furthermore, the device (1500) may be implemented as a wearable device, such as a watch, glasses, hair band, or ring, equipped with communication and data processing capabilities.

[0261] Embodiments according to the present invention may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. In this case, the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memories.

[0262] Meanwhile, the computer program may be specifically designed and constructed for the present invention, or may be one known and available to those skilled in the computer software field. Examples of computer programs may include not only machine language code, such as that generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

[0263] According to one embodiment, the method according to various embodiments of the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0264] Unless the steps constituting the method according to the present invention are explicitly described in a specific order or are otherwise described in a different order, the steps may be performed in any appropriate order. The present invention is not necessarily limited to the order in which the steps are described. The use of all examples or exemplary terms (e.g., “for example,” “etc.”) in the present invention is merely intended to illustrate the present invention in more detail, and the scope of the present invention is not limited by the examples or exemplary terms unless otherwise defined by the claims. Furthermore, those skilled in the art will appreciate that various modifications, combinations, and variations can be configured according to design conditions and factors within the scope of the appended claims or their equivalents.

[0265] Therefore, the idea of ​​the present invention should not be limited to the embodiments described above, and not only the scope of the patent claims described below but also all scopes equivalent to or equivalently modified from the scope of the patent claims are considered to fall within the scope of the idea of ​​the present invention.

Claims

1. As an autonomous docking control method, A step of receiving image and posture data collected through a camera; A step of calculating obstacle information around the ship based on the image and the posture data; A step of calculating a contact area based on the above obstacle information; A step of displaying the above eye contact area; including, method.

2. In paragraph 1, Step of receiving a docking point and docking method; Including more, The above eye contact area is, Calculated based on the above-mentioned docking point and the above-mentioned docking method, method.

3. In paragraph 1, A step of calculating control logic based on the above contact area; and A step of generating a control command based on the above control logic; including more, method.

4. In paragraph 1, The above eye contact area is, Updated according to a preset cycle, method.

5. In paragraph 1, The image above is, Comprising one or more images collected by one or more cameras having different viewpoints centered on the vessel, method.

6. In paragraph 5, The steps for generating obstacle information are: generating probability data for obstacles corresponding to each of the one or more images; and A step of generating the obstacle information by matching the above probability data; including, method.

7. In paragraph 1, The step of calculating the above contact area is: A step of determining a berth area based on the above obstacle information; and A step of creating a boundary area including two obstacle points corresponding to the berth entrance based on the above berth area; including, method.

8. In paragraph 7, The step of calculating the above contact area is: A step of calculating an outer boundary area, which is a safety area secured in the opposite direction to the boundary area and includes two obstacle points corresponding to the entrance of the above-mentioned berth; including more, method.

9. In paragraph 1, The step of displaying the above eye contact area is. A step of marking the berthing area using a square reflecting the size of the vessel so that the center of the square corresponds to the berthing point; including more, method.

10. In paragraph 2, a step of determining whether an obstacle exists between the vessel and the berthing point; and A step of stopping autonomous docking based on determining that an obstacle exists between the vessel and the docking point; including more, method.

11. In paragraph 2, The step of receiving the above-mentioned docking point and docking method is as follows: A step of storing information on a berthing target point including the latitude and longitude of the berthing point while the vessel is berthed; A step in which information about the above target point of contact is displayed to the user as a candidate target point of contact; and comprising a step of receiving a selection input from a user regarding any one of the candidate contact points; method.

12. In paragraph 2, The step of receiving the above-mentioned docking point and docking method is as follows: A step in which the berthing area is displayed to the user as a candidate berthing point while the vessel is in operation; A step of receiving a selection input from a user regarding any one of the candidate contact points; and A step of storing the latitude and longitude of the selected candidate docking point as information about the docking target point; including; method.

13. In paragraph 1, Based on the completion of the docking, if information about the point where the docking is completed is not stored, a step of storing at least some of the latitude and longitude of the point where the docking is completed, the type of berth, the entrance direction of the berth, the docking direction, and the docking method; including more, method.

14. As an autonomous eye control device, memory in which at least one program is stored; and A processor that operates by executing at least one program; The above processor, Receive one or more images and attitude data of the vessel collected through one or more cameras having different viewpoints centered on the vessel, By matching one or more images based on the above detailed data, obstacle information around the vessel is calculated, Based on the above obstacle information, a berthing area including two obstacle points corresponding to the berth entrance is calculated, Indicating the above eye contact area, device.

15. A computer-readable recording medium recording a program for executing the method according to paragraph 1 on a computer.

Images