Apparatus and method for situational awareness of ship, and system for identifying unnavigable zones

Abstract

An apparatus and a method for situational awareness of a ship, and a system for identifying unnavigable zones are disclosed. The apparatus for situational awareness of a ship, according to an embodiment of the present invention, can be provided, the apparatus comprising: an input unit for inputting sensing data received from at least one sensor disposed in a marine environment; a processing unit for detecting at least one object on the basis of on the sensing data, matching the object, and fusing location information of the object on the basis of the matching result; and a storage unit for storing situational awareness information of the object in the marine environment on the basis of the fusion result.

Description

Ship's surroundings recognition device, method, and no-navigation zone recognition system

[0001] The present invention relates to a device, method, and system for recognizing the surroundings of a ship, and more particularly, to a device, method, and system for recognizing a non-navigable area for recognizing the surroundings of a ship that can precisely recognize the surroundings in a maritime environment.

[0002] Generally, the safe operation of a vessel has always been a crucial task, given the complexity of the maritime environment and the constantly changing surroundings. Traditionally, crews navigated primarily through visual observation via telescopes, gathering information from radar and Automatic Identification Systems (AIS) to understand their surroundings and avoid collisions. In particular, during nighttime navigation, onboard lights were used to identify other vessels.

[0003] However, these traditional methods of perception have several limitations. First, in situations where visibility is extremely limited, such as in adverse maritime weather conditions (e.g., fog, heavy rain, high waves) or at night, visual perception of the surroundings becomes significantly more difficult. Consequently, crews become increasingly reliant on onboard equipment such as radar and AIS. Furthermore, concerns have been consistently raised that fatigue or temporary inattention during long voyages can lead to human errors, such as failure to recognize obstacles, thereby increasing the risk of maritime accidents.

[0004] In this regard, Republic of Korea Patent Publication No. 10-2023-0131723 (2023.09.14) was previously published.

[0005] Meanwhile, relying on a single sensor has significant limitations. Radar has difficulty detecting noise, non-metallic objects, and small objects, and AIS cannot detect vessels or floating objects without the device installed. This makes it difficult to fully grasp the surrounding environment with a single sensor, a major factor in compromising vessel stability.

[0006] In addition, in the case of autonomous ships using sensors such as cameras and radar, continuous detection and identification (recognition) of objects around the ship and real-time information analysis (tracking) of dynamic objects are required. However, due to the strengths / weaknesses and uncertainties of sensors, there are limitations in recognizing all situations with just one sensor.

[0007] In addition to obstacles like ships and buoys, the surrounding environment also presents areas that are difficult to navigate, such as land or islands, fishing boat groups, and fishing nets. However, conventional passive perception methods or those relying on single sensors have clear limitations in complex and ever-changing maritime environments, especially when areas of impassability not recorded on electronic charts occur.

[0008] Therefore, there is an urgent need for research and development of new situational awareness technologies and systems that can dramatically improve maritime navigation safety by resolving these issues and recognizing areas that are difficult to navigate in bad weather or at night.

[0009] The present embodiment aims to provide a device and method for recognizing the surrounding situation of a ship, which can provide more robust and accurate situation recognition results than a single sensor by recognizing surrounding obstacles by fusing sensors such as a camera, radar, AIS (automatic identification system), and GPS (global positioning system).

[0010] The present embodiment aims to provide a system for recognizing a no-navigation zone of a ship that can accurately recognize a no-navigation zone around a ship in real time by complementing and integrating the uncertainty of radar and AIS data.

[0011] This embodiment aims to provide a system for recognizing a no-navigation area of ​​a ship that can maximize navigation safety by precisely identifying the location of a no-navigation area that is difficult to confirm with the naked eye and not recorded on an electronic chart through the fusion of radar and AIS.

[0012] The present embodiment aims to provide a system for recognizing a vessel's unnavigable area, which supports efficient route decision-making based on recognized unnavigable areas and safely modifies existing routes, thereby further improving navigational safety in complex waters.

[0013] The present embodiment aims to provide a system for recognizing a no-navigation zone for a ship, which can prevent the risk of collision in poor visibility conditions in advance by determining the risk of approaching a no-navigation zone, generating a warning signal or alarm signal, and immediately transmitting the signal to an operator.

[0014] According to one aspect of the present invention, a device for recognizing the surroundings of a ship includes: an input unit for inputting sensing data received from at least one sensor placed in a maritime environment; a processing unit for detecting at least one object based on the sensing data, performing matching processing on the object, and fusing location information of the object based on a result of the matching; and a storage unit for storing situational awareness information of the object in the maritime environment based on a result of the fusion.

[0015] The above sensing data may include image data, radio wave data, and identification data.

[0016] The processing unit may include a detection processing unit that classifies the image data based on artificial intelligence learning to detect first object information, generates a PPI (plane position indicator) image of the radio wave data to detect second object information, and decodes the identification data to detect third object information; a matching processing unit that matches the first object information, the second object information, and the third object information; and a fusion processing unit that applies an image-based object tracking algorithm to the matching result of the matching processing unit to fuse the first object information, the second object information, and the third object information.

[0017] The above detection processing unit can detect the first object information by correcting distortion of the image data, and can detect the second object information by filtering the radio wave data.

[0018] The above alignment processing unit can determine the bearing and range of the object by aligning the position of the object based on the first object information and the second object information.

[0019] The above fusion processing unit can estimate the direction of movement (course of ground, COG) and average speed (speed of ground, SOG) of the object by fitting the previous position of the object to an ellipse based on the third object information.

[0020] According to an embodiment of the present invention, a ship can be provided, including: at least one sensor disposed in a maritime environment; a ship's surrounding situation recognition device that inputs sensing data received from the at least one sensor, detects at least one object based on the sensing data, performs matching processing on the object, and fuses location information of the object based on a result of the matching; and an output device that outputs situation recognition information of the object in a maritime environment based on a result of the fusion.

[0021] The at least one sensor may include an image sensor that obtains an image of the object through a vision camera; a radio sensor that obtains echo data of the object through RADAR (radio detection and ranging); and an identification sensor that obtains the location of the object through AIS (automatic identification system) and GPS (global positioning system).

[0022] The surrounding situation recognition device of the above ship can classify and detect the object by restoring the distortion of the image acquired from the image sensor, and can filter the echo data acquired from the radio wave sensor to generate a PPI image.

[0023] According to an embodiment of the present invention, a method for recognizing the surroundings of a vessel in a maritime environment for an autonomous vessel can be provided, including: a step of inputting sensing data received from at least one sensor disposed in the maritime environment; a step of detecting at least one object based on the sensing data; a step of matching the object; and a step of fusing location information of the object based on a result of the matching.

[0024] The sensing data may include image data, radio wave data, and identification data, and the detecting step may include: a step of classifying the image data based on artificial intelligence learning to detect first object information; a step of generating a PPI image of the radio wave data to detect second object information; and a step of decoding the identification data to detect third object information.

[0025] The above matching processing step may include a step of matching the first object information, the second object information, and the third object information, and the fusing step may include a step of fusing the first object information, the second object information, and the third object information by applying an image-based object tracking algorithm to the result of the matching step.

[0026] The method may further include a step of detecting the first object information by correcting distortion of the image data.

[0027] Meanwhile, according to another aspect of the present invention, a method for recognizing the surroundings of a ship includes the steps of: receiving a photographed image of the surroundings of the ship from a camera to detect one or more first objects; receiving a radar signal of the surroundings of the ship from a radar to detect one or more second objects; matching corresponding objects among the one or more first objects and the one or more second objects as similar objects through a similarity comparison; and tracking the matched similar objects and displaying them on an electronic chart.

[0028] The step of detecting the one or more first objects includes the steps of: obtaining a binarized image from the photographed image; determining one or more object candidates through clustering of a plurality of pixels in the binarized image; selecting the one or more first objects from the one or more object candidates based on preset constraints; and setting a bounding box for each of the one or more first objects and estimating a distance for each bounding box.

[0029] The step of estimating the distance for each bounding box includes the step of determining the lower center point of the bounding box as a point of contact with the sea surface; and the step of estimating the distance between the magnetic pole and the bounding box based on the height from the sea surface to the camera.

[0030] The step of determining the one or more object candidates includes the step of removing object candidates having a reference size or larger from among the one or more object candidates based on the constraints; and the step of removing object candidates located on an average horizontal line from among the one or more object candidates based on the constraints.

[0031] The method for recognizing the surrounding situation of the present embodiment includes the steps of obtaining self-positioning information from a GPS; and the steps of projecting the one or more first objects onto a virtual radar image based on the self-positioning information to determine an azimuth for each of the one or more first objects.

[0032] The step of detecting the one or more second objects includes: removing noise in a radar image generated from the radar signal; determining one or more object candidates through clustering of a plurality of clutters in the radar image; selecting the one or more second objects from among the one or more object candidates based on preset constraints; and setting a bounding box including an azimuth and a distance for each of the one or more second objects.

[0033] The step of determining the one or more object candidates includes the step of removing object candidates having a reference size or larger from among the one or more object candidates based on the constraints; and the step of removing object candidates located in an area outside the field of view of the camera from among the one or more object candidates based on the constraints.

[0034] The step of matching with the above similar objects includes the step of extracting a pair of objects in which the azimuth difference between the at least one first object and the at least one second object is less than or equal to a first reference value; and the step of matching the pair of objects as similar objects if the distance difference between the at least one first object and the at least one second object is less than or equal to a second reference value.

[0035] The step of displaying on the electronic chart includes the step of calculating a moving direction and an average speed based on the previous point-in-time location information of the similar object; the step of calculating a latitude and longitude based on the azimuth of the similar object; and the step of displaying the similar object on the electronic chart based on the latitude and longitude, and displaying the direction and speed of the displayed similar object based on the moving direction and the average speed.

[0036] A device for recognizing the surroundings of a vessel navigating at night according to an embodiment of the present invention includes a memory storing a situation recognition program; and a processor for, when a photographed image and a radar signal around the vessel are received from a camera and a radar, respectively, executing the situation recognition program to detect one or more first objects from the photographed image, detecting one or more second objects from the radar signal, matching corresponding objects among the one or more first objects and the one or more second objects as similar objects through a similarity comparison, and tracking the matched similar objects to display them on an electronic chart.

[0037] The processor obtains a binarized image from the photographed image, determines one or more object candidates through clustering of a plurality of pixels in the binarized image, selects one or more first objects from among the one or more object candidates based on preset constraints, and sets a bounding box for each of the one or more first objects to estimate a distance for each bounding box.

[0038] The processor determines the lower center point of the bounding box as a point of contact with the sea surface, and estimates the distance between the magnet and the bounding box based on the height from the sea surface to the camera.

[0039] The processor removes object candidates having a reference size or larger and object candidates located on an average horizontal line from among the one or more object candidates based on the constraints.

[0040] The processor obtains self-positioning information from a GPS, and projects the one or more first objects onto a virtual radar image based on the self-positioning information to determine an azimuth for each of the one or more first objects.

[0041] The processor removes noise in a radar image generated from the radar signal, determines one or more object candidates through clustering of a plurality of clutters in the radar image, selects one or more second objects from among the one or more object candidates based on preset constraints, and sets a bounding box including an azimuth and a distance for each of the one or more second objects.

[0042] The processor removes object candidates having a reference size or larger from among the one or more object candidates and object candidates located outside the field of view of the camera based on the constraints.

[0043] The processor extracts a pair of objects in which the azimuth difference between the one or more first objects and the one or more second objects is less than or equal to a first reference value, and matches the pair of objects as similar objects if the distance difference between the pair of objects is less than or equal to a second reference value.

[0044] The processor calculates the direction of movement and average speed based on the previous point-in-time location information of the similar object, converts the azimuth of the similar object into latitude and longitude, displays the similar object on the electronic chart based on the latitude and longitude, and displays the direction and speed of the similar object displayed based on the direction of movement and average speed.

[0045] Meanwhile, according to another aspect of the present invention, a system for recognizing a no-navigation zone of a ship includes: a data receiving unit that receives radar data from a marine radar and AIS data from an automatic ship identification device; a preprocessing unit that preprocesses the radar data and AIS data received through the data receiving unit; a probability distribution generating unit that generates a probability distribution for each of the radar data and AIS data preprocessed through the preprocessing unit; a probability distribution fusion unit that fuses the probability distributions generated through the probability distribution generating unit into one space; and an no-navigation zone output unit that determines a no-navigation zone based on the probability distribution fused through the probability distribution fusion unit and outputs the result to the outside.

[0046] The above data receiving unit may include a radar receiving module that receives radar data from the marine radar; and an AIS receiving module that receives AIS data from the ship automatic identification device.

[0047] The above preprocessing unit may include a radar data filtering module that performs filtering on the radar data; and an AIS data projection module that projects the AIS data into the same space as the radar data.

[0048] The above radar data filtering module may be configured to remove noise included in the radar data through temporal filtering.

[0049] The above AIS data projection module may be arranged to project the AIS data onto the spatial coordinate system of the radar data using the ship's position and azimuth information.

[0050] The above probability distribution generation unit may include a radar probability distribution generation module that generates a probability distribution based on the intensity value of the preprocessed radar data; and an AIS probability distribution generation module that generates a probability distribution based on the projection position of the preprocessed AIS data.

[0051] The above AIS probability distribution generation module may be configured to generate a virtual circle (circular region) based on the projection location of the AIS data, and generate a probability distribution for an internal region of the virtual circle.

[0052] The above AIS probability distribution generation module may be arranged to adjust the diameter of the virtual circle according to the distance from the target recognized by the marine radar.

[0053] The above-mentioned impossible area output unit may include a cluster generation module that generates clusters from a probability distribution fused through the probability distribution fusion unit; a cluster selection module that selects clusters determined to be impossible areas among the clusters generated through the cluster generation module; and an outline extraction module that extracts outlines of the clusters selected through the cluster selection module and determines them as impossible areas.

[0054] The above cluster selection module may be configured to select multiple ships anchored near a port as a single ship cluster.

[0055] The above cluster selection module may be arranged to select a single cluster by merging the overlapping areas when the previously recognized non-navigable areas and the non-navigable areas determined through the outline extraction module spatially overlap.

[0056] The above outline extraction module may be arranged to determine the final non-navigable area based on the outline of the non-navigable area corresponding to the direction of movement of the charity.

[0057] According to another aspect of the present invention, a ship is provided including the above-described ship's non-navigation zone recognition system.

[0058] The above vessel may further include a route determination unit that automatically plans or modifies the vessel's route using information on an unnavigable area output through the unnavigable area output unit.

[0059] The above vessel may further include a warning output unit that generates warning information about an unnavigable area outputted through the unnavigable area output unit and outputs it to the outside.

[0060] The above probability distribution generation unit may be arranged to adjust the radar probability distribution value generated through the radar probability distribution generation module and the AIS probability distribution value generated through the AIS probability distribution generation module by reflecting a correction factor provided through the electronic chart.

[0061] The above cluster selection module is designed to calculate a risk index and select a non-navigable area based on the cluster size and risk index, and may be designed to adjust the grid spacing and risk index calculation method according to the operating environment conditions.

[0062] The above-mentioned impossible navigation area output section may be arranged to provide output information including a warning signal or an alarm signal when the selected impossible navigation area overlaps with the expected navigation route of the vessel.

[0063] The above-mentioned impossible area output unit may be arranged to calculate a reliability index by synthesizing the risk index provided from the cluster selection module and the radar probability distribution value and AIS probability distribution value calculated by the probability distribution generation unit, and output the reliability index together to an electronic chart display system (ECDIS) or an automatic radar plotting apparatus (ARPA).

[0064] According to embodiments of the present invention, continuous detection and identification (recognition) of objects surrounding an autonomous vessel using sensors such as cameras and radar, as well as real-time information analysis (tracking) of dynamic objects, can be performed more precisely and reliably. In particular, the present invention recognizes surrounding obstacles by integrating sensors such as cameras, radar, AIS, and GPS, thereby providing more robust and accurate situational awareness results than single sensors.

[0065] The ship surrounding situation recognition device of the present invention can detect the presence of objects such as other ships around the ship by fusing information provided from various sensors such as cameras, GPS, or radar installed on the ship, track the detected objects, and display them to a user who controls ship operation through an electronic chart.

[0066] Accordingly, the present invention can accurately determine the presence of other vessels around a vessel's route during night navigation, thereby improving the stability of the vessel's night navigation.

[0067] The system for recognizing a non-navigable area of ​​a ship according to this embodiment can precisely recognize a non-navigable area around a ship in real time by compensating for and integrating the uncertainty of radar and AIS data.

[0068] The vessel navigational restriction area recognition system according to this embodiment can maximize navigational safety by precisely identifying the location of navigational restriction areas that are difficult to confirm with the naked eye and not recorded on electronic charts through the fusion of radar and AIS.

[0069] The system for recognizing a vessel's unnavigable area according to the present embodiment can support efficient route decision-making based on the recognized unnavigable area and safely modify the existing route, thereby further improving the safety of navigation in complex waters.

[0070] The system for recognizing a no-navigation zone of a ship according to this embodiment can prevent the risk of collision in advance in a situation of poor visibility by determining the risk of approaching a no-navigation zone, generating a warning signal or an alarm signal, and immediately transmitting the signal to the operator.

[0071] FIG. 1 is a schematic functional block diagram of an autonomous ship including a ship's surroundings recognition device according to an embodiment of the present invention.

[0072] FIG. 2 is a block diagram illustrating the function of a ship's surrounding situation recognition device according to an embodiment of the present invention, for example, a surrounding situation recognition device of a surrounding vessel included in an autonomous navigation vessel of FIG. 1.

[0073] Figure 3 is a block diagram explaining the specific functions of the processing unit of the surrounding situation recognition device of the surrounding vessel of Figure 2.

[0074] Figure 4 is a flowchart exemplarily explaining a method for recognizing the surroundings of a ship according to an embodiment of the present invention.

[0075] Fig. 5 is a drawing exemplarily explaining a preprocessing process in the method for recognizing the surrounding situation of a ship in Fig. 4.

[0076] Fig. 6 is a drawing exemplarily explaining an object detection process in the method for recognizing the surrounding situation of a ship in Fig. 4.

[0077] Fig. 7 is a drawing exemplarily explaining the position alignment process in the method for recognizing the surrounding situation of the ship in Fig. 4.

[0078] Fig. 8 is a drawing exemplarily explaining the fusion process in the method for recognizing the surrounding situation of the ship in Fig. 4.

[0079] FIGS. 9 to 12 are drawings exemplarily illustrating experimental results for object tracking for environmental awareness for an autonomous ship according to an embodiment of the present invention.

[0080] Figure 13 shows the average course over ground (COG) / speed over ground (SOG) error between the tracking results and AIS.

[0081] Fig. 14 is a drawing showing a device for recognizing the surroundings of a ship according to an embodiment of the present invention.

[0082] Figure 15 is a diagram showing the function of the situation recognition program of Figure 14.

[0083] Fig. 16 is a drawing showing a method for recognizing the surroundings of a vessel navigating at night according to an embodiment of the present invention.

[0084] Figures 17 to 19 are drawings specifically showing a method for recognizing the surroundings of a vessel navigating at night according to the present invention.

[0085] Figure 20 shows the configuration of a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention.

[0086] Fig. 21 shows the configuration of a ship according to another embodiment of the present invention.

[0087] Figure 22 illustrates the process of fusion and outline determination of radar data and AIS data of a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention.

[0088] Figure 23 illustrates the process of fusion and outline determination of radar data, AIS data, and electronic chart data of a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention.

[0089] Figure 24 shows the detection results for the cluster outline of the system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention.

[0090] Figure 25 shows the result of recognition of a non-navigable area by a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention, displayed on an electronic chart.

[0091] Figure 26 shows the results of visualizing the unrecognized navigational area on radar and electronic charts through the system for recognizing the unrecognized navigational area of ​​a ship according to an embodiment of the present invention.

[0092] FIG. 27 is a diagram showing a radar echo adaptation threshold and a generated PPI image in a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention.

[0093] FIG. 28 is a diagram showing a spatiotemporal filtering process of a radar PPI image in a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention.

[0094] Figure 29 is a diagram showing data results before and after filtering (left: before, right: after) in a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention.

[0095] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments are provided to sufficiently convey the spirit of the present invention to those skilled in the art. The present invention is not limited to the embodiments presented herein and may be embodied in other forms. To clarify the present invention, the drawings may omit portions irrelevant to the description, and the sizes of components may be slightly exaggerated to facilitate understanding.

[0096] 1. Description of the device and method for recognizing the surroundings of a ship according to the first embodiment.

[0097] FIG. 1 is a schematic functional block diagram of an autonomous ship (1) including a ship's surroundings recognition device (100) according to an embodiment of the present invention.

[0098] As illustrated in FIG. 1, an autonomous ship (1) may include at least one sensor (10), an output device (20), and a ship's surroundings recognition device (100).

[0099] At least one sensor (10) may include navigation equipment or sensors (10) of an autonomous ship (1), and may provide current status information of the autonomous ship (1) to a ship's surroundings recognition device (100).

[0100] These sensors (10) may include an image sensor (10-1), a radio wave sensor (10-2), and an identification sensor (10-3).

[0101] The image sensor (10-1) may include, for example, a vision camera and may acquire an image of any object in a maritime environment.

[0102] The radio sensor (10-2) may include, for example, a RADAR (radio detection and ranging) and may acquire echo data of any object in the maritime environment.

[0103] The identification sensor (10-3) may include, for example, an automatic identification system (AIS) and a global positioning system (GPS), and may obtain location information of any object in the maritime environment.

[0104] The output device (20) can output situational awareness information in the maritime environment of the ship's surrounding situational awareness device (100). Such situational awareness information may include, for example, an object's track identification (ID), bearing, range, course of ground (COG), and average speed (SOG). Here, the track ID may mean, for example, a tracking number of an image-based object tracking algorithm, and the image-based object tracking algorithm may include an algorithm that tracks by fusing a predicted object location and an actual object location using a Kalman filter-based algorithm.

[0105] A ship's surrounding situation recognition device (100) can input sensing data received from at least one sensor (10), detect at least one object based on the sensing data, perform object matching, and fuse-process the object's location information based on the result of the matching. Here, the sensing data can include, for example, an object image through a vision camera from an image sensor (10-1), object echo data through RADAR from a radio sensor (10-2), and object location information through AIS and GPS from an identification sensor (10-3).

[0106] FIG. 2 is a block diagram illustrating the function of a ship's surroundings recognition device in a maritime environment for an autonomous ship according to an embodiment of the present invention, for example, a ship's surroundings recognition device (100) included in an autonomous ship (1) of FIG. 1.

[0107] As shown in Fig. 2, the ship's surrounding situation recognition device (100) may include an input unit (110), a processing unit (120), and a storage unit (130).

[0108] The input unit (110) can input sensing data received from at least one sensor placed in the maritime environment. The at least one sensor may include an image sensor (10-1), a radio wave sensor (10-2), and an identification sensor (10-3) as illustrated in FIG. 1, and the input unit (110) can input image data, radio wave data, and identification data received from these sensors (10). The input unit (110) may include a communication device for connecting the sensors (10) and the ship's surrounding situation recognition device (100). The communication device may include at least one of the Internet, a 4th generation or higher broadband wireless communication device, a short-range wireless communication device such as UWB (ultra-wideband), Wi-Fi, Bluetooth, etc., and is not limited to a specific wireless device as long as it can receive status information of the sensor (10) or transmit control information of the ship's surrounding situation recognition device (100).

[0109] The processing unit (120) detects at least one object in response to receiving at least one of image data, radio wave data, and identification data input through the input unit (110), performs matching processing on the detected object, and fuses and processes the location information of the object based on the result of the matching processing. This processing unit (120) may include, for example, a microprocessor-based control unit, and the specific functions and operations of the processing unit (120) will be described in more detail with reference to FIGS. 3 and 4.

[0110] The storage unit (130) can store situational awareness information based on the data fusion results processed by the processing unit (120), such as the track ID, azimuth, distance, moving direction, average speed, etc. of the object in the maritime environment. This storage unit (120) can include, for example, a RAM (random access memory), a ROM (read only memory), etc., and the information stored in the storage unit (130) can be selectively loaded by the processing unit (120), or specific information can be selectively recorded by the processing unit (120).

[0111] Figure 3 is a block diagram explaining the specific function of the processing unit (120) of the ship's surrounding situation recognition device (100) of Figure 2.

[0112] As shown in FIG. 3, the processing unit (120) may include a detection processing unit (122), a matching processing unit (124), and a fusion processing unit (126).

[0113] The detection processing unit (122) can classify the image data of the image sensor (10-1) based on artificial intelligence learning to detect first object information, generate a PPI (plane position indicator) image of the radio wave data of the radio wave sensor (10-2) to detect second object information, and decode the identification data of the identification sensor (10-3) to detect third object information. Here, the detection processing unit (122) can detect the first object information by correcting the distortion of the image data, and detect the second object information by filtering the radio wave data.

[0114] The alignment processing unit (124) can align the first object information, second object information, and third object information detected by the detection processing unit (122). Here, the alignment processing unit (124) can align the position of the object based on the first object information and the second object information and determine the bearing and range of the object.

[0115] The fusion processing unit (126) can fuse and process the first object information, the second object information, and the third object information by applying an image-based object tracking algorithm to the matching result of the matching processing unit (124). Here, the fusion processing unit (126) can estimate the object's moving direction (course of ground, COG) and average speed (speed of ground, SOG) by fitting the previous position of the object to an ellipse based on the third object information.

[0116] Hereinafter, a method for recognizing the surrounding situation in a maritime environment according to an embodiment of the present invention, together with the above-described configuration, will be described in more detail with reference to the attached FIGS. 4 to 12.

[0117] First, FIG. 4 is a flowchart exemplarily explaining a method for recognizing the surroundings in a maritime environment according to an embodiment of the present invention.

[0118] The method for recognizing the surroundings of a vessel's surroundings recognition device (100) according to an embodiment of the present invention may include a step (S100) of inputting sensing data received from at least one sensor (10) placed in a maritime environment and preprocessing the input sensing data; a step (S102) of detecting at least one object based on the preprocessed sensing data; a step (S104) of performing matching processing on the detected object; and a step (S106) of fusing the object's location information based on the result of the matching.

[0119] The preprocessing step (S100) may include a process of detecting first object information by correcting distortion of image data.

[0120] Fig. 5 is a drawing exemplarily explaining a preprocessing step (S100) in the surrounding situation recognition method of Fig. 4.

[0121] The preprocessing step (S100) is a process performed once for the first time, and the radar space (radar PPI image) can be defined as the reference space.

[0122] As shown in Fig. 5, if the radar space is assumed to be the sea surface and the sea surface is assumed to be flat, the projection relationship between the camera and the radar space can be corrected based on the actual measurement information of the camera installation location (height) from the sea surface.

[0123] Such correction may include, for example, a process of pre-calculating distance values ​​for each pixel of the image via a pinhole-camera projection model, and may be expressed as [Mathematical Formula 1] below.

[0124]

[0125] In [Mathematical Formula 1], K is the camera intrinsic parameter matrix (focal length, principle point), and [R|t] represents the camera pose (rotation, translation).

[0126] The camera selects objects with AIS and detects their 3D sea surface locations, which are then projected onto radar space. The radar can then select objects with AIS through connected component labeling in radar data. AIS can then use GPS as the object location to project the data onto the PPI image, which is radar space.

[0127] With the camera aligned in radar space, the sensor can be calibrated to minimize the distance error between the AIS object and the radar object.

[0128] FIG. 6 is a drawing exemplarily explaining the object detection step (S102) in the surrounding situation recognition method of FIG. 4.

[0129] The object detection step (S102) may include an object detection and classification process using artificial intelligence after distortion correction for an input image acquired through a camera.

[0130] The radar can be applied with echo data filtering based on adaptive threshold, image processing-based radar PPI image filtering, and then an object detection process of a certain size can be applied.

[0131] AIS can obtain the position, COG, and SOG of other objects using an NMEA (National Marine Electronics Association) decoder. Additionally, GPS can obtain the vessel's position and heading information using an NMEA decoder.

[0132] Fig. 7 is a drawing exemplarily explaining the position alignment step (S104) in the surrounding situation recognition method of Fig. 4.

[0133] The lower center point (p) of the object (bounding box) detected by the camera x , p y ) and can be projected onto the radar space (radar PPI image) using the pre-calculated depth information, which can be expressed as in the following [Mathematical Formula 2].

[0134]

[0135] At this time, the farther the object is, the more uncertain the distance information may be due to the pinhole camera projection model. Therefore, after searching along the pixel ray from the projected position, the radar object with the most overlap is selected as the matching object. r ) and the search distance can be proportional to the distance of the object.

[0136] Based on these alignment objects, the azimuth and distance of objects can be determined.

[0137] Fig. 8 is a drawing exemplarily explaining the fusion processing step (S106) in the surrounding situation recognition method of Fig. 4.

[0138] The location information of the matching object is tracked using an image-based object tracking algorithm, and the previous location information of the tracking matching object can be stored in latitude and longitude.

[0139] The previous positions of the tracked object can be fitted to an ellipse to estimate the direction of movement (COG, degree) and average speed (SOG, knot / s), and the long axis of the ellipse can be used to calculate the ratio (1.994 = meter to knot).

[0140] The final estimated output may include track ID, bearing, range, direction of movement (COG), and average speed (SOG). Here, track ID refers to the tracking number of the image-based object tracking algorithm.

[0141] FIGS. 9 to 12 are drawings exemplarily illustrating experimental results for object tracking for situational awareness in a maritime environment according to an embodiment of the present invention.

[0142] Figure 9 shows the results of multi-object tracking using sensor fusion, and Figure 10 is a graph comparing the tracking results of object 1 with AIS. The tracking results are compared with the COG and SOG of AIS. (AIS is assumed as ground truth)

[0143] Figure 11 is a graph comparing the object tracking result of object 7 with AIS, and Figure 12 is a graph comparing the object tracking result of object 17 with AIS.

[0144] The left pictures in FIGS. 9 to 12 show the results of obstacle detection by the camera, and the right pictures show the results of obstacle recognition through sensor fusion by the radar.

[0145] If the color of the box detected as an obstacle in the left picture and the color of the circle on the right are the same, they can be recognized as the same obstacle, and in Figures 10 to 12, the green line is the movement path of the obstacle, and the red line represents an arrow drawn in the direction and speed of movement of the obstacle.

[0146] Performance comparison can be confirmed with the graphs of Figs. 10 to 12, and since AIS directly transmits the sensor values ​​of the relative obstacle, it can know the exact values ​​of the moving direction and moving speed.

[0147] Therefore, it is possible to compare the movement direction and movement speed calculated by the proposed method with those of AIS to see how similar they are.

[0148] Figure 10 shows the results of tracking an obstacle 2 km away, and Figures 11 and 12 show the results of tracking an obstacle 8 km away. The results show an average error of less than 3%.

[0149] The experimental results are the result of tracking the same obstacle for approximately 5 minutes, and because the proposed method successfully detected and tracked the obstacle without any intermediate failures, it was possible to calculate the accurate moving direction and moving speed.

[0150] Meanwhile, Fig. 13 shows the average course over ground (COG) / speed over ground (SOG) error between the tracking results and AIS.

[0151] That is, Fig. 13 is a graph showing the average COG / SOG error after simultaneously tracking four vessels for one hour to verify the reliability of long-term tracking performance.

[0152] This experiment was conducted to evaluate how accurately the tracking system could predict vessel movements using only other sensors (e.g., cameras and radar) without integrating AIS information. As shown in Figure 13, all four vessels demonstrated very low COG and SOG errors. COG errors ranged from 0.45 to 1.96 degrees on average, and SOG errors ranged from 0.18 to 0.32 knots on average, demonstrating stable performance. This demonstrates that the present invention can reliably identify the movements of surrounding vessels without AIS information, even in long-term operating environments.

[0153] According to the embodiments of the present invention described above, continuous detection and identification (recognition) of objects around an autonomous vessel using sensors such as cameras and radar, as well as real-time information analysis (tracking) of dynamic objects, can be performed more precisely and reliably. In particular, the present invention is implemented to provide more robust and accurate situational awareness results than a single sensor by recognizing surrounding obstacles through the fusion of sensors such as cameras, radar, AIS, and GPS.

[0154] Meanwhile, the combinations of each block in the attached block diagram and each step in the flowchart may be performed by computer program instructions. These computer program instructions may be installed in a processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, so that the instructions, executed by the processor of the computer or other programmable data processing equipment, create a means for performing the functions described in each block in the block diagram.

[0155] These computer program instructions may also be stored in a computer-usable or computer-readable recording medium (or memory) that can direct a computer or other programmable data processing equipment to implement a function in a specific manner, so that the instructions stored in the computer-usable or computer-readable recording medium (or memory) can also produce a manufactured item that includes instruction means for performing the function described in each block of the block diagram.

[0156] And, since the computer program instructions can also be installed on a computer or other programmable data processing equipment, a series of operation steps are performed on the computer or other programmable data processing equipment to create a process that is executed by the computer, so that the instructions that execute the computer or other programmable data processing equipment can also provide steps for executing the functions described in each block of the block diagram.

[0157] Additionally, each block may represent a module, segment, or portion of code that includes at least one executable instruction for performing a specific logical function(s). It should also be noted that in some alternative embodiments, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may actually be executed substantially concurrently, or the blocks may sometimes be executed in reverse order, depending on their respective functions.

[0158] 2. Description of the device and method for recognizing the surroundings of a ship according to the second embodiment.

[0159] Fig. 14 is a drawing showing a device for recognizing the surroundings of a ship according to an embodiment of the present invention.

[0160] Referring to FIG. 14, the surrounding situation recognition device (100) of the present embodiment can detect the surrounding situation of a ship sailing at night, for example, an object such as another ship around the ship, and notify a user, for example, a pilot controlling the operation of the ship, of the detection result.

[0161] This surrounding situation recognition device (100) can recognize the surrounding situation of the ship based on information provided from at least one of one or more sensors of the sensor unit (200) installed on the ship, such as a camera (210), GPS (220), and radar (230).

[0162] The surrounding situation recognition device (100) may include an input / output unit (110), a processor (120), a display unit (130), and a memory (140).

[0163] The input / output unit (110) can receive sensing information about the vessel, i.e., the own ship, from the sensor unit (200). For example, the input / output unit (110) can receive a photographed image of the surroundings of the own ship captured by the camera (210) of the sensor unit (200). In addition, the input / output unit (110) can receive location information of the own ship, such as latitude, longitude, and heading direction of the own ship's location, from the GPS (220) of the sensor unit (200). The input / output unit (110) can receive radar signals about the surroundings of the own ship from the radar (230). At this time, the radar signals can include approximately 1024 signals at each azimuth interval around the own ship.

[0164] Additionally, the input / output unit (110) can output the detection results for objects around the magnet detected by the processor (120) to the outside.

[0165] The processor (120) can detect one or more objects around the ship from sensing information provided from the sensor unit (200) through the input / output unit (110), such as a photographed image of the ship around the ship, a radar signal, and the ship's location information, using the situational awareness program (150) of the memory (140), and can display and output the detected objects on an electronic chart.

[0166] Here, the processor (120) can output an electronic chart showing the detected object to the display unit (130) to display it to the user or output it to an external device, such as a control device that controls ship operation, through the input / output unit (110).

[0167] The memory (140) can store a situation awareness program (150) and information necessary for its execution. The situation awareness program (150) receives sensing information through the input / output unit (110), fuses the sensing information to detect one or more objects, such as other vessels around the ship, and tracks the movement direction, speed, etc. of the detected objects to map them onto an electronic chart and output them.

[0168] Accordingly, the processor (120) can detect one or more objects around the charity using a situation recognition program (150), map the detected objects onto an electronic map, and output them through a display unit (130).

[0169] Figure 15 is a diagram showing the function of the situation recognition program of Figure 14.

[0170] Referring to FIG. 15, the situation awareness program (150) of the present embodiment may include an image object detection unit (151), a location information acquisition unit (152), a radar object detection unit (153), an object matching unit (154), and an object tracking unit (155).

[0171] The image object detection unit (151) can obtain a photographed image of the surroundings of the charity from the camera (210) and detect one or more image objects, for example, a first object, of the surroundings of the charity from the photographed image.

[0172] Here, the surrounding situation recognition device (100) of the present embodiment detects objects around the ship while the ship is navigating at night. Accordingly, the captured image received by the image object detection unit (151) of the present embodiment is an image captured by the camera (210) at night of the ship, i.e., the surroundings of the ship itself. Such captured image may include images of external lights of other ships, lights of fixed objects such as lighthouses, lights such as moonlight depending on weather conditions, and reflected lights of the aforementioned lights reflected on the sea surface.

[0173] Accordingly, the image object detection unit (151) can detect one or more first objects, such as other objects, around the charity from a photographed image including images for various lighting conditions.

[0174] To this end, the image object detection unit (151) can obtain a binary image from the received photographed image.

[0175] As described above, the captured image may include multiple lighting images with different brightness levels. Accordingly, the image object detection unit (151) may convert the captured image into a binary image to detect lighting above a certain brightness level, such as external lighting from another line, as a first object in the captured image.

[0176] For example, the image object detection unit (151) may convert pixels having a grayscale level higher than the threshold value among a plurality of pixels of a photographed image to a grayscale level of 255, for example, the maximum brightness, based on a preset threshold value, and may convert pixels having a grayscale level lower than the threshold value to a grayscale level of 0, for example, the minimum brightness. Accordingly, the image object detection unit (151) may convert the photographed image into a binary image composed of pixels having a grayscale level of 255 and pixels having a grayscale level of 0.

[0177] In addition, the image object detection unit (151) can cluster pixels having a 255 gray level among multiple pixels in a binary image and determine each of these one or more clusters as a first object candidate.

[0178] Next, the image object detection unit (151) can select one or more first objects from one or more first object candidates according to preset constraints.

[0179] Here, the constraints may be constraints on the size and position of the first object. For example, among one or more first object candidates, any object candidate whose size exceeds the reference size set in the constraint may be eliminated. Additionally, among one or more first object candidates, any object candidate located above the average horizontal line set in the constraint may be eliminated.

[0180] Additionally, the image object detection unit (151) can set a bounding box for each of one or more first objects selected according to constraints. Then, the image object detection unit (151) can estimate the distance between the charity and the corresponding bounding box, i.e., the corresponding first object, based on the set bounding box.

[0181] For example, the image object detection unit (151) can determine the lower center point of the bounding box as the point of contact with the sea surface. Then, the image object detection unit (151) can estimate the distance between the ship and the first object through triangulation based on the position, i.e., height, of the camera (210) installed on the ship and the point of contact with the predetermined sea surface.

[0182] The location information acquisition unit (152) can acquire self-location information from GPS (220) and determine the azimuth of the first object detected based on the self-location information.

[0183] Here, the charity location information is in the form of an NMEA sentence provided by GPS (220), and among the provided information, the latitude, longitude, and heading of the current location of the charity may be included.

[0184] The location information acquisition unit (152) can project the first object into a virtual radar image space and determine an azimuth based on the own location for the projected first object based on the own location information.

[0185] Accordingly, the bounding box of the first object detected by the image object detection unit (151) may include the estimated distance and azimuth of the first object with respect to the charity.

[0186] The radar object detection unit (153) can obtain radar signals from the radar (230) surrounding the vehicle and generate a radar image based on the radar signals. The radar object detection unit (153) can detect one or more radar objects, for example, one or more second objects, in the radar image.

[0187] For example, the radar object detection unit (153) can filter and remove noise from the radar image. Then, the radar object detection unit (153) can perform clustering on a plurality of clutters in the radar image from which noise has been removed to determine one or more second object candidates.

[0188] Additionally, the radar object detection unit (153) can select one or more second objects from one or more second object candidates according to preset constraints.

[0189] Here, the constraints may be conditions regarding the size and location of the second object. For example, among one or more second object candidates, an object candidate whose size exceeds the reference size set in the constraint may be removed. Additionally, among one or more second object candidates, an object candidate whose location is outside the field of view of the camera (210) set in the constraint may be removed.

[0190] Accordingly, the radar object detection unit (153) can select one or more second object candidates from which some object candidates have been removed according to constraints as the second object, and set a bounding box for the one or more selected second objects.

[0191] Here, the bounding box of the second object may include the distance and azimuth of the second object to the charity.

[0192] The object matching unit (154) can compare the similarity between the first object and the second object that have been detected and match a pair of corresponding objects as similar objects.

[0193] For example, the bounding box of the first object may include the distance and azimuth to the first object, and the bounding box of the second object may include the distance and azimuth to the second object.

[0194] The object matching unit (154) can extract a pair of objects, i.e., the first object and the second object, in which the absolute value of the azimuth difference between the first object and the second object is less than or equal to a preset first reference value. Here, the first reference value may be 5.

[0195] For example, if the azimuth of the first object is 20° and the azimuth of the second object is 22°, the absolute value of the azimuth difference between the two objects is 2, so the object matching unit (154) can extract the two objects.

[0196] In addition, the object matching unit (154) can compare the absolute value of the distance difference between a pair of extracted objects with a preset second reference value and match the pair of objects as similar objects accordingly. Here, the second reference value may be 1 nautical mile (NM).

[0197] For example, if the distance of the extracted first object is 3.5 NM and the distance of the second object is 3.1 NM, the absolute value of the distance difference between the two objects is 0.4 NM, which is smaller than the second reference value. Therefore, the object matching unit (154) can match the extracted first object and the second object as similar objects.

[0198] Here, the object matching unit (154) can determine a bounding box of a relatively large size as the bounding box of the similar object when the sizes of the bounding boxes of the first object and the second object matched as similar objects are different.

[0199] The object tracking unit (155) can track similar objects and display and output the results on an electronic map.

[0200] Here, the object tracking unit (155) can calculate the object's moving direction and average speed based on the previous point-in-time location information for the similar object. In addition, the object tracking unit (155) can calculate the object's latitude and longitude based on the azimuth of the similar object.

[0201] Accordingly, the object tracking unit (155) can display a similar object on the electronic chart according to the calculated latitude and longitude, and display the direction and speed of the displayed similar object based on the moving direction and average speed. Here, the direction of the similar object can be displayed as an image such as an arrow, and the speed can be displayed in the form of text.

[0202] Accordingly, the processor (120) can receive an electronic chart output from the object tracking unit (155), i.e., an electronic chart indicating the position, direction, and speed of an object detected around the ship, and output it to the display unit (130) for display. In addition, the processor (120) can output the electronic chart to the outside through the input / output unit (110).

[0203] In this way, the surrounding situation recognition device (100) of the present embodiment can detect the presence of objects around the ship by fusing information provided from various sensors installed on the ship, such as a camera (210), GPS (220), and radar (230), and track the detected objects to display them to a user who controls ship operation through an electronic chart.

[0204] Accordingly, the present invention can accurately determine the situation around the ship's route, i.e., the presence of other ships, during nighttime navigation of a ship, thereby increasing the stability of nighttime navigation of the ship.

[0205] FIG. 16 is a drawing showing a method for recognizing the surroundings of a vessel navigating at night according to an embodiment of the present invention, and FIGS. 17 to 19 are drawings specifically showing a method for recognizing the surroundings of a vessel navigating at night according to the present invention.

[0206] Referring to FIG. 16, when the surrounding situation recognition device (100) of the present embodiment receives sensing information about the surroundings of a ship from the sensor unit (200), it can detect the presence of an object around the ship using the situation recognition program (150) and display it through an electronic chart.

[0207] The image object detection unit (151) of the surrounding situation recognition device (100) can obtain a photographed image of the surroundings of the ship, i.e., the ship's own ship, from the camera (210) and detect one or more first objects around the ship's own ship from the photographed image.

[0208] The image object detection unit (151) can obtain a binary image from a photographed image based on a preset threshold value.

[0209] For example, the image object detection unit (151) can obtain a binary image composed of pixels of a 255-level grayscale and pixels of a 0-level grayscale by converting pixels having a grayscale level higher than the threshold among a plurality of pixels of a photographed image to a 255-level grayscale and converting pixels having a grayscale level lower than the threshold to a 0-level grayscale based on a preset threshold.

[0210] Next, the image object detection unit (151) can cluster pixels having a 255 gray level among multiple pixels in a binary image and determine each of one or more clusters as a first object candidate (S120).

[0211] Continuing, the image object detection unit (151) can select one or more first objects from one or more first object candidates according to constraints on the size and position of the preset first object (S130).

[0212] And, the image object detection unit (151) can set a bounding box for each of one or more selected first objects and estimate the distance between the charity and the first object based on the bounding box (S140).

[0213] The location information acquisition unit (152) can acquire self-location information from GPS (220) and determine the azimuth of the first object detected based on the self-location information (S20).

[0214] Here, the location information acquisition unit (152) can project the first object into a virtual radar image space and determine the azimuth for the projected first object based on the self-referential location information. Accordingly, the bounding box of the first object can include the estimated distance and azimuth.

[0215] Next, the radar object detection unit (153) obtains a radar signal from the radar (230) around the charity to generate a radar image, and can detect one or more second objects in the radar image (S30).

[0216] The radar object detection unit (153) can filter and remove noise from the radar image (S210).

[0217] Next, the radar object detection unit (153) can perform clustering on multiple clutters in the noise-removed radar image to determine one or more second object candidates (S220).

[0218] Next, the radar object detection unit (153) can select one or more second objects from one or more second object candidates according to constraints on the size and location of the preset second object (S230).

[0219] Next, the radar object detection unit (153) can set a bounding box including the distance and azimuth to the charity for one or more selected second objects (S240).

[0220] The object matching unit (154) can compare the similarity between the first object and the second object that have been detected and match a pair of corresponding objects as similar objects (S40).

[0221] The object matching unit (154) can calculate the absolute value of the azimuth difference between the first object and the second object, and compare the calculated value with the first reference value (Ref1) (S310).

[0222] Here, if the calculated value, i.e., the absolute value of the azimuth difference, is less than or equal to the first reference value (Ref1), the object matching unit (154) can extract a corresponding pair of objects, i.e., the first object and the second object. On the other hand, if the calculated value exceeds the first reference value (Ref1), the detection step (S10) of the first object through the aforementioned image object detection unit (151) can be re-performed.

[0223] Next, when a pair of objects is extracted based on a comparison with the first reference value, the object matching unit (154) can calculate an absolute value for the difference between the distance of the extracted first object and the distance of the second object, and compare the calculated value with the second reference value (Ref2) (S320).

[0224] Here, if the calculated value, i.e., the absolute value of the distance difference, is less than or equal to the second reference value (Ref2), the object matching unit (154) can match the first object and the second object as similar objects (S330). On the other hand, if the calculated value exceeds the second reference value (Ref2), the detection step (S10) of the first object through the aforementioned image object detection unit (151) can be re-performed.

[0225] Here, the object matching unit (154) can determine a bounding box of a relatively large size as the bounding box of the similar object when the sizes of the bounding boxes of the first object and the second object matched as similar objects are different.

[0226] The object tracking unit (155) can track similar objects and display the results on an electronic map (S50).

[0227] Here, the object tracking unit (155) can calculate the moving direction and average speed of the object based on the position information of the previous point in time for the similar object, and can calculate the latitude and longitude of the object based on the azimuth of the similar object.

[0228] Accordingly, the object tracking unit (155) can display the object and its direction and speed on the electronic chart based on the latitude, longitude, direction of movement and average speed of the similar object.

[0229] The combinations of each block in the block diagram and each step in the flowchart described above may be performed by computer program instructions. These computer program instructions may be installed in a processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, so that the instructions, executed by the processor of the computer or other programmable data processing equipment, create a means for performing the functions described in each block in the block diagram.

[0230] These computer program instructions may also be stored in a computer-usable or computer-readable recording medium (or memory) that can direct a computer or other programmable data processing equipment to implement a function in a specific manner, so that the instructions stored in the computer-usable or computer-readable recording medium (or memory) can also produce a manufactured item that includes instruction means for performing the function described in each block of the block diagram.

[0231] And, since the computer program instructions can also be installed on a computer or other programmable data processing equipment, a series of operation steps are performed on the computer or other programmable data processing equipment to create a process that is executed by the computer, so that the instructions that execute the computer or other programmable data processing equipment can also provide steps for executing the functions described in each block of the block diagram.

[0232] Additionally, each block may represent a module, segment, or portion of code that includes at least one executable instruction for performing a specific logical function(s). It should also be noted that in some alternative embodiments, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may actually be executed substantially concurrently, or the blocks may sometimes be executed in reverse order, depending on their respective functions.

[0233] 3. Description of the vessel's navigational restriction area recognition system

[0234] FIG. 20 shows the configuration of a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention, FIG. 21 shows the configuration of a ship according to another embodiment of the present invention, FIG. 22 shows the process of fusion and outline determination of radar data and AIS data of a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention, FIG. 23 shows the process of fusion and outline determination of radar data, AIS data, and electronic chart data of a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention, FIG. 24 shows the detection result of a cluster outline of a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention, FIG. 25 shows the result of recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention on an electronic chart, FIG. 26 shows the result of visualizing a non-navigable area recognized by a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention on radar and an electronic chart, FIG. 27 is a diagram showing a radar echo adaptation threshold and a generated PPI image in a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention, and FIG. 28 shows the process of spatiotemporal filtering of a radar PPI image in a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention. It is a drawing, and FIG. 29 is a drawing showing the data results before and after filtering (left: before, right: after) in a system for recognizing a non-navigable area of ​​a ship according to an embodiment of the present invention.

[0235] Referring to FIGS. 20 to 28, a system (100) for recognizing a non-navigable area of ​​a ship according to the present embodiment includes a data receiving unit (110) that receives radar data from a marine radar and AIS data from an automatic identification system (AIS); a preprocessing unit (120) that preprocesses the radar data and AIS data received through the data receiving unit (110); a probability distribution generating unit (130) that generates a probability distribution for each of the radar data and AIS data preprocessed through the preprocessing unit (120); a probability distribution fusion unit (140) that fuses the probability distributions generated through the probability distribution generating unit (130) into one space; and an non-navigable area output unit (150) that determines a non-navigable area based on the probability distribution fused through the probability distribution fusion unit (140) and outputs the result to the outside.

[0236] The vessel's no-navigation zone recognition system (100) can be configured to precisely recognize the no-navigation zone around the vessel in real time by fusing data received from marine radar and AIS (Automatic Identification System).

[0237] The data receiving unit (110) may be provided to perform an important role of acquiring raw data about the environment around the ship at the forefront of the ship's navigation-free zone recognition system (100).

[0238] Specifically, the data receiving unit (110) may include a radar receiving module (111) that receives radar data from the marine radar; and an AIS receiving module (112) that receives AIS data from the ship automatic identification device.

[0239] First, the radar receiving module (111) may be configured to receive radar echo data from the surrounding environment via the marine radar antenna. The radar receiving module (111) emits radio waves and analyzes the reflected signals to obtain physical detection information regarding the general shape of the surrounding environment. This radar data can have the advantage of enabling wide-range detection even in environments with limited visibility, such as inclement weather or at night. This allows the detection of obstacles around the vessel even in conditions where visual recognition is difficult.

[0240] In addition, the AIS receiving module (112) may be arranged to receive automatic identification system (AIS) data from an AIS device. The AIS receiving module (112) receives digital data voluntarily transmitted by surrounding vessels, and such AIS data may include quantitative information such as the vessel's unique identification information (MMSI), precise location (latitude, longitude), average speed (SOG), direction of movement (COG), type and size of the vessel, depending on the implementation. In addition, AIS data is particularly effective in detecting the presence of small vessels (e.g., fishing boats, yachts, etc.) or non-metallic objects that are difficult for radar to detect, and has the characteristics of high accuracy and reliability of the transmitted information.

[0241] In this way, the data receiving unit (110) can separate radar data and AIS data with different physical characteristics and strengths from the initial stage and receive them separately, thereby maximizing the strengths of each and improving efficiency in the subsequent processing stage and stability of data processing.

[0242] The preprocessing unit (120) may be provided to remove noise, errors, uncertainties, etc. contained in the raw radar data and AIS data received from the data receiving unit (110), and to refine and standardize the data into a form suitable for subsequent probability distribution generation and fusion processing.

[0243] Specifically, the preprocessing unit (120) may include a radar data filtering module (121) that performs filtering on the radar data; and an AIS data projection module (122) that projects the AIS data into the same space as the radar data.

[0244] The radar data filtering module (121) can be provided to solve the problem that radar data frequently generates noise or causes irregular flickering of echo signals due to unnecessary diffuse reflections (clutter) caused by waves, tsunamis, precipitation, or land or ship structures due to the characteristics of the maritime environment.

[0245] The above radar data filtering module (121) may be configured to remove noise contained in the radar data through temporal filtering. Specifically, the radar data filtering module (121) may effectively remove instantaneous noise components by applying a temporal filtering technique that averages multiple radar echo data accumulated over a certain period of time.

[0246] That is, the radar data filtering module (121) receives GPS (Global Positioning System) data in real time, considering that the actual position of radar echo data may slightly change depending on the real-time movement of the ship (e.g., turning movement, forward movement, backward movement). Therefore, the radar data filtering module (121) corrects the echo data by the amount of change in the ship's position between the time each echo data was acquired and the current system's reference time using the received GPS information, and then performs filtering, thereby improving the accuracy of the data and enabling more consistent recognition of the positions of actual terrain features or fixed obstacles. In addition, in order to remove fine noise or spike noise that may remain even after temporal filtering, image processing techniques (e.g., morphological operation, Gaussian blurring, median filtering, etc.) may be additionally applied to improve the final precision of the radar data. In other words, by establishing a foundation for generating a more reliable probability distribution in the subsequent stage, it can contribute to improving the recognition accuracy of the entire system.

[0247] Meanwhile, the AIS data projection module (122) may be configured to place the AIS data received through the AIS receiving module (112) on the same spatial coordinate system as the radar data. The AIS data may be received primarily in the form of latitude and longitude for obstacles such as surrounding vessels according to the NMEA-0183 protocol.

[0248] In addition, the AIS data projection module (122) may be configured to project the AIS data onto the spatial coordinate system of the radar data using the ship's own position and azimuth information. That is, by utilizing the ship's real-time GPS position and azimuth (heading) information, the acquired latitude and longitude coordinates of the AIS-based obstacle can be projected onto a two-dimensional plane space such as a PPI (Plan Position Indicator) image coordinate system in which the radar echo is expressed.

[0249] Furthermore, spatial synchronization through the AIS data projection module (122) can contribute to increasing the efficiency and accuracy of probability distribution generation and fusion by logically integrating data with different physical characteristics and enabling fusion processing in a single space. This is particularly essential for accurately identifying the locations of real-time obstacles not recorded on electronic charts (e.g., groups of fishing vessels in operation, moving buoys, etc.), and can contribute to precisely securing location information on non-metallic vessels and small buoys that are difficult to identify with radar alone.

[0250] The probability distribution generation unit (130) may be configured to perform the role of converting the radar data and AIS data refined from the preprocessing unit (120) into probabilistic information according to the characteristics of each sensor.

[0251] Specifically, the probability distribution generation unit (130) may include a radar probability distribution generation module (131) that generates a probability distribution based on the intensity value of the preprocessed radar data; and an AIS probability distribution generation module (132) that generates a probability distribution based on the projection position of the preprocessed AIS data.

[0252] The radar probability distribution generation module (131) may be configured to generate a Gaussian probability distribution based on the intensity value of the reflected signal of each pixel or position data in the radar PPI (Plan Position Indicator) image. Since the radar signal intensity varies depending on the size, material, distance, etc. of the obstacle, it can be determined that the higher the intensity value, the higher the probability that an actual obstacle exists at the corresponding location. The radar probability distribution generation module (131) generates a Gaussian distribution with the intensity value as the average and an appropriate variance value, thereby probabilistically expressing the uncertainty of radar data and representing the possibility of the existence of an obstacle as a continuous value. Through this, it is possible to quantify the degree of possibility of the existence of an obstacle as well as simply determine the presence or absence thereof, thereby enabling more flexible and accurate judgment when determining an unnavigable route through the unnavigable area output unit (150).

[0253] The AIS probability distribution generation module (132) can be designed to overcome the disadvantage of providing only point-type location information due to the nature of AIS data and not directly providing information on physical size or shape, as in radar.

[0254] That is, the AIS probability distribution generation module (132) may be configured to generate a virtual circle (circular region) based on the projection position of the AIS data, and generate a probability distribution for the internal region of the virtual circle. Specifically, based on the position of the AIS vessel projected through the AIS data projection module (122), the vessel may be recognized as an obstacle of a certain size rather than a point, and a virtual circle may be generated. This virtual circle may be set using a diameter, such as the size of an obstacle recognized on average by the radar, and a probability distribution may be generated for the internal region of this circle.

[0255] In addition, the AIS probability distribution generation module (132) may be arranged to adjust the diameter of the virtual circle according to the distance from the target recognized by the marine radar.

[0256] That is, in the AIS probability distribution generation module (132), the variable values ​​used for probability calculation or the criteria for the diameter of the virtual circle can be set as adaptive variables that dynamically change according to the recognition distance set in the radar system. In addition, the AIS probability distribution generation module (132) can reflect uncertainty by setting the virtual circle diameter larger when recognizing a long-distance obstacle, and can enable precise recognition by setting a smaller diameter when recognizing a short-distance obstacle, thereby generating an optimized probability distribution under various recognition distance conditions. This can have the effect of comprehensively expanding the recognition range of the no-navigation zone to include ships that are not visible to radar but have AIS (for example, ships hidden by the radar shadow at the rear of a ship). Therefore, the AIS probability distribution generation module (132) can complement the uncertainty of AIS data, increase the effectiveness in an actual navigation environment, and maximize the synergy of sensor fusion.

[0257] The probability distribution fusion unit (140) may be provided to perform a key role of fusion of the radar probability distribution and AIS probability distribution generated by the probability distribution generation unit (130) into a single integrated spatial probability distribution. That is, the probability distribution fusion unit (140) may perform a role of recognizing complex navigation environment information that is difficult to grasp with a single sensor more accurately and reliably by combining the unique advantages of each sensor data (e.g., wide detection range and physical shape detection of radar, accurate identification and location information of AIS).

[0258] This probability distribution fusion unit (140) can generate an integrated probability map by combining the two probability distributions of the probability distribution generation unit (130) in an overlay manner, or by applying an advanced fusion algorithm such as a weighted average, a product rule, or Bayes filtering. That is, when AIS information exists in an area where radar signals are weak, the AIS probability distribution generated by the AIS probability distribution generation module (132) can reduce the uncertainty of the area and reinforce the probability of obstacle existence, thereby increasing recognition accuracy. Conversely, small vessels without AIS signals or irregular obstacles such as fishing nets can be effectively recognized through the radar probability distribution generated by the radar probability distribution generation module (131).

[0259] In particular, in addition to common obstacles such as ships and buoys, the environment around a vessel may also contain areas that are impassable, such as land, islands, schools of fishing vessels in operation, and fishing nets. While existing electronic navigational charts contain impassable areas, this information is primarily recorded statically and may lack real-time functionality. Therefore, the present invention addresses the aforementioned lack of real-time functionality of electronic navigational charts by fusing radar and AIS data through a probability distribution fusion unit (140), and is remarkably effective in recognizing previously unrecorded real-time impassable areas. The integrated spatial probability distribution generated through this fusion process provides the most up-to-date and reliable probabilistic recognition information on all types of impassable areas around a vessel, which maximizes the accuracy of the next step, determining impassable areas.

[0260] The impossible navigation area output unit (150) may be configured to determine the final impossible navigation area based on the integrated spatial probability distribution generated by the probability distribution fusion unit (140) and provide it to the ship's navigation system and operator.

[0261] The above-mentioned impossible area output unit (150) may include a cluster generation module (151) that generates clusters from a probability distribution fused through the probability distribution fusion unit (140); a cluster selection module (152) that selects clusters determined to be impossible areas among the clusters generated through the cluster generation module (151); and an outline extraction module (153) that extracts outlines of the clusters selected through the cluster selection module (152) and determines them as impossible areas.

[0262] The cluster generation module (151) may be configured to cluster adjacent pixels or regions with a probability exceeding a preset threshold value on the fused probability distribution map. This threshold value may be dynamically set based on various factors, such as maritime safety regulations, vessel type, and operating environment characteristics. The sensitivity of the recognition may be controlled by adjusting the threshold value. For example, this may be implemented by recognizing regions with a probability value of 0.7 or higher as potential obstacle clusters. This may be essential for distinguishing between uncertain signals and clear obstacle signals.

[0263] The cluster selection module (152) may be configured to select, from among the multiple clusters generated by the cluster generation module (151), meaningful clusters that are judged to be threatening to actual navigation or to be unnavigable areas requiring avoidance. This process may utilize cluster size (area), shape, density, etc. as criteria. For example, clusters smaller than a preset minimum size or deemed to have no impact on navigation safety may be filtered out.

[0264] In addition, the cluster selection module (152) may be configured to select multiple vessels anchored near a port as a single vessel cluster. There are cases where multiple vessels anchored near a port are recognized as a single, large cluster in radar and AIS fusion data. In this case, the cluster selection module (152) selects the cluster as a "ship cluster" (an unnavigable area) rather than a single object, thereby preventing problems such as radar instability or deterioration of tracking performance due to calculation errors that may occur when tracking a single object, thereby generating a more stable and realistic avoidance route.

[0265] In addition, the cluster selection module (152) may be configured to merge the overlapping areas and select them as a single cluster when the previously recognized non-navigable areas and the non-navigable areas determined through the outline extraction module (153) spatially overlap. In other words, this prevents redundant recognition and optimizes the recognition efficiency of the system.

[0266] The contour extraction module (153) accurately extracts the contours of the unnavigable clusters finally confirmed by the cluster selection module (152), and makes the final decision to designate these as "unnavigable areas" where ships should not approach. This contour information can be extracted in vector or raster form, thereby realizing a high degree of accuracy in which the contours of the actual terrain and the contours of the perceived area are detected similarly.

[0267] Furthermore, the contour extraction module (153) may be arranged to determine the final non-navigable area based on the contour of the non-navigable area corresponding to the direction of movement of the own ship. This is in consideration of the characteristic that in the case of radar data, since the reflected area is detected, an occluded area may occur behind the area where the signal is reflected, resulting in inaccurate data. Therefore, the contour extraction module (153) includes a process of determining the final visible contour based on the vector relationship between the center of the detected contour and the own ship, thereby preventing unnecessary area recognition and increasing the efficiency of generating an avoidance route.

[0268] In this way, the present invention transmits information on a navigational area in real time to a ship's navigation system so that it can be utilized for automatic route planning and correction, and at the same time, it can be clearly and visually displayed on a display device in the ship's wheelhouse so that the ship operator can intuitively recognize the surrounding environment and contribute to safe navigation.

[0269] According to another aspect of the present invention, a vessel (200) including the vessel's non-navigable area recognition system (100) described above can be provided. The vessel (200) can include additional components for maximizing navigational safety by utilizing the vessel's non-navigable area recognition system (100).

[0270] That is, the vessel (200) may include a route determination unit (210) that automatically plans or modifies the vessel's route using information on the navigation impossibility area output through the navigation impossibility area output unit (150).

[0271] The route decision unit (210) calculates an optimal avoidance route in real time based on the identified unnavigable area and, if necessary, safely modifies the existing route. This significantly enhances vessel safety by generating accurate and stable avoidance routes, particularly in narrow coastal navigation or complex waters. Furthermore, the route decision unit (210) can continuously update the route by reflecting dynamic changes in the unnavigable area (such as the movement of fishing vessel groups), thereby maintaining an optimal navigation strategy.

[0272] In addition, the vessel (200) may further include a warning output unit (220) that generates warning information about the navigation impossibility area output through the impossibility area output unit (150) and outputs it to the outside.

[0273] The warning output unit (220) comprehensively determines the approach distance, speed, or collision risk of a no-navigation area, and generates various warning information such as a signal alarm sound, warning messages on the display, and color changes, thereby immediately conveying this to the ship operator, thereby increasing awareness of dangerous situations and prompting a swift response. This plays a crucial role in preemptively recognizing obstacles such as fishing nets that are difficult to see with the naked eye, especially during nighttime navigation or in poor visibility conditions, and in preventing collision risks in advance, ultimately maximizing the safety of ship operation.

[0274] Meanwhile, the vessel navigation-free zone recognition system (100) according to one embodiment of the present invention, in addition to the warning output unit (220) described above, does not simply fuse radar and AIS data, but also reflects various environmental factors and risk assessment techniques, thereby maximizing usability in actual operating situations.

[0275] First, referring to FIG. 23, a probability distribution generation unit (130) according to an embodiment of the present invention may be arranged to adjust radar probability distribution values ​​and AIS probability distribution values ​​by reflecting a correction factor. Here, the correction factor may be electronic chart data provided from an electronic chart (ENC), and the electronic chart data includes structural navigation restriction information such as depth information, navigation prohibited areas, fishing nets and fish farms, buoys, etc.

[0276] The electronic chart data provided from such electronic charts can be received through the electronic chart receiving module (113), processed in the preprocessing unit (123), and then provided to the probability distribution generation unit (133). In addition, the electronic chart data can be projected into the same spatial coordinate system as the radar data using the ship's position and azimuth information, and then transmitted to the probability distribution generation unit (133).

[0277] Accordingly, the probability distribution generation unit (130) can produce a more precise risk distribution that reflects actual navigational restrictions by considering not only radar reflection intensity or AIS location information, but also electronic chart data. In other words, the probability distribution generation unit (130) can provide a more realistic and precise risk distribution by comprehensively considering environmental and structural data.

[0278] In addition, referring to FIG. 28, a cluster selection module (152) according to one embodiment of the present invention is provided to calculate a risk index and select a non-navigable area through the size of the cluster and the risk index.

[0279] Here, the risk indicators can include quantitative factors such as relative distance, approach angle, time to collision, and evasive maneuver margin, and can be updated in real time based on data accumulated over time.

[0280] Furthermore, the cluster selection module (152) can adjust the grid spacing and risk index calculation method according to the operating environment. For example, in a sea area where ships are concentrated, such as near a port, the grid spacing can be set finely so that even small clusters can be recognized, and in an open sea area, such as the ocean, the grid spacing can be set wide to improve computational efficiency.

[0281] In addition, referring to FIG. 29, the impossible area output unit (160) according to one embodiment of the present invention may be arranged not only to simply display the outline of the recognized impossible navigation area, but also to determine whether it overlaps with the expected route of the ship.

[0282] If the navigation impassable area overlaps with the navigation route of the ship, the impassable area output unit (160) can provide a visual, auditory, or tactile warning signal or alarm signal to the navigator through an external output unit.

[0283] For example, risk areas could be outlined in red on electronic navigation charts, a warning sound could be generated in the wheelhouse, or a warning message could be sent through the ship's control unit interface. These output functions can go beyond simple information provision and serve as a direct aid to operational decision-making.

[0284] Additionally, the unobstructed area output unit (160) can provide warnings along with quantitative indicators such as the expected time to collision (Time to Collision) and the distance to avoid maneuvers, as well as whether the vessel is overlapping with the route. For example, messages such as "Collision expected within 5 minutes" or "Approach within 500 meters" can be displayed, allowing navigators to intuitively understand the situation and respond quickly.

[0285] In addition, a reliability index can be calculated by synthesizing the risk index calculated from the cluster selection module (152) and the radar probability distribution value and AIS probability distribution value calculated from the probability distribution generation unit (130).

[0286] The above reliability index is provided to the electronic chart display system (ECDIS) or automatic radar plotting apparatus (ARPA) along with the output information, so that the navigator can intuitively check the reliability of the result along with the recognized unnavigable area.

[0287] That is, the system (100) for recognizing a vessel's navigation impossibility area according to one embodiment of the present invention can function as an intelligent decision support system beyond the existing single-sensor-based object detection technology, as the probability distribution generation unit (130) reflects a correction factor to produce a more precise risk distribution, the cluster selection module (152) evaluates the risk based on a risk index, and the navigation impossibility area output unit (160) determines whether there is an overlap with the vessel's expected route and provides a quantitative warning signal.

[0288] Therefore, the vessel's navigation impossibility area recognition system (100) according to one embodiment of the present invention can support the immediate response judgment of the navigator by having the impossibility area output unit (160) linked to the electronic chart to comprehensively display the impossibility area and the vessel's expected route, and can maximize operational efficiency and safety, especially in an operating environment requiring a high level of situational awareness, such as a port entry section, a densely populated fishing boat area, a narrow waterway, etc.

[0289] Furthermore, the vessel navigation impassability zone recognition system (100) according to one embodiment of the present invention can comprehensively reflect environmental changes and structural navigational constraints compared to existing simple sensor-based recognition technologies, as the probability distribution generation unit (130), cluster selection module (152), and impassability zone output unit (160) organically operate each other. Accordingly, the system can assist navigators' judgment in situations with a high risk of collision, such as narrow waterways, port entry sections, and waters densely populated with fishing vessels, thereby significantly improving the accuracy and reliability of collision avoidance judgments.

[0290] In addition, the reliability index calculated by the no-navigation area output unit (160) is output in conjunction with the electronic chart display system (ECDIS) or automatic radar plotting apparatus (ARPA), so that the navigator can comprehensively check the ship's expected route, no-navigation area, risk index, and reliability index on a single screen. In particular, since the above results are updated at a cycle of 1 second, the navigator can intuitively recognize the surrounding situation that is changing in real time and make immediate response decisions.

[0291] Accordingly, the present invention can provide an effect that can significantly improve the response speed and safety in emergency situations while increasing the efficiency of situational awareness in an actual operating environment.

[0292] In this way, the vessel navigation impossibility area recognition system (100) according to the present invention can accurately recognize a navigation impossibility area existing at a long distance in real time by effectively fusing radar data and AIS data, and can also improve navigation safety by including unrecorded real-time navigation impossibility areas (groups of fishing vessels, fishing nets, etc.) even in situations where visibility is limited, such as in bad weather or at night.

[0293] In addition, the vessel (200) according to the present invention can plan and modify an optimal route based on recognized navigational impassability area information, and can ultimately greatly contribute to the prevention of maritime accidents and safe navigation by inducing a quick response from the vessel operator through warnings of dangerous situations.

Claims

1. An input unit for inputting sensing data received from at least one sensor placed in a marine environment; A processing unit that detects at least one object based on the sensing data, performs alignment processing on the object, and fuses location information of the object based on the result of the alignment; and A storage unit that stores situational awareness information of the object in the maritime environment based on the result of the above fusion; A device that recognizes the ship's surroundings.

2. In paragraph 1, The above sensing data includes image data, radio wave data and identification data. A device that recognizes the ship's surroundings.

3. In paragraph 2, The above processing unit, A detection processing unit that classifies the image data based on artificial intelligence learning to detect first object information, generates a PPI (plane position indicator) image of the radio wave data to detect second object information, and decodes the identification data to detect third object information; A matching processing unit that matches the first object information, the second object information, and the third object information; and A fusion processing unit that applies an image-based object tracking algorithm to the matching result of the above matching processing unit to fuse the first object information, the second object information, and the third object information; A device that recognizes the ship's surroundings.

4. In paragraph 3, The above detection processing unit detects the first object information by correcting the distortion of the image data, and detects the second object information by filtering the radio wave data. A device that recognizes the ship's surroundings.

5. In paragraph 3, The above alignment processing unit aligns the position of the object based on the first object information and the second object information to determine the bearing and range of the object. A device that recognizes the ship's surroundings.

6. In paragraph 3, The above fusion processing unit estimates the direction of movement (course of ground, COG) and average speed (speed of ground, SOG) of the object by fitting the previous position of the object to an ellipse based on the third object information. A device that recognizes the ship's surroundings.

7. A step of receiving a photographed image of the surrounding area from a camera and detecting one or more first objects; A step of receiving a radar signal from the radar around the charity and detecting one or more second objects; A step of matching corresponding objects among the one or more first objects and the one or more second objects as similar objects through a similarity comparison; and A step of tracking the matched similar object and displaying it on an electronic map. How to recognize the surroundings of a ship.

8. In paragraph 7, The step of detecting one or more first objects comprises: A step of obtaining a binary image from the above photographed image; A step of determining one or more object candidates through clustering of multiple pixels in the above binary image; A step of selecting one or more first objects from among the one or more object candidates based on preset constraints; and A step of setting a bounding box for each of the one or more first objects and estimating a distance for each bounding box is included. How to recognize the surroundings of a ship.

9. In paragraph 8, The step of estimating the distance for each bounding box is: A step of determining the lower center point of the above bounding box as the contact point with the sea surface; and A step of estimating the distance between the charity and the bounding box based on the height from the sea surface to the camera. How to recognize the surroundings of a ship.

10. In paragraph 8, The step of determining one or more object candidates is: A step of removing object candidates having a size greater than a reference size from among the one or more object candidates based on the above constraints; and A step of removing an object candidate located on an average horizontal line among the one or more object candidates based on the above constraints. How to recognize the surroundings of a ship.

11. In paragraph 7, A step of obtaining charity location information from GPS; and A step of projecting the one or more first objects onto a virtual radar image based on the above charity location information to determine an azimuth for each of the one or more first objects. How to recognize the surroundings of a ship.

12. In paragraph 7, The step of detecting one or more second objects comprises: A step of removing noise in a radar image generated from the above radar signal; A step of determining one or more object candidates through clustering of multiple clutters in the radar image; A step of selecting one or more second objects from among the one or more object candidates based on preset constraints; and A step of setting a bounding box including an azimuth and a distance for each of the one or more second objects. How to recognize the surroundings of a ship.

13. In paragraph 12, The step of determining one or more object candidates is: A step of removing object candidates having a size greater than a reference size from among the one or more object candidates based on the above constraints; and A step of removing an object candidate located outside the field of view of the camera among the one or more object candidates based on the above constraints. How to recognize the surroundings of a ship.

14. In paragraph 7, The step of matching with the above similar object is: A step of extracting a pair of objects in which the azimuth difference between the one or more first objects and the one or more second objects is less than or equal to a first reference value; and If the distance difference between the pair of objects is less than or equal to the second reference value, the step of matching the pair of objects as similar objects is included. How to recognize the surroundings of a ship.

15. In paragraph 7, The steps indicated on the electronic chart above are: A step of calculating the movement direction and average speed based on the previous point-in-time location information of the above similar object; A step of calculating latitude and longitude based on the azimuth of the similar object; and A step of displaying the similar object on the electronic chart based on the latitude and longitude, and displaying the direction and speed of the displayed similar object based on the moving direction and average speed. How to recognize the surroundings of a ship.

16. A data receiving unit that receives radar data from a marine radar and AIS data from a ship automatic identification device; A preprocessing unit that preprocesses radar data and AIS data received through the above data receiving unit; A probability distribution generation unit that generates a probability distribution for each of the radar data and AIS data preprocessed through the above preprocessing unit; A probability distribution fusion unit that fuses probability distributions generated through the above probability distribution generation unit into one space; and An impossibility zone output unit that determines an impossibility zone based on the probability distribution fused through the above probability distribution fusion unit and outputs it to the outside; A system for recognizing a vessel's no-navigation zone.

17. In paragraph 16, The above data receiving unit, A radar receiving module for receiving radar data from the above marine radar; and An AIS receiving module that receives AIS data from the above ship automatic identification device; A system for recognizing a vessel's no-navigation zone.

18. In paragraph 16, The above preprocessing unit, A radar data filtering module that performs filtering on the above radar data; and An AIS data projection module that projects the AIS data into the same space as the radar data; A system for recognizing a vessel's no-navigation zone.

19. In paragraph 18, The above radar data filtering module, It is designed to remove noise contained in the above radar data through temporal filtering. A system for recognizing a vessel's no-navigation zone.

20. In paragraph 18, The above AIS data projection module, It is arranged to project the AIS data onto the spatial coordinate system of the radar data using the ship's position and azimuth information. A system for recognizing a vessel's no-navigation zone.

21. In paragraph 16, The above probability distribution generation unit, A radar probability distribution generation module that generates a probability distribution based on the intensity value of the above-mentioned preprocessed radar data; and An AIS probability distribution generation module that generates a probability distribution based on the projection position of the above-mentioned preprocessed AIS data; A system for recognizing a vessel's no-navigation zone.

22. In paragraph 21, The above AIS probability distribution generation module is, A virtual circle is generated based on the projection position of the above AIS data, and a probability distribution is generated for the inner area of ​​the virtual circle. A system for recognizing a vessel's no-navigation zone.

23. In paragraph 22, The above AIS probability distribution generation module is, The diameter of the above virtual circle is adjusted according to the distance from the target recognized by the above marine radar. A system for recognizing a vessel's no-navigation zone.

24. In paragraph 16, The above-mentioned impossible area output section is, A cluster generation module that generates clusters from the probability distribution fused through the above probability distribution fusion unit; A cluster selection module that selects a cluster determined to be an unnavigable area among the clusters generated through the cluster generation module; and An outline extraction module that extracts the outlines of the clusters selected through the above cluster selection module and determines them as non-navigable areas; A system for recognizing a vessel's no-navigation zone.

25. In paragraph 24, The above cluster selection module, It is designed to select multiple ships anchored near a port as a single ship group. A system for recognizing a vessel's no-navigation zone.

26. In paragraph 24, The above cluster selection module, In cases where previously recognized non-navigable areas and non-navigable areas determined through the outline extraction module overlap, the overlapping areas are merged and selected as a single cluster. A system for recognizing a vessel's no-navigation zone.

27. In paragraph 24, The above outline extraction module, The final no-navigation zone is determined based on the outline of the no-navigation zone corresponding to the direction of the charity's progress. A system for recognizing a vessel's no-navigation zone.

28. In paragraph 21, The above probability distribution generation unit The radar probability distribution value generated through the above radar probability distribution generation module and the AIS probability distribution value generated through the above AIS probability distribution generation module are adjusted by reflecting the correction factor provided through the electronic chart. A system for recognizing a vessel's no-navigation zone.

29. In paragraph 24, The above cluster selection module It is designed to calculate the risk index and select the unnavigable area through the size of the cluster and the risk index, and to adjust the grid spacing and risk index calculation method according to the operating environment conditions. A system for recognizing a vessel's no-navigation zone.

30. In paragraph 29, The above-mentioned impossible area output section If the selected no-navigation area overlaps with the expected route of the ship, it is arranged to provide output information including a warning signal or alarm signal. A system for recognizing a vessel's no-navigation zone.

31. In paragraph 30, The above-mentioned impossible area output section A reliability index is calculated by synthesizing the risk index provided from the above cluster selection module and the radar probability distribution value and AIS probability distribution value produced by the above probability distribution generation unit, and the reliability index is output together to an electronic chart display system (ECDIS) or an automatic radar plotting device (ARPA). A system for recognizing a vessel's no-navigation zone.

Images