Image-based floating marine object detection and distance measurement system for small vessel

Abstract

An image-based floating marine object detection and distance measurement system for a small vessel of less than 10 tons is disclosed. The system comprises: an infrared stereo camera and an IR projector provided on a vessel; and an image display system which is connected to the infrared stereo camera and the IR projector through a control API, includes an image correction filter for correcting blurriness due to light reflection, shadow, and sea fog, detects the floating marine object (a discarded fishing net, a rope, a buoy, and marine litter) by a deep learning-based object detection model on the basis of pre-training data of the floating marine object, measures a distance between a camera of a vessel of less than 10 tons and the floating marine object, and displays an image to prevent vessel collision, damage to a propulsion shafting system, and an accident of entanglement between a vessel and floating marine debris through a forward hazard situation alert and a corresponding response. The system uses an infrared stereo camera and an IR projector provided on a small vessel of less than 10 tons in which hull motion, sea fog, and light reflection due to wind and wave heights exist on the sea, so as to detect a floating marine object posing a hazard during vessel navigation, by a vision recognition technology using a floating marine object recognition distance measurement algorithm and a deep learning-based floating marine object recognition model using an AI technology, prevents vessel collision, damage to a propulsion shafting system, and an accident of entanglement between a vessel and floating marine debris through a hazard situation alert and a corresponding response, monitors marine litter, and is used as a marine safe navigation assistance system for a small vessel that is vulnerable to the floating marine object.

Description

Image-based marine floating object detection and distance measurement system for small vessels

[0001] The present invention relates to an image-based marine floating object detection and distance measurement system for small vessels under 10 tons. More specifically, it utilizes an infrared stereo camera and an IR illuminator equipped on a small vessel under 10 tons where hull shaking due to wind and wave height, sea fog, and light reflection exist at sea, and measures the distance between the vessel and the marine floating object using a marine floating object recognition object distance measurement algorithm based on a deep learning-based marine floating object object recognition model using AI technology based on learning data of the marine floating object, detects marine floating objects (discarded fishing nets, ropes, grills, marine debris, etc.) that pose a risk during the operation of a small vessel under 10 tons using vision recognition technology, prevents vessel collisions, damage to the propulsion shaft system, and accidents of entanglement between the vessel and floating objects (discarded fishing nets) by providing alarm warnings and responses to forward danger situations in the vessel's course, and monitors marine debris. It is utilized as a marine safety navigation assistance system for small vessels under 10 tons that are vulnerable to marine floating objects.

[0002] According to recent marine accident statistics, small vessels under 10 tons account for the majority of marine accidents at 70%, with damage to propulsion shaft systems and entanglement by floating objects caused by floating bodies accounting for the largest share at 18%, and the share reaching 32% when including ship collision accidents. A lack of forward visibility is a major cause of collision accidents, and with the recent increase in ships equipped with automatic steering gears, situations involving a lack of forward visibility are becoming more frequent.

[0003] Figure 1 is a diagram showing the current status and types of marine accidents and the status of entanglement accidents (Ministry of Oceans and Fisheries).

[0004] Figure 2 is a photograph showing the detection limits of major ship safety navigation aids and radars. Although there are many existing commercial products for safe ship navigation, such as radio radars and plotters, these are equipment based on radio radar or ultrasound (depth measurement) and can only detect large objects such as ships, navigational aid buoys, and reefs. While effective in preventing ship accidents such as collisions and groundings, they have limitations in being applied to prevent types of maritime accidents such as entanglement of floating bodies by small ropes or buoys, or damage to propulsion shaft systems.

[0005] Currently, most related technology development focuses on Level 3 or higher autonomous operation for large vessels, and it is realistically difficult to introduce this to small vessels under 10 tons.

[0006] In Korea, the "Autonomous Ship Technology Development Project," jointly promoted by the Ministry of Trade, Industry and Energy and the Ministry of Oceans and Fisheries in 2020, is currently being developed under government leadership, while private companies such as Hyundai Heavy Industries (HiNAS), Samsung Heavy Industries (SAS), and Daewoo Shipbuilding & Marine Engineering (DAN-V) are independently developing autonomous ships.

[0007] Although everyone is aware of the risks associated with floating structures on small vessels under 10 tons, accidents frequently occur due to inattention caused by looking straight ahead, and there is a need for a maritime safety assistance system that supports intuitive decision-making tailored to site characteristics (Autonomy Level 1).

[0008] As prior art related to this, Patent No. 10-1545276 is registered as 'a ship-mounted omnidirectional monitoring system using a PTZ camera and a ship-mounted omnidirectional monitoring method using the same'.

[0009] Figure 1 is a schematic diagram showing a ship-mounted omnidirectional monitoring system using a conventional PTZ camera.

[0010] A ship-mounted omnidirectional monitoring system using a PTZ camera

[0011] A camera unit (110) including a plurality of PTZ cameras installed around the vessel and acquiring surrounding images according to the operation mode of the vessel through a PTZ (pan-tilt-zoom) function;

[0012] A camera control unit (120) that controls the overall operation of each of the plurality of PTZ cameras;

[0013] An image registration unit (130) that extracts feature points from image information provided by each of the plurality of cameras, extracts the relationship between feature points of an overlapping area, and then sequentially performs a geometric transformation process, a color normalization process, and an edge blending process to register them into a single image;

[0014] An image overlay unit (140) that outputs an overlay image by overlaying the matching image provided by the image matching unit (130) and an auxiliary image required for the operation mode of the vessel;

[0015] An image information display unit (150) that displays an all-around image of the vessel to the navigator by aligning the images provided by the image overlay unit (140) according to the position of the PTZ camera;

[0016] It includes an operation mode processing unit (160) that provides different auxiliary image information to the image overlay unit (140) according to the operation mode of the above vessel, and

[0017] The above operation mode processing unit (160) includes an image control console (161) that stores and controls auxiliary images according to the operation mode of the vessel and provides them to the image overlay unit.

[0018] When the above image control console (161) is operated in navigation mode, it provides the above auxiliary image information, which displays at least one of the ship's course, speed, safe waters, navigation markers, and object information required for navigation mode, to the image overlay unit (140).

[0019] When the above vessel is operated in a docking mode, the above auxiliary image information, which displays at least one of the vessel's course, bow / stern speed, distance to the quay, and direction information required for the above docking mode, is provided to the image overlay unit (140).

[0020] An omnidirectional monitoring system for ships using a PTZ camera and an omnidirectional monitoring method for ships using the same have the advantage of being able to provide the navigator with real-time omnidirectional images of the surroundings required for each mode when the ship is operating in navigation mode and docking / undocking mode, respectively.

[0021] In addition, when the vessel is operating in navigation mode, a long-distance omnidirectional image can be superimposed with an auxiliary image displaying the vessel's course, speed, safe zone, navigational aids, and landmark information required for navigation mode, allowing the navigator to perform navigation more comfortably.

[0022] In addition, when the vessel is operating in undocking / docking mode, a near-field omnidirectional image can be superimposed with an auxiliary image displaying the vessel's course, bow / stern speed, distance to the quay, and direction information required for the undocking / docking mode, allowing the navigator to perform the vessel's undocking / docking process more easily.

[0023] As related prior art 2, Patent Registration No. 10-1729725 discloses a 'real-time monitoring system for ships and offshore plants,' and

[0024] We provide integrated control technology for ships and offshore plants that utilizes international standard protocols for vessels to monitor various sensor devices in real time. In the event of an anomaly, this technology integrates seamlessly with an Internet Protocol-based CCTV system to digitize sensor data and supplementary information along with CCTV footage, transmitting this content to a land-based control center. This enables more accurate status verification, rapid response to failure situations, and proactive prevention of accidents.

[0025] A real-time monitoring system for ships and offshore plants comprises, in the case of a marine operation system, a sensor unit equipped with a wind direction and speed sensor, an azimuth sensor, an AIS sensor, a GPS sensor, etc., operating based on the international standard IEC 61162 for ships; a sensor server unit serving as a standardized information exchange platform between heterogeneous devices within the ship; an event server unit that notifies a shore control center of failure situations via a wireless network; multiple Internet Protocol cameras installed in key areas of the ship and offshore plant; a CCTV recording server that receives, processes, controls, and manages video data captured by the multiple Internet Protocol cameras; a video transcoding transmitter that recompresses video data when transmitting video to a remote location at low bandwidth; and multiple monitoring clients that perform real-time monitoring, playback of recorded video, and two-way voice communication through the CCTV recording server.

[0026] The ground control system consists of an integrated event server that receives event alerts regarding failure situations from the event server and propagates the emergency situation to the operator; and an integrated monitoring client that displays re-encoded video received from the video transcoding transmitter and displays status information of sensor data.

[0027] However, existing systems only provide omnidirectional monitoring functions using cameras, and do not provide a system that prevents ship collisions, propulsion shaft damage, and entanglement accidents by detecting marine floating objects (discarded fishing nets, ropes, grills, marine debris, etc.) that pose a risk during the operation of small vessels, and by issuing alarms and responding to dangerous situations. This is achieved by using deep learning-based image recognition technology to detect marine floating objects (discarded fishing nets, ropes, grills, marine debris, etc.) that pose a risk during operation of small vessels equipped with infrared stereo cameras and IR illuminators on small vessels under 10 tons at sea, which do not rely on existing radar or sensors, and by using vision recognition technology that uses a deep learning-based marine floating object object recognition model with AI technology and a marine floating object recognition object distance measurement algorithm, without relying on existing radar or sensors.

[0028] [Prior Art Literature]

[0029] (Patent Document 1) Patent Registration No. 10-1545276 (Registration Date August 11, 2015), 'Omnidirectional monitoring system for ships using a PTZ camera and omnidirectional monitoring method for ships using the same', Korea Institute of Ocean Science and Technology

[0030] (Patent Document 2) Patent Registration No. 10-1729725 (Registration Date April 18, 2017), 'Real-time Monitoring System for Ships and Offshore Plants', FG Electric Co., Ltd.

[0031] The objective of the present invention to solve the above problems is to provide an image-based marine floating object detection and distance measurement system for small vessels, which is utilized as a marine safety navigation assistance system for small vessels under 10 tons that are vulnerable to marine floating objects, by using an infrared stereo camera and an IR illuminator equipped on a small vessel under 10 tons where hull shaking due to wind and waves, sea fog, and light reflection exist at sea, and by using a deep learning-based marine floating object object recognition model using AI technology based on training data to measure the distance between the vessel and the marine floating object, and by using vision recognition technology to detect approaching vessels and marine floating objects (discarded fishing nets, ropes, grills, marine debris) that pose a risk during the operation of small vessels under 10 tons, and by preventing vessel collisions, damage to the propulsion shaft system, and accidents of entanglement between the vessel and floating objects (discarded fishing nets) through alarm warnings and responses to forward danger situations in the vessel's course, and by monitoring marine debris.

[0032] To achieve the objective of the present invention, an image-based offshore floating object detection and distance measurement system for small vessels includes an infrared stereo camera and an IR illuminator equipped on the vessel; and an image display system that is connected to the infrared stereo camera and the IR illuminator via a control API and is equipped with an image correction filter that corrects blur caused by light reflection, shadows, and sea fog, detects the offshore floating object by a deep learning-based object detection model based on prior training data of the offshore floating object, measures the distance between the camera of the vessel and the offshore floating object, and displays images to prevent vessel collision, propulsion shaft damage, and entanglement accidents between the vessel and the offshore floating object through alarm warnings and responses to forward danger situations on the vessel's route.

[0033] The image-based marine floating object detection and distance measurement system and method for small vessels according to the present invention utilizes an infrared stereo camera and an IR illuminator equipped on a small vessel of less than 10 tons where hull shaking due to wind and wave height, sea fog, and light reflection exist at sea, measures the distance between the vessel and the marine floating object using a marine floating object recognition object distance measurement algorithm using a deep learning-based marine floating object object recognition model using AI technology based on learning data of the marine floating object, detects marine floating objects (discarded fishing nets, ropes, grills, marine debris) that pose a risk during the operation of a small vessel of less than 10 tons using vision recognition technology, prevents vessel collisions, damage to the propulsion shaft system, and accidents of entanglement between the vessel and marine floating objects (discarded fishing nets) through alarm warnings and responses to dangerous situations in the vessel's course, monitors marine debris, and is utilized as a marine safety navigation assistance system for small vessels of less than 10 tons that are vulnerable to marine floating objects, and can be used in large vessels, fishing vessel CCTVs, and shipyards.

[0034] This product equips vessels under 10 tons at sea with an infrared stereo camera and an IR illuminator. It does not rely on conventional radar or sensors, but uses deep learning-based image recognition technology to detect in real time not only large floating objects such as ships but also small floating objects (ropes, discarded fishing nets, etc.) that are vulnerable to the safety of small vessels under 10 tons. Through stereo image analysis, it is possible to measure the distance of floating objects at sea and display the distance between the vessel and the hazardous object in real time. This product incorporates various image preprocessing technologies to respond to nighttime conditions, currents, light reflections, and hull movement through an infrared stereo camera system designed for the marine environment. Unlike radar or radio wave-based equipment, where accuracy is determined by the hardware performance of the sensor due to system characteristics, this product can improve performance solely through deep learning-based software updates.

[0035] Figure 1 is a diagram showing the current status and types of marine accidents and the status of entanglement accidents (Ministry of Oceans and Fisheries).

[0036] Figure 2 is a photograph showing the detection limits of the main ship safety navigation aids and radar.

[0037] Figure 3 is a configuration diagram showing a ship-mounted omnidirectional monitoring system using a conventional PTZ camera.

[0038] FIG. 4 is a configuration diagram of an image-based marine floating body detection and distance measurement system for small vessels under 10 tons according to the present invention.

[0039] Figure 5 is a conceptual diagram of an infrared stereo camera system for a marine vessel.

[0040] Figure 6 is a photograph showing image preprocessing techniques for improving object recognition rates of infrared stereo cameras: an image correction filter for correcting light reflection, shadows, fog, etc., and an image stabilization algorithm for correcting ship shaking.

[0041] Figure 7 is a conceptual diagram of 3D information extraction using stereo images.

[0042] Figure 8 is a photograph showing images of floating objects on the sea (Examples 1 and 2).

[0043] Figure 9 is a diagram comparing the performance of different versions of the deep learning object detection algorithm YOLO.

[0044] Figure 10 is a diagram showing a distance measurement algorithm for a floating object at sea through a dense disparity map between stereo images.

[0045] Figure 11 is a photograph showing the video monitoring function of a ship at sea and the GUI of the danger situation alarm and video display system (basic settings, night mode, video recording, screen capture, warning alarm off / on, etc.).

[0046] Figure 12 is a photograph showing the test operation site testbed, Saemangeum Seawall, or the test vessel at Gyeokpo Port in Buan-gun.

[0047] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings, including the structure and operation of the invention.

[0048] The present invention is not limited to the disclosed embodiments and may be implemented in various different forms by those skilled in the art. In the description of the present invention, detailed descriptions of related known technologies or configurations are omitted if it is determined that such detailed descriptions may unnecessarily obscure the essence of the invention. Additionally, the same drawing numbers are assigned in different drawings when indicating the same configuration.

[0049] This study is not limited to specific embodiments and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0050]

[0051] The image-based marine floating object detection and distance measurement system for small vessels under 10 tons according to the present invention uses an infrared stereo camera and an IR illuminator equipped on a small vessel under 10 tons where hull shaking, sea fog, and light reflection occur due to wind and waves at sea. It measures the distance between a navigating vessel and a marine floating object using a deep learning-based marine floating object object recognition model that uses AI technology based on training data and a marine floating object recognition object distance measurement algorithm. It detects marine floating objects (discarded fishing nets, ropes, grills, marine debris) that pose a risk during the operation of small vessels under 10 tons using vision recognition technology, prevents vessel collisions, damage to the propulsion shaft system, and entanglement accidents between the vessel and floating objects (discarded fishing nets, nets, etc.) by providing forward danger situation alarms and emergency response measures in the vessel's course, monitors marine debris, and serves as a marine safety navigation assistance system for small vessels under 10 tons that are vulnerable to marine floating objects. It can also be used in large vessels, fishing vessel CCTVs, and shipyards.

[0052] In the event of detecting discarded fishing nets, the vessel's emergency measure refers to the act of shifting the engine gear to neutral to prevent small buoys or discarded nets from becoming entangled in the vessel's propeller.

[0053] This technology product is a "smart safety navigation assistance system for small vessels" that uses image recognition technology to detect, warn, and respond to floating objects at sea that pose a risk during the operation of small vessels under 10 tons in real time, thereby ensuring forward visibility (assistance) and preventing vessel collisions, damage to the propulsion shaft system, and accidents caused by entanglement of floating objects.

[0054] This product equips vessels under 10 tons at sea with an infrared stereo camera and an IR illuminator. It does not rely on conventional radar or sensors, but utilizes deep learning-based image recognition technology to detect in real time not only large floating objects such as ships but also small floating objects (ropes, discarded fishing nets, etc.) that are vulnerable to the safety of small vessels under 10 tons. Through stereo image analysis, it is possible to measure the distance between the vessel and floating objects at sea and display the distance between the vessel and dangerous objects in real time. This product incorporates various image preprocessing technologies to respond to nighttime conditions, currents, light reflections, and hull movement through an infrared stereo camera system designed for the marine environment. Unlike radar or radio wave-based equipment, where accuracy is determined by the hardware performance of the sensor due to system characteristics, this product can improve performance solely through deep learning-based software updates.

[0055] FIG. 4 is a configuration diagram of an image-based offshore floating body detection and distance measurement system for small vessels under 10 tons according to the present invention, and a conceptual diagram of an infrared stereo camera system for an offshore vessel.

[0056] The image-based marine floating object detection and distance measurement system for small vessels under 10 tons is a system that enables emergency measures by rapidly detecting and alerting marine floating objects based on images when small vessels under 10 tons are in operation. It applies various image preprocessing and deep learning technologies to quickly and accurately detect marine floating objects of various shapes and types (vehicles, ropes, nets, buoys, etc.), and when a danger alert is issued, the image is immediately displayed on the monitor along with an alarm sound.

[0057] - Infrared stereo camera system considering marine environment (salt, moisture, hull movement, etc.)

[0058] - Image correction filter preprocessing technology for improving object recognition rates

[0059] - Provides a deep learning-based marine object detection model

[0060] - Application of object distance algorithm for recognizing floating marine objects

[0061] - Dangerous situation alarm alert and video display system

[0062] - Test operation and performance verification

[0063] The image-based marine floating object detection and distance measurement system for small vessels according to the present invention comprises: an infrared stereo camera (100) and an IR illuminator (110) equipped on the vessel; and an image display system (130) connected to the infrared stereo camera (100) and the IR illuminator (120) via a control API (120), and equipped with an image correction filter that corrects blur caused by light reflection, shadows, and sea fog in the small vessel navigating the sea at night, low tide, cloudy weather, or sea fog, and detects marine floating objects by a deep learning-based object detection model based on prior training data, measures the distance to the marine floating objects, and displays images to prevent vessel collision, propulsion shaft damage, and entanglement accidents between the vessel and marine floating objects (nets) through alarm warnings and responses to forward danger situations on the vessel's route.

[0064] The camera can use one infrared camera or an infrared stereo camera.

[0065] The infrared stereo camera (100) above uses two infrared (IR) cameras to photograph the sea even at night or in low-light environments and detects floating bodies on the sea.

[0066] The infrared stereo camera (100) is composed of two infrared cameras and photographs floating objects on the sea. It uses two infrared (IR) cameras to detect floating objects on the sea even at night or in low-light environments. The IR cameras use a spatial resolution of 1,920×1,080 or higher based on stereo standards and a temporal resolution of 30 FPS or higher. The infrared stereo camera (100) and the IR illuminator (110) each have a waterproof / dustproof rating of IP65 or higher in consideration of the marine environment, and a camera housing made of a material resistant to salt and moisture is applied for installation and operation on a ship.

[0067] The infrared (IR) illuminator (110) is an infrared illuminator used in conjunction with an infrared stereo camera (100) and emits infrared light so that the camera can accurately detect the floating body in the sea even in low-light conditions such as night or cloudy weather in a marine environment where visible light is scarce. This enables accurate detection of the floating body in a marine environment where visible light is scarce.

[0068] The above control API (120) provides automatic adjustment of environmental variables (amplification ratio, exposure time, etc.) according to illumination when acquiring an image, as well as scheduling and preprocessing functions.

[0069] The above-mentioned marine floating body (300) is classified by type of marine floating body that poses a threat to the hull, and includes marine debris such as ships, fishing net buoys, discarded nets, vinyl, and styrofoam.

[0070]

[0071] The above image display system (130) is equipped with an image correction filter to correct blur caused by light reflection, shadows, and sea fog in order to improve object recognition rate, and

[0072] The above image correction filter 1) improves clarity by correcting the blur caused by fog in the sea observation image using the Dark Channel Prior (DCP) algorithm when the image appears blurry due to sea fog at sea, and 2) reduces reflected light from the water surface and filters out parts distorted by shadows by using the Water Filling algorithm to detect areas of light reflection or shadows in the camera image at sea and accurately correcting them to restore the surface information of the actual object.

[0073] The above deep learning-based object detection model uses the YOLO (You Only Look Once) algorithm to detect floating objects within the bounding box of a camera image.

[0074] The above image display system (130) extracts 3D distance information by calculating the parallax of all pixels in a pair of stereo images using a Dense Stereo algorithm to measure the distance of a floating object recognized from an image captured by the infrared stereo camera (100).

[0075] The above video display system (130) uses an embedded PC and provides video monitoring functions for a ship at sea and dangerous situation alarm alerts, and the GUI of the above video display system includes basic settings, night mode, video recording, screen capture, and turning warning alarms on / off, and monitors marine debris to prevent ship collisions, propulsion shaft damage, and entanglement accidents between the ship and marine floating objects (discarded fishing nets) through detection of floating objects at sea and response to dangerous situation alarm alerts in the ship's course.

[0076] A dense disparity map generation module that compares left and right image pairs acquired from the infrared stereo camera (100), calculates the disparity of each pixel, and generates a dense disparity map including three-dimensional distance and position information of an object based on the disparity information;

[0077] A distance measuring module that precisely calculates / measures the distance between a floating marine body and a camera based on the above-mentioned stereoscopic parallax map, and uses the calculated measured distance to warn of hazards on the vessel's route in real time; and

[0078] It includes a feature point extraction module that analyzes the above-mentioned stereoscopic parallax map to extract feature points of the shape and size of the floating body, and the extracted feature points are used for classification and risk level assessment of the floating body through AI deep learning-based analysis.

[0079] The above-described image display system (130) further includes a danger situation alarm warning unit that provides real-time distance information between the current location of the vessel and the floating object in a situation where there is marine debris or a dangerous object on the vessel's route, avoids obstacles on the vessel's route, and analyzes the size, location, and movement of the object based on the distance information to accurately detect and respond to dangerous situations by providing a danger alarm warning in advance of accidents such as collision with the vessel, damage to the propulsion shaft system, and entanglement of the vessel with the marine floating object (discarded net).

[0080] A danger situation warning system may further include an emergency broadcasting device and a speaker for marine safety connected to the above-mentioned video display system (130).

[0081] The infrared stereo camera system for a marine vessel acquires image data and is equipped with an IR (Infra-Red) camera, an IR transmitter, a control API (S / W), and an image display system.

[0082] o Camera system design considering the marine environment (salt, moisture, hull movement, etc.)

[0083] - Video data acquisition consists of an IR camera (H / W), an IR illuminator (H / W), a control API (S / W), and a video display system.

[0084] - The IR camera's spatial resolution is set to 1,920x1,080 or higher (stereo), and the temporal resolution is set to 30 FPS or higher.

[0085] - Application of IR illuminators to identify offshore floating objects even at night and on cloudy days

[0086] - IR cameras and IR floodlights are equipped with a waterproof and dustproof rating of IP65 or higher, considering the marine environment.

[0087] - Camera housing made of a material resistant to salt and moisture for installation and operation on ships

[0088] - The optimal combination of wavelengths for the IR camera and IR illuminator is determined after field testing.

[0089] - The control API provides automatic adjustment of environmental variables (amplification ratio, exposure time, etc.) based on illumination during image acquisition, as well as scheduling and preprocessing functions.

[0090] Figure 6 is a photograph showing image preprocessing techniques for improving object recognition rates of infrared stereo cameras: an image correction filter for correcting light reflection, shadows, fog, etc., and an image stabilization algorithm for correcting ship shaking.

[0091] o Development of image preprocessing technology to improve object recognition rates

[0092] - Design of image correction filters for correcting light reflections, shadows, fog, etc.

[0093] - Application of image stabilization algorithms for rock correction of small vessels under 10 tons

[0094] o Deep learning-based object detection model

[0095] - The large-scale training data applied to this development is collected in parallel with public data (Ai-Hub) and direct collection (operation of two or more vessels at Saemangeum Seawall or Gyeokpo Port in Buan-gun)

[0096] - To prevent model performance degradation caused by domain shift between public data and directly collected data, adjust the composition ratio of training data and perform transfer learning based on directly collected data.

[0097] - When class bias occurs in the training data, eliminate overfitting and data bias through data augmentation

[0098] - Label annotation information with bounding boxes and classify marine floating objects that threaten the hull by type (vehicles, fishing net buoys, marine debris (discarded nets, plastic, styrofoam, etc.)).

[0099] - As shown in Fig. 9, a pool of optimized model candidates (YOLO family) is selected in real-time, and the deep learning-based object detection model YOLO, most suitable for this system, is applied considering computing device specifications and processing speed.

[0100] - Hardware (embedded PC) and software optimization including GPU acceleration to enable real-time image processing of infrared stereo cameras

[0101] The above deep learning-based object detection model uses the YOLO (You Only Look Once) algorithm to detect floating objects within the bounding box of a camera image.

[0102] The above image display system extracts 3D distance information by calculating the parallax of all pixels in a pair of stereo images using a Dense Stereo algorithm to measure the distance between the ship's camera and the offshore floating object recognition object from an image captured by the infrared stereo camera.

[0103] The above Dense Stereo algorithm is

[0104] (a) Stereo image acquisition step: The same scene is captured from different left and right angles using a stereo camera, and the difference between the two images provides distance information of the object.

[0105] (b) Disparity calculation step for each pixel: Matching pixels are found in the images to the left and right of each pixel, and the positional difference between those pixels is calculated; this difference is the disparity. The process of finding a corresponding pixel in another image for each pixel in the stereo image is performed, and methods such as Sum of Absolute Differences (SAD), Normalized Cross-Correlation (NCC), or Semi-Global Matching (SGM) are used. In this process, the optimal matching point is found for all pixels, and methods such as Graph Cuts or Dynamic Programming are used to optimize disparity in a way that minimizes cost to minimize errors occurring during the matching process.

[0106] (c) Disparity map generation step: A disparity map is generated based on the disparity values ​​calculated for all pixels, and in this disparity map, bright areas indicate where the object is close to the camera, and dark areas indicate where the object is far from the camera, and

[0107] (d) Depth (distance) information calculation step: The actual distance (depth) of the object is calculated through disparity, and the distance between the camera and the object is calculated using the following formula,

[0108]

[0109] Here, f is the camera's focal length, B is the baseline between the two infrared stereo cameras, and d is the disparity between the two images, and

[0110] In this way, the Dense Stereo algorithm and the Dense Disparity Map contain depth information for each pixel, accurately detect the location and characteristics of various marine debris floating in the marine environment, and the floating body measures the distance to the ship's camera.

[0111] The above-described video display system provides real-time distance information between the vessel at the current location and the floating object in situations where marine debris or hazardous objects are present, enabling the vessel to bypass obstacles by diverting from its course. Furthermore, by analyzing the size, location, and movement of the object based on the distance information, it enables more accurate detection and allows for response by issuing a danger alarm warning in advance regarding hazardous situations, such as vessel collision, propulsion shaft damage, and entanglement of the vessel with marine floating objects (discarded fishing nets). The system also includes a hazardous situation warning system for marine safety connected to the above-described video display system.

[0112] (1) Dark Channel Prior algorithm

[0113] When images appear blurry due to sea fog at sea, the Dark Channel Prior (DCP) algorithm is used to correct the blur caused by fog in the marine observation images, thereby improving clarity. The Dark Channel Prior algorithm removes fog by utilizing the characteristic that a 'dark channel' exists in most of the image, possessing minimal light intensity. As a technique for restoring images blurred by fog, haze, or fine dust, the Dark Channel Prior (DCP) algorithm is highly effective in enhancing the clarity of images captured at sea. In this invention, the Dark Channel Prior algorithm is used to correct the blur caused by fog in marine observation images.

[0114] DCP Algorithm Operation Principle: Most natural images have at least one color channel with a very low value for a specific pixel. This is called the "Dark Channel." In other words, the Dark Channel is where pixels with the minimum amount of light are located within the image. In foggy environments, light is scattered by the fog, causing the entire image to become blurry and the Dark Channel tends to disappear. The DCP algorithm leverages this phenomenon by analyzing the Dark Channel information of the image and then restoring the original clear image.

[0115] Step 1: Extract Dark Channel: Extract the lowest value (dark channel) among the RGB channels from each pixel of the original image.

[0116] Step 2: Estimate the distribution of fog based on the dark channels. Since the dark channels of the image disappear as the fog deepens, the fog density is calculated based on this.

[0117] Step 3: This step restores the original accurate image based on the estimated fog density. It removes light scattering to enhance image contrast and sharpens the outlines of blurry objects. By using the DCP algorithm, the impact of fog occurring during river water level observation can be reduced, allowing for clearer and more accurate water level images to be obtained.

[0118] (2) Water Filling Algorithm

[0119] The Water Filling algorithm is a technique that detects areas of light reflection or shadows in an image and precisely corrects them to restore the surface information of the actual object. It is primarily used to reduce reflected light from the water surface and filter out parts distorted by shadows.

[0120] Step 1: Identify the areas in the image that appear very bright due to reflected light. These areas are where light, such as sunlight or electric lights, is directly reflected from the water surface.

[0121] Step 2: Identify areas in the image that appear very dark, that is, areas where shadows are cast and the outline of the actual object is not visible.

[0122] Step 3: Apply correction filters to the detected light reflections and shadow areas. This process balances the image by reducing brightness values ​​exceeding a specific threshold for light reflections and clearly illuminating dark areas for shadows.

[0123] Step 4: This is the process of restoring the image with light reflections and shadows removed. During this process, the position of the water surface distorted by reflected light or the shape of objects obscured by shadows becomes clearly visible. By using the Water Filling algorithm, the image of the water distorted by light reflections and shadows is corrected, enabling more accurate measurements.

[0124] IR illuminators emit infrared light in dark or low-light environments, helping infrared stereo cameras capture objects more accurately.

[0125] o Maritime Floating Object Recognition Object Distance Algorithm

[0126] - Provides object distance information through a dense disparity map between stereo images

[0127] - A camera calibration process to estimate the intrinsic and extrinsic parameters of the two infrared stereo cameras, and a feature point matching process to extract corresponding locations on the images acquired from the two cameras are required.

[0128] - Align the two images collinearly using external parameters to match them with the epipolar lines calculated in the matching step, then align them, and finally construct a stereoscopic parallax map between the two images using the Dense Stereo Algorithm.

[0129] Figure 9 is a diagram comparing the performance of different versions of the deep learning object detection algorithm YOLO.

[0130] Figure 10 is a diagram showing a distance measurement algorithm for a floating object at sea through a dense disparity map between stereo images.

[0131] The dense disparity map generation module compares left and right image pairs acquired from an infrared stereo camera to calculate the disparity of each pixel, and generates a dense disparity map containing three-dimensional distance and position information of an object based on the disparity information.

[0132] The distance measurement module precisely calculates the distance between the floating body and the camera based on the generated stereoscopic parallax map, and the calculated distance is used to warn of hazards on the ship's route in real time.

[0133] The feature point extraction module analyzes the stereoscopic parallax map to extract feature points such as the shape and size of the floating body, and the extracted feature points are used for the classification of the floating body and the assessment of the risk level through AI deep learning-based analysis.

[0134]

[0135] (3) Dense Stereo algorithm

[0136] The Dense Stereo algorithm is a method that extracts 3D distance information by calculating the parallax of every pixel in a pair of stereo images. Unlike the Sparse Stereo algorithm (which matches only specific feature points), the Dense Stereo algorithm provides higher density depth information because it deals with every pixel in the image.

[0137] Figure 7 is a conceptual diagram of 3D information extraction using stereo images.

[0138] * Stereo Image Acquisition: The same scene is captured from different left and right angles using stereo cameras. The difference between the two images serves as the basis for providing distance information about objects. * Calculation of Disparity for Each Pixel: Matching pixels are found in the images to the left of each pixel, and the positional difference between those pixels is calculated; this difference is the disparity. The process involves finding the corresponding pixel in the other image for each pixel in the stereo image. Various algorithms can be used for this purpose, but generally, methods such as Sum of Absolute Differences (SAD), Normalized Cross-Correlation (NCC), or Semi-Global Matching (SGM) are used. The goal of Dense Stereo is to find the optimal matching point for every pixel during this process. To minimize errors that may occur during the matching process, disparity is optimized using methods that minimize cost. Methods such as Graph Cuts or Dynamic Programming are primarily used.

[0139] * Disparity Map Generation: A disparity map is generated based on the disparity values ​​calculated for every pixel. In this disparity map, bright areas indicate where the object is close to the camera, while dark areas indicate where the object is far from the camera.

[0140] * Calculation of depth (distance) information: The actual distance (depth) of an object can be calculated through disparity. The distance between the camera and the object is calculated using the following formula.

[0141]

[0142] Here, f is the focal length of the camera, B is the baseline between the two cameras of the infrared stereo camera, and d is the disparity between the two images.

[0143] The Dense Disparity Map obtained in this way contains depth information for each pixel, allowing for the accurate measurement of the distance an object is from the camera. The Dense Stereo algorithm plays a crucial role in the detection of floating objects at sea. In situations involving marine debris or hazardous objects, it provides real-time distance information, enabling ships to avoid obstacles in their routes. Furthermore, by analyzing the size, location, and movement of objects based on this distance data, it is possible to build more accurate detection and warning systems. Through these Dense Stereo algorithms and Disparity Maps, the locations and characteristics of various floating marine debris can be accurately identified, making a significant contribution to maritime safety and environmental protection.

[0144] o Function of the IR transmitter

[0145] IR illuminators play an important role because they are used in conjunction with IR cameras. This is because IR cameras are sensitive to the infrared (IR) spectrum. Specifically, the reasons for using IR illuminators are as follows:

[0146] Since infrared cameras use the infrared spectrum instead of visible light to capture images, they do not function properly if there is insufficient infrared light in the surroundings. In such cases, an IR illuminator projects infrared light into the environment, allowing the camera to detect and photograph floating objects at sea even in dark areas or situations with low visibility. This is particularly useful at night, in dark seas, and in cloudy weather.

[0147] Because IR illuminators emit infrared light rather than visible light, they are less affected by interference from visible light, such as fog, dust, and cloudy weather. IR cameras utilize illuminators to detect objects even in environments where conventional cameras cannot see, and they detect floating marine objects in marine environments.

[0148] o Development of a hazardous situation alarm, warning, and video display system

[0149] - After analyzing the acquired image, visualize alarms and information via the display in the event of an alarm.

[0150] - To ensure the intuitiveness of alarms, only minimal intuitive information is displayed, and the GUI (Graphical User Interface) is configured to include basic video monitoring functions (default settings, night mode, video recording, screen capture, turning warning alarms on / off, etc.).

[0151] Figure 8 is a photograph showing images of floating objects (Example 1, Example 2).

[0152]

[0153] Figure 11 is a photograph showing the video monitoring function of a ship at sea and the GUI of the danger situation alarm and video display system (basic settings, night mode, video recording, screen capture, warning alarm off / on, etc.).

[0154] o Test operation and performance testing

[0155] - The trial operation consists of two phases; the first phase involves establishing the camera system, collecting training data necessary for building the deep learning model, and conducting performance tests.

[0156] - Once the initial product prototype is completed, conduct product testing and improvement work through a second operation.

[0157] The testbed for the trial operation will be the Saemangeum Seawall or the waters off Gyeokpo Port in Buan-gun, and we plan to recruit two or more vessels to conduct data collection and testing.

[0158] Figure 12 is a photograph showing a test vessel at the test operation site testbed, Saemangeum Seawall, or Gyeokpo Port in Buan-gun.

[0159] Representative similar commercial products from abroad include the AI-RIS product from Sea Machines Robotics of the United States. This developed product incorporates proprietary elemental technologies (such as ship motion stabilization, blur and light reflection correction, and stereo object distance extraction) and applies model construction based on floating bodies collected from the domestic marine environment, thereby enabling the securing of technological competitiveness tailored to domestic environmental conditions. Furthermore, considering that the unit price of the competitor's AI-RIS product is approximately 45 million won, we plan to set the unit price of this product at around 10 million won, ensuring price competitiveness. Additionally, most domestic and international competitors are developing technologies focused on the autonomous operation of large vessels, and there are virtually no commercial products specialized for small vessels.

[0160] o Commercialization Plan

[0161] The global digital maritime traffic information industry market is projected to grow from approximately 106 trillion won in 2023 to 156 trillion won in 2028, with an annual average growth rate of 8%. The total number of registered fishing vessels in Korea is estimated at 66,970 (2022 statistics from the Ministry of Oceans and Fisheries), of which small vessels under 10 tons, representing the actual demand sources, account for over 95% with 63,380 vessels. Considering the unit price of the developed product (approximately 10 million won), the domestic market size is estimated at 633.8 billion won.

[0162] - The target market for the product is estimated at 633.8 billion KRW, considering the number of target fishing vessels and the estimated unit price of the developed product. Currently, there are very few related competitors or similar products overseas (two or fewer). By pioneering a new market that does not exist domestically, we aim to secure a market preemption through proactive product development.

[0163] - Commercialization of product technology utilizing domestic AI technology, currently in the state of academic research.

[0164]

[0165] - The target market share is approximately 0.14% (about 900 million KRW) within three years of development completion, aiming for expansion into the large vessel market and entry into the global market through product advancement (functional diversification).

[0166] - Step-by-step commercialization strategy

[0167] Step 1: Product Promotion and Identification of Improvements through Experience Ship Operations

[0168] Phase 2: Implementation of direct sales (B2C) targeting fishermen, fishing communities, etc.

[0169] Step 3: Expansion of distribution channels (partnerships with shipyards, fishing vessel CCTV, and steering gear suppliers)

[0170] Step 4: Expanding Revenue Models (B2G), Building and Selling Public Good Data

[0171] o Expected effects

[0172] - Securing an exclusive AI-based video solution targeting the detection and warning of floating objects (ships, nets, ropes, etc.) that are fatal to small vessels compared to existing maritime safety systems.

[0173] - Securing market share in a new domestic niche market worth approximately 633.8 billion KRW and generating revenue

[0174] - Reduction of casualties and damage to vessels by ensuring the safety of small vessels vulnerable to floating structures

[0175] 1) Technical aspects

[0176] This technology product has independently secured an AI-based video solution targeting the detection of marine floating objects (ships, nets, ropes, etc.) that are fatal to small vessels under 10 tons and the warning of dangerous situations (ship collisions, net entanglement accidents, etc.). By introducing core technologies of the 4th Industrial Revolution (video analysis, IoT, drones, AI, etc.), it presents a new product technology paradigm in the field of maritime safety surveillance and has the effect of reducing casualties and damage to vessels by ensuring the safety of small vessels vulnerable to marine floating objects.

[0177] 2) Economic and industrial aspects

[0178] - Securing market share in a new domestic niche market worth approximately 633.8 billion KRW and generating revenue

[0179] - Due to the nature of the technology, securing training data is key; thus, securing a competitive advantage through proactive technology development.

[0180] - Improved work efficiency for captains and crew through AI-based detection of floating objects at sea and reduced burden in assessing dangerous situations such as ship collisions and net entanglement accidents.

[0181] - Maritime safety management of small vessels under 10 tons based on Fourth Industrial Revolution technologies

[0182] 3) Growth potential and technological spillover effects

[0183] This technology product contributes to the local economy through sustainable corporate growth and the creation of local jobs resulting from subsequent investments, while laying the foundation for the activation of new growth engines in the field of maritime safety. Furthermore, this technology has ripple effects applicable not only to safe ship navigation but also to marine environmental fields such as marine debris monitoring; when applied to unmanned military vessels and small military ships, it can be utilized to secure domestic maritime defense technology.

[0184]

[0185] Embodiments according to the present invention are implemented in the form of program instructions that can be executed through various computer means and may be recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, and data structures alone or in combination. The computer-readable recording medium may include magnetic media such as storage, servers, hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices configured to store and execute program instructions on storage media such as ROM, RAM, flash memory, storage, etc. Examples of program instructions may include machine code as well as high-level language code that can be executed by a computer using an interpreter. The hardware device may be configured to operate as one or more software modules to perform the operation of the present invention.

[0186] The method of the present invention can be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, memory card, hard disk, magneto-optical disk, storage device, etc.) in a form that can be read using computer software.

[0187] As described above, although the present invention has been explained with reference to specific embodiments, the present invention is not limited to the same configuration and operation as the specific embodiments to illustrate the technical concept as described above, and may be implemented with various modifications within the limits of the technical concept and scope of the present invention, and the scope of the present invention shall be determined by the claims set forth below.

[0188] The image-based marine floating object detection and distance measurement system and method for small vessels according to the present invention utilizes an infrared stereo camera and an IR illuminator equipped on a small vessel of less than 10 tons where hull shaking due to wind and wave height, sea fog, and light reflection exist at sea, and uses vision recognition technology that employs a marine floating object recognition object distance measurement algorithm using a deep learning-based marine floating object object recognition model based on AI technology based on training data to measure the distance between the vessel and the marine floating object, detects marine floating objects (discarded fishing nets, ropes, grills, marine debris) that pose a risk during the operation of a small vessel of less than 10 tons, prevents vessel collisions, damage to the propulsion shaft system, and accidents involving the vessel and marine floating objects entanglement through alarm warnings and responses to dangerous situations, monitors marine debris, and is utilized as a marine safety navigation assistance system for small vessels of less than 10 tons that are vulnerable to marine floating objects, and can be used in large vessels, fishing vessel CCTVs, and shipyards.

Claims

1. An infrared stereo camera and an IR illuminator equipped on a vessel; and An image display system connected to the infrared stereo camera and the IR illuminator via a control API, equipped with an image correction filter that corrects blur caused by light reflection, shadows, and sea fog; detects the floating object by a deep learning-based object detection model based on prior training data of the floating object; measures the distance between the camera of the vessel and the floating object; and displays images to prevent vessel collision, propulsion shaft damage, and entanglement accidents between the vessel and the floating object through alarm alerts and responses to forward danger situations on the vessel's route. Image-based marine floating object detection and distance measurement system for small vessels including 2. In Paragraph 1, The above-described infrared stereo camera uses two infrared (IR) cameras to photograph the sea even in cloudy weather, at night, or in low-light environments to detect floating objects on the sea, and uses a camera with a spatial resolution of 1,920×1,080 or higher and a temporal resolution of 30 FPS or higher for the IR camera, and the above-described infrared stereo camera and the above-described IR illuminator each have a waterproof / dustproof rating of IP65 or higher considering the marine environment, and are equipped with a camera housing made of a material resistant to salt and moisture for installation and operation on a ship, an image-based floating object detection and distance measurement system for small vessels.

3. In Paragraph 2, The above IR illuminator is an infrared illuminator used in conjunction with the above infrared stereo camera, and is an image-based marine floating object detection and distance measurement system for small vessels that emits infrared light so that the camera can accurately detect the marine floating object in low light conditions such as at night or in cloudy weather in a marine environment where visible light is insufficient.

4. In Paragraph 1, The above-mentioned marine floating object is a small vessel image-based marine floating object detection and distance measurement system that includes marine debris, including ships, fishing net buoys, discarded fishing nets, and vinyl and styrofoam, classified by type of marine floating object that poses a threat to the hull.

5. In Paragraph 1, The above deep learning-based object detection model is an image-based marine floating object detection and distance measurement system for small vessels that uses the YOLO (You Only Look Once) algorithm to detect marine floating objects within the bounding box of a camera image.

6. In Paragraph 1, The above video display system uses an embedded PC and provides video surveillance functions for a vessel at sea and alarm alerts for dangerous situations. The GUI of the above video display system is equipped with basic settings, night mode, video recording, screen capture, and on / off alarms. It also monitors marine debris to prevent vessel collisions, propulsion shaft damage, and entanglement accidents between the vessel and the floating object at sea through detection of floating objects at sea, alarm alerts for dangerous situations on the vessel's course, and response. A dense disparity map generation module that compares left and right image pairs acquired from the infrared stereo camera, calculates the disparity of each pixel, and generates a dense disparity map including three-dimensional distance and position information of an object based on the disparity information; A distance measuring module that precisely calculates the distance between a floating body and a camera based on the above-mentioned stereoscopic parallax map, and uses the calculated measured distance to warn of hazards on the vessel's route in real time; and An image-based offshore floating body detection and distance measurement system for small vessels, comprising a feature point extraction module that analyzes the above-mentioned stereoscopic parallax map to extract feature points of the shape and size of the offshore floating body, wherein the extracted feature points are utilized for the classification of the floating body and risk level assessment by AI deep learning-based analysis.

7. In Paragraph 6, The above-described image display system is equipped with an image correction filter for correcting light reflection, shadows, and sea fog to improve object recognition rates, and The above image correction filter is a technique that 1) improves clarity by correcting the blurriness caused by fog in the marine observation image using a Dark Channel Prior (DCP) algorithm when the image appears blurry due to sea fog at sea, and 2) reduces reflected light generated from the water surface and filters out parts distorted by shadows by using a Water Filling algorithm to detect areas of light reflection or shadows in the marine camera image and accurately correcting them to restore surface information of the actual object, thereby reducing reflected light generated from the water surface and filtering out parts distorted by shadows, for a small vessel image-based marine floating object detection and distance measurement system.

8. In Paragraph 1, The above image display system is a small vessel image-based offshore floating object detection and distance measurement system that extracts 3D distance information by calculating the parallax of all pixels in a pair of stereo images using a Dense Stereo algorithm to measure the distance between the ship's camera and the offshore floating object recognition object from an image captured by the infrared stereo camera.

9. In Paragraph 8, The above Dense Stereo algorithm is (a) Stereo image acquisition step: The same scene is captured from different left and right angles using a stereo camera, and the difference between the two images provides distance information of the object. (b) Disparity calculation step for each pixel: Matching pixels are found in the images to the left and right of each pixel, and the positional difference between those pixels is calculated; this difference is the disparity. The process of finding a corresponding pixel in another image for each pixel in the stereo image is performed, and methods such as Sum of Absolute Differences (SAD), Normalized Cross-Correlation (NCC), or Semi-Global Matching (SGM) are used. In this process, the optimal matching point is found for all pixels, and methods such as Graph Cuts or Dynamic Programming are used to optimize disparity in a way that minimizes cost to minimize errors occurring during the matching process. (c) Disparity map generation step: A disparity map is generated based on the disparity values ​​calculated for all pixels, and in the disparity map, bright areas indicate where the object is close to the camera, and dark areas indicate where the object is far from the camera. (d) Depth (distance) information calculation step: The actual distance (depth) of the object is calculated through disparity, and the distance between the camera and the object is calculated using the following formula, Here, f is the camera's focal length, B is the baseline between the two infrared stereo cameras, and d is the disparity between the two images, and In this manner, the Dense Stereo algorithm and the Dense Disparity Map contain depth information for each pixel, and the image-based marine floating object detection and distance measurement system for small vessels accurately detects the location and characteristics of various marine debris floating in a marine environment and measures the distance of the marine floating object to the ship's camera.

10. In Paragraph 1, The above-described image display system is a small vessel image-based marine floating object detection and distance measurement system that further includes a danger situation notification warning unit that provides real-time distance information between the vessel and the floating object in situations where marine debris or dangerous objects are present on the vessel's course, avoids obstacles on the vessel's course, analyzes the size, location, and movement of the object based on the distance information to enable accurate detection and response to dangerous situations by issuing a danger alarm warning in advance for accidents such as collision with the vessel, damage to the propulsion shaft system, and entanglement of the vessel with the floating object or net.

Images