A method for preventing collisions of vessels and electronic devices for performing thereof

Abstract

A method for preventing a collision of a ship using a ship collision prevention device according to one embodiment of the present application comprises: a step of acquiring image data by photographing the surroundings of the ship through an ultra-low light camera; a step of acquiring motion data regarding the motion state of the ship with six degrees of freedom using an inertial measurement unit; a step of generating a stabilized image by calculating the inter-frame displacement of the image data based on the motion data; a step of generating an improved image by improving the quality of the stabilized image through fog removal and noise removal processing; a step of detecting an object from the improved image and calculating the distance between the detected object and the ship by applying a monocular depth estimation algorithm; and a step of determining the collision risk between the detected object and the ship based on the distance.

Description

Technology Field

[0001] The present application relates to a method for preventing ship collisions and an electronic device for performing the same. Background Technology

[0003] Collisions between vessels at sea can lead to serious accidents resulting in casualties and property damage. To prevent such accidents, technology capable of detecting surrounding vessels or obstacles during navigation and assessing the likelihood of collision in advance is critically required.

[0004] Generally, navigation equipment such as the Automatic Identification System (AIS) is used to determine the navigation status of a vessel. AIS supports the identification of the navigation status of surrounding vessels by transmitting and receiving navigation information, such as a vessel's position, speed, and direction of travel. However, since AIS can only provide information to vessels equipped with the system, it has limitations in that it is difficult to detect small vessels not equipped with AIS, floating objects, or maritime obstacles.

[0005] To address these issues, radar or image-based surveillance technologies are sometimes utilized. In particular, image-based technologies using cameras have the advantage of allowing for intuitive perception of the surrounding environment. However, in maritime environments, accurate object recognition can be difficult due to image shaking caused by the rolling and pitching of vessels, as well as reduced visibility often resulting from sea fog or low-light conditions.

[0006] Therefore, there is an increasing need for technology capable of stably detecting surrounding objects while compensating for image shaking and low-light issues occurring in marine environments, and determining the distance and collision risk between detected objects and vessels. Prior art literature

[0008] (Patent Document 0001) KR 10-2809777 B The problem to be solved

[0009] The problem that the present invention aims to solve is to provide a ship collision prevention technology capable of detecting objects around a ship and calculating the distance between the objects and the ship using image data capturing the surrounding environment of the ship.

[0010] In addition, the problem that the present invention aims to solve is to provide a technology that improves the accuracy of object detection by correcting image shaking caused by the movement of a ship and image quality degradation occurring in sea fog and low-light environments.

[0011] In addition, the problem that the present invention aims to solve is to provide a ship collision prevention method and an electronic device capable of determining the collision risk between a ship and an object by utilizing image-based object information and navigation data together.

[0012] The problems that the present invention aims to solve are not limited to those described above, and problems not mentioned will be clearly understood by those skilled in the art from this specification and the attached drawings. means of solving the problem

[0014] A method for a ship collision prevention device according to an embodiment of the present invention to prevent a collision of a ship may include: a step of acquiring image data by photographing the surroundings of the ship through an ultra-low light camera; a step of acquiring motion data for the 6-degree-of-freedom motion state of the ship using an inertial measurement unit; a step of generating a stabilized image by calculating the inter-frame displacement of the image data based on the motion data; a step of generating an improved image by improving the image quality of the stabilized image through fog removal and noise removal processing; a step of detecting an object from the improved image and calculating the distance between the detected object and the ship by applying a monocular depth estimation algorithm; and a step of determining the collision risk between the detected object and the ship based on the distance.

[0015] An electronic device for preventing collision of a ship according to one embodiment of the present invention comprises: an image acquisition unit that acquires image data by photographing the surroundings of the ship through an ultra-low light camera; a sensing unit that acquires motion data regarding the motion state of the ship with six degrees of freedom using an inertial measurement unit; a memory that stores at least one instruction; and an edge AI control unit including a processor that executes the at least one instruction. The edge AI control unit can generate a stabilized image by calculating the inter-frame displacement of the image data based on the motion data, generate an improved image by improving the quality of the stabilized image through fog removal and noise removal processing, detect an object from the improved image, calculate the distance between the detected object and the ship by applying a monocular depth estimation algorithm, and determine the collision risk between the detected object and the ship based on the distance.

[0016] The means for solving the problem of the present invention are not limited to the means for solving the problem described above, and unmentioned means for solving the problem will be clearly understood by those skilled in the art from this specification and the attached drawings. Effects of the invention

[0018] According to one embodiment of the present invention, by acquiring images of the surroundings of a vessel using an ultra-low light camera and applying image-based object detection technology, objects such as small vessels not equipped with navigation equipment or floating objects can also be detected.

[0019] In addition, by correcting image shake based on the ship's motion state and performing fog removal and noise removal processing, the accuracy of object recognition can be improved even in foggy or low-light environments.

[0020] In addition, by utilizing image-based object information and navigation data together to analyze the distance and relative motion state between the vessel and the object, the collision risk can be determined, and by visualizing and providing this, it is possible to support the vessel operator's decision to avoid collisions. Brief explanation of the drawing

[0022] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present disclosure, provide embodiments of the present disclosure and explain the technical features of the present disclosure together with the detailed description. FIG. 1 is a drawing illustrating an example of an electronic device for preventing ship collisions according to one embodiment of the present application. FIG. 2 is a block diagram briefly illustrating the configuration of an electronic device for preventing ship collisions according to one embodiment of the present application. FIG. 3 is a block diagram briefly illustrating the configuration of an edge AI control unit of an electronic device according to one embodiment of the present application. FIG. 4 is a flowchart illustrating a method for preventing ship collisions according to one embodiment of the present application. Specific details for implementing the invention

[0023] The aforementioned objectives, features, and advantages of the present application will become more apparent from the following detailed description in conjunction with the accompanying drawings. However, as the present application is subject to various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail below.

[0024] Throughout the specification, identical reference numbers generally represent identical components. Additionally, components with identical functions within the same scope of concept appearing in the drawings of each embodiment are described using the same reference numeral, and redundant descriptions thereof are omitted.

[0025] If it is determined that a detailed description of known functions or configurations related to this application could unnecessarily obscure the essence of this application, such detailed description is omitted. Furthermore, numbers used in the description of this specification (e.g., First, Second, etc.) are merely identifiers to distinguish one component from another.

[0026] Furthermore, the suffixes "module" and "part" for components used in the following embodiments are assigned or used interchangeably solely for the ease of drafting the specification, and do not inherently possess distinct meanings or roles.

[0027] In the following examples, singular expressions include plural expressions unless the context clearly indicates otherwise.

[0028] In the following embodiments, terms such as "include" or "have" mean that the features or components described in the specification are present, and do not preclude the possibility that one or more other features or components may be added.

[0029] In the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily depicted for convenience of explanation, and the present invention is not necessarily limited to what is illustrated.

[0030] Where an embodiment can be implemented differently, the order of a particular process may be performed differently from the order described. For example, two processes described consecutively may be performed substantially simultaneously or proceed in the reverse order of the description.

[0031] In the following embodiments, when components are described as being connected, the case includes not only instances where the components are directly connected but also instances where components are indirectly connected by interposing them in between.

[0032] For example, when it is stated in this specification that components, etc. are electrically connected, it includes not only cases where the components, etc. are directly electrically connected, but also cases where components, etc. are interposed in between and are indirectly electrically connected.

[0034] Hereinafter, with reference to FIGS. 1 to 4, the method for preventing ship collisions and the electronic device for performing the same according to the present application will be described.

[0036] FIG. 1 is a drawing illustrating an example of an electronic device for preventing ship collisions according to one embodiment of the present application. Referring to FIG. 1, a ship (100) may be equipped with a communication unit (110) for transmitting and receiving navigation data, a sensing unit (120) for measuring the motion state of the ship, an image acquisition unit (130) for capturing the surrounding environment of the ship, an edge AI control unit (140) for processing data, an output unit (150) for outputting a collision risk level, and a memory (160).

[0037] The communication unit (110) can receive navigation data from other vessels or transmit data to an external device, and may include an Automatic Identification System (AIS) or other navigation communication interfaces. The communication unit (110) can transmit the received navigation data to the edge AI control unit (140).

[0038] The sensing unit (120) can measure six degrees of freedom of motion states including rolling, pitching, and yawing of the vessel (100) by including an inertial measurement unit (IMU). Motion data acquired from the sensing unit (120) can be transmitted to the edge AI control unit (140) and used for image stabilization processing.

[0039] The image acquisition unit (130) can acquire images of the surroundings of the vessel, including an ultra-low light camera, and can be positioned at least one of the forward, rear, port, and starboard sides of the vessel. If the image acquisition unit (130) is equipped with multiple cameras, it can be positioned to photograph the surroundings of the vessel from multiple angles. Additionally, the image acquisition unit (130) may be composed of a 360-degree camera capable of 360-degree shooting and one channel of a pan-tilt camera.

[0040] For example, the ultra-low light camera included in the image acquisition unit (130) may be equipped with a large image sensor of 1 / 1.8 inches or larger and a large-aperture bright lens of F1.5 or smaller to overcome power constraints (battery capacity limits) and poor marine lighting environments, thereby enabling 'Full color night vision' that allows color identification without infrared (IR) illumination even in an environment of 0.001 Lux. To prevent corrosion caused by the salt in seawater, the housing of the ultra-low light camera included in the image acquisition unit (130) may be made of stainless steel of SUS316L grade or higher, or may be designed to be explosion-proof. Additionally, a hydrophobic coating or a wiper system may be applied to the front of the lens of the ultra-low light camera to physically remove visibility obstruction caused by wave foam or rain.

[0041] The edge AI control unit (140) may be implemented as a computing device including at least one processor and can perform image stabilization, image quality improvement, object detection, monocular depth estimation, and collision risk determination based on image data acquired from the image acquisition unit (130), motion data acquired from the sensing unit (120), and navigation data received from the communication unit (110). Additionally, the edge AI control unit (140) can perform the above functions by executing program instructions stored in the memory (160).

[0042] The edge AI control unit (140) is equipped with NMEA 0183 and NMEA 2000 standard interface ports to link with the ship's existing navigation equipment when necessary, and can receive AIS (Automatic Identification System) and V-PASS (Vessel Position Transmitter) data and utilize it for AI calculations. In addition, considering the narrow wheelhouse environment and vibrations, a fanless heat dissipation case structure that is as fanless as possible may be applied.

[0043] The memory (160) can store program commands, video data, exercise data, and risk assessment results, and supports the operation of the edge AI control unit (140).

[0044] The output unit (150) can output the collision risk determined by the edge AI control unit (140) as a visual, auditory, or tactile signal. For example, the output unit (150) can visualize the location of the detected object and the expected collision point through an electronic chart display, or generate a warning sound.

[0045] FIG. 1 illustrates an example of the physical arrangement of the above components, and the arrangement location, number, and connection structure of each component may be varied within the scope of the technical concept of the present application.

[0047] FIG. 2 is a simplified block diagram illustrating the configuration of an electronic device for preventing ship collisions according to one embodiment of the present application. Referring to FIG. 2, the electronic device (100) for preventing ship collisions may include a communication unit (110), a sensing unit (120), an image acquisition unit (130), an edge AI control unit (140), an output unit (150), and a memory (160).

[0049] The communication unit (110) can support the establishment of a direct communication channel or a wireless communication channel between the device (100) and an external electronic device, and the transmission and reception of data through the established communication channel. The communication unit (110) can operate independently of the edge AI control unit (140) and may include one or more communication processors that support direct communication or wireless communication.

[0050] The communication unit (110) may be configured to receive navigation data from other vessels, and said navigation data may include the vessel's location information, speed information, direction of travel information, and identification information. The communication unit (110) may transmit such navigation data to the edge AI control unit (140).

[0051] Additionally, the communication unit (110) can transmit the collision risk determination result calculated by the edge AI control unit (140) to an external device as needed. For example, the communication unit (110) can transmit the collision risk information to an external control system, another vessel, or a remote monitoring device.

[0052] The communication unit (110) can operate according to a communication protocol or control command stored in memory (160) and can perform data transmission and reception under the control of the edge AI control unit (140).

[0054] The sensing unit (120) may be configured to measure the motion state of the vessel. Specifically, the sensing unit (120) may include an inertial measurement unit (IMU) composed of a gyroscope sensor and an accelerometer sensor, and the inertial measurement unit through Motion data for 6-degree-of-freedom motion states including rolling, pitching, and yawing of the vessel can be acquired.

[0055] The sensing unit (120) may be configured to generate motion data by sampling the angular velocity and acceleration information of the vessel at a preset period, or to output the generated motion data at a fixed period. The sensing unit (120) may transmit the acquired motion data to the edge AI control unit (140).

[0056] The sensing unit (120) can be positioned inside the ship's wheelhouse or at a location where vibrations of the hull are transmitted, and can detect the ship's motion state in real time to provide basic data that can be used for image stabilization and collision risk assessment.

[0057] The sensing unit (120) can acquire motion data regarding the capsizing, collision, and sinking of a ship through an inertial measurement unit (IMU) composed of a gyroscope sensor and an accelerometer sensor embedded within the sensing unit (120). By doing so, it can detect the accident status of the ship in real time and provide basic data that can be used to determine the risk of a ship accident.

[0058] The image acquisition unit (130) may be configured to acquire image data by capturing the surrounding environment of the ship. The image acquisition unit (130) may include an ultra-low light camera and may generate an image that can identify objects around the ship even at night or in an environment with low light intensity (e.g., 0.001 Lux or less).

[0059] The ultra-low light camera may include a high-sensitivity image sensor and may be configured to acquire color images in low-light environments without separate infrared illumination. The image sensor may include a CMOS sensor and may be combined with a large-aperture lens to secure sufficient light even under low illumination conditions.

[0060] The image acquisition unit (130) may be positioned at least one of the front, rear, port, and starboard sides of the vessel, and if multiple cameras are installed, the surroundings of the vessel may be photographed from multiple angles. Alternatively, a 360-degree camera or a pan-tilt camera may be applied.

[0061] The image acquisition unit (130) can convert the captured image into digital image data and transmit it through an internal interface. The generated image data can subsequently be used as input data for stabilization processing, image quality improvement, object detection, and distance calculation.

[0062] The image acquisition unit (130) can be designed with a waterproof and corrosion-resistant structure suitable for a marine environment, and a coating or physical removal device can be applied to the lens surface to minimize obstruction of view caused by salt and water droplets.

[0064] The edge AI control unit (140) may include at least one processor and can perform a ship collision prevention function by executing program instructions stored in memory (160). The processor can control the communication unit (110), the sensing unit (120), the image acquisition unit (130), and the output unit (150) by executing stored software, and can perform various operations and data processing on the received data. The processor can store navigation data or image data received through the communication unit in memory, process the stored data, and then control the result to be stored back in memory or output through the output unit.

[0065] The processor can be implemented as a central processing unit (AP) and, if necessary, may include or operate in cooperation with a neural network processing unit (NPU), an image signal processing unit (ISP), a sensor hub processor, or a dedicated communication processor. In particular, inference computations for artificial intelligence models can be performed by auxiliary processors such as an NPU.

[0066] Artificial intelligence models can be generated through machine learning, and learning can be performed on the electronic device itself or on an external server. Learning methods may include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Artificial intelligence models can be implemented as deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), transformer-based neural networks, or combinations thereof, and can be configured to include hardware and software structures.

[0067] The edge AI control unit (140) may be configured to process image data transmitted from the image acquisition unit (130), motion data transmitted from the sensing unit (120), and navigation data received through the communication unit (110). Specifically, the edge AI control unit (140) may calculate the inter-frame displacement of the image data based on the motion data transmitted from the sensing unit (120) and correct the shaking of the image to generate a stabilized image. The motion data may represent a 6-degree-of-freedom motion state including rolling, pitching, and yawing, and may perform electronic image stabilization (EIS) or optical image stabilization (OIS) through lens driving by reflecting these motion components at a preset period. For example, the edge AI generation unit (140) can sample the rolling, pitch, and yawing components of the vessel included in the motion data at a preset period (e.g., 100ms or less) and perform electronic image stabilization by moving the region of interest of the image sensor of the ultra-low light camera based on the sampled motion data. Alternatively, the edge AI generation unit (140) can perform image stabilization by driving the lens of the ultra-low light camera based on the motion data.

[0068] The edge AI control unit (140) can generate an improved image with improved image quality by performing fog removal and noise removal processing on the stabilized image. Specifically, the edge AI control unit (140) can estimate the fog concentration by analyzing the histogram of the stabilized image. Then, the edge AI control unit (140) can perform dehazing processing on the stabilized image with an intensity corresponding to the estimated concentration. The edge AI control unit (140) can generate an improved image by applying a multi-frame-based 3D digital noise removal or a deep learning-based denoising filter to the stabilized image on which dehazing processing has been performed.

[0069] The edge AI control unit (140) can detect objects from the enhanced image. For example, the edge AI control unit (140) can detect objects from the enhanced image by combining a coordinate attention mechanism with an object detection model. By detecting objects by combining a coordinate attention mechanism with an object detection model, the performance of detecting small objects in a marine environment can be improved.

[0070] The edge AI control unit (140) can calculate the distance between the detected object and the vessel by applying a monocular depth estimation algorithm to the detected object. Specifically, the edge AI control unit (140) can extract depth values ​​for multiple pixels inside the bounding box of the detected object and calculate the representative depth of the object based on the extracted depth values. The edge AI control unit (140) can calculate the distance between the vessel and the object by converting the representative depth of the object into actual distance units using the ultra-low light camera internal parameters.

[0071] The edge AI control unit (140) can determine the collision risk between the detected object and the vessel based on the distance between the detected object and the vessel. Specifically, the edge AI control unit (140) can determine the distance between the detected object and the vessel, the change in distance, the relative speed, etc., and can determine the collision risk between the object and the vessel based on at least some of the distance, the change in distance, and the relative speed. For example, the edge AI control unit (140) can determine the relative speed by utilizing navigation data of another vessel (e.g., AIS (Automatic Identification System)) received through the communication unit (110). Here, the navigation data may include data about the detected object if the detected object is equipped with AIS, and may not include information about the detected object in the navigation data if the detected object is not equipped with AIS (e.g., a vessel without AIS or a floating object). The edge AI control unit (140) can classify a detected object as a communication object or a non-communication object based on the received navigation data, and if classified as a non-communication object, it can raise the monitoring level of the object to reflect it in the risk judgment. If the navigation data includes information corresponding to the detected object, the edge AI control unit (140) can calculate the relative speed based on the navigation data, and if the navigation data does not include information of the object, it can calculate the relative speed based on the change in the object's position between consecutive frames.

[0072] The edge AI control unit (140) can calculate the shortest approach distance (DCPA) and the shortest approach time (TCPA) based on the calculated distance and relative speed, and can determine the collision risk based on the calculation results.

[0073] The edge AI control unit (140) can control the output unit (150) based on the determined collision risk to output a warning signal. For example, it can visualize and display the location, direction of movement, and expected collision point of the detected object on an electronic chart, or generate a voice or warning sound.

[0074] The edge AI control unit (140) can predict the state of the ship at the time of capsizing, collision, or sinking based on motion data obtained through an inertial measurement unit (IMU) composed of a gyroscope sensor and an accelerometer sensor, and can determine the risk of accident based on the predicted result.

[0075] In this way, the edge AI control unit (140) can prevent collisions of ships by integrally performing image stabilization, image quality improvement, object detection, monocular distance calculation, relative speed determination based on navigation data, shortest approach distance and time calculation, collision risk determination and visualization.

[0077] The output unit (150) may be configured to provide collision risk information calculated by the edge AI control unit (140) to the user. The output unit (150) may include a visual output device, an auditory output device, or a tactile output device. For example, the output unit (150) may include an electronic chart display and may visualize and display the location, direction of movement, and expected collision point of a detected object on the screen. Additionally, if the collision risk is above a threshold value, it may display a warning message or generate a warning sound. If necessary, it may also provide a tactile warning, such as a vibration signal. The output unit (150) may support the operator in intuitively perceiving the surrounding situation and performing appropriate avoidance measures.

[0079] The memory (160) may be configured to store program instructions and data for the operation of the electronic device (100). The memory (160) may include volatile memory or non-volatile memory and may store image data, motion data, navigation data, and collision risk assessment results. The memory (160) may store artificial intelligence models, image processing algorithms, and risk assessment logic executed by the edge AI control unit (140). Additionally, it may store object detection models, monocular depth estimation models, and parameters required for calculating collision risk. The memory (160) may be implemented as a non-transient computer-readable recording medium, and the edge AI control unit (140) may perform a ship collision prevention method by executing instructions stored in the memory (160).

[0081] FIG. 3 is a block diagram briefly illustrating the configuration of an edge AI control unit of an electronic device according to one embodiment of the present application. Referring to FIG. 3, the edge AI control unit (140) may include an image correction unit (141), an object analysis unit (143), and a risk determination unit (145).

[0083] The image correction unit (141) can be configured to correct image data input from the image acquisition unit (130) into a form suitable for a marine environment, thereby generating an improved image that can be used for object detection and monocular depth estimation. The image correction unit (141) can process image stabilization and image quality improvement.

[0084] The image correction unit (141) can generate a stabilized image by correcting image shaking caused by rolling, pitching, and yawing of the vessel using motion data provided by the sensing unit (120). Specifically, the image correction unit (141) can generate a stabilized image by calculating the displacement between image frames by reflecting the angular velocity or attitude change included in the motion data, and inversely correcting the frames according to the calculated displacement.

[0085] For example, the image correction unit (141) may support electronic image stabilization (EIS). Specifically, the image correction unit (141) may mitigate screen tilting and shaking by moving the region of interest of the image sensor or cropping the output frame to correct the viewpoint within the frame. Additionally, the image correction unit (141) may perform optical image stabilization (OIS) if the camera supports lens driving.

[0086] The image correction unit (141) can generate an improved image by improving the quality of the stabilized image through fog removal and noise removal processing on the stabilized image. The image correction unit (141) can improve visibility in foggy or low-light environments by generating an improved image. Specifically, the image correction unit (141) can estimate the fog concentration by analyzing the brightness distribution or histogram features of the stabilized image. The image correction unit (141) can perform dehazing processing on the stabilized image with an intensity corresponding to the estimated fog concentration. During the dehazing process, contrast restoration, color distortion correction, brightness correction, etc., may be performed together.

[0087] The image correction unit (141) can perform noise removal after dehazing processing. Specifically, the image correction unit (141) can perform noise removal on the dehazing processed image using a multi-frame-based 3D digital noise removal (3D-DNR) method, or apply a deep learning-based denoising filter to the dehazing processed image to reduce sensor noise occurring in low light conditions. When performing a multi-frame-based method, the image correction unit (141) can reduce noise by aligning multiple temporally adjacent frames.

[0088] The image correction unit (141) can provide an improved image with image stabilization and image quality improvement to the object analysis unit (143). The improved image can be configured to maintain the outlines of objects such as ships and floating objects more clearly, and can be used as an input to reduce the error in monocular depth estimation.

[0090] The object analysis unit (143) may be configured to analyze the improved image transmitted from the image correction unit (141) to detect objects around the ship and to calculate the distance between the detected objects and the ship. The object analysis unit (143) can process object detection and distance calculation.

[0091] The object analysis unit (143) can detect objects present around a vessel from an enhanced image. Specifically, the object analysis unit (143) can detect objects using a learned object detection model and can generate a bounding box indicating the location of the object and class information indicating the type of object as a result of the detection. For example, the object detection model can be implemented with a structure combined with a Coordinate Attention mechanism to improve object detection performance in a marine environment. The Coordinate Attention mechanism supports the stable detection of objects such as small vessels or floating objects even in situations where environmental noise, such as sea surface reflections, waves, or sea fog, is present, by maintaining spatial location information of the feature map and adjusting the importance between channels. The object analysis unit (143) can generate a bounding box for each detected object and use the area of ​​the bounding box as an input for a subsequent distance calculation step.

[0092] The object analysis unit (143) can calculate depth information by applying a monocular depth estimation algorithm to the detected object. Monocular depth estimation can be performed by estimating the depth distribution of the scene from a single camera image, thereby generating a depth map containing depth values ​​corresponding to each pixel in the image. The object analysis unit (143) can select multiple pixels included within the bounding box of the detected object from the generated depth map and extract the depth values ​​of the selected pixels. The extracted depth values ​​can be used to calculate a representative depth that represents the actual distance of the object. The representative depth can be calculated in various ways; for example, the object analysis unit (143) can calculate the representative depth using the average, median, or weighted average of the pixel depth values ​​within the bounding box. Through this method, the impact of depth estimation errors present in some pixels can be reduced.

[0093] The object analysis unit (143) can calculate the actual distance between the vessel and the object using the calculated representative depth. In the distance calculation process, intrinsic parameters of the ultra-low light camera may be used, and the intrinsic parameters may include focal length, sensor size, and image resolution information. Specifically, the object analysis unit (143) can calculate the distance between the vessel and the object by converting the representative depth into actual distance units using the camera intrinsic parameters. The calculated distance information can be transmitted to the risk judgment unit (145) and used to determine the collision risk.

[0094] The object analysis unit (143) can transmit location information of the detected object, the type of object, and distance information between the vessel and the object to the risk judgment unit (145).

[0096] The risk determination unit (145) may be configured to determine the possibility of collision between a vessel and surrounding objects and to calculate the collision risk based on object information and distance information transmitted from the object analysis unit (143). The risk determination unit (145) may perform navigation data-based analysis, relative speed calculation, approach risk calculation, and risk output processing.

[0097] The risk determination unit (145) can determine the navigation status of surrounding vessels using navigation data of other vessels received through the communication unit (110). The navigation data may include the vessel's position, speed, direction of travel, and identification information.

[0098] The risk determination unit (145) can determine whether an object is a communication object possessing navigation data by comparing the object detected by the object analysis unit (143) with the received navigation data. For example, if the location of the detected object matches the location of the vessel included in the navigation data within a certain range, the risk determination unit (145) can determine that the object is a communication object. On the other hand, if there is no navigation data corresponding to the detected object, the risk determination unit (145) can classify the object as a non-communication object. A non-communication object may be a floating object, a small vessel, or an object that is not equipped with navigation equipment. The risk determination unit (145) can increase the monitoring grade for objects classified as non-communication objects so that they are considered with higher importance in the collision risk determination process.

[0099] The risk determination unit (145) can calculate the relative speed between the vessel and the object. If the object is classified as a communication object, the risk determination unit (145) can calculate the relative speed of the object using speed information and direction of travel information included in the navigation data. If the object is classified as a non-communication object, the risk determination unit (145) can calculate the relative speed using image-based tracking. For example, the risk determination unit (145) can track the position change of the same object in consecutive frames and calculate the movement speed and direction of the object using the amount of position change and the time interval between frames.

[0100] The risk judgment unit (145) can calculate the risk of approach between a vessel and an object using distance information transmitted from the object analysis unit (143) and the calculated relative speed. For example, the risk judgment unit (145) can calculate the Distance at Closest Point of Approach (DCPA) and the Time to Closest Point of Approach (TCPA). DCPA is the distance at which the two objects get closest while maintaining the current route, and TCPA is the time to reach that point. The risk judgment unit (145) can calculate the collision risk based on the calculated DCPA and TCPA values. For example, if the DCPA is less than or equal to a preset safety distance and the TCPA is within a preset time range, the collision risk can be determined to be high.

[0101] The risk judgment unit (145) can transmit the calculated collision risk to the output unit (150). The output unit (150) can display a warning message based on the risk information or visualize the location of the object and the expected collision point on the electronic chart.

[0102] The risk judgment unit (145) can predict the state of the vessel at the time of capsizing, collision, or sinking transmitted from the object analysis unit (143), and can transmit the accident risk level to the output unit (150) based on the predicted result. The output unit (150) can display an emergency risk situation through a warning sound, etc., based on the risk level information.

[0104] The risk judgment unit (145) can be configured to generate warning information for high-risk objects first, thereby enabling the operator to quickly recognize a collision risk situation and take appropriate avoidance measures.

[0106] FIG. 4 is a flowchart illustrating a method for preventing a ship collision according to an embodiment of the present application. The operations in FIG. 4 are not limited in order, and additional operations may be performed between two adjacent operations. Furthermore, at least some of the operations in FIG. 4 may be omitted. In the present invention, the expression that an electronic device (100) performs a specific operation may mean that an edge AI control unit (140) of the electronic device (100) performs a specific operation, or that the edge AI control unit (140) controls other hardware to perform a specific operation.

[0108] Referring to FIG. 4, the electronic device (100) can acquire image data by photographing the surroundings of the vessel through an ultra-low light camera (S1000). Here, the ultra-low light camera can be configured to generate images that can identify objects around the vessel even in a low-light environment, and the captured images can be processed in the form of digital image data.

[0110] The electronic device (100) can acquire motion data for the 6-degree-of-freedom motion state of the vessel using an inertial measurement device (S2000). The inertial measurement device may include a gyroscope sensor and an accelerometer sensor, and can generate motion data representing information on the attitude change of the vessel by detecting motion components including rolling, pitching, and yawing of the vessel.

[0112] The electronic device (100) can generate a stabilized image by calculating the inter-frame displacement of the image data based on motion data (S3000). For example, the electronic device (100) can generate a stabilized image by calculating the inter-frame displacement to correct the tilting and shaking of the image due to changes in the ship's attitude, and correcting the image frames according to the calculated displacement.

[0114] The electronic device (100) can generate an improved image by improving the quality of the stabilized image through image noise removal processing (S4000). For example, the electronic device (100) can estimate the fog concentration by analyzing the histogram information of the image and perform dehazing processing with an intensity corresponding to the estimated fog concentration. In addition, image noise occurring in a low-light environment can be reduced by applying a multi-frame-based 3D digital noise removal or a deep learning-based denoising filter.

[0116] The electronic device (100) can detect objects from an enhanced image and calculate the distance between the detected objects and the vessel by applying a monocular depth estimation algorithm (S5000). Specifically, the electronic device (100) can detect objects around the vessel using a learned object detection model and can generate a bounding box indicating the location of the object and class information indicating the type of object as a result of the detection. In addition, depth information of the image can be calculated using a monocular depth estimation algorithm, and a representative depth of the object can be calculated based on depth values ​​for multiple pixels inside the bounding box. The electronic device (100) can calculate the distance between the vessel and the object by converting the representative depth into actual distance units using the intrinsic parameters of the camera.

[0118] The electronic device (100) can determine the collision risk between the detected object and the vessel based on the calculated distance (S6000). For example, the electronic device (100) can calculate the relative speed using navigation data of another vessel received through a communication unit, and if navigation data is not available, it can estimate the relative speed using the change in the object's position in a continuous frame. In addition, based on the calculated distance and relative speed, the shortest approach distance (DCPA) and the shortest approach time (TCPA) can be calculated, and the collision risk can be determined based on the calculated values. The determined collision risk can be displayed on an electronic chart or output in the form of warning information.

[0120] The features, structures, effects, etc. described in the embodiments above are included in at least one embodiment of the present invention and are not necessarily limited to only one embodiment. Furthermore, the features, structures, effects, etc. exemplified in each embodiment may be combined or modified and implemented in other embodiments by a person skilled in the art to which the embodiments belong. Accordingly, details regarding such combinations and modifications should be interpreted as being included within the scope of the present invention.

[0121] Furthermore, although the embodiments have been described above, this is merely illustrative and does not limit the invention. Those skilled in the art will understand that various modifications and applications not exemplified above are possible within the scope of the essential characteristics of the embodiments. In other words, each component specifically shown in the embodiments may be modified and implemented. Differences related to such modifications and applications should be interpreted as being included within the scope of the invention as defined in the appended claims.

Claims

Claim 1 A method for preventing a collision of a ship using a ship collision prevention device comprises: a step of acquiring image data by photographing the surroundings of the ship through an ultra-low light camera; a step of acquiring motion data regarding the motion state of the ship with six degrees of freedom using an inertial measurement unit; a step of generating a stabilized image by calculating the inter-frame displacement of the image data based on the motion data; a step of generating an improved image by improving the image quality of the stabilized image through image noise removal processing; a step of detecting an object from the improved image and calculating the distance between the detected object and the ship by applying a monocular depth estimation algorithm; and a step of determining the collision risk between the detected object and the ship based on the distance. Claim 2 A method for preventing ship collisions according to claim 1, wherein the step of generating the stabilized image comprises: a step of sampling the rolling, pitching, and yawing components of the ship included in the motion data at a preset period; and a step of performing electronic image stabilization by moving the region of interest of the image sensor of the ultra-low light camera based on the sampled motion data. Claim 3 A method for preventing ship collisions according to claim 1, wherein the step of generating the stabilized image comprises the step of performing image stabilization by driving the lens of the ultra-low light camera based on the motion data. Claim 4 A method for preventing ship collisions according to claim 1, wherein the step of generating the improved image comprises: a step of estimating fog concentration based on the histogram of the stabilized image; and a step of applying a dehazing algorithm to the stabilized image with an intensity corresponding to the fog concentration. Claim 5 A method for preventing ship collisions according to claim 4, wherein the step of generating the improved image further comprises the step of applying a multi-frame-based 3D digital noise removal or a deep learning-based denoising filter to the stabilized image to which the dehazing algorithm is applied. Claim 6 A method for preventing ship collisions according to claim 1, wherein the step of calculating the distance comprises: a step of detecting an object from the enhanced image by combining a coordinate attention mechanism with an object detection model; a step of extracting depth values ​​for a plurality of pixels inside the bounding box of the detected object; a step of calculating a representative depth of the object based on the depth values; and a step of converting the representative depth of the object into actual distance units based on the internal parameters of the ultra-low light camera. Claim 7 A method for preventing ship collisions according to claim 1, wherein the step of determining the collision risk comprises: receiving navigation data of at least one other vessel; determining the relative speed with the detected object based on the received navigation data if the navigation data includes navigation data of the object, and determining the relative speed with the detected object based on the change in the position of the object between consecutive frames if the navigation data does not include navigation data of the object; calculating the shortest approach distance and the shortest approach time based on the distance between the detected object and the vessel and the determined relative speed; and calculating the collision risk based on the calculated shortest approach distance and the shortest approach time. Claim 8 A method for preventing ship collisions according to claim 7, wherein the step of determining the collision risk further comprises: a step of classifying the detected object into a communication object and a non-communication object based on the received navigation data; and a step of upgrading the monitoring grade of the object when the object is classified as a non-communication object. Claim 9 A method for preventing ship collisions according to claim 1, further comprising the step of visualizing and outputting the location of the object and the expected collision point on an electronic chart based on the collision risk determined above. Claim 10 A non-transient computer-readable recording medium comprising instructions to perform the method according to paragraph 1. Claim 11 An electronic device for preventing collisions of ships comprises: an image acquisition unit that acquires image data by photographing the surroundings of the ship through an ultra-low light camera; a sensing unit that acquires motion data regarding the motion state of the ship with six degrees of freedom using an inertial measurement unit; a memory that stores at least one instruction; and an edge AI control unit that includes a processor that executes the at least one instruction. The edge AI control unit generates a stabilized image by calculating the inter-frame displacement of the image data based on the motion data, generates an improved image by improving the quality of the stabilized image through fog removal and noise removal processing, detects an object from the improved image, calculates the distance between the detected object and the ship by applying a monocular depth estimation algorithm, and determines the collision risk between the detected object and the ship based on the distance.

Images