Vessel identification system and method for satellite or aerial imagery utilizing AIS(Automatic Identification System) information

Abstract

A ship identification system using satellite or aerial imagery utilizing AIS information according to an embodiment of the present invention may include: a receiving unit that receives satellite or aerial imagery and AIS (Automatic Identification System) data; a processing unit that preprocesses the AIS data and image received from the receiving unit, calculates AIS data for the time of capture (ts) of the image, and matches the calculated corrected AIS data with a ship detected in the image to align information; and an output unit that outputs the information of the ship aligned by the processing unit as an image. Additionally, the processing unit may use a linear interpolation method or an interpolation method considering speed and direction of movement by user selection so that the received AIS data is corrected to the value of the time of capture (t).

Description

Technology Field

[0001] The present invention relates to a ship identification system and method using satellite or aerial imagery utilizing AIS (Automatic Identification System) information. Background Technology

[0003] AIS (Automatic Identification System, hereinafter referred to as “AIS”) is an automatic ship identification system. Developed as a tool to prevent ship collisions, AIS enables the identification of the locations and travel paths of different vessels and allows for mutual identification.

[0004] Meanwhile, according to the International Maritime Organization (IMO) Ship Safety Act, ships exceeding a certain weight are required to install AIS.

[0005] Generally, the above AIS transmits the collected information to ground receivers or satellite receivers at intervals of seconds to minutes, depending on the vessel's speed of movement.

[0006] The information transmitted by the above-mentioned AIS (hereinafter referred to as “AIS information” or “AIS data”) includes not only static information such as the type, width, length, height, name, and IMO number (International Maritime Organization), but also dynamic information such as the vessel's position, time of information reception, course, and speed.

[0007] Meanwhile, visually identifying whether an object visible in an image is an actual vessel or noise, such as waves, floating debris, or other marine structures, solely from images captured by satellites or aircraft (hereinafter referred to as "satellite images") is significantly limited.

[0008] Therefore, if the aforementioned AIS information can be fused or combined when identifying a vessel in the satellite imagery above, more accurate and precise vessel identification can be achieved.

[0009] However, there is a problem in that not all vessels are transmitting or receiving AIS information, as vessels can manually turn AIS signals on or off. Therefore, it is necessary to identify vessels with AIS turned off using satellite or aerial imagery to compensate for this.

[0010] In addition, AIS information may differ from the actual ship location due to reasons such as radio interference and signal delay.

[0011] In addition, since AIS has a non-periodic transmission pattern at intervals ranging from seconds to minutes, it may be difficult to obtain AIS information (e.g., position, speed, heading, etc.) at any given point in time.

[0012] In this regard, conventional technology simply matches the closest or closest AIS information to the ship information in satellite imagery. However, since this conventional method does not take into account the ship's speed of movement or direction, the matching accuracy may be low in images where ships are detected at high density in a small space.

[0013] In addition, if the time of satellite imagery capture is between the transmission time intervals of AIS, then if AIS information is matched to the vessel in the satellite imagery, errors such as position are inevitable.

[0014] Therefore, an estimation relationship that considers more precise and diverse variables is required to acquire AIS information during the time interval between the above AIS transmissions. In other words, if AIS information can be fused or aligned with satellite imagery more precisely and accurately, a more improved ship identification system can be implemented.

[0015] To do this, if there is a difference between the reception time of AIS information and the time of satellite image capture, it is necessary to fuse the information by matching them.

[0016] In this regard, Korean Published Patent Application KR 10-2020-0050736 A ("Method and apparatus for correcting image sensor misalignment using ship identification information") discloses a method for correcting the misalignment (i.e., line of sight angle) of an image sensor when the ship coordinates from an image and the ship coordinates from AIS are received with a time difference. Briefly, the prior art document discloses a method for calculating a correction coordinate by shifting the second coordinate, which is the AIS-based ship coordinate, by the difference in observation time (Δτ) between the first coordinate of the ship calculated from the ship image and the second coordinate of the ship calculated from the ship identification information (AIS), thereby synchronizing the time with the first coordinate, and correcting the misalignment value of the image sensor using the line of sight angle of the image sensor looking at the calculated correction coordinate.

[0017] However, the aforementioned prior art document merely discloses a method for correcting the line-of-sight angle of an image sensor under the premise that the vessel in the satellite image and the vessel based on AIS information are the same vessel. In other words, the aforementioned prior art document does not provide any technical motivation or implications regarding which AIS data among multiple AIS data received with a time difference should be matched to a designated vessel in the image.

[0018] Furthermore, the aforementioned prior art documents merely disclose that AIS-based ship coordinates are moved by the difference in observation time to synchronize the time with the ship coordinates in the image, and do not disclose at all how the AIS-based ship coordinates (second coordinates) are moved to the corrected coordinates using any method or relationship.

[0019] In addition, the AIS-based ship correction coordinates, which are time-synchronized in the aforementioned prior art document (see Figure 3), have a different position from the ship coordinates in the image, and the line-of-sight angle of the image sensor is corrected based on this. That is, the aforementioned prior art document merely discloses a method for correcting when the ship coordinates in the image and the AIS-based ship correction coordinates differ solely due to misalignment of the image sensor. Therefore, the aforementioned prior art document has a problem in that it fails to reflect at all the possibility of error or error in the AIS-based correction coordinates.

[0020] Furthermore, the aforementioned prior art literature has a problem in that it does not reflect at all the possibility that the ship coordinates in the image may be subject to errors caused by environmental factors other than the image sensor.

[0021] Therefore, in order to realize more precise and reliable remote sensing images by reflecting various environmental factors, a new system and method are required to determine the reliability of both the vessel coordinates in the image and the vessel coordinates based on AIS information, and to supplement the results accordingly. Prior art literature

[0022] KR 10-2020-0050736 A (2020.05.12.), Method and apparatus for correcting image sensor misalignment using vessel identification information The problem to be solved

[0023] The objective of the present invention is to provide a system and method for identifying ships in satellite imagery using AIS information that solves the problems of the aforementioned prior art.

[0024] Another objective of the present invention is to provide a ship identification system and a method capable of acquiring AIS data at the time of satellite and aerial image capture in order to accurately perform ship identification within satellite or aerial imagery.

[0025] Another objective of the present invention is to provide a ship identification system and a method that can resolve the difference caused by the time of observation (capture) of satellite or aerial imagery and the time of reception of AIS data.

[0026] Another objective of the present invention is to provide a ship identification system and a method that corrects AIS data to have information at the time of satellite or aerial image capture, and evaluates the reliability thereof to provide more precise real-time ship information.

[0027] Another objective of the present invention is to provide a ship identification system and a method capable of distinguishing between a ship in satellite or aerial imagery and a ship based on AIS information in the same area (or coordinates) when they are different, and matching accurate AIS information for each individual ship. means of solving the problem

[0029] To achieve the above objective, a ship identification system using satellite or aerial imagery utilizing AIS information according to an embodiment of the present invention may include: a receiving unit that receives satellite or aerial imagery and AIS (Automatic Identification System) data; a processing unit that preprocesses the AIS data and image received from the receiving unit, calculates AIS data for the time of capture (ts) of the image, and matches the calculated corrected AIS data with a ship detected in the image to align information; and an output unit that outputs the information of the ship aligned by the processing unit as an image.

[0030] In addition, the processing unit may use a linear interpolation method or an interpolation method considering speed and direction of movement by user selection so that the received AIS data is corrected to the value of the image capture time (t).

[0031] In addition, the above linear interpolation method can approximate the ship moving in a straight line at a constant speed in the user's region of interest (ROI) to calculate AIS data at the time of image capture (ts).

[0032] In addition, the above linear interpolation method can calculate AIS data at the image capture time (ts) by the following mathematical formula 1.

[0033] Mathematical formula 1:

[0034]

[0035]

[0036] (t_s: time of video capture, X_s: X-plane position at the time of video capture, Y_s: Y-plane position at the time of video capture, X_(AIS-1): X-plane position on AIS data immediately before the time of video capture, Y_(AIS-1): Y-plane position on AIS data immediately before the time of video capture, X_(AIS+1): X-plane position on AIS data immediately after the time of video capture, Y_(AIS+1): Y-plane position on AIS data immediately after the time of video capture, t_(AIS-1): time of reception of AIS data immediately before the time of video capture, t_(AIS+1): time of reception of AIS data immediately after the time of video capture (ts))

[0037] In addition, the interpolation method considering the speed and direction of movement can calculate the AIS data at the time of video recording using only the AIS data received immediately before the time of video recording.

[0038] In addition, the interpolation method considering the speed and direction of movement above can calculate AIS data at the time of video capture (ts) by the following mathematical formula 2.

[0039] Mathematical formula 2:

[0040]

[0041]

[0042] (t_s: time of video capture, X_s: X-plane position at the time of video capture, Y_s: Y-plane position at the time of video capture, X_(AIS-1): X-plane position on AIS data immediately prior to video capture, Y_(AIS-1): Y-plane position on AIS data immediately prior to video capture, t_(AIS-1): time of reception of AIS data immediately prior to video capture)

[0043] In addition, the processing unit may include a data processing unit that processes the received AIS data; and an image processing unit that processes the received image.

[0044] In addition, the image processing unit can provide a matched image to the output unit by rotating and scaling the image using the AIS data at the time of image capture calculated by the data processing unit.

[0045] In addition, the processing unit can extract an error (d) through Euclidean distance calculation when the position of the AIS data corrected to the shooting time (ts) of the image for matching with the vessel detected in the image is different from the position of the vessel detected in the image.

[0046] In addition, the processing unit compares the extracted error (d) with a threshold value (a) based on the evaluation of the reliability of the AIS data and the reliability of the image, and if the extracted error (d) is greater than the threshold value (a), it aligns with the position value of the calculated corrected AIS data, and if the extracted error (d) is less than or equal to the threshold value (a), the position value of the vessel derived from the image can be used as is in the image.

[0047] In addition, the reliability of the above AIS data can be evaluated by the following formula.

[0048]

[0049] (C_AIS: Result value representing the reliability of AIS data in the range of 0 to 1, Xs and Ys: Interpolated AIS data X,Y coordinates, Xv and Yv: X,Y coordinates of vessels detected in the image, σ_GPS is the GPS error range value)

[0050] In addition, the reliability of the above image can be evaluated by the following formula.

[0051]

[0052] (C_VID: Result value representing image reliability in the range of 0 to 1, Xs and Ys are interpolated AIS data X,Y coordinates, Xv and Yv: X,Y coordinates of detected vessels in the image, ??_VID is the value for the total image reliability error)

[0053] In addition, the value of the total error for the image reliability mentioned above can be defined as the sum of geometric error values ​​due to satellite shooting angle or Earth curvature, hardware error values ​​due to sensor noise or lens distortion, error values ​​due to ship speed, and error values ​​due to shooting environment.

[0054] In addition, the above threshold value (a) is, Equation It can be defined by.

[0055] In another aspect, a method for identifying a ship in satellite or aerial imagery using AIS information according to an embodiment of the present invention may include: receiving satellite or aerial imagery and AIS data; correcting the received AIS data to the AIS data at the time of image capture (ts); searching for a ship in the image and setting it as a target; and matching and aligning the corrected AIS data to the target-set ship.

[0056] In addition, the step of correcting the AIS data at the time of the image capture (ts) can calculate the corrected AIS data by considering the speed and direction of movement at the location of the AIS data received immediately before the time of the image capture (ts-1).

[0057] In addition, if a difference is identified between the position of the target in the above image and the position of the above-mentioned corrected AIS data, the error (d) between the two positions can be extracted through Euclidean distance calculation.

[0058] In addition, the error (d) is compared with a threshold value (a) defined by a formula that reflects the reliability of the AIS data and the reliability of the image, and based on the result of the comparison between the error (d) and the threshold value (a), the information value with higher reliability can be matched to the data of the target in the image. Effects of the invention

[0060] According to the present invention, although it is relatively difficult to accurately and precisely identify information about each vessel using only satellite imagery, accurate and precise vessel information can be provided within the imagery by fusing (or matching) AIS information.

[0061] According to the present invention, ships can be identified more accurately in satellite and aerial images by utilizing AIS information, and errors caused by the difference between the time of receiving the image and the AIS information can be effectively and reliably corrected.

[0062] According to the present invention, since AIS information can be fused (or aligned) with ships in satellite and aerial images, the problems that occur when identifying ships solely from satellite images in the past and the problems that occur when identifying ships solely from AIS information in the past can be mutually resolved, thereby providing more precise and accurate output results.

[0063] According to the present invention, since the vessel in the image includes unique identification information (e.g., IMO number), speed, and direction of movement among the AIS information, precise tracking of the actual vessel in the image is possible, and false positives can be reduced.

[0064] According to the present invention, for satellite and aerial images captured at a specific point in time, information at the captured specific point in time is obtained from the received AIS information using an interpolation technique, thereby offsetting errors in the ship's position and enabling more precise matching of the ship with the AIS information when a dense ship is detected in a narrow area of ​​interest.

[0065] According to the present invention, the Euclidean distance between ships in AIS information and satellite imagery is calculated to analyze the error, and a more precise result can be output by comparing it with a threshold value set to evaluate reliability. Brief explanation of the drawing

[0067] FIG. 1 is a diagram exemplarily showing the configuration of a ship identification system according to an embodiment of the present invention. FIG. 2 is a flowchart exemplarily showing a ship identification method according to an embodiment of the present invention. Figure 3 is a flowchart showing a method for correcting an error that occurs between the position of a ship in an image and the position of corrected AIS data in a ship identification method according to an embodiment of the present invention. FIG. 4 is a diagram showing a screen for searching for and setting a target (ship) in an area of ​​interest (ROI) in a satellite image according to an embodiment of the present invention. FIG. 5 is a diagram showing a screen in which a target (ship) in a satellite image and AIS data matched to the target (ship) are fused (or aligned) according to an embodiment of the present invention. Specific details for implementing the invention

[0068] Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. It should be noted that in assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the embodiments of the present invention, if it is determined that a detailed description of related known components or functions would hinder understanding of the embodiments of the present invention, such detailed description is omitted.

[0069] In addition, terms such as first, second, A, B, (a), (b), etc., may be used when describing the components of the embodiments of the present invention. These terms are intended only to distinguish the components from other components, and the essence, order, or sequence of the components is not limited by the terms. Where it is stated that a component is "connected," "combined," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but that another component may also be "connected," "combined," or "connected" between each component.

[0070] FIG. 1 is a diagram exemplarily showing the configuration of a ship identification system according to an embodiment of the present invention.

[0071] For the sake of convenience of explanation, satellite or aerial images may be referred to as 'images', and information transmitted and received by the AIS (Automatic Identification System) may be referred to as 'AIS data or AIS information'.

[0072] Referring to FIG. 1, a ship identification system (1) for satellite or aerial imagery according to an embodiment of the present invention may include a receiving unit (10) that receives AIS data and imagery.

[0073] The above receiving unit (10) may include an image receiving unit (11) for receiving satellite or aerial images and a data receiving unit (13) for receiving AIS data.

[0074] The above image receiving unit (11) can receive the shooting time (ts) of the received image together. That is, the above image receiving unit (11) can acquire the shooting time (ts) of the image that serves as a time reference for acquiring AIS data.

[0075] In addition, the image receiving unit (11) can receive an image of a region of interest (ROI) that the user is interested in. For example, the image receiving unit (11) can receive a specific area of ​​the sea where multiple vessels are navigating. It can also receive the time of capture (ts) of the received image.

[0076] The data receiving unit (13) can receive AIS data for vessels in an area corresponding to the area of ​​interest (ROI) of the user. Additionally, the data receiving unit (13) can receive AIS data at a time point before and after the time of video capture (ts). For example, the data receiving unit (13) can receive AIS data at a time point ts±n (n=1,2,3…) based on the time of video capture (ts). Preferably, to reflect fast calculation speeds and real-time changing vessel information, AIS data at a time point ts-1 based on the time of video capture (ts) can be received.

[0077] In addition, the above ts±1 time point can be understood as the AIS data immediately before and immediately after ts.

[0078] Meanwhile, depending on the user's needs, AIS data of the target (vessel) after the shooting time (ts) of the above video may be received and used for analysis, so AIS data at time ts+n may also be received.

[0079] As described above, since AIS data is transmitted and received at intervals of seconds to minutes, there may be a difference between the time of video capture (ts) and the time of transmission and reception of AIS data. Therefore, the data receiving unit (13) can receive AIS data before and after the time of video capture (ts).

[0080] In addition, as described above, AIS data may include not only static information such as the type, width, length, height, name, and IMO number (International Maritime Organization), but also dynamic information such as the location, time of information reception, course, and speed of the vessel.

[0081] The above-mentioned satellite or aerial image ship identification system (1) may further include a processing unit (20) that processes received AIS data and images.

[0082] The above processing unit (20) can perform a preprocessing process to remove noise or abnormal values ​​from AIS data at a time before or after the time of shooting (ts) of satellite and aerial images.

[0083] For example, the received AIS data may contain errors such as duplicate unique information of the vessel or positional errors caused by GPS equipment malfunction. The processing unit (20) can perform a preprocessing process to verify and process such errors in the AIS data.

[0084] As another example, if the unique identification information of a ship is transmitted from two or more ships due to duplication of unique identification information caused by human error or intentional manipulation of information, the processing unit (20) can perform a preprocessing process to separate the information of the ships with duplicated unique identification information into information for each individual ship based on information such as location, route, and speed, and to extract data for the ship corresponding to the user's area of ​​interest (ROI).

[0085] Meanwhile, AIS data positioning errors may occur due to temporary errors or malfunctions in the vessel's GPS equipment.

[0086] According to the following embodiment of the present invention, the processing unit (20) evaluates the reliability of the AIS data and reflects information with higher reliability in the final output result. Therefore, it is possible to easily distinguish fundamental AIS data errors caused by factors such as the aforementioned GPS equipment, rather than errors caused simply by differences in shooting time and time, and to remove or correct them from the output result.

[0087] Furthermore, the processing unit (20) can detect a vessel in the image within the user's region of interest (ROI) for which AIS data is not received. In this case, the processing unit (20) can feed back to the preprocessing process to set the vessel as an unregistered vessel or a Black ship and process it to output vessel information obtained only from the image.

[0088] Additionally, the above image may contain noise caused by floating objects, structures, waves, etc. that may appear in a marine environment. Therefore, the processing unit (20) can perform a preprocessing process to remove such noise.

[0089] In addition, the processing unit (20) can perform a process of correcting the time difference between the video capture time (ts) and the time difference caused by the AIS data transmission and reception interval.

[0090] That is, the processing unit (20) can correct the AIS data. Specifically, the processing unit (20) can process the AIS data at the video shooting time (ts) based on the AIS data received before and after the video shooting time (ts).

[0091] For example, the processing unit (20) can calculate AIS data at the time of video capture (ts) by applying a linear interpolation or velocity-based interpolation method. A detailed method for calculating this will be described later.

[0092] In addition, the processing unit (20) can search for the user's target (vessel) in the received user's region of interest (ROI) image and identify the location of the target.

[0093] In addition, the processing unit (20) can match the target and the corrected AIS data to align (or may be expressed as 'fusion') the vessel and the AIS data at the time of video capture (ts) within the image.

[0094] In addition, if there is a difference (i.e., error) between the position of the target in the image and the position of the corrected AIS data, the processing unit (20) can extract the difference (error) value through Euclidean distance calculation and calculate and compare the reliability of the image and the AIS data.

[0095] In addition, the processing unit (20) can determine whether the difference (error) exceeds a threshold value and apply a highly reliable data value (e.g., location) based on the result. Therefore, the final output image can be provided with more precise and accurate AIS data for each vessel.

[0096] Meanwhile, the processing unit (10) may include a data processing unit (22) that preprocesses and corrects the received AIS data (22) described above, an image processing unit (24) that preprocesses the received image described above and searches for and sets targets (ships) in the user's area of ​​interest (ROI), and a matching unit (26) that matches and aligns the AIS data corrected by the data processing unit (22) with targets in the user's area of ​​interest (ROI).

[0097] The above data processing unit (22) can also filter out reception errors, such as location errors caused by signal interference, from the received AIS data during the preprocessing process.

[0098] The above data processing unit (22) can calculate AIS data (especially dynamic information) at the time of video capture (ts) by using a linear interpolation or velocity-based interpolation method when the interval of the received AIS data is not constant.

[0099] Since the above linear interpolation is performed when the ship moves in a straight line at a constant speed, it can quickly derive a result value, making it convenient for users to utilize in large-capacity and wide-area images.

[0100] The above processing unit (20) can calculate AIS data at the time of image capture (ts) using linear interpolation with the following mathematical formula 1 when a vessel in a designated window in the user's region of interest (ROI) can be evaluated as moving in a straight line at a constant speed.

[0101]

[0102] Here, t_s is the time of video recording, X_s and Y_s are the positions (X,Y) at the time of video recording (ts), X_(AIS-1) and Y_(AIS-1) are the (X,Y) positions of the vessel in the AIS data immediately before the time of video recording (ts), X_(AIS+1) and Y_(AIS+1) are the (X,Y) positions of the vessel in the AIS data immediately after the time of video recording (ts), t_(AIS-1) is the time of reception of the AIS data received immediately before the time of video recording (ts), and t_(AIS+1) is the time of reception of the AIS data received immediately after the time of video recording (ts).

[0103] According to the above mathematical formula 1, AIS data (especially dynamic information) at the time of video recording (ts) can be calculated using AIS data immediately before and after the time of video recording (ts).

[0104] The above interpolation considering speed and direction of movement is a method that considers the ship's speed and direction of movement based on AIS data at time t-1, and may be desirable for users who require more precise analysis and matching.

[0105] The above processing unit (20) can calculate AIS data at the time of image capture (ts) by Velocity-Based Interpolation considering speed and direction of movement using the following mathematical formula 2 in order to precisely track or analyze a target (ship) in the user's area of ​​interest (ROI).

[0106]

[0107] Here, t_s is the time of video recording, X_s and Y_s are the positions (X,Y) at the time of video recording (ts), X_(AIS-1) and Y_(AIS-1) are the (X,Y) positions of the vessel in the AIS data immediately before the time of video recording (ts), V_(AIS-1) is the vessel speed (m / s) in the AIS data immediately before the time of video recording (ts), and θ_(AIS-1) is the direction of movement (azimuth, degrees) in the AIS data immediately before the time of video recording (ts).

[0108] The above mathematical formula 2 can be corrected to the position of the satellite image point (ts) considering the direction of movement of the target (ship) by calculating the variables cos(θ) and sin(θ).

[0109] Accordingly, according to the above mathematical formula 2, the AIS data (in particular, dynamic information) at the time of video capture (ts) can be calculated using the AIS data immediately prior to the time of video capture (ts).

[0110] The image processing unit (24) can provide an image aligned with a target (ship) in the image by performing rotation, scaling, etc. on the AIS data at the image capture time (ts) calculated (or corrected) by the data processing unit (22).

[0111] The above matching unit (26) can identify cases where there is a difference between the position of the target in the image and the position of the corrected AIS data.

[0112] In addition, the above-mentioned matching unit (26) can match more reliable data to the target by extracting the error (d) between two locations through Euclidean distance calculation and performing a reliability evaluation process.

[0113] The above-mentioned satellite or aerial image ship identification system (1) may further include an output unit (30) and a storage unit (40).

[0114] The output unit (30) can output the result processed by the processing unit (20) as an image. For example, the output unit (30) can output an image in which corrected AIS data is aligned for each target (ship) at the time of image capture (ts).

[0115] The storage unit (40) can store the results of each process performed by the processing unit (20). Additionally, information stored in the storage unit (40) can be loaded at the request of the processing unit (20).

[0116] FIG. 2 is a flowchart exemplarily showing a method for identifying a ship in satellite or aerial imagery using AIS information according to an embodiment of the present invention.

[0117] Referring to FIG. 2, a method for identifying a ship in an image according to an embodiment of the present invention will be described.

[0118] First, a step (S10) of receiving satellite or aerial imagery can be performed. The satellite or aerial imagery can be received by the receiving unit (10) along with the time of image capture (ts). The receiving unit (10) can receive imagery of a region of interest to the user, i.e., a user's area of ​​interest (ROI). For example, the image receiving unit (10) can receive imagery of an optical satellite, synthetic aperture radar (SAR), etc., of the user's area of ​​interest (ROI) and receive the time of image capture (ts).

[0119] In addition, a step (S20) of receiving and preprocessing AIS data can be performed simultaneously with or after securing an image of the user's region of interest (ROI).

[0120] The above AIS data can be received by the receiving unit (10) for all vessels existing in the user's area of ​​interest (ROI).

[0121] In addition, a preprocessing step can be performed to remove noise or anomalies from AIS data at points in time before or after the video capture time (ts). For example, position errors caused by signal interference in the received AIS data can also be filtered out during the preprocessing step.

[0122] In addition, a preprocessing process for the image may also be performed in step S20 as needed, such as for noise. As described above, the preprocessing process can be performed in the processing unit (20).

[0123] Meanwhile, vessels within the user's Region of Interest (ROI) in the image that do not receive AIS data can be detected. In this case, the vessel for which AIS data is not received can be fed back to the preprocessing step (S20) to be designated as an unregistered vessel or a Black ship. The unregistered vessel is information obtained only through satellite or aerial imagery and can be processed during AIS data matching.

[0124] After the above AIS data is received and / or preprocessed, a step (S30) of correcting the AIS data can be performed.

[0125] The above AIS data correction step (S30) can be understood as a step of correcting the time difference between the image capture time (ts) and the time difference caused by the AIS data transmission and reception interval for each vessel, as described above.

[0126] For example, step S30 can calculate AIS data (especially dynamic information) at the image capture time (ts) by using a linear interpolation or velocity-based interpolation method based on AIS data received before or after the image capture time (ts).

[0127] In step S30, it can be determined whether a vessel within a window designated as a user region of interest (ROI) can be evaluated as moving in a uniform linear motion.

[0128] For example, depending on the area covered by the video of the user's region of interest (ROI) (e.g., window size) or if the time interval for tracking the target (ship) is larger than a preset standard, the ship in the video can be approximated as moving in a straight line at a constant speed and calculated as AIS data at the time of video capture (ts), thereby improving the processing speed of large volumes.

[0129] Therefore, when the above vessel is evaluated as moving in a constant linear motion, the AIS data at the time of image capture (ts) can be calculated using Equation 1, which is the linear interpolation method described above.

[0130] Meanwhile, if the above linear interpolation method cannot be used in step S30, or if one wishes to calculate AIS data at the time of image capture (ts) with relative precision by considering the ship's speed and direction of movement, the AIS data at the time of image capture (ts) can be calculated using Equation 2, which is a velocity-based interpolation method that considers the speed (including the direction of movement) described above.

[0131] In this case, based on the AIS data at the time point (t-1) immediately preceding the image capture time point (ts), the AIS data at the image capture time point (ts) can be calculated considering the ship's speed and direction of movement.

[0132] AIS data calculated using the above method can be named “corrected AIS data.”

[0133] After the above AIS data correction step (S30) is completed, a step (S40) of searching for and setting a target (vessel) within the image of the user's region of interest (ROI) can be performed.

[0134] That is, in step S40, vessels existing within the user's Region of Interest (ROI) are searched for and extracted, and targets can be set for vessels that the user wishes to track or vessels that meet preset conditions.

[0135] For example, the image processing unit (24) can search for and extract a ship within the image of the user's region of interest (ROI). Here, the search and extraction of the ship image within the image can be performed using a known deep learning (machine learning) method.

[0136] Additionally, in step S40, a target (vessel) for matching the corrected AIS data can be set. For example, the image processing unit (24) can set a target for matching the corrected AIS data among the vessels searched within the image of the user's region of interest (ROI).

[0137] When the above target is set, the step (S50) of matching the corrected AIS data to the target can be performed.

[0138] According to the above-described step S30, AIS information (i.e., corrected AIS data) for each vessel at the time of video capture (ts) can be obtained, and accordingly, the position (coordinates) of each vessel based on the corrected AIS data at the time of video capture (ts) can be obtained. Therefore, the target vessel searched and set in the image can be matched with the AIS data calculated as the position (coordinates) of the same area.

[0139] For example, the processing unit (20) can find a vessel in an image of a user's region of interest (ROI) corresponding to a location (coordinates) based on the corrected AIS data. The processing unit (20) can also match the corrected AIS data to the vessel.

[0140] And for a vessel in an image matched with corrected AIS data, a step (S60) of aligning two types of data (e.g., data from a satellite and corrected AIS data) into information about one identical vessel based on the time of shooting (ts) can be performed.

[0141] Vessels with matched data can have more accurate position coordinates and can provide static and dynamic information via AIS.

[0142] That is, a step (S70) can be performed in which the result of matching the corrected AIS data for each target (vessel) in the video is output.

[0143] According to this, since the AIS data of the vessel in the image is aligned to output not only static but also dynamic information, more accurate and precise identification can be achieved when identifying the vessel in the image. Furthermore, by aligning the time difference between the reception of the AIS information and the capture time of the satellite image, the two pieces of information regarding the same vessel can be fused into one accurate piece of information.

[0144] Meanwhile, based on the above-mentioned video capture time (ts), there is a possibility that the position (coordinates) of the vessel shown in the corrected AIS data may not match very accurately due to errors that may occur in the video and / or errors that may occur in the AIS data.

[0145] Therefore, in step S60, which aligns the corrected AIS data with the vessel in the image, it is necessary to determine whether the information regarding the vessel in the image is reliable or whether the corrected AIS data information is more reliable, taking into account the aforementioned error influence.

[0146] In this regard, a detailed explanation will be provided based on Figure 3 below.

[0147] Figure 3 is a flowchart showing a method for correcting an error that occurs between the position of a ship in an image and the position of corrected AIS data in a ship identification method according to an embodiment of the present invention.

[0148] As described above, when the processing unit (20) identifies a difference between the position of the target in the image and the position of the corrected AIS data, the processing unit (20) can extract and store the difference between the two positions (hereinafter referred to as “error (d)”) through Euclidean distance calculation.

[0149] And by determining whether the above error (d) is greater or smaller than the threshold value reflecting the reliability evaluation formula, more reliable data can be aligned with the target.

[0150] In this regard, referring to FIG. 3, if information regarding two positions (coordinates) is generated in the step (S60) of aligning corrected AIS data with the target (vessel), the position value (coordinates by plane) of the image of the target (vessel) and the position value (coordinates by plane) based on the corrected AIS data can be compared. (S610)

[0151] The above comparison can be understood as comparing the location of the corrected AIS data with the location of the target (vessel) detected in the image.

[0152] And through the above comparison, the error (d) between the positions (coordinates) provided by the two data can be calculated. (S620)

[0153] The above error (d) can be calculated as the Euclidean distance between the corrected AIS data and the target (ship) position in the image, as shown in Equation 3 below.

[0154]

[0155] Here, X_im and Y_im represent position coordinates per plane within the image, and X_AIS Data and Y_AISData represent position coordinates per xy plane of the AIS data corrected for the image capture time (ts).

[0156] After the above error (d) is calculated, it can be determined whether the above error (d) exceeds a threshold value (a). (S630)

[0157] Hereinafter, a threshold value (a) according to an embodiment of the present invention is described in detail.

[0158] Sources of error in AIS data include GPS errors, such as multipath errors, ionospheric and tropospheric refraction errors, ephemeris errors, internal noise of the GPS receiver, and calculation errors.

[0159] In this regard, the reliability of AIS data can be evaluated using the following [Equation 4].

[0160]

[0161] Here, C_AIS is a result value representing the reliability of AIS data in the range of 0 to 1, Xs and Ys are the AIS data coordinates after interpolation, Xv and Yv are the coordinates of the vessel detected in the image, and σ_GPS is the GPS error range value (e.g., standard GPS receiver = 3m, RTK GPS using high-precision positioning = 1m).

[0162] In addition, causes of error in satellite or aerial images include geometric distortion of the satellite and aerial images, the viewing angle of the satellite, geometric correction errors of the sensor, hardware errors (sensor noise, optics error), distortion of the camera lens, CMOS sensor noise, the influence of atmospheric conditions and sea conditions, and the influence of clouds, haze, waves, etc.

[0163] In this regard, mathematical formulas 5 to 7 can be defined as follows to evaluate the reliability of the image.

[0164]

[0165] Here, σ_VID is the value for the total error of image reliability, σ_g is the geometric error value such as satellite shooting angle and Earth curvature, σ_hw is the hardware error value such as sensor noise and lens distortion, σ_motion is the error value due to ship speed, and σ_camera is the error value due to shooting environment.

[0166] Meanwhile, the above σ_motion follows the following mathematical formula 6.

[0167]

[0168] Here, Vs is the speed of the vessel (m / s) and Δt is the time error for capturing the image (sec).

[0169] In addition, σ_camera follows the following mathematical formula 7 as an error value based on the shooting angle (Oblique Angle) and exposure time.

[0170]

[0171] Here, H is the satellite's shooting altitude (m), ? is the satellite's shooting angle (degree), Vs is the ship's speed (m / s), and t_exp is the camera's exposure time (sec).

[0172] Based on the above mathematical formulas 5 to 7, the reliability of the image (C_VID) can be evaluated using the following mathematical formula 8.

[0173]

[0174] Here, C_VID is a result value representing image reliability in the range of 0 to 1, Xs and Ys are AIS data coordinates after interpolation, Xv and Yv are the coordinates of the vessel detected in the image, and C_VID is a value for the total error of image reliability.

[0175] And the above threshold value (a) can be defined by the following mathematical formula 9.

[0176]

[0177] Therefore, the above threshold value is a value that considers AIS data reliability and image reliability, and can be used as a criterion to determine whether the difference between the position of a vessel in the image and the corrected AIS data position is due to an error of the same vessel or different vessels.

[0178] If the above error (d) exceeds the above threshold value (a), the position (coordinates) of the corrected AIS data can be applied to the ship information in the image. (S640) This is because the corrected AIS data is more reliable than the ship information derived from the image.

[0179] Meanwhile, if the above error (d) is smaller than the above threshold value (a), it can be considered that the risk due to the error in the corrected AIS data is higher than the risk due to the error in the image. Therefore, it can be interpreted that the data of the vessel based on the image is more reliable.

[0180] Therefore, if the above error (d) is equal to or smaller than the threshold value (a), the position (coordinates) value of the vessel in the image can be applied as is. (S650)

[0181] In addition, if the error (d) is equal to or smaller than the threshold value (a), the data regarding the vessel in the image is more reliable, so the position difference (error) between the position of the target (vessel) in the image and the corrected AIS data can be determined as an error caused by the AIS of the same vessel.

[0182] According to this, through a reliability evaluation formula for AIS data and data of a vessel in the image, if the position coordinates of the corrected AIS data and the position coordinates of the vessel in the image are different, it is easy to determine whether it is due to an error of the same vessel or different vessels.

[0183] Meanwhile, according to the embodiment of the present invention described above, when the error (d) between the two ship positions exceeds a threshold value, a corrected AIS data value is applied, and at this time, static information of the ship among the AIS data can be matched.

[0184] Therefore, if the two vessels are different, the vessel whose corrected AIS data (especially static information) in the image is matched will be excluded from the search and target setting of the S40 step that proceeds simultaneously or subsequently, while the other vessel may still be searched in the image and classified as an unregistered vessel, or the attempt to match and align with another AIS data may be repeated.

[0185] FIG. 4 is a diagram showing a screen for searching for and setting a target (ship) in a region of interest (ROI) in a satellite image according to an embodiment of the present invention, and FIG. 5 is a diagram showing a screen in which a target (ship) in a satellite image and AIS data matched to the target (ship) are fused (or aligned) according to an embodiment of the present invention.

[0186] Figure 4 is an image of a user's region of interest (ROI) for a specific port, captured by a synthetic aperture radar. Referring to Figure 4, the vessels detected in the image (IPs, defined by blue squares) are defined, and a point (orange) that can activate information about each vessel is defined.

[0187] Meanwhile, in the aforementioned step S40, one of the vessels illustrated in FIG. 4 can be set as a target.

[0188] In addition, if the user selects the above point, information about the vessel (such as location coordinates) derived from the satellite imagery can be displayed.

[0189] Referring to Fig. 5, when one of the vessels discovered in Fig. 4 is set as a target, a screen (W1) in which the corrected AIS data for the set target (T1) is matched can be seen.

[0190] For example, the information of the above target (T1) vessel includes information such as time (point in time), IMO number, vessel number, location (latitude, longitude), altitude, etc. Explanation of the symbols

[0192] 10: Receiver 20: Processing unit 30: Output section 40: Storage section

Claims

Claim 1 A satellite or aerial image ship identification system comprising: a receiving unit for receiving satellite or aerial imagery and AIS (Automatic Identification System) data; a processing unit for preprocessing the AIS data and image received from the receiving unit, calculating AIS data for the time of shooting (ts) of the image, and matching the calculated corrected AIS data with a ship detected in the image to align information; and an output unit for outputting the information of the ship aligned by the processing unit as an image, wherein the processing unit utilizes a linear interpolation method or an interpolation method considering speed and direction of movement by user selection so that the received AIS data is corrected to the value of the time of shooting (t). Claim 2 A ship identification system for satellite or aerial imagery according to claim 1, characterized in that the linear interpolation method approximates the ship as moving in a straight line at a constant speed in the user's region of interest (ROI) to calculate AIS data at the time of image capture (ts). Claim 3 In Clause 2, the linear interpolation method is Equation 1: A ship identification system for satellite or aerial imagery that calculates AIS data at the image capture time (ts) by means of (t_s: image capture time, X_s: X-plane position at the image capture time, Y_s: Y-plane position at the image capture time, X_(AIS-1): X-plane position on AIS data immediately before the image capture time, Y_(AIS-1): Y-plane position on AIS data immediately before the image capture time, X_(AIS+1): X-plane position on AIS data immediately after the image capture time, Y_(AIS+1): Y-plane position on AIS data immediately after the image capture time, t_(AIS-1): time of reception of AIS data received immediately before the image capture time, t_(AIS+1): time of reception of AIS data received immediately after the image capture time (ts). Claim 4 A ship identification system for satellite or aerial imagery according to claim 1, wherein the interpolation method considering the speed and direction of movement calculates the AIS data at the time of image capture using only the AIS data received immediately before the time of image capture. Claim 5 In Clause 4, the interpolation method considering the speed and direction of movement is Equation 2: A ship identification system for satellite or aerial imagery that calculates AIS data at the image capture time (ts) by means of (t_s: image capture time, X_s: X-plane position at the image capture time, Y_s: Y-plane position at the image capture time, X_(AIS-1): X-plane position on AIS data immediately prior to the image capture time, Y_(AIS-1): Y-plane position on AIS data immediately prior to the image capture time, t_(AIS-1): time of reception of AIS data received immediately prior to the image capture time). Claim 6 A ship identification system for satellite or aerial imagery according to claim 1, wherein the processing unit comprises: a data processing unit that processes the received AIS data; and an image processing unit that processes the received image, wherein the image processing unit provides a matched image to an output unit by rotating and scaling the AIS data at the time of image capture calculated by the data processing unit. Claim 7 A ship identification system for satellite or aerial imagery according to claims 1 to 6, wherein the processing unit extracts an error (d) through Euclidean distance calculation when the position of AIS data corrected to the shooting time (ts) of the image for matching to the ship detected in the image is different from the position of the ship detected in the image. Claim 8 A ship identification system for satellite or aerial imagery according to claim 7, wherein the processing unit compares the extracted error (d) with a threshold value (a) based on an evaluation of the reliability of the AIS data and the reliability of the image, and if the extracted error (d) is greater than the threshold value (a), aligns it with the position value of the calculated corrected AIS data, and if the extracted error (d) is less than or equal to the threshold value (a), aligns the position value of the ship derived from the image with the image as is. Claim 9 In Clause 8, the reliability of the AIS data is, Formula: A ship identification system for satellite or aerial imagery evaluated by (C_AIS: result value representing the reliability of AIS data in the range of 0 to 1, Xs and Ys: X,Y coordinates of AIS data after interpolation, Xv and Yv: X,Y coordinates of ships detected in the image, σ_GPS is the GPS error range value). Claim 10 In Clause 9, the reliability of the above image is, Formula: A ship identification system for satellite or aerial imagery, evaluated by (C_VID: result value representing image reliability in the range of 0 to 1, Xs and Ys are interpolated AIS data X,Y coordinates, Xv and Yv: X,Y coordinates of the vessel detected in the image, ??_VID is the value for the total error of image reliability). Claim 11 A ship identification system for satellite or aerial imagery according to claim 10, wherein the value for the total error of image reliability is defined as the sum of geometric error values ​​for satellite shooting angle or Earth curvature, hardware error values ​​for sensor noise or lens distortion, error values ​​due to ship speed, and error values ​​due to shooting environment. Claim 12 In claim 11, the threshold value (a) is, formula: A ship identification system of satellite or aerial imagery defined by Claim 13 A method for identifying a ship in satellite or aerial imagery, comprising: receiving satellite or aerial imagery and AIS data; correcting the received AIS data to the AIS data at the time of image capture (ts); searching for a ship in the image and setting it as a target; and matching and aligning the corrected AIS data to the targeted ship, wherein the step of correcting to the AIS data at the time of image capture (ts) calculates the corrected AIS data by considering the speed and direction of movement at the location of the AIS data received immediately before the time of image capture (ts-1), and if a difference is identified between the location of the target in the image and the location of the corrected AIS data, extracts an error (d) between the two locations through Euclidean distance calculation, compares the error (d) with a threshold value (a) defined by a formula reflecting the reliability of the AIS data and the reliability of the image, and aligns the information value with the higher reliability to the data of the target in the image based on the result of the comparison between the error (d) and the threshold value (a).

Images