AIS data cleaning method based on cubic spline interpolation method

Abstract

The invention discloses an AIS data cleaning method based on a cubic spline interpolation method, and the method comprises the steps: collecting AIS original data, cleaning and removing noise, and carrying out the standardization processing of the data; abnormal data points are detected through a set rule, and all the abnormal data points are marked for restoration; normal data points are calculated through a cubic spline interpolation method, a smooth curve is generated, and in the difference calculation process, the time sequence characteristic of AIS data is considered, and it is ensured that the interpolation result is consistent with the time change trend of the data; replacing an abnormal data point by using a difference result, checking the continuity and accuracy of the data, and ensuring that no new exception is introduced; and storing the cleaned AIS output or returning the cleaned AIS output to other systems for further processing through an API (Application Program Interface). According to the method, abnormal points are smoothed through the cubic spline interpolation technology, error data are corrected, and therefore the accuracy and reliability of the data are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data cleaning, and more specifically to an AIS data cleaning method based on a cubic spline interpolation method. Background Art

[0002] Cubic Spline Interpolation is an interpolation method widely used in numerical analysis and computing, especially for smoothing the transition between data points. The basic idea is to construct a function consisting of multiple cubic polynomials between data points to ensure that the function is continuous at the data points and that the first and second order derivatives are continuous. Cubic spline interpolation is often used in signal processing, data cleaning, image processing and other fields. It can smooth irregular data and avoid overfitting problems.

[0003] For AIS data cleaning, cubic spline interpolation has been used for smoothing and interpolation, but it is mostly applied to conventional signal processing or physical measurement data. For AIS data, the existing application of cubic spline interpolation is not yet common, and often fails to fully consider the particularity of AIS data (such as ship motion trajectory, timestamp dependency, long time series, etc.).

[0004] Moreover, there are several problems when the existing technical methods are applied to AIS data: 1) In traditional interpolation methods, the time series characteristics of the data are often ignored. In AIS data, the change of each data point is not only related to the spatial position, but also closely related to the time change. For example, the speed and direction of the ship will change over time. Ignoring this will lead to deviations in the interpolation results; 2) Traditional interpolation methods (such as linear interpolation and weighted moving average) usually rely on simple neighborhood data, and for some serious outliers, they often cannot be effectively repaired. Although cubic spline interpolation is smooth, it may produce unsatisfactory interpolation effects when dealing with drastic abnormal changes. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an AIS data cleaning method based on cubic spline interpolation, which can eliminate abnormal points, ensure the accuracy and continuity of data, and improve the reliability of subsequent analysis and decision-making.

[0006] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows.

[0007] An AIS data cleaning method based on cubic spline interpolation method includes the following steps:

[0008] S1. Collect AIS raw data, clean and remove noise, and standardize the data;

[0009] S2. Detect abnormal data points by setting rules and mark all abnormal data points for repair;

[0010] S3. The normal data points are calculated using cubic spline interpolation to generate a smooth curve. In the difference calculation process, the time series characteristics of AIS data are considered to ensure that the interpolation result is consistent with the time change trend of the data;

[0011] S4. Use the difference results to replace abnormal data points, check the continuity and accuracy of the data, and ensure that no new anomalies are introduced;

[0012] S5. Store the cleaned AIS output or return it to other systems through the API interface for further processing.

[0013] To further optimize the technical solution, in step S1, the original data includes the reporting time, longitude and latitude, and length / width / course of the ship.

[0014] To further optimize the technical solution, the method for standardizing the original data is as follows: the method for processing the reporting time of the original data is as follows: aggregating the data according to the nine-digit code of the fishing vessel, and deleting the data with the same reporting time; the method for processing the latitude and longitude of the original data is as follows: deleting empty row data and retaining 6 decimal places; the method for processing the length / width / course of the original data is as follows: retaining 2 decimal places.

[0015] To further optimize the technical solution, in step S2, the abnormal data points include abnormal: abnormal stop points, abnormal acceleration points and abnormal drift points.

[0016] To further optimize the technical solution, the method for detecting the abnormal stop point is as follows: for the AIS sequence, if the speed of the i-th point is greater than 2 knots, and the coordinates, speed v, and heading C data of the i+1-th point are the same as those of the i-th point, then the i+1-th point is the abnormal stop point.

[0017] To further optimize the technical solution, the abnormal acceleration point detection method is as follows: the acceleration of a ship accelerating to its design speed at a distance of 10 times the length of the ship can be used as its theoretical maximum acceleration; the acceleration of decelerating from its design speed to a stop at a distance of 8 times the length of the ship can be used as its theoretical minimum acceleration; assuming that the length of the ship is L and the design speed is V d , the maximum acceleration is a max , the minimum acceleration is a min , the acceleration time is t 1 The deceleration time is t 2 , and if the ship is moving with uniform acceleration or uniform deceleration, then:

[0018] V d =a max ×t 1 =-amin ×t 2

[0019]

[0020] From the above relationship, the maximum acceleration a of the ship can be obtained: max and minimum acceleration a min for:

[0021]

[0022] For the AIS sequence, the acceleration between the i-th point and the i+1-th point can be calculated based on the speed and time difference between the two points. If the calculated acceleration is greater than the maximum acceleration or less than the minimum acceleration, the i+1-th point is determined to be an abnormal acceleration point.

[0023] To further optimize the technical solution, the abnormal drift point detection method is as follows: the ship starts from point i and first performs maximum acceleration until it reaches a certain time t m After reaching the maximum, it starts to decelerate, so that the speed at point i+1 is exactly v i+1 , at this time, the ship's travel distance is s max , which can be expressed as follows:

[0024]

[0025] Due to the influence of wind and waves, the actual journey of the ship between point i and point i+1 is generally not a strict straight line. Therefore, the straight-line distance d(i,i+1) between track point i and track point i+1 must be less than s max , the straight-line distance d(i, i+1) is greater than s max This is the abnormal drift point.

[0026] To further optimize the technical solution, in step S3, the cubic spline interpolation mathematical algorithm ensures the continuity of the first-order and second-order derivatives at the data points, performs smoothing on the scattered data, and uses CubicSpline in the scipy library to perform cubic spline interpolation on the time and longitude and latitude data.

[0027] To further optimize the technical solution, in step S4, the abnormal data are missing reporting time, repeated reporting time, empty longitude data / empty latitude data, longitude and latitude values ​​exceeding the geographical range, and abnormal point data. By deleting the abnormal data, interpolation processing is performed on the data with reporting time of two adjacent groups of data less than 10 minutes.

[0028] Due to the adoption of the above technical scheme, the technical progress achieved by the present invention is as follows.

[0029] The present invention provides an AIS data cleaning method based on cubic spline interpolation, which smoothes abnormal points and corrects erroneous data through cubic spline interpolation technology, thereby improving the accuracy and reliability of data. This technical solution not only focuses on the implementation of mathematical models, but also takes into account the particularity of AIS data and optimizes the traditional cubic spline interpolation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart of the present invention;

[0031] Figure 2 This is the cleaning data comparison diagram example 1 of the present invention;

[0032] Figure 3 This is Example 2 of the cleaning data comparison diagram of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] A method for cleaning AIS data based on cubic spline interpolation, combined with Figure 1 As shown, it includes the following steps:

[0035] S1. Data Preprocessing

[0036] The AIS raw data is collected and cleaned to remove noise, and the data is standardized. At this time, the timestamp and vessel coordinates will be converted into a unified format for subsequent processing.

[0037] The original data includes reporting time, longitude and latitude, and length / width / course of the ship.

[0038] The method for processing the reporting time is to aggregate the data according to the nine-digit code of the fishing vessel, delete the data with the same reporting time, and format the time as %Y / %m / %d%H:%M:%S. The format example is: 2024-11-0108:00:32.

[0039] The processing method for longitude and latitude is: delete the empty row data and retain 6 decimal places;

[0040] The processing method for ship length / ship width / course is: keep 2 decimal places.

[0041] S2. Outlier Detection

[0042] By setting rules to detect abnormal data points and marking all abnormal data points for repair, statistical methods (such as standard deviation) or machine learning methods can be used to determine outliers to prepare for subsequent difference repair.

[0043] Abnormal points include abnormal stopping points, abnormal acceleration points and abnormal drift points.

[0044] The detection method of abnormal stop point is as follows: for the AIS sequence, if the speed of the i-th point is greater than 2 knots (1 knot = 1 nautical mile / hour), and the coordinates (lon, lat), speed v, and heading C data of the i+1-th point are the same as those of the i-th point, then the i+1-th point is an abnormal stop point.

[0045] The detection method of abnormal acceleration point is as follows: the acceleration of a ship accelerating to its design speed at a distance of 10 times the length of the ship can be used as its theoretical maximum acceleration; the acceleration of decelerating from its design speed to a stop at a distance of 8 times the length of the ship can be used as its theoretical minimum acceleration. Let the length of the ship be L and the design speed be V d , the maximum acceleration is a max , the minimum acceleration is a min , the acceleration time is t 1 The deceleration time is t 2 , and if the ship is moving with uniform acceleration or uniform deceleration, then:

[0046] V d =a max ×t 1 =-a min ×t 2

[0047]

[0048] From the above relationship, the maximum acceleration a of the ship can be obtained: max and minimum acceleration a min for:

[0049]

[0050] For the AIS sequence, the acceleration between the i-th point and the i+1-th point can be calculated based on the speed and time difference between the two points. If the calculated acceleration is greater than the maximum acceleration or less than the minimum acceleration, the i+1-th point is determined to be an abnormal acceleration point.

[0051] The detection method of abnormal drift point is as follows: the ship starts from point i and first performs maximum acceleration until it reaches a certain time t m After reaching the maximum, it starts to decelerate, so that the speed at point i+1 is exactly v i+1 , at this time, the ship's travel distance is s max , which can be expressed as follows:

[0052]

[0053] Due to the influence of wind and waves, the actual journey of the ship between point i and point i+1 is generally not a strict straight line. Therefore, the straight-line distance d(i,i+1) between track point i and track point i+1 must be less than smax , the straight-line distance d(i, i+1) is greater than s max This is the abnormal drift point.

[0054] S3. Cubic spline difference calculation

[0055] The normal data points are calculated using cubic spline interpolation to generate a smooth curve, ensuring that the first-order and second-order derivatives of the curve are continuous at each data point. When performing interpolation calculations, the time series characteristics of AIS data are considered to ensure that the interpolation results are consistent with the time variation trend of the data.

[0056] The cubic spline interpolation mathematical algorithm ensures the continuity of the first-order and second-order derivatives at the data points, smoothes the scattered data, and uses CubicSpline in the scipy library to perform cubic spline interpolation on the time and longitude and latitude data.

[0057] S4. Data repair and reconstruction

[0058] The difference results are used to replace abnormal data points and reconstruct the entire data set. After the repair, the continuity and accuracy of the data need to be rechecked to ensure that no new anomalies are introduced.

[0059] Abnormal data include missing reporting time, repeated reporting time, empty longitude data / empty latitude data, longitude and latitude values ​​exceeding the geographical range, and abnormal point data. By deleting the abnormal data, interpolation processing is performed on the data with reporting time of two adjacent groups of data less than 10 minutes.

[0060] S5. Output cleaned AIS data

[0061] The cleaned AIS output is stored or returned to other systems through the API interface for further processing. Figure 2 and Figure 3 shown.

[0062] When the present invention adopts the cubic spline interpolation method, the continuity of the timestamp and position data is optimized to ensure that the interpolation calculation can adapt to the changing law of the data in the time dimension, so as to maintain the natural transition and accuracy of the repair result. The cubic spline interpolation method ensures a smooth transition between data points. It uses a cubic polynomial for interpolation in each interval to ensure that the interpolation curve is not only continuous at the data point, but also continuous in the derivative and second-order derivative, avoiding the transition mutation that may occur in the traditional method.

[0063] Through precise interpolation calculation and outlier repair strategy, the present invention can generate smooth trajectory data, ensuring that the repaired data not only conforms to actual physical laws, but also maintains the temporal and spatial continuity of the original data to the maximum extent, thereby improving the overall reliability of the data.

Claims

1. An AIS data cleaning method based on cubic spline interpolation method, characterized in that: It includes the following steps: S1. Collect AIS raw data, clean and remove noise, and standardize the data; S2. Detect abnormal data points by setting rules and mark all abnormal data points for repair; S3. The normal data points are calculated using cubic spline interpolation to generate a smooth curve. In the difference calculation process, the time series characteristics of AIS data are considered to ensure that the interpolation result is consistent with the time change trend of the data; S4. Use the difference results to replace abnormal data points, check the continuity and accuracy of the data, and ensure that no new anomalies are introduced; S5. Store the cleaned AIS output or return it to other systems through the API interface for further processing.

2. The AIS data cleaning method based on cubic spline interpolation method according to claim 1, characterized in that: In step S1, the original data includes the reporting time, longitude and latitude, and length / width / course of the ship.

3. The AIS data cleaning method based on cubic spline interpolation according to claim 2, characterized in that: The method for standardizing the original data is as follows: the method for processing the reporting time of the original data is to aggregate the data according to the nine-digit code of the fishing vessel, and delete the data with the same reporting time; the method for processing the latitude and longitude of the original data is to delete the empty row data and retain 6 decimal places; the method for processing the length / width / course of the original data is to retain 2 decimal places.

4. The AIS data cleaning method based on cubic spline interpolation according to claim 1, characterized in that: In step S2, the abnormal data points include abnormal: abnormal stop points, abnormal acceleration points and abnormal drift points.

5. The AIS data cleaning method based on cubic spline interpolation according to claim 4 is characterized in that: The detection method of the abnormal stop point is: for the AIS sequence, if the speed of the i-th point is greater than 2 knots, and the coordinates, speed v, and heading C data of the i+1-th point are the same as those of the i-th point, then the i+1-th point is the abnormal stop point.

6. The AIS data cleaning method based on cubic spline interpolation according to claim 4 is characterized in that: The abnormal acceleration point detection method is as follows: the acceleration of a ship at a distance of 10 times the length of the ship to its design speed can be used as its theoretical maximum acceleration; The acceleration of decelerating from its design speed to a stop at a distance of 8 times the length of the ship can be taken as its theoretical minimum acceleration; set up The length of the ship is L and the design speed is V d , the maximum acceleration is a max , the minimum acceleration is a min , the acceleration time is t1, the deceleration time is t2, and if the ship is moving in a uniformly accelerated or decelerated motion, then: In d =a max ×t1=-a min ×t2 From the above relationship, the maximum acceleration a of the ship can be obtained: max and minimum acceleration a min for: For the AIS sequence, the acceleration between the i-th point and the i+1-th point can be calculated based on the speed and time difference between the two points. If the calculated acceleration is greater than the maximum acceleration or less than the minimum acceleration, the i+1-th point is determined to be an abnormal acceleration point.

7. The AIS data cleaning method based on cubic spline interpolation according to claim 4 is characterized in that: The abnormal drift point detection method is as follows: the ship starts from point i and first performs maximum acceleration until it reaches a certain time t m After reaching the maximum, it starts to decelerate, so that the speed at point i+1 is exactly v i+1 , at this time, the ship's travel distance is s max , which can be expressed as follows: Due to the influence of wind and waves, the actual journey of the ship between point i and point i+1 is generally not a strict straight line. Therefore, the straight-line distance d(i,i+1) between track point i and track point i+1 must be less than s max , the straight-line distance d(i, i+1) is greater than s max This is the abnormal drift point.

8. The AIS data cleaning method based on cubic spline interpolation according to claim 1, characterized in that: In step S3, the cubic spline interpolation mathematical algorithm ensures the continuity of the first-order and second-order derivatives at the data points, performs smoothing on the scattered data, and uses CubicSpline in the scipy library to perform cubic spline interpolation on the time and longitude and latitude data.

9. The AIS data cleaning method based on cubic spline interpolation according to claim 1, characterized in that: In step S4, abnormal data includes missing reporting time, repeated reporting time, empty longitude data / empty latitude data, longitude and latitude values ​​exceeding the geographical range, and abnormal point data. By deleting the abnormal data, interpolation processing is performed on the data with reporting time of two adjacent groups of data less than 10 minutes.

Images