Clustering and anomaly analysis method for ship trajectory data

Abstract

The invention discloses a clustering and anomaly analysis method for ship trajectory data. Before clustering and anomaly analysis are carried out, trajectory data are classified in advance according to ship types, and as group characteristics of ship trajectories are often related to operation characteristics, draft and the like of the ship trajectories, subsequent clustering and anomaly analysis are established in pre-classified sample subsets, so that the mining analysis performance and the result robustness are greatly improved; as ship type noise often exists in the original trajectory data, the technology further uses a data-driven type sample pre-clustering means, thereby overcoming the problem of unbalanced type samples in the original trajectory data.

Description

Technical Field

[0001] The invention relates to a cluster analysis and anomaly analysis technology for massive ship trajectory data, belonging to the field of information technology. Background Art

[0002] Mining and analyzing massive spatiotemporal data of ships is a broad research topic, and cluster analysis and anomaly analysis are the basic issues. Cluster analysis aims to mine group characteristics from massive data, while anomaly analysis aims to find individual samples that are different from the majority of massive data. Based on the results of cluster analysis, we can further conduct secondary analysis such as target motion law extraction and interest point mining; the results of anomaly analysis are usually objects worthy of attention in massive trajectory data, which can be used for further in-depth research on target behavior, intentions, etc.

[0003] Both spatiotemporal data clustering technology and anomaly detection technology are fields with a deep research foundation. Among them, the most widely used clustering technology is the K-means clustering technology for spatiotemporal data and the spatial density noise clustering technology. However, when these two are directly applied to large-scale ship trajectory data, there are certain defects.

[0004] The ordinary K-means clustering technology needs to calculate the distance between the candidate center trajectory and all trajectories in each iteration process. It has poor performance in the scenario of massive trajectory data, and its clustering result is sensitive to the distance measurement method used for spatiotemporal data. The spatial density noise clustering technology needs to determine the intra-cluster distance accessibility threshold as a hyperparameter of the algorithm. The selection of this parameter requires prior knowledge of the actual field, because the data quality factors of the actual trajectory samples (far / near sea, operation type, sampling frequency, etc.) and the distance measurement of the spatiotemporal data used vary greatly.

[0005] In terms of spatiotemporal trajectory anomaly detection, directly using spatial noise-based clustering technology for noise trajectory detection not only has the same hyperparameter selection problem as in cluster analysis, but also has the problem of uneven distribution of actual trajectory data sample space, type, nationality, etc., which may lead to a large number of anomaly misjudgments. Other anomaly detection technologies often require additional a priori static anomaly discovery rules, and a priori domain knowledge is often not universal.

[0006] Based on the above problems, it is necessary to propose a new technical solution for clustering analysis and anomaly analysis technology of extensive ship trajectory data to improve the shortcomings of existing technologies. Summary of the invention

[0007] The purpose of the present invention is to provide an improved clustering analysis and anomaly analysis method for a wide range of ship trajectory data in view of the shortcomings of the prior art in mining clustering analysis and anomaly analysis for massive ship spatiotemporal data, such as poor performance, great influence of data quality, and misjudgment caused by uneven data distribution.

[0008] In order to achieve the above object, the technical solution of the present invention is to disclose a clustering and anomaly analysis method for ship trajectory data, which is characterized by comprising the following steps:

[0009] Step 1: Classify the ship trajectory data according to their ship types, and the results of cluster analysis and anomaly analysis of ship trajectories are limited to a certain ship type;

[0010] Step 2: Preprocess the classified sub-category trajectories, including stop point detection, trajectory segmentation, and median filter smoothing;

[0011] Step 3: Perform K-means time series clustering based on soft dynamic time warping on the segmented ship trajectory sets of each type. K-means time series clustering is an extension of K-means clustering technology on the distance measurement of time series data, which specifically includes the following steps:

[0012] Step 3.1: Randomly select K trajectories as candidate center trajectories of K clusters;

[0013] Step 3.2: Calculate the soft time-reduced distances from other trajectories in the cluster to the candidate cluster center, and take the cluster center trajectory with the closest distance as the clustering result of the trajectory;

[0014] Step 3.3: Calculate the distance matrix of each clustering result trajectory cluster, and obtain the trajectory with the smallest sum of distances within each cluster as the new candidate trajectory for the cluster center;

[0015] Step 3.4: Loop steps 3.2 and 3.3 until the cluster center trajectory converges or the maximum number of iterations is reached;

[0016] Step 4: In each trajectory cluster of the K-means clustering result, find the time series average centroid of the trajectory cluster to obtain the cluster analysis result of each ship type;

[0017] Step 5: In each result trajectory cluster of the K-means clustering result, perform spatial density noise clustering to obtain the result of anomaly analysis;

[0018] Step 6: Perform DBSCAN spatial density noise clustering in each cluster and take the isolated trajectories in the clustering results as the results of anomaly analysis.

[0019] Preferably, the step 1 comprises the following steps:

[0020] Step 1.1: Count the occurrence frequencies of all ship types and retain the 20 most frequently occurring types as classification targets;

[0021] Step 1.2: For the remaining ship types that do not appear frequently enough, classify them into the high-frequency ship types with the most similar type names according to the edit distance between their original type strings and the matching type strings with higher occurrence frequencies;

[0022] Step 1.3: Ship trajectories whose edit distance between the original type and the matching type string exceeds half of the string length are classified as ship type noise in the trajectory data and are not subject to subsequent analysis.

[0023] Preferably, step 2 comprises the following steps:

[0024] Step 2.1: Identify the segmentation points of the ship’s track points according to the ship’s ID number:

[0025] First, the trajectory points are sorted by time, and the time difference between two adjacent points is calculated; when the time difference between adjacent trajectory points is greater than the specified threshold, the first segmentation is performed, and the movement speed of each point in the trajectory segment after the preliminary segmentation is calculated, and the trajectory points with too low movement speed are further used as the second segmentation points;

[0026] Step 2.2: Perform median filtering and smoothing on all segmented sub-trajectory segments, wherein the median filter samples the median value of the coordinates within a fixed window range.

[0027] Preferably, in step 3.2, the soft time-reduced distance soft_dtw between two points i and j in any two trajectories is i,j

[0028] The calculation is done using the following formula:

[0029]

[0030] Where euclidean(i, j) is the Euclidean distance between points i and j, and γ is the smoothing coefficient.

[0031] Preferably, step 4 comprises the following steps:

[0032] Step 4.1: Randomly select a trajectory within the cluster as the candidate centroid;

[0033] Step 4.2: Calculate the soft time-reduced distance from other trajectories in the cluster to the candidate centroid, and obtain the best alignment position from other trajectories to the candidate centroid;

[0034] Step 4.3: Calculate the coordinate average of all temporal positions of the candidate centroid as the new trajectory cluster centroid candidate;

[0035] Step 4.4: Loop steps 4.2 and 4.3 until the coordinates of the candidate centroid no longer change or the maximum number of iterations is reached.

[0036] Preferably, step 5 comprises the following steps:

[0037] Step 5.1: Calculate the dynamic time warping distance matrix between trajectories in each result cluster;

[0038] Step 5.2: Calculate the mean μ and variance σ of the distance matrix respectively, and use μ+0.5*σ as the domain accessibility threshold parameter based on spatial density noise clustering.

[0039] Preferably, step 6 comprises the following steps:

[0040] Step 6.1: Select a random core point that has enough neighboring points within its neighborhood radius, and then evaluate each point in the neighborhood of the current core point to determine whether it has enough neighboring points within the neighborhood distance;

[0041] Step 6.2: If satisfied, the current core point becomes another core point and the cluster will expand; if not satisfied, the current core point becomes a boundary point;

[0042] Step 6.3: When the cluster is surrounded by boundary points, the cluster has been searched completely, a new random point is selected, and steps 6.1 to 6.2 are repeated to identify the next cluster.

[0043] Compared with the prior art solutions, the present invention has the following beneficial effects:

[0044] 1. Before clustering and anomaly analysis, the trajectory data is pre-classified by ship type. Since the group characteristics of ship trajectories are often related to their operating characteristics, draft depth, etc., the subsequent clustering and anomaly analysis are based on the pre-classified sample subset, which greatly improves the performance of mining analysis and the robustness of the results. Since there is often ship type noise in the original trajectory data, this technology further uses data-driven type sample pre-clustering means to overcome the problem of unbalanced type samples in the original trajectory data.

[0045] 2. The trajectory clusters formed by K-means clustering are further extracted based on dynamic time warping as the result of cluster analysis. It is smoother and more robust than the traditional method of directly using cluster center trajectories to represent clusters, which is especially obvious when the time series of cluster center trajectories is long.

[0046] 3. Anomaly detection Using the anomaly detection technology based on spatial density noise on the result cluster based on K-means clustering can greatly reduce the memory consumption caused by spatial density noise clustering on the one hand, and on the other hand, it is beneficial for the algorithm to dynamically select its domain accessibility threshold. The present invention considers the mean and variance of the dynamic time warping matrix of the trajectory in each cluster to select the threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart;

[0048] Figure 2 A sample of the original trajectory is illustrated;

[0049] Figure 3 The smoothed trajectory samples are shown;

[0050] Figure 4 The effect diagram of K-means clustering (manually selected K=5) for the "Bulk Carrier" ship type;

[0051] Figure 5 This is the cluster analysis effect diagram for the "Bulk Carrier" ship type;

[0052] Figure 6 Illustration of the original K-means clustering trajectory clusters to be analyzed for anomalies

[0053] Figure 7 This is the anomaly detection effect diagram for this cluster. DETAILED DESCRIPTION

[0054] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0055] The present invention provides a cluster analysis and anomaly analysis technology for massive ship trajectory data, such as Figure 1The method includes: classifying the original marine ship trajectory data according to the ship type; pre-processing the motion trajectories of various types of ships by deduplication, smoothing, segmentation, filtering, etc.; pre-clustering the processed ship trajectories of each sub-category based on K-means of soft dynamic time warping; in each trajectory cluster after pre-clustering, calculating the time series centroid based on dynamic time warping as the trajectory clustering analysis result of this type of ship; then calculating the dynamic time warping values ​​of the trajectories in the same cluster two by two, and calculating the mean and variance of the dynamic time warping matrix respectively; using the mean and variance of the dynamic time warping distance matrix of each trajectory cluster as the trajectory domain distance threshold, further performing spatial density noise clustering in the trajectory cluster, and finding the noise trajectory in the cluster as the result of the abnormal analysis of this type of ship.

[0056] This example combines the ship trajectory data samples from the southeastern coastal waters of my country from April 2021 to May 2021 (the original trajectory samples are shown in Figure 2 ), introduces the implementation process of ship trajectory cluster analysis and anomaly analysis (the invention process is shown in Figure 1 ):

[0057] Step 1: Classify the ship trajectory data according to their ship types. The results of cluster analysis and anomaly analysis of ship trajectories are limited to a certain ship type. The classification process of ship types is as follows:

[0058] Step 1.1: Count the occurrence frequencies of all ship types. Keep the 20 types with the highest occurrence frequencies as classification targets. In this embodiment, the types with the highest occurrence frequencies are: "Bulk Carrier", "Container Ship", "Chemical / Oil Products", "General Cargo Ship", etc.

[0059] Step 1.2: For the remaining ship types that do not appear frequently enough, they are classified into the high-frequency ship types with the most similar type names according to the edit distance between their original type strings and the matching type strings with higher occurrence frequencies. In this embodiment, the "Carrier" type is absorbed by the "Bulk Carrier" type, and the "Cargo Ship" is absorbed by the "General Cargo Ship", etc.

[0060] Step 1.3: Ship trajectories whose edit distance between the original type and the matching type string exceeds half of the string length are classified as ship type noise in the trajectory data and are not subject to subsequent analysis.

[0061] Step 2: Perform preprocessing such as stop point detection, trajectory segmentation, median filter smoothing, etc. on the classified sub-category trajectories.

[0062] Step 2.1: First, identify the segmentation points of the track points according to the ship ID number. First, sort the track points by time and calculate the time difference between two adjacent points. The first segmentation is performed when the time difference between adjacent track points is greater than the specified threshold. The movement speed of each point in the track segment after the preliminary segmentation is calculated, and the track points with too low movement speed are further used as the second segmentation points.

[0063] Step 2.2: Perform median filtering and smoothing on all segmented sub-trajectory segments. The median filter samples the median of the coordinates within a fixed window range, and the calculation process is as follows (the window length is 2N):

[0064] x i =median(x i-N , x i-N+1 ..., x i+N-1 , x i+N )

[0065] y i = median(y i-N ,y i-N+i ..., y i+N-1 ,y i+N )

[0066] In this example, “Bulk Carrier” is selected as the ship type to be analyzed, and the smoothed trajectory samples are shown in Figure 3 .

[0067] Step 3: Perform K-means time series clustering based on soft dynamic time warping on the segmented ship trajectory sets of each type. K-means time series clustering is an extension of K-means clustering technology on the time series data distance measurement (soft dynamic time warping). The specific steps are as follows:

[0068] Step 3.1: Randomly select K trajectories as candidate center trajectories of K clusters.

[0069] Step 3.2: Calculate the soft time-reduced distance from other trajectories in the cluster to the candidate cluster center, and take the cluster center trajectory with the closest distance as the clustering result of the trajectory.

[0070] Step 3.3: Calculate the distance matrix of each clustering result trajectory cluster, and obtain the trajectory with the smallest intra-cluster distance sum as the new cluster center candidate trajectory.

[0071] Step 3.4: Repeat steps 3.2-3.3 until the cluster center trajectory converges or the maximum number of iterations is reached.

[0072] Soft dynamic time warping is a differentiable measurement method for time series data distance. The calculation method is as follows:

[0073]

[0074] Where i and j are two points in the two trajectories, euclidean(i, j) is the Euclidean distance between points i and j, and γ is the smoothing coefficient.

[0075] The effect of K-means clustering (manually selecting K=5) for the "Bulk Carrier" ship type is shown in Figure 4 , each color represents a cluster.

[0076] Step 4: In each trajectory cluster of the K-means clustering result, find the time series average centroid of the trajectory cluster to obtain the cluster analysis result of each ship type. The specific steps are as follows:

[0077] Step 4.1: Randomly select a trajectory within the cluster as the candidate centroid.

[0078] Step 4.2: Calculate the soft time-reduced distance from other trajectories in the cluster to the candidate centroid, and obtain the best alignment position from other trajectories to the candidate centroid.

[0079] Step 4.3: Calculate the coordinate average of all temporal positions of the candidate centroid as the new trajectory cluster centroid candidate.

[0080] Step 4.4: Loop steps 4.2-4.3 until the coordinates of the candidate centroid no longer change or the maximum number of iterations is reached.

[0081] The cluster analysis results for the “Bulk Carrier” ship type are shown in Figure 5 , each line represents the spatiotemporal centroid of a particular cluster.

[0082] Step 5: In each trajectory cluster of the K-means clustering result, perform DBSCAN to obtain the results of anomaly analysis. The specific steps are as follows:

[0083] Step 5.1: Calculate the dynamic time warping distance matrix between trajectories in each result cluster.

[0084] Step 5.2: Calculate the mean μ and variance σ of the distance matrix respectively, and use μ+0.5*σ as the domain accessibility threshold (∈) parameter of DBSCAN.

[0085] The original K-means clustering trajectory clusters selected for anomaly analysis in this example are shown in Figure 6 .

[0086] Step 6: Perform DBSCAN spatial density noise clustering in the cluster and use the isolated tracks in the clustering results as the results of anomaly analysis. The specific steps are as follows:

[0087] Step 6.1: Select a random core point that has enough neighbors within its neighborhood radius. Then evaluate each point in the neighborhood of the core point to determine whether it has enough neighbors within the neighborhood distance.

[0088] Step 6.2: If satisfied, it will become another core point and the cluster will expand.

[0089] Step 6.3: If not satisfied, it becomes a boundary point.

[0090] Step 6.4: When a cluster is surrounded by boundary points, the cluster has been searched completely because there are no more points within the distance. Select a new random point and repeat steps 6.1 to 6.3 to identify the next cluster.

[0091] The anomaly detection effect on this cluster is shown in Figure 7 , red is the abnormal trajectory, and yellow is the non-abnormal trajectory.

Claims

1. A clustering and anomaly analysis method for ship trajectory data, characterized in that: The following steps are involved: Step 1: Classify the ship trajectory data according to their ship types, and the results of cluster analysis and anomaly analysis of ship trajectories are limited to a certain ship type; Step 2: Preprocess the classified sub-category trajectories, including stop point detection, trajectory segmentation, and median filter smoothing; Step 3: Perform K-means time series clustering based on soft dynamic time warping on the segmented ship trajectory sets of each type. K-means time series clustering is an extension of K-means clustering technology on the distance measurement of time series data, which specifically includes the following steps: Step 3.1: Randomly select K trajectories as candidate center trajectories of K clusters; Step 3.2: Calculate the soft time-reduced distances from other trajectories in the cluster to the candidate cluster center, and take the cluster center trajectory with the closest distance as the clustering result of the trajectory; Step 3.3: Calculate the distance matrix of each clustering result trajectory cluster, and obtain the trajectory with the smallest sum of distances within each cluster as the new candidate trajectory for the cluster center; Step 3.4: Loop steps 3.2 and 3.3 until the cluster center trajectory converges or the maximum number of iterations is reached; Step 4: In each trajectory cluster of the K-means clustering result, find the time series average centroid of the trajectory cluster to obtain the cluster analysis result of each ship type; Step 5: In each result trajectory cluster of the K-means clustering result, perform spatial density noise clustering to obtain the result of anomaly analysis; Step 6: Perform DBSCAN spatial density noise clustering in each cluster and take the isolated trajectories in the clustering results as the results of anomaly analysis.

2. A clustering and anomaly analysis method for ship trajectory data according to claim 1, characterized in that: The step 1 comprises the following steps: Step 1.1: Count the occurrence frequencies of all ship types and retain the 20 most frequently occurring types as classification targets; Step 1.2: For the remaining ship types that do not appear frequently enough, classify them into the high-frequency ship types with the most similar type names according to the edit distance between their original type strings and the matching type strings with higher occurrence frequencies; Step 1.3: Ship trajectories whose edit distance between the original type and the matching type string exceeds half of the string length are classified as ship type noise in the trajectory data and are not subject to subsequent analysis.

3. A clustering and anomaly analysis method for ship trajectory data according to claim 1, characterized in that: The step 2 comprises the following steps: Step 2.1: Identify the segmentation points of the ship’s track points according to the ship’s ID number: First, the trajectory points are sorted by time, and the time difference between two adjacent points is calculated; when the time difference between adjacent trajectory points is greater than the specified threshold, the first segmentation is performed, and the movement speed of each point in the trajectory segment after the preliminary segmentation is calculated, and the trajectory points with too low movement speed are further used as the second segmentation points; Step 2.2: Perform median filtering and smoothing on all segmented sub-trajectory segments, wherein the median filter samples the median value of the coordinates within a fixed window range.

4. The clustering and anomaly analysis method for ship trajectory data according to claim 1, characterized in that: In step 3.2, the soft time-reduced distance soft_dtw between two points i and j in any two trajectories i,j The calculation is done using the following formula: Where euclidean(i, j) is the Euclidean distance between points i and j, and γ is the smoothing coefficient.

5. The clustering and anomaly analysis method for ship trajectory data according to claim 1, characterized in that: The step 4 comprises the following steps: Step 4.1: Randomly select a trajectory within the cluster as the candidate centroid; Step 4.2: Calculate the soft time-reduced distance from other trajectories in the cluster to the candidate centroid, and obtain the best alignment position from other trajectories to the candidate centroid; Step 4.3: Calculate the coordinate average of all temporal positions of the candidate centroid as the new trajectory cluster centroid candidate; Step 4.4: Loop steps 4.2 and 4.3 until the coordinates of the candidate centroid no longer change or the maximum number of iterations is reached.

6. The clustering and anomaly analysis method for ship trajectory data according to claim 1, characterized in that: The step 5 comprises the following steps: Step 5.1: Calculate the dynamic time warping distance matrix between trajectories in each result cluster; Step 5.2: Calculate the mean μ and variance σ of the distance matrix respectively, and use μ+0.5*σ as the domain accessibility threshold parameter based on spatial density noise clustering.

7. The method for clustering and anomaly analysis of ship trajectory data according to claim 1, characterized in that: The step 6 comprises the following steps: Step 6.1: Select a random core point that has enough neighboring points within its neighborhood radius, and then evaluate each point in the neighborhood of the current core point to determine whether it has enough neighboring points within the neighborhood distance; Step 6.2: If satisfied, the current core point becomes another core point and the cluster will expand; if not satisfied, the current core point becomes a boundary point; Step 6.3: When the cluster is surrounded by boundary points, the cluster has been searched completely, a new random point is selected, and steps 6.1 to 6.2 are repeated to identify the next cluster.

Images