Ship AIS data feature point online identification method based on change point detection

Abstract

The application discloses a ship AIS data feature point online identification algorithm based on a variable point detection, and comprises the following steps: S1, AIS feature processing: extracting AIS data features from AIS data; the AIS data features include a time stamp, a longitude, a latitude, a speed and a destination; S2, variable point identification: detecting the AIS data features by using a stable statistical method or an online Bayesian variable point detection method, so as to identify the AIS data features with mutations; S3, feature point identification: calculating the difference between each data point in the AIS data features with mutations and the target value by using a CUSUM algorithm, accumulating the difference value, and then judging whether the deviation degree of the AIS data sequence reaches a preset threshold value. Advantages are as follows: the application is helpful to reduce the data size, is easy to manage for storage and transmission, improves the storage reading efficiency, and thus saves the cost. The application highlights and saves the key information in the AIS data, reduces the redundancy and noise, and improves the quality and reliability of data analysis.

Description

Technical Field

[0001] This invention relates to the field of ship automatic identification technology, and in particular to an online identification algorithm for ship AIS data feature points based on change point detection. Background Technology

[0002] The Automatic Identification System (AIS) is a technology used for radio communication between ships and between ships and land. The main purpose of AIS is to provide real-time information on a ship's position, identification, and navigation status to enhance maritime traffic safety and efficiency. Due to its standardized, real-time, transparent, and reliable characteristics, it is widely used in maritime traffic management, ship positioning, navigation safety, and port management.

[0003] The periodic transmission of AIS messages by numerous ships generates a massive amount of real-time AIS data stream, especially in busy waterways such as ports and canals. Processing this real-time data stream requires sufficient computing and storage resources; delays can lead to data accumulation and latency, posing a threat to the safe operation of ships. Therefore, an online feature point recognition algorithm that compresses the amount of ship AIS data while preserving the ship's operational characteristics is essential.

[0004] In actual production, due to issues such as AIS jump points, signal loss, MMSI serial numbers, equipment malfunctions, and manual data entry errors, combined with changes in the ship's position, speed, heading, and destination during actual operation, the trend of the actual AIS trajectory can "change." This change generally manifests as two different operating states before and after a certain point of change. An AIS point of change refers to a point where the parameters of an AIS data item undergo a sudden change.

[0005] The characteristic points of a ship are location information in its AIS data that represents the dynamic and static characteristics of its route operation, such as changes in speed, direction, destination, and estimated time of arrival. For example... Figure 1 As shown, a feature point is a set of points where the AIS information of a ship changes abruptly during its journey from one port to another. The set of feature points is a subset of the ship's AIS data.

[0006] The AIS system continuously generates streaming data, transmitting ship-related information, which flows into the system in real time. High-efficiency processing and real-time performance are required during AIS transmission. Due to the uncontrollable input time and data item quality of streaming data, the identification of outliers and velocity feature points requires an online, robust outlier detection capability. Therefore, this invention proposes an online feature point identification algorithm for ship AIS data based on change point detection. Summary of the Invention

[0007] The purpose of this invention is to provide an online identification algorithm for feature points in ship AIS data based on change point detection, thereby solving the aforementioned problems in the prior art.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] An online feature point recognition algorithm for ship AIS data based on change point detection includes the following steps:

[0010] S1, AIS Feature Processing:

[0011] Extract AIS data features from AIS data; the AIS data features include timestamp, longitude, latitude, air speed, and destination;

[0012] S2, Variable Point Recognition:

[0013] The stable statistical method or the online Bayesian change point detection method is used to detect the features of AIS data, thereby identifying the AIS data features that have undergone abrupt changes;

[0014] S3, Feature Point Recognition:

[0015] The CUSUM algorithm is used to calculate the difference between each data point in the AIS data feature that has undergone mutation and its target value. The difference values ​​are accumulated to determine whether the deviation of the AIS data sequence has reached a preset threshold.

[0016] Preferably, AIS data includes static data and dynamic data. The destination is selected from the static data, and the timestamp, longitude, latitude, and speed are selected from the dynamic data as the five main features of AIS. The timestamp, longitude, latitude, and speed are integers or floating-point numbers, and the destination is a text description that needs to be vectorized for description.

[0017] Preferably, the destination of the text representation is vectorized using the cosine frequency of the term or the TF-IDF algorithm.

[0018] Preferably, to determine the time when a vessel changes its destination, the time when the vessel updates its destination information but does not change its destination is ignored. The determination is made in the following way:

[0019] Calculate the cosine similarity or TF-IDF of the word frequencies between the two points. If the word similarity between the two points is greater than 0.5, it is considered that the destination has been changed.

[0020] Establish a correspondence between the English name of a port and its five-letter code to prevent ships from being mistakenly identified as changing their destination when updating their destination name to the five-letter code.

[0021] Preferably, the stable statistical method identifies and removes outliers through outlier detection.

[0022] Preferably, online Bayesian variable detection specifically includes the following:

[0023] Initialize prior distribution: Given an observed data sequence, initialize the prior distribution, which represents the distribution of the points of change before any points of change are observed;

[0024] Process new observation data sequentially: For each new observation data, calculate the posterior distribution based on the current prior distribution and likelihood function;

[0025] Update model parameters: Based on the newly observed data, update the model parameters, that is, update the estimated values ​​of the location and properties of the change points;

[0026] Calculate the probability of a change point: Based on the current posterior distribution, calculate the probability of a change point to determine whether a change point has been detected;

[0027] Update the prior distribution: Use the current posterior distribution as the prior distribution for the next time step.

[0028] Preferably, step S3 specifically includes the following:

[0029] S31. Set target values ​​and thresholds;

[0030] S32. For each data point, calculate its difference from the target value and add the difference value to the cumulative sum.

[0031] S33. If the cumulative sum exceeds the set threshold, it is determined that the AIS data sequence has changed or is abnormal.

[0032] The beneficial effects of this invention are as follows: 1. In terms of AIS feature selection, statistical analysis of AIS data is performed to select representative features, and natural language processing methods are used to construct word vectors for extracting ship destinations. In terms of change point identification, both offline and online algorithms are used to analyze the changing trends of AIS data, accurately identifying change points in the data, i.e., two data points belonging to two different ships. In terms of feature point extraction, the CUSUM method is used to extract feature points from AIS data, thereby achieving the purpose of data segmentation, data volume compression, and capturing key information of the data. 2. Extracting ship feature points from AIS data helps reduce data size, making it easier to manage for storage and transmission, improving storage and retrieval efficiency, and thus achieving cost savings. Compressing data volume can significantly reduce computation time and accelerate analysis, decision-making, and response time. The algorithm highlights and preserves key information in AIS data, reduces redundancy and noise, improves the quality and reliability of data analysis, and achieves more accurate trend analysis, anomaly detection, and predictive modeling. Attached Figure Description

[0033] Figure 1 It is a diagram showing the ship's AIS location information and ship feature points;

[0034] Figure 2 This is a flowchart of the algorithm in an embodiment of the present invention;

[0035] Figure 3 This is a histogram of word frequency cosine values ​​and TFIDF values ​​in an embodiment of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0037] Example 1

[0038] like Figure 2 As shown in this embodiment, an online feature point recognition algorithm for ship AIS data based on change point detection is provided, which specifically includes the following steps:

[0039] I. AIS Feature Processing:

[0040] Extract AIS data features from AIS data; the AIS data features include timestamp, longitude, latitude, air speed, and destination.

[0041] AIS data consists of static and dynamic data. The frequency of dynamic data is related to the ship's speed, and timestamps, longitude, latitude, and speed are selected as features from the dynamic data. Static data consists of ship length, beam, draft, destination, etc., and the destination is selected as a feature from the static data. Based on the actual selection, timestamps, longitude, latitude, speed, and destination are the five main features of AIS. Timestamps, longitude, latitude, and speed are all integers or floating-point numbers, while the destination field is text; therefore, the destination field needs to be vectorized to convert it into a numerical representation of the feature.

[0042] The static data in AIS is manually entered, and the "Vessel Destination" field lacks standardized formatting, resulting in instances of abbreviations, vague entries, spelling errors, compound symbols, and omissions. The vessel destination and the update time of this data are crucial points in the shipping process, potentially directly impacting route selection and average speed. Therefore, change point detection also requires discussion of the "Vessel Destination" field.

[0043] II. Change Point Identification:

[0044] Stable statistical methods or online Bayesian change point detection methods are used to detect AIS data features, thereby identifying AIS data features that have undergone mutations.

[0045] Stable statistics is a statistical approach used to handle data containing outliers or non-normally distributed data. Its goal is to reduce the impact of outliers on statistical results and improve the stability and reliability of statistical analysis. Methods reduce the impact of outliers through more stable estimation techniques and utilize quantile methods to adapt to the statistical analysis of non-normally distributed data. Stable statistics identifies and removes outliers through outlier detection methods, such as using median absolute deviation or robust loss functions.

[0046] Online Bayesian change point detection is a statistical method for detecting change points in a data sequence, combining Bayesian inference and online learning. This algorithm can detect changes in data in real time and update the model based on newly observed data, making it suitable for handling large-scale and continuous streaming data. By continuously observing new data and updating model parameters, the algorithm can detect change points in the data sequence in real time and adapt to data changes. The core idea of ​​the algorithm is to use Bayesian inference to estimate the location and properties of change points and to use the recursive Bayes formula for online learning. The process involves: initializing the prior distribution: given the observed data sequence, initializing the prior distribution to represent the distribution of change points before any change points are observed; processing new observations sequentially: for each new observation, calculating the posterior distribution based on the current prior distribution and likelihood function; updating model parameters: updating the model parameters based on the new observations, i.e., updating the estimated values ​​of the location and properties of change points; calculating the change point probability: calculating the probability of the change point based on the current posterior distribution to determine whether a change point has been detected; and updating the prior distribution: using the current posterior distribution as the prior distribution for the next time step.

[0047] III. Feature Point Recognition:

[0048] The CUSUM algorithm is used to calculate the difference between each data point in the AIS data feature that has undergone a mutation and its target value. The difference values ​​are accumulated to determine whether the deviation of the AIS data sequence reaches a preset threshold, thereby determining whether the AIS data sequence deviates from the normal state.

[0049] The CUSUM algorithm, based on the concept of cumulative summation, determines whether a data sequence has deviated from its normal state by calculating the difference between each data point and the expected target value and accumulating these differences. The CUSUM algorithm first sets a target value and a threshold. The target value is the desired data state, and the threshold is the critical value for judging data deviation. The cumulative sum is initialized to 0. For each data point, its difference from the target value is calculated, and the difference is added to the cumulative sum. If the cumulative sum exceeds the set threshold, the data sequence is judged to have changed or is abnormal; if the cumulative sum is below a set negative threshold, a change in the data sequence can also be judged.

[0050] Example 2

[0051] In this embodiment, the main function of AIS data feature processing is to convert the destination field into vector values ​​using an algorithm, with the aim of effectively identifying when a ship is changing its destination. Therefore, destination text analysis and algorithm selection are necessary.

[0052] The "Vessel Destination" data field generally follows the format of the English name of the destination port, such as "ANTALAHA" referring to the port of Taraha, Madagascar. It may also use the port's five-letter code "MGAHA," "MGAHA-001," or "MGAHA%001" to represent the destination and its berth number. During voyages, some vessels fill in both the port of origin and port of destination in the "Vessel Destination" field, such as "DONGNAI—ANTALAHA" to indicate a journey from Dong Nai, Vietnam to Taraha, Madagascar. Therefore, the "Vessel Destination" data field needs to be cleaned.

[0053] Conventional processing methods include encoding, hot tagging, and word vectorization. Word vectorization is a concept derived from the field of natural language processing. Its purpose is to quantify and compare the differences between words. The method maps "words" from a set to a vector space R. n Words mapped to the vector space can be quantified and compared. In this embodiment, word frequency cosine values ​​and TF-IDF are used to compare some port names and port five-letter codes.

[0054] Table 1. Calculation of Destination Port Keywords in AIS

[0055]

[0056] Word vectorization methods are worth considering for determining whether a ship has changed its destination. Based on cosine frequency and TF-IDF methods, we discuss the following two scenarios.

[0057] (1) The destination was not changed, but the port berth number and other information were updated, and letters were added or deleted. The similarity between the combination of "port name + berth number" and "port name" is greater than 0.5, and the similarity between "origin port - destination port" and "destination port" is greater than 0.5. The similarity between "port name" and "port name + symbol" is approximately 1.

[0058] (2) Without changing the destination, the port name is changed to the port code, and the similarity is 0.

[0059] (3) If the destination is changed and the names before and after are the full pinyin of the port name, the similarity is 0; if the names before and after are the five-character codes of the port name, the similarity is 0.

[0060] Regarding the determination of the port of destination, this embodiment aims to successfully identify the moment when the vessel changes its port of destination, ignoring moments when the vessel updates its destination information even if it hasn't changed its destination. Therefore, both parts of the data need to be updated.

[0061] (1) Calculate the cosine value (or TF-IDF) of the word frequency between the two points. If the similarity between the two words is greater than 0.5, it is considered that the destination has been changed.

[0062] (2) Establish a correspondence between the English name of the port and the five-letter code of the port to avoid the mistaken belief that the ship has changed its destination when it updates the name of the port of destination to the five-letter code.

[0063] A cross-comparison was performed on port names and their five-letter codes (including 200 English port names and 111 five-letter port codes) to calculate their similarity index. Data items with an index of 0 were removed, and the non-zero data items were visualized using a histogram. (See appendix for details.) Figure 3 .

[0064] Using both cosine similarity (CNS) and TF-IDF methods to calculate the similarity between different port names and their five-letter codes, 95% of the values ​​were 0, indicating a high degree of differentiation between port names and codes. For data pairs with non-zero similarity, the CNS results showed that a large number of pairs had similarity below 0.5, while the TF-IDF results showed similarity below 0.4. Therefore, this demonstrates that identifying ship destination changes using word vector methods is feasible. Furthermore, based on actual data measurements, for a destination in AIS data, if the similarity between the preceding and following word vectors is within 0.5, the ship is considered to have changed its destination; otherwise, it is considered that its destination has not been adjusted.

[0065] In this embodiment, the challenges posed by issues such as multiple ships sharing the same number and jump points to AIS data acquisition and cleaning necessitate parameter calibration and evaluation of the use of change point identification algorithms in AIS data. This embodiment fully considers the characteristics of AIS and conducts experiments from five aspects: experimental objects, data dimensions, number of change points, time frequency, and algorithm type. It discusses the change point identification and feature point extraction problems of AIS data under different conditions in various scenarios. See Table 2 for details; the following describes the experiments in different aspects.

[0066] Table 2 Experimental Design for Changing Point Recognition

[0067] Serial Number Data Dimensions Transformation point Time frequency Algorithm type 1 4*N 1 [High, Low], Equal Length Offline 2 4*N 1 [High, Low], Equal Length Online 3 4*N [4,8,12] [High, Low], Unequal Length Offline 4 4*N [4,8,12] [High, Low], Unequal Length Online

[0068] (1) Experimental data

[0069] The AIS data of vessels using cloned license plates makes it difficult to directly determine the change point location. Therefore, AIS data from several vessels were selected and stitched together to form a sample of vessels using cloned license plates. The data was divided by the change point, and the mean and variance of various parameters differed among the different vessels.

[0070] (2) Data Dimensions

[0071] A total of five dimensions were selected: "timestamp", "longitude", "latitude", "air speed", and "destination". The timestamp was used as the index.

[0072] (3) Transformation point

[0073] In the change point recognition task, if there is only one change point, the algorithm considers the entire sequence; otherwise, a window function needs to be set to perform change point recognition under different window lengths. Therefore, change point recognition is divided into four change point cases: 1, 4, 8, and 12, for data experiments.

[0074] (4) Time frequency

[0075] Considering the different sampling frequencies of AIS data, the time frequency of the data experiment was set to reflect the high and low sampling patterns under conditions of equal and unequal lengths.

[0076] (5) Algorithm type

[0077] Given the real-time and time-sensitive requirements of AIS data processing, we will consider testing offline (stable statistical method) and online algorithms (online Bayesian change point detection).

[0078] (6) Evaluation indicators

[0079] This embodiment uses AIS data under different parameter sets to experiment with both online and offline algorithms to discover suitable algorithms for AIS data change point extraction and feature point extraction. Therefore, a feasible evaluation method needs to be designed for the experiment.

[0080] The number of change points, sequence length, mean, and variance of the data allow us to obtain the change point locations as labels. Comparing the algorithm output with these change point location labels indicates a better algorithm performance; the greater the similarity, the better the algorithm's effect. For real-world data, this embodiment stitches together several normal AIS trajectories to obtain the actual change point locations, which are then evaluated using the same metric.

[0081] In this embodiment, the AIS data item consists of four data items: timestamp longitude, latitude, airspeed, and destination. The data range is aligned with the actual range. The longitude range is between -180 and 180, the latitude range is between -90 and 90, and the airspeed range is between 0 and 20. According to Table 3, the accuracy of different models was tested under the data experiment.

[0082] Table 3 shows the data from the data experiment in change point identification.

[0083]

[0084]

[0085] Table 3 shows that the accuracy of identifying a single transform point is higher than that of identifying multiple transform points. For both high and low frequency data, the stable statistical method achieves high accuracy, exceeding 88%; while the Bayesian transform point detection method is lower than the offline method. This is because the offline method examines every point in the sample, while the online algorithm only processes current and historical data. Regarding the identification of multiple transform points, when the point frequency is high, both the stable statistical method and the Bayesian transform point detection method achieve accuracy above 74%.

[0086] In this embodiment, starting with the practical problems existing in the real-time storage and computation of AIS data, statistical methods and word vector extraction methods were used to organize the AIS data from three aspects: AIS feature selection, change point identification, and feature point extraction. Data experiments were conducted under different conditions, and the experimental results are given. The conclusions are as follows.

[0087] 1. Experiments were conducted on the English names and five-letter codes of ports using the word frequency cosine value and TFIDF value calculation methods. The results show that both methods can accurately reflect whether the destination port of a ship has changed.

[0088] 2. The accuracy of variable point detection in AIS data is similar regardless of the number of variable points. Therefore, in subsequent experiments, the accuracy can be considered to be independent of the number of variable points.

[0089] 3. By using both the stable statistical method and Bayesian change point detection, a change point detection method for AIS data can be implemented. The accuracy of the offline method is about 7.3 percentage points higher than that of the online method.

[0090] By adopting the above-disclosed technical solution of this invention, the following beneficial effects are obtained:

[0091] This invention provides an online feature point recognition algorithm for ship AIS data based on change point detection. In terms of AIS feature selection, statistical analysis is performed on the AIS data to select representative features, and natural language processing methods are used to construct word vectors for ship destinations. For change point recognition, both offline and online algorithms are employed to analyze the changing trends of the AIS data, accurately identifying change points where two consecutive data points belong to two different ships. In feature point extraction, the CUSUM method is used to extract feature points from the AIS data, thereby achieving data segmentation, data compression, and the capture of key information. Extracting ship feature points from AIS data helps reduce data size, making it easier to manage for storage and transmission, improving storage and retrieval efficiency, and thus saving costs. Compressing data significantly reduces computation time, accelerating analysis, decision-making, and response times. The algorithm highlights and preserves key information in the AIS data, reduces redundancy and noise, improves the quality and reliability of data analysis, and achieves more accurate trend analysis, anomaly detection, and predictive modeling.

[0092] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1.A method for online identification of feature points of ship AIS data based on change point detection, characterized in that: The method comprises the following steps, S1, AIS feature processing: extracting AIS data features from AIS data; the AIS data features include timestamp, longitude, latitude, speed and destination; S2, change point identification: using stable statistical method or online Bayesian change point detection method to detect AIS data features, so as to identify the AIS data features with mutation; S3, feature point identification: using CUSUM algorithm to calculate the difference between each data point in the AIS data features with mutation and its target value, and accumulating the difference value, and then judging whether the deviation degree of the AIS data sequence reaches the preset threshold value; AIS data includes static data and dynamic data, destination is selected from static data, and timestamp, longitude, latitude and speed are selected from dynamic data as five main features of AIS; The timestamp, longitude, latitude and speed are integers or floating point numbers, and the destination is a text representation, which needs to be vectorized for description; The text representation of the destination is vectorized by using word frequency cosine value or TF-IDF algorithm; In order to judge the time when the ship changes the destination, the time when the ship updates the destination information without changing the destination is ignored, and the following method is used for judgment: Calculate the word frequency cosine value or TF-IDF of the two points, if the word similarity of the two points is greater than 0.5, it is considered that the destination is changed; Establish the correspondence between the English name of the port and the five-character code of the port to avoid that the ship updates the destination name to the five-character code and is mistaken for changing the destination; The stable statistical method identifies and removes outliers through outlier detection method; Online Bayesian change point detection specifically includes the following contents, Initialize the prior distribution: given the observed data sequence, initialize the prior distribution, which represents the distribution of change points before any change point is observed; Process new observation data in turn: for each new observation data, calculate the posterior distribution according to the current prior distribution and likelihood function; Update model parameters: update the model parameters according to the new observed data, that is, update the estimated value of the position and nature of the change point; Calculate the change point probability: calculate the probability of change point according to the current posterior distribution, which is used to judge whether the change point is detected; Update the prior distribution: the current posterior distribution is used as the prior distribution of the next time step; Step S3 specifically includes the following contents, S31, set the target value and threshold value; S32, for each data point, calculate the difference between it and the target value, and accumulate the difference value in the cumulative sum; S33, if the cumulative sum exceeds the set threshold value, it is judged that the AIS data sequence has changed or is abnormal.

Images