Ship path search and arrival time prediction method based on AIS historical data

Abstract

The invention discloses a ship path search and arrival time prediction method based on AIS historical data. The method comprises the following steps: S1, setting grid feature codes of AIS data; s2, constructing an AIS track network and weights; and S3, searching and simplifying a ship path. The method has the advantages that the abnormal data is removed based on the geographic space features and the track features, and the quality of ship data and the path searching effect in the global range are improved. An AIS data gridding method based on a Google S2 spatial coding principle is provided, and the grid size is intelligently planned according to the density of AIS data. The dynamic weight strategy is introduced into the edge weight endowing of the AIS trajectory network, the weight is dynamically adjusted according to the daily density condition of the ship, the method adapts to the change of ship distribution in different time periods, and the path search accuracy is improved. A directed path search optimization method based on a Bellman-Ford algorithm and a path trajectory simplification method based on Kalman filtering are provided, the efficiency of the path search algorithm is improved, and efficient expression of the path trajectory is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of ship path search, and in particular to a ship path search and arrival time prediction method based on AIS historical data. Background Art

[0002] Automatic Identification System (AIS) is a technology used for radio communication between ships and between ships and land. The purpose of AIS is to provide real-time information such as ship position, identification, navigation status, etc. to enhance the safety and efficiency of maritime traffic. Due to its standardization, real-time, openness and reliability, it is widely used in maritime traffic management, ship positioning, navigation safety, port management and other aspects.

[0003] In the field of ship path search, a suitable algorithm is crucial for effectively planning and predicting the trajectory of the ship. There are three common methods: path search based on network models, path search based on historical trajectories, and path search based on machine learning.

[0004] 1. Path search based on network model

[0005] Path search based on network models often uses graph theory methods. Channels and water areas are modeled as graphs, with nodes representing locations and edges representing paths, so that graph algorithms can be applied to find the optimal path. Common path search algorithms include: (1) Dijkstra algorithm: Finds the shortest path by calculating the shortest path between nodes. Suitable for path planning in static environments. (2) Bellman-Ford algorithm: An algorithm used to calculate the shortest path from a single source, widely used in graph theory and network routing algorithms. This algorithm can process graphs with negative weight edges and can detect whether there are negative weight loops in the graph. (3) A* algorithm: Combines heuristic search and Dijkstra algorithm, taking into account both search speed and path quality, and is suitable for path search in dynamic environments.

[0006] 2. Path search based on historical trajectory

[0007] The habitual path judgment based on path search of historical trajectories can identify and predict the path that the ship may choose in the future by analyzing the ship's past trajectory data. The required steps are roughly as follows: (1) Data collection: Obtain the ship's historical trajectory data, which can be obtained through the AIS system or other ship tracking systems. The historical data should include information such as the ship's location, speed, and heading. (2) Path clustering: Cluster the historical trajectory data to identify different habitual paths. Clustering algorithms can help identify similar ship behavior patterns, thereby determining the ship's regular route in specific waters. (3) Real-time path update: As new real-time data is generated, the habitual path information is continuously updated to ensure that the path prediction can reflect the latest ship behavior. (4) Path display and user feedback: The prediction results of the habitual path are displayed to the user in a visual form, and the user is allowed to provide feedback so that the system can better adapt to the actual situation.

[0008] 3. Path search based on machine learning

[0009] In recent years, machine learning technology has made significant progress in the field of path search. By learning the historical trajectory of the ship and environmental information, the model can predict the most likely future path. Commonly used machine learning models include: (1) spatiotemporal sequence model: using models such as recurrent neural network (RNN) or long short-term memory network (LSTM) to consider the temporal characteristics of ship movement. (2) Deep reinforcement learning: using deep neural network combined with reinforcement learning algorithm, the model can learn the optimal path from the interaction with the environment.

[0010] The current arrival time prediction model (ETA) is a combination of path search based on historical trajectories and path search based on machine learning. It does not extract customary routes, but directly uses AIS data + machine learning methods to complete the prediction. The calculation is between port pairs, and the calculation range is small. Based on the existing technical capabilities, the arrival time prediction model (ETA) has requirements for the timeliness of data prediction. Assuming that the upper limit of the sum of the calculation time, transmission time and display time of ETA is about 10 seconds, the current technology cannot provide a fast tool to realize historical path search. Summary of the invention

[0011] The object of the present invention is to provide a method for searching ship paths and predicting arrival times based on AIS historical data, thereby solving the above-mentioned problems existing in the prior art.

[0012] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0013] A method for searching ship paths and predicting arrival time based on AIS historical data comprises the following steps:

[0014] S1. Grid feature encoding settings for AIS data:

[0015] AIS data is obtained by establishing a connection between PySpark and ClickHouse databases. AIS data is cleaned based on geographic spatial features and trajectory features to remove data anomalies. AIS data gridding method based on Google S2 spatial coding principle is used to divide the earth's surface into small blocks with variable precision, and gridding statistics are performed in combination with the spatiotemporal characteristics of AIS data.

[0016] S2. AIS trajectory network and weight construction:

[0017] The neighborhood points of a given grid point are determined by the dynamic neighborhood selection method based on the KNN algorithm, and they are connected into edges according to their location information to construct the AIS trajectory network. The edge weight determination method based on the dynamic weight strategy is used to introduce a dynamic weight strategy in the edge weight assignment of the AIS trajectory network, and the weight is dynamically adjusted according to the daily density of ships.

[0018] S3. Ship path search and simplification:

[0019] The directed path search optimization method based on the Bellman-Ford algorithm is used to realize the shortest path search for ships, and the path trajectory simplification method based on Kalman filtering is used to remove the fluctuation points and invalid points of the path trajectory to realize the simplification of the path trajectory.

[0020] Preferably, in step S1, obtaining AIS data specifically includes the following contents:

[0021] S111. Establish a connection between PySpark and ClickHouse database and configure the connection parameters, including the host address, port, and user credentials of ClickHouse;

[0022] S112, achieving parallel access to AIS data in the database through time-index-based partitioning; dividing the AIS data into appropriate time windows to improve data retrieval efficiency;

[0023] S113, preprocessing the acquired AIS data, including processing missing values, cleaning abnormal data, and performing feature engineering on the data;

[0024] S114. Use GIS visualization tools to gain in-depth understanding of the distribution, trends, and correlations of pre-processed AIS data.

[0025] Preferably, in step S1, data cleaning based on geographic spatial features is specifically as follows:

[0026] The AIS data are cleaned based on the global continental shelf edge contour data, and the AIS data within the continental shelf contour are deleted.

[0027] Preferably, the data cleaning based on trajectory features includes cleaning of position drift abnormal points, and the cleaning of position drift abnormal points is specifically as follows:

[0028] When the ship drifts at a certain track point, it will be different from the previous and next track points P i-1 and P i+1 The smaller the angle, the smaller the angle. If the ship is at point P i If the front and rear trajectory segments at satisfy the following formula, it is considered to be an abnormal point caused by position drift and is removed;

[0029] Δθ Pi <δ θ ,Δt Pi <δ t

[0030] Among them, Δθ Pi is the trajectory point P i and the front and rear trajectory points P i-1 and P i+1 The angle of the formed trajectory segment; δ θ is the angle threshold; Δt Pi is the time interval between three trajectory points; δ t is the time interval threshold.

[0031] Preferably, the data cleaning based on trajectory features includes cleaning of abnormal ship speed points, and the cleaning of abnormal ship speed points is specifically as follows:

[0032] Let trajectory point P i Speed ​​V i Only the speed V of the first two adjacent track points i-2 and V i-1 Related, t i-2 ,t i-1 ,t i are the sampling times of the three trajectory points respectively, then the trajectory point P i The predicted speed V i ′ is expressed as,

[0033] V′ i =V i-1 +(V i-1 -V i-2 ) / (t i-1 -t i-2 )×(t i -t i-1 )

[0034] If the predicted speed value V′ iWith the original recorded value V i The error between them is greater than the speed change threshold, i.e. |V′ i -V i |>δ θ ′, then the ship is considered to be at the trajectory point P i The speed value is an abnormal value, and the predicted speed V′ i Replaces the original speed V i ; Among them, δ θ ′ is the speed change threshold, which is estimated by the speed of the previous trajectory point, that is, δ′ θ = k × V i-1 , the k value range is (0, 1).

[0035] Preferably, in step S1, the AIS data gridding method based on the Google S2 spatial coding principle specifically includes the following contents:

[0036] S121, based on the Google S2 spatial coding principle, the global network is gridded and encoded at intervals of preset distances;

[0037] S122, on the basis of sorting the AIS data by time and segmenting the track, counting each segment of the AIS track data by ship and segment; in the counting process, based on the statistical result of the previous AIS track point, only several grids around the previous grid are called to count the next AIS track;

[0038] S123, during the statistical process, the number of AIS track points in the grid is determined. When the data of the points in the grid exceeds the set threshold and the size of the grid is greater than the threshold for grid refinement, the grid is subdivided according to the Google S2 spatial coding principle, and the number of AIS track points in each grid is re-counted;

[0039] S124, repeat steps S122 to S123 until all AIS data are counted;

[0040] S125, selecting grid center points in which the number of AIS track points in the grid is greater than a preset number to form a grid matrix.

[0041] Preferably, in step S2, the point neighborhood dynamic selection method based on the KNN algorithm specifically includes the following contents:

[0042] S211, selecting any grid center point in the grid matrix as a judgment point;

[0043] S212, based on the spatial coding of the grid matrix, selecting the grid center points adjacent to the judgment point as the initial judgment group;

[0044] S213, calculating the Euclidean distance between the starting point and all points in the initial judgment group, and selecting the three points with the smallest distance as the connecting points of the judgment points, and selecting any one of them as a new judgment;

[0045] S214. Repeat steps S212 to S213 until all points that can be connected are connected to form an AIS trajectory network.

[0046] Preferably, in step S2, the edge weight determination method based on the dynamic weight strategy specifically includes the following five methods:

[0047] (1) For two connected points, the edge weight is the inverse of the sum of the weights of the two points;

[0048] (2) For two connected points, the edge weight is the mean of the weights divided by the sum of the weights of the two points;

[0049] (3) For two connected points, if the point weight is greater than [50, 100, 300, 500], reset its weight to [50, 100, 300, 500], and the edge weight is the inverse of the sum of the two point weights;

[0050] (4) Perform Max-Min normalization on the weights of the points. For two connected points, the edge weight is the sum of the normalized weights of the two points.

[0051] (5) Perform Z-score normalization on the point weights. For two connected points, the edge weight is the inverse of the sum of the normalized weights of the two points.

[0052] Preferably, in step S3, the directed path search optimization method based on the Bellman-Ford algorithm specifically includes the following contents:

[0053] S311, initializing the distances of all nodes except the source node to infinity, and initializing the distance of the source node to 0;

[0054] S312, for each node in the graph, perform relaxation operation in turn; the relaxation operation refers to trying to shorten the path distance through the edge from one node to another node, if the path through a node is shorter than the path directly reaching the target node, then update the distance of the target node to the smaller path distance;

[0055] S313, repeating the iteration operation N-1 times, where N is the number of vertices in the graph; in each iteration, performing a relaxation operation on each edge;

[0056] S314, after completing N-1 iterations, perform the Nth iteration operation; if relaxation operation can still be performed in the Nth iteration, it means that there is a negative weight cycle in the graph; if a negative weight cycle is detected, the output result of the algorithm is invalid;

[0057] S315. The final distance array is the shortest path from the source node to all other nodes.

[0058] Preferably, in step S3, the path trajectory simplification method based on Kalman filtering specifically includes the following contents:

[0059] S321, data initialization, including initialization of initial state vector, state transfer matrix, measurement matrix, noise parameters and initial error covariance matrix;

[0060] S322, selecting path points in sequence, and calculating a state prediction matrix and a covariance matrix according to the Kalman gain, the measurement matrix and the strategy amount;

[0061] S323, looping through step S322 until all points are traversed to obtain a path smoothing trajectory point set;

[0062] S324, sequentially calculating the direction of the path smooth trajectory point set and the rate of change of the direction;

[0063] S325, determining whether there is a situation in the point set where the direction change rate is continuously less than the direction change threshold, if so, retaining only the first and last two points, deleting the remaining points, and obtaining a simplified path trajectory.

[0064] The beneficial effects of the present invention are as follows: 1. The method of the present invention connects the ClickHouse database through PySpark, accesses the AIS data in the database in parallel based on the time index, and provides large-scale, high-temporal and spatial resolution marine motion information for network construction. 2. AIS data cleaning is realized based on geographic spatial features and trajectory features, abnormal data is eliminated, and the quality of ship data and the effect of path search are improved worldwide. 3. By dynamically adjusting weights and carefully processing abnormal points, the actual distribution of ships can be better reflected, providing more accurate support for path search. 4. An AIS data gridding method based on the Google S2 spatial coding principle is proposed. According to the density of AIS data, the grid size is intelligently planned, and combined with the characteristics of the continuity of AIS trajectories, a suitable number of grids are called to participate in statistics during the statistical process, which has the advantages of uniform division, variable precision support and efficient spatial operation. 5. By introducing a dynamic weight strategy in the edge weight assignment of the AIS trajectory network, the weight is dynamically adjusted according to the daily density of the ship, and the changes in ship distribution are adapted to different time periods to improve the accuracy of path search. 6. A directed path search optimization method based on the Bellman-Ford algorithm is proposed to improve the efficiency of the path search algorithm. 7. A path trajectory simplification method based on Kalman filtering is proposed to effectively remove the fluctuation points and invalid points of the path trajectory and achieve efficient expression of the path trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a flow chart of a prediction method in an embodiment of the present invention;

[0066] Figure 2 is a schematic diagram of a GIS tool processing anomaly points in an embodiment of the present invention;

[0067] Figure 3 It is a spatial index map (partial) of the S2 algorithm based on AIS data in an embodiment of the present invention;

[0068] Figure 4 is a network edge construction graph (Mediterranean Sea) based on the KNN algorithm in an embodiment of the present invention;

[0069] Figure 5 It is a network edge construction graph (partial) based on the KNN algorithm in an embodiment of the present invention;

[0070] Figure 6 is the path search based on the Bellman-Ford algorithm in the embodiment of the present invention (light gray line);

[0071] Figure 7 It is a schematic diagram of interface calling in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0073] like Figure 1 As shown, in this embodiment, a method for searching ship paths and predicting arrival times based on AIS historical data is provided. The method of the present invention constructs a network model (not a route network, but a network based on the geocoding of ship locations), where the nodes of the network represent the locations of ships and the edges represent the probability of selecting a path for the ship's navigation. Given the starting and ending points, a path selection result with a time complexity of O(V*E) is quickly formed, and the navigation time is calculated based on the historical speed. The method of the present invention mainly includes the following parts:

[0074] 1. Grid feature coding setting of AIS data

[0075] By establishing a connection between PySpark and ClickHouse database, AIS data is obtained; data cleaning based on geographic spatial features and trajectory features is performed on AIS data in turn to remove data anomalies; AIS data gridding method based on Google S2 spatial coding principle is used to divide the earth's surface into small blocks with variable precision, and gridding statistics are performed based on the spatiotemporal characteristics of AIS data. Specifically, the following contents are included:

[0076] 1.1 AIS data acquisition

[0077] The use of AIS historical data requires sufficient AIS data, so the ability to quickly access AIS data is formed by combining the big data platform. PySpark is connected to the ClickHouse database, and the AIS data in the database is accessed in parallel based on the time index, providing large-scale, high-temporal and spatial resolution marine motion information for network construction.

[0078] (1) Data connection and configuration: Establish a connection between PySpark and ClickHouse database. Configure connection parameters, including ClickHouse host address, port, and user credentials.

[0079] (2) Time index partitioning: Through time index-based partitioning, parallel access to AIS data in the database is achieved. Dividing data into appropriate time windows can improve the efficiency of data retrieval, avoid full table scans, and speed up the data acquisition process.

[0080] (3) Data preprocessing: After obtaining AIS data, data preprocessing is required to ensure the network is effective. Methods include processing missing values, cleaning abnormal data, and performing feature engineering on the data. Here, three conditions are used for screening: MMSI length is 9, the first digit of SHIP_TYPE_ID is 6, 7, 8, and the Long and Lat fields are not empty;

[0081] (4) Data exploration and visualization: Based on data acquisition and preprocessing, data exploratory analysis was performed. Through GIS visualization tools, the distribution, trend, and correlation of AIS data were deeply understood, which helped to provide guidance for the design of subsequent network models.

[0082] After visual exploration and analysis, it was found that there were too many noise points on a global scale. The abnormal points were not only distributed on the continent, but also distributed in three belts on a global scale.

[0083] 1.2 Data Cleaning

[0084] Based on the two problems found in 1.1 Data Acquisition, the data is further cleaned. Combining the analysis of space and time dimensions, it aims to improve the quality of ship data and the effect of path search on a global scale. By dynamically adjusting weights and carefully handling outliers, the actual distribution of ships can be better reflected, providing more accurate support for path search.

[0085] (1) Data cleaning based on geospatial features

[0086] The original AIS data is cleaned based on the global continental shelf edge contour data, and the AIS data within the continental shelf contour is deleted to quickly eliminate the abnormal points of the AIS data (such as Figure 2 shown). However Figure 1 The three strip-shaped outliers are not easy to remove.

[0087] (2) Data cleaning based on trajectory features

[0088] Due to the strong inertia of the ship itself and the weak viscous resistance of the water body carrying the ship, the maneuverability of the ship is often poor (except for a few special ships), and the motion characteristics of the ship's trajectory points often change slowly or remain almost unchanged. Therefore, based on the spatiotemporal characteristics of the ship's trajectory, the abnormal points that were not deleted in 1.1 can be removed. The specific cleaning is mainly divided into the following two aspects:

[0089] A. Cleaning of abnormal position drift points

[0090] Since the update time of AIS ship track is short, the track segments are usually smoother under normal circumstances, and the angles between adjacent track segments are generally larger. However, when the ship drifts at a certain track point, it will be inconsistent with the previous and next track points P.i-1 and P i+1 The smaller the angle, the smaller the angle. If the ship is at point P i If the front and rear trajectory segments at satisfy the following formula, it is considered to be an abnormal point caused by position drift and is removed;

[0091] Δθ Pi <δ θ ,Δt Pi <δ t

[0092] Among them, Δθ Pi is the trajectory point P i and the front and rear trajectory points P i-1 and P i+1 The angle of the formed trajectory segment; δ θ is the angle threshold; Δt Pi is the time interval between three trajectory points; δ t is the time interval threshold.

[0093] B. Cleaning of abnormal ship speed points

[0094] Considering that the ship speed changes slowly, assuming that the trajectory point P i Speed ​​V i Only the speed V of the first two adjacent track points i-2 and V i-1 Related, t i-2 ,t i-1 ,t i are the sampling times of the three trajectory points respectively, then the trajectory point P i The predicted speed V i ′ is expressed as,

[0095] V′ i =V i-1 +(V i-1 -V i-2 ) / (t i-1 -t i-2 )×(t i -t i-1 )

[0096] If the predicted speed value V′ i With the original recorded value V i The error between them is greater than the speed change threshold, i.e. |V′ i -V i |>δ θ ′, then the ship is considered to be at the trajectory point P i The speed value is an abnormal value, and the predicted speed V′ i Replaces the original speed V i ; Among them, δ θ′ is the speed change threshold, which is estimated by the speed of the previous trajectory point, that is, δ′ θ = k × V i-1 , the k value range is generally (0, 1), and here k is 0.5.

[0097] 1.3. Encoding feature engineering

[0098] For the path search network, if each AIS latitude and longitude point generated by the ship is used as the center point of the network, the number of grids will be hundreds of millions, resulting in extremely low path search efficiency. In addition, the traditional AIS data gridding method requires traversing the pre-formed grid for each statistics or editing, especially for large-area AIS data statistics such as global scope, the work efficiency is low.

[0099] Therefore, the present invention proposes a method for gridding AIS data based on the Google S2 spatial coding principle. Based on the principle of Google S2 spatial coding, the earth's surface is divided into small blocks with variable precision, and gridding statistics are performed in combination with the spatiotemporal characteristics of AIS data. This method supports efficient spatial operations and is particularly suitable for applications that require indexing and querying in geographic space, such as maps, location services, etc. The specific steps are as follows:

[0100] (1) Based on the Google S2 spatial coding principle, the global network is divided into grids and coded at intervals of preset distances (e.g., 10 kilometers);

[0101] (2) After sorting the AIS data by time and dividing the track into segments, the AIS track data of each segment is counted ship by ship. During the statistical process, based on the statistical result of the previous AIS track point, only several grids around the previous grid are called to count the next AIS track.

[0102] (3) During the statistical process, the number of AIS track points in the grid is determined. When the data of the points in the grid exceeds the set threshold and the size of the grid is larger than the threshold for grid refinement, the grid is subdivided according to the Google S2 spatial coding principle, and the number of AIS track points in each grid is re-counted;

[0103] (4) Repeat steps (2) to (3) until all AIS data are counted;

[0104] (5) Select the grid center points whose number of AIS track points in the grid is greater than the preset number (e.g., 100) to form a grid matrix. The calculation results of Google S2 are shown in Table 1.

[0105] Table 1 Google S2 calculation results

[0106] S2 Long Lat weight speed std 31da1f 103.86801 1.21417 3011 10 3 31da2f 104.14661 1.28741 3007 10.6 3.2 31da21 103.96082 1.21369 3007 10.5 3.4 31da2d 104.2396 1.28688 2988 11 3.2 31dbd3 104.33265 1.28635 2978 11.6 3.1 31dbcf 104.42576 1.36057 2959 12 3.1 31dbc7 104.51893 1.43485 2900 12.2 3 31dbc9 104.42576 1.43545 2816 11 2.8 31d0ad 103.40495 1.21656 2815 11.3 3.1 31d0b1 103.31254 1.292 2780 11.8 3 31d0b9 103.22019 1.36763 2776 12 3 31dbcd 104.33265 1.36113 2752 10.6 3 31d0c1 103.12791 1.44345 2750 12.2 3 31da25 104.05368 1.28793 2744 9.5 3.1 31d19b 102.30059 1.90433 2740 13 2.7 31d0e9 103.0357 1.51946 2717 12.9 3 31d103 102.75949 1.67265 2655 12.6 2.8 31d9f7 103.77527 1.13997 2599 10.5 2.9 31dbb9 104.51893 1.50985 2576 11.4 2.8 31dbbf 104.61215 1.50921 2536 12.3 3.1 31d1a1 102.48393 1.82667 2502 12.8 2.8 31d0e5 102.94356 1.59565 2498 13.3 2.8 31da23 103.96082 1.28845 2444 7.8 2.5 31d109 102.57571 1.74987 2442 12.1 2.8 31dbbd 104.61215 1.5843 2384 11.8 2.9 31da1d 103.77527 1.21466 2202 8.9 2.6

[0107] In Table 1, S2 is the unique grid code; Long is the longitude of the center point of S2 grid; Lat is the latitude of the center point of S2 grid; weight is the number of ships in the grid; speed is the average value of the speed item in the AIS data of ships in the grid (excluding speeds below 5 nautical miles / hour and above 30 nautical miles / hour). std is the variance of speed.

[0108] like Figure 3 As shown in the figure, compared with the traditional AIS data gridding method, the present invention proposes an AIS data gridding method based on the Google S2 spatial coding principle, which can intelligently plan the grid size according to the density of AIS data, and combine the characteristics of AIS trajectory continuity to call an appropriate number of grids to participate in the statistics during the statistical process. Therefore, the method proposed by the present invention has the advantages of uniform division, variable precision support, and efficient spatial operation.

[0109] 2. AIS trajectory network and weight construction

[0110] The neighborhood points of a given grid point are determined by the point neighborhood dynamic selection method based on the KNN algorithm, and they are connected into edges according to their location information to construct the AIS track network; the edge weight determination method based on the dynamic weight strategy is used to introduce a dynamic weight strategy in the edge weight assignment of the AIS track network, and the weight is dynamically adjusted according to the daily density of ships. This part is mainly to solve the problem of how the grid rich in AIS ship information obtained by the S2 algorithm should be organized into a network. Specifically, it includes the following contents,

[0111] 2.1 Point Neighborhood Selection

[0112] Points cannot be connected together in disorder, otherwise the path search after forming a network is likely to be skipped and unreasonable. Therefore, the present invention proposes a point neighborhood dynamic selection method based on the KNN algorithm. This method can be used to find the three nearest neighboring points to a given point (such as Figure 4 and Figure 5 ), and connect them into edges based on their location information for further processing. The specific operation steps of the method are as follows:

[0113] (1) Select any grid center point in the grid matrix as the judgment point;

[0114] (2) Based on the spatial encoding of the grid matrix, the grid center points adjacent to the judgment point are selected as the initial judgment group;

[0115] (3) Calculate the Euclidean distance between the starting point and all points in the initial judgment group, select the three points with the smallest distance as the connection points of the judgment points, and select any one of them as the new judgment;

[0116] (4) Repeat steps (2) and (3) until all points that can be connected are connected to form an AIS trajectory network.

[0117] 2.2 Edge Weight Selection

[0118] The center point of the grid in the AIS track network formed above represents the position of the ship, and the continuity between the center points of the grid represents the movement direction of the ship, that is, the ship can move from the current grid to the adjacent grid connected by the connecting line.

[0119] Based on the analysis of ship navigation habits, areas with dense ship tracks often represent areas with better hydrological and meteorological conditions, shorter distances, and more suitable for ship navigation. Therefore, when planning ship track paths, it is generally necessary to give priority to grid points with a large number of historical AIS track data. Therefore, in the process of ship path optimization, the probability of each edge in the AIS track network being selected is different, that is, the possible movement direction of the ship is biased.

[0120] In summary, the present invention proposes a method for determining edge weights based on a dynamic weight strategy. A dynamic weight strategy is introduced in the edge weight assignment of the AIS trajectory network, and the weights are dynamically adjusted according to the daily density of ships. Different time periods are adapted to changes in ship distribution and path search accuracy is improved. The present invention proposes five edge weight assignment ideas, which are as follows:

[0121] 1) For two connected points, the edge weight is the inverse of the sum of the weights of the two points;

[0122] 2) For two connected points, the edge weight is the mean of the weights divided by the sum of the weights of the two points;

[0123] 3) For two connected points, if the point weight is greater than [50, 100, 300, 500], reset its weight to [50, 100, 300, 500], and the edge weight is the inverse of the sum of the two point weights;

[0124] 4) Perform Max-Min normalization on the weight of the points. If for two connected points, the edge weight is the sum of the normalized weights of the two points;

[0125] 5) Perform Z-score normalization on the point weights. If for two connected points, the edge weight is the inverse of the sum of the normalized weights of the two points.

[0126] After testing, the fifth method is used to weight the edges of AIS trajectory networks over a large area such as the global network, and better results are obtained.

[0127] 3. Ship path search and simplification

[0128] The directed path search optimization method based on the Bellman-Ford algorithm is used to realize the shortest path search for ships, and the path trajectory simplification method based on the Kalman filter is used to remove the fluctuation points and invalid points of the path trajectory to simplify the path trajectory. This part is mainly to solve how to calculate the shortest path of the grid rich in AIS ship information obtained by the S2 algorithm. It includes the following contents,

[0129] 3.1 Optimization of path search algorithm

[0130] Considering that the proposed edge weight determination method based on the dynamic weight strategy may have negative weights when assigning weights. Although the traditional Bellman-Ford algorithm can handle negative weight edges, it needs to traverse all nodes, which results in a large amount of computing time and computing memory being wasted for global ship path planning. Therefore, the present invention proposes a directed path search optimization method based on the Bellman-Ford algorithm to improve the efficiency of the path search algorithm (such as Figure 6 The specific operation steps are as follows:

[0131] (1) Initialize the distances of all nodes except the source node to infinity, and initialize the distance of the source node to 0;

[0132] (2) For each node in the graph (AIS trajectory network), relaxation operations are performed in turn. The relaxation operation refers to the edge from one node to another, trying to shorten the path distance. If the path through a node is shorter than the path directly to the target node, then the distance of the target node is updated to the smaller path distance;

[0133] (3) Repeat the iteration operation N-1 times, where N is the number of vertices in the graph. In each iteration, relax each edge;

[0134] (4) After completing N-1 iterations, perform the Nth iteration. If relaxation can still be performed in the Nth iteration, it means that there is a negative weight cycle in the graph. Negative weight cycles can reduce the total cost indefinitely, so if a negative weight cycle is detected, the output of the algorithm is invalid;

[0135] (5) The final distance array is the shortest path from the source node to all other nodes.

[0136] 3.2 Path Trajectory Simplification

[0137] When planning ship paths for the global AIS network, the planned path is long and passes through many network nodes. Therefore, after using the Bellman-Ford algorithm for path search, the point sequence given is too long, resulting in a tortuous and uneven path after visualization, which is not conducive to the application of path planning results. Therefore, the present invention proposes a path trajectory simplification method based on Kalman filtering. First, the Kalman filtering method is used to smooth the point sequence of the path, and then the point sequence of the path is simplified based on the directional change rate of the point sequence of the path. The specific operation steps are as follows:

[0138] (1) Data initialization, mainly including the initial state vector, state transfer matrix, measurement matrix, noise parameters, and initial error covariance matrix;

[0139] (2) Select path points in sequence and calculate the state prediction matrix and covariance matrix based on the Kalman gain, measurement matrix and strategy quantity;

[0140] (3) Run (2) repeatedly until all points are traversed and a set of smooth trajectory points is obtained;

[0141] (4) Calculate the direction of the path smooth trajectory point set and the rate of change of the direction in sequence;

[0142] (5) Determine whether there is a situation in the point set where the direction change rate is continuously less than the direction change threshold. If so, only the first and last points are retained, and the remaining points are deleted to obtain a simplified path trajectory. The path trajectory simplification method based on Kalman filtering can effectively remove the fluctuation points and invalid points of the path trajectory. Achieve efficient expression of the path trajectory. After continuous testing, the current process noise parameter value is 100, the covariance matrix of the measurement noise value is 2, and the direction change threshold is 3°, which has a good processing effect.

[0143] 4. Implementation of forecasting methods

[0144] 4.1、Model cloud formation interface (see the attached Figure 7 )

[0145] (1) The model completion files, parameter update code, and path search code have been placed in the provided Ubuntu server.

[0146] (2) Based on the flask library provided by Python, the interface of the existing model is encapsulated and intranet calls are provided.

[0147] Input: starting point longitude and latitude, end point longitude and latitude;

[0148] Output: predicted path latitude and longitude points, predicted nautical miles, predicted arrival time.

[0149] 4.2 Dynamic update of model parameters

[0150] Based on the dynamic weight adjustment and encoding feature engineering proposed above, the dynamic update of model parameters is realized and the automation script is basically completed.

[0151] (1) Generate and maintain intermediate files based on the big data platform

[0152] Automatic statistics of Google S2 based on Crontab. Statistics include five data items: S2 code, average speed, speed standard deviation, longitude and latitude, and number of ships. The results are saved in the HDFS directory. The calculation time is about 15-18 minutes, and the generated file size is about 80M. Assuming that 30 days of data are saved, the total file size is about 2.3G.

[0153] Automatic maintenance of HDFS files. The script automatically checks the directory length. When the directory length exceeds 30 (configurable), the system automatically deletes the oldest file to ensure that the directory maintains a shorter history, thereby effectively managing and maintaining HDFS files.

[0154] (2) Dynamic adjustment and maintenance of network parameters based on cloud platform

[0155] HDFS automatically pulls files, reads and merges the original weights and the weights of the latest day, calculates and saves the updated weights based on the data cleaning based on trajectory features in 1.2. Automatic maintenance includes automatic saving of snapshots and automatic deletion of data beyond 30 days.

[0156] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0157] The present invention provides a ship path search and arrival time prediction method based on AIS historical data. The method of the present invention connects the ClickHouse database through PySpark, accesses the AIS data in the database in parallel based on the time index, and provides large-scale, high-temporal and spatial resolution marine motion information for network construction. AIS data cleaning is implemented based on geographic spatial features and trajectory features, abnormal data is eliminated, and the quality of ship data and the effect of path search are improved worldwide. By dynamically adjusting weights and carefully processing abnormal points, the actual ship distribution can be better reflected, providing more accurate support for path search. An AIS data gridding method based on the Google S2 spatial coding principle is proposed. According to the density of AIS data, the grid size is intelligently planned, and combined with the characteristics of AIS trajectory coherence, a suitable number of grids are called to participate in statistics during the statistical process, which has the advantages of uniform division, variable precision support and efficient spatial operation. By introducing a dynamic weight strategy in the edge weight assignment of the AIS trajectory network, the weight is dynamically adjusted according to the daily density of the ship, and the changes in ship distribution are adapted to different time periods to improve the accuracy of path search. A directed path search optimization method based on the Bellman-Ford algorithm is proposed to improve the efficiency of the path search algorithm. A path trajectory simplification method based on Kalman filtering is proposed to effectively remove the fluctuation points and invalid points of the path trajectory and achieve efficient expression of the path trajectory.

[0158] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A method for searching ship paths and predicting arrival times based on AIS historical data, characterized in that: The following steps are included: S1. Grid feature encoding settings for AIS data: AIS data is obtained by establishing a connection between PySpark and ClickHouse databases. AIS data is cleaned based on geographic spatial features and trajectory features to remove data anomalies. AIS data gridding method based on Google S2 spatial coding principle is used to divide the earth's surface into small blocks with variable precision, and gridding statistics are performed in combination with the spatiotemporal characteristics of AIS data. S2. AIS trajectory network and weight construction: The neighborhood points of a given grid point are determined by the point neighborhood dynamic selection method based on the KNN algorithm, and they are connected into edges according to their location information to construct the AIS trajectory network. The edge weight determination method based on the dynamic weight strategy is used to introduce a dynamic weight strategy in the edge weight assignment of the AIS trajectory network, and the weight is dynamically adjusted according to the daily density of ships. S3. Ship path search and simplification: The directed path search optimization method based on the Bellman-Ford algorithm is used to realize the shortest path search for ships, and the path trajectory simplification method based on Kalman filtering is used to remove the fluctuation points and invalid points of the path trajectory to realize the simplification of the path trajectory.

2. The method for ship path search and arrival time prediction based on AIS historical data according to claim 1, characterized in that: In step S1, obtaining AIS data specifically includes the following contents: S111. Establish a connection between PySpark and ClickHouse database and configure the connection parameters, including the host address, port, and user credentials of ClickHouse; S112, achieving parallel access to AIS data in the database through time-index-based partitioning; dividing the AIS data into appropriate time windows to improve data retrieval efficiency; S113, preprocessing the acquired AIS data, including processing missing values, cleaning abnormal data, and performing feature engineering on the data; S114. Use GIS visualization tools to gain in-depth understanding of the distribution, trends, and correlations of pre-processed AIS data.

3. The method for ship path search and arrival time prediction based on AIS historical data according to claim 1, characterized in that: In step S1, data cleaning based on geographic spatial features is specifically as follows: The AIS data are cleaned based on the global continental shelf edge contour data, and the AIS data within the continental shelf contour are deleted.

4. The method for ship path search and arrival time prediction based on AIS historical data according to claim 1, characterized in that: Data cleaning based on trajectory features includes cleaning of position drift anomaly points. The cleaning of position drift anomaly points is specifically as follows: When the ship drifts at a certain track point, it will be different from the previous and next track points P i-1 and P i+1 The smaller the angle, the smaller the angle. If the front and rear track segments of the ship at point Pi satisfy the following formula, it is considered to be an abnormal point caused by position drift and will be removed. Dth Pi <d θ ,Δt Pi <d t Among them, Δθ Pi is the trajectory point P i and the front and rear trajectory points P i-1 and P i+1 The angle of the formed trajectory segment; δ θ is the angle threshold; Δt Pi is the time interval between three trajectory points; δ t is the time interval threshold.

5. The method for ship path search and arrival time prediction based on AIS historical data according to claim 1, characterized in that: Data cleaning based on trajectory features includes cleaning of abnormal ship speed points. The specific cleaning of abnormal ship speed points is as follows: Let trajectory point P i Speed ​​V i Only the speed V of the first two adjacent track points i-2 and V i-1 Related, t i-2 ,t i-1 ,t i are the sampling times of the three trajectory points respectively, then the trajectory point P i The predicted speed V i ′ is expressed as, V′ i =V i-1 +(V i-1 -V i-2 ) / (t i-1 -t i-2 )×(t i -t i-1 ) If the predicted speed value V′ i With the original recorded value V i The error between them is greater than the speed change threshold, i.e. |V′ i -V i |>δ θ ′, then the ship is considered to be at the trajectory point P i The speed value is an abnormal value, and the predicted speed V′ i Replaces the original speed V i ; Among them, δ θ ′ is the speed change threshold, which is estimated by the speed of the previous trajectory point, that is, δ′ θ = k × V i-1 , the k value range is (0, 1).

6. The method for ship path search and arrival time prediction based on AIS historical data according to claim 1, characterized in that: In step S1, the AIS data gridding method based on the Google S2 spatial coding principle specifically includes the following contents: S121, based on the Google S2 spatial coding principle, the global network is gridded and encoded at intervals of preset distances; S122, on the basis of sorting the AIS data by time and segmenting the track, counting each segment of the AIS track data on a ship-by-ship basis; During the statistics process, based on the statistical results of the previous AIS track point, only several grids around the previous grid are called to perform statistics on the next AIS track; S123, during the statistical process, the number of AIS track points in the grid is determined. When the data of the points in the grid exceeds the set threshold and the size of the grid is greater than the threshold for grid refinement, the grid is subdivided according to the Google S2 spatial coding principle, and the number of AIS track points in each grid is re-counted; S124, repeat steps S122 to S123 until all AIS data are counted; S125, selecting grid center points in which the number of AIS track points in the grid is greater than a preset number to form a grid matrix.

7. The method for ship path search and arrival time prediction based on AIS historical data according to claim 1, characterized in that: In step S2, the point neighborhood dynamic selection method based on the KNN algorithm specifically includes the following contents: S211, selecting any grid center point in the grid matrix as a judgment point; S212, based on the spatial coding of the grid matrix, selecting the grid center points adjacent to the judgment point as the initial judgment group; S213, calculating the Euclidean distance between the starting point and all points in the initial judgment group, and selecting the three points with the smallest distance as the connecting points of the judgment points, and selecting any one of them as a new judgment; S214. Repeat steps S212 to S213 until all points that can be connected are connected to form an AIS trajectory network.

8. The method for ship path search and arrival time prediction based on AIS historical data according to claim 1, characterized in that: In step S2, the edge weight determination method based on the dynamic weight strategy specifically includes the following five methods: (1) For two connected points, the edge weight is the inverse of the sum of the weights of the two points; (2) For two connected points, the edge weight is the mean of the weights divided by the sum of the weights of the two points; (3) For two connected points, if the point weight is greater than [50, 100, 300, 500], reset its weight to [50, 100, 300, 500], and the edge weight is the inverse of the sum of the two point weights; (4) Perform Max-Min normalization on the weights of the points. For two connected points, the edge weight is the sum of the normalized weights of the two points. (5) Perform Z-score normalization on the point weights. For two connected points, the edge weight is the inverse of the sum of the normalized weights of the two points.

9. The method for ship path search and arrival time prediction based on AIS historical data according to claim 1, characterized in that: In step S3, the directed path search optimization method based on the Bellman-Ford algorithm specifically includes the following contents: S311, initializing the distances of all nodes except the source node to infinity, and initializing the distance of the source node to 0; S312, for each node in the graph, perform relaxation operation in turn; the relaxation operation refers to trying to shorten the path distance through the edge from one node to another node, if the path through a node is shorter than the path directly reaching the target node, then update the distance of the target node to the smaller path distance; S313, repeating the iteration operation N-1 times, where N is the number of vertices in the graph; in each iteration, performing a relaxation operation on each edge; S314, after completing N-1 iterations, perform the Nth iteration operation; if relaxation operation can still be performed in the Nth iteration, it means that there is a negative weight cycle in the graph; if a negative weight cycle is detected, the output result of the algorithm is invalid; S315. The final distance array is the shortest path from the source node to all other nodes.

10. The method for ship path search and arrival time prediction based on AIS historical data according to claim 1, characterized in that: In step S3, the path trajectory simplification method based on Kalman filtering specifically includes the following contents: S321, data initialization, including initialization of initial state vector, state transfer matrix, measurement matrix, noise parameters and initial error covariance matrix; S322, selecting path points in sequence, and calculating a state prediction matrix and a covariance matrix according to the Kalman gain, the measurement matrix and the strategy amount; S323, looping through step S322 until all points are traversed to obtain a path smoothing trajectory point set; S324, sequentially calculating the direction of the path smooth trajectory point set and the rate of change of the direction; S325, determining whether there is a situation in the point set where the direction change rate is continuously less than the direction change threshold, if so, retaining only the first and last two points, deleting the remaining points, and obtaining a simplified path trajectory.

Images