Ship abnormal trajectory real-time discrimination method based on stream-oriented computation

Abstract

The invention discloses a ship abnormal trajectory real-time discrimination method based on stream-oriented computation, which is applied to identification of ship navigation abnormal trajectories in a specific area of an inland river basin based on ship AIS data. Comprising the following steps: a ship AIS data preprocessing module receives AIS data and performs operations such as de-duplication, analysis and filtering, and the processed data is sent to a track acquisition module and an abnormal track identification module; the ship track acquisition module realizes acquisition and storage of ship navigation tracks in different electronic fence areas by judging ship navigation states; the ship track clustering module is used for clustering tracks of different ships on the basis of an improved DBSCAN (Density Based Spatial Clustering of Applications with Noise), and counting the navigation course and speed of the ships according to a clustering result and grids; and the abnormal trajectory recognition module recognizes the abnormal trajectory of the ship in real time based on the clustering and grid statistical results. Monitoring of the ship navigation trajectory and identification of the abnormal trajectory are realized, and improvement of the management efficiency of water traffic and reduction of the occurrence rate of water traffic accidents are facilitated.

Description

Technical Field

[0001] The present invention relates to the application in the field of intelligent monitoring of inland vessels, and specifically to a real-time identification method of abnormal ship trajectories based on streaming computing. Background Art

[0002] As an important source of ship supervision data, the Automatic Identification System (AIS) contains key information such as the ship's real-time position, speed, and heading. Real-time tracking and identification of ship trajectories is one of the key technologies to improve the level of intelligent maritime management due to its important application value.

[0003] Real-time identification of abnormal ship trajectories is a key issue that needs to be addressed in the intelligent supervision of inland ships, especially in some important waterway areas. This problem is a typical computationally intensive task and it is difficult to meet real-time requirements. Traditional methods are based on offline computing and cannot meet the needs of real-time supervision. This method is based on streaming computing, combined with offline batch processing for ship clustering calculation, and realizes abnormal ship trajectory identification through stream-batch fusion. Streaming computing realizes real-time collection of ship trajectories and identification of abnormal trajectories. The ship trajectory clustering module realizes ship trajectory clustering and classifies normal ship trajectories and abnormal ship trajectories. Improving the accuracy and efficiency of identifying abnormal ship trajectories and improving the level of intelligent ship supervision through stream-batch fusion has urgent needs and broad application prospects in the field of smart maritime.

[0004] Therefore, it is necessary to provide a real-time identification method of abnormal ship trajectories based on streaming computing to solve the above technical problems. Summary of the invention

[0005] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0006] The purpose of the present invention is to make up for the shortcomings of streaming computing that cannot perform complex calculations and batch processing in real time through stream-batch fusion, that is, to ensure the real-time performance of ship abnormal trajectory identification and improve the accuracy and efficiency of ship abnormal trajectory identification.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a real-time identification method of abnormal ship trajectory based on streaming computing, comprising the following steps:

[0008] Step 1: AIS data preprocessing, pushing the ship's navigation status and AIS data in the electronic fence area to the ship track acquisition module and the ship track anomaly recognition module;

[0009] Step 2: Ship trajectory collection;

[0010] Step 3: Ship trajectory clustering;

[0011] Step 4: Identification of abnormal ship trajectories.

[0012] As a preferred embodiment of the method for real-time identification of abnormal ship trajectories based on streaming computing of the present invention, step 1 comprises the following steps:

[0013] Step 1a: Use a streaming computing-based AIS data cleaning and indicator data analysis method to achieve data deduplication, parsing, and erroneous data filtering.

[0014] Step 1b: Use a method based on a stream processing engine to analyze and calculate the pre-processed ship position data to determine whether the ship is in the electronic fence area to determine the current state of the ship; if the ship is not in the electronic fence area and the current state is not sailing in the fence, execute step 1a to continue receiving the next packet of AIS data. Otherwise, execute step 1c;

[0015] Step 1c: To update the ship status and ship position information, the longitude and latitude need to be converted into a rectangular coordinate system;

[0016] Step 1d: The current state of the ship, the fence number and the ship trajectory are pushed to the trajectory collection task and the abnormal trajectory identification task through the message queue.

[0017] As a preferred embodiment of the method for real-time identification of abnormal ship trajectories based on streaming computing of the present invention, step 2 comprises the following steps:

[0018] Step 2a: Get the ship's navigation status data and AIS data from the message queue;

[0019] Step 2b: Determine the navigation status of the ship. If it is a newly entered state, call step 2c; if it is a navigation state, call step 2d; if it is a departure state, call step 2e;

[0020] Step 2c: Add a new record for the current ship in the cache, including the electronic fence number, the ship MMSI code and the ship track. The ship track includes the intersection point between the ship and the electronic fence and the current position of the ship;

[0021] Step 2d: Add the current ship's position data to the ship track list, and jump to step 2a to continue receiving and processing data;

[0022] Step 2e: Store the current record in the offline data warehouse and delete the current record from the cache.

[0023] As a preferred embodiment of the method for real-time identification of abnormal ship trajectories based on streaming computing of the present invention, step 3 comprises the following steps:

[0024] Step 3a: retrieve the trajectory set of the ship sailing in the electronic fence area from the data warehouse according to the electronic fence number, take the intersection point of the ship entering the electronic fence with the fence boundary or the position where the ship changes from the moored state to the sailing state as the first trajectory point, and take the intersection point of the ship leaving the electronic fence area with the fence boundary or the position of the ship after mooring as the last trajectory point;

[0025] Step 3b: Since the number of trajectory points of each ship trajectory is inconsistent, in order to unify the length of each trajectory sample and evenly distribute it on the trajectory, 10 points are sampled for each trajectory according to the mean value, and the sampling results are stored in the cache with the electronic fence number + ship MMSI as the key value and the sampled trajectory as the value.

[0026] Step 3c: Take all the ship trajectories in the fence area from the cache according to the electronic fence number, and cluster the converted sample trajectory set based on the improved DBScan algorithm. When the data is initial, mark all sampled trajectories as unprocessed, randomly select a trajectory object trajectory, and check whether other trajectories are within the scope of the area with the object as the center;

[0027] Step 3d: Analyze the navigation characteristics of ships in each cluster, and count the average, maximum and minimum navigation direction and speed of ships in the cluster by grid;

[0028] Step 3e: Store the clustering results and the navigation characteristics of each cluster in the database.

[0029] As a preferred embodiment of the method for real-time identification of abnormal ship trajectories based on stream computing of the present invention, step 4 comprises the following steps:

[0030] Step 4a: Receive the ship's heading status and navigation data from the message queue;

[0031] Step 4b: Identify the ship status and determine whether the ship status is out of the fence or anchored. If yes, execute step 4e;

[0032] Step 4c: According to the electronic fence area and the grid where the ship is located, when the heading of the ship and the heading of the normal cluster of the grid do not exceed the threshold or there are insufficient track points, jump to 4a;

[0033] Step 4d: predicting the ship's trajectory based on the ship's existing trajectory points;

[0034] Step 4e: Convert the predicted ship trajectory into 10 sampling points and compare them with the trajectory in the normal navigation ship trajectory cluster. If it does not belong to the normal trajectory cluster, the ship trajectory is considered abnormal and an alarm information is pushed to the message queue.

[0035] The beneficial effects of the present invention are as follows: (1) The present invention constructs a four-level processing architecture of AIS data preprocessing, ship trajectory collection, ship trajectory clustering, and ship trajectory anomaly identification based on streaming computing technology, and adopts distributed parallel processing, so that the present invention has the advantages of high efficiency and easy software and hardware implementation, and can realize the process of real-time identification of ship trajectory anomalies.

[0036] (2) In the present invention, a real-time and offline integrated processing architecture is adopted. The offline module analyzes the ship's navigation trajectory based on the improved DBScan method to analyze the normal navigation status of the ship; the real-time module identifies the ship's trajectory status in real time through streaming computing technology; based on offline computing resources and storage capabilities, it can store and process trajectory information within a longer time range, ensuring the accuracy of offline analysis; the real-time module only compares with the analysis results to ensure rapid response and real-time performance. Through the real-time and offline integrated processing architecture, both the accuracy and real-time performance of the method are guaranteed. The present invention utilizes an offline and real-time integrated architecture, based on a streaming computing architecture, which can make up for the shortcomings of complex algorithms, ensuring the real-time performance of ship abnormal trajectory identification, and improving the accuracy and efficiency of ship abnormal trajectory identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0038] in:

[0039] Figure 1 This is a system architecture diagram of the method for real-time identification of abnormal ship trajectories based on streaming computing according to the present invention;

[0040] Figure 2 It is an overall flow chart of the real-time identification method of abnormal ship trajectory based on streaming calculation according to the present invention;

[0041] Figure 3 A flow chart of ship navigation status update of the method for real-time identification of abnormal ship trajectories based on streaming computing according to the present invention;

[0042] Figure 4 The AIS data format of the real-time identification method of abnormal ship trajectory based on streaming calculation described in the present invention;

[0043] Figure 5 The ship trajectory collection process in the electronic fence area of ​​the real-time identification method of abnormal ship trajectory based on streaming computing of the present invention;

[0044] Figure 6 It is a flow chart of ship trajectory clustering of the real-time identification method of abnormal ship trajectory based on streaming computing according to the present invention;

[0045] FIG7 is a diagram showing the ship trajectory clustering result of the real-time identification method for abnormal ship trajectory based on streaming computing according to the present invention;

[0046] Figure 8 It is a flow chart of ship track abnormality identification of the real-time ship track abnormality identification method based on streaming computing according to the present invention;

[0047] Fig. 9 This is a diagram of the ship trajectory recognition result of the real-time identification method of abnormal ship trajectory based on streaming computing described in the present invention. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, but not all of the embodiments.

[0049] Example 1

[0050] As attached Figure 1As shown in the figure, it is a system architecture diagram of an implementation scheme of a real-time identification method of ship trajectory anomaly based on streaming computing of the present invention, including a data acquisition layer, a data computing layer, a data storage layer and a data visualization layer. The data acquisition layer realizes data collection and information transmission between modules through the Kafka message queue. The data computing layer is divided into real-time computing and offline computing. The real-time computing realizes three tasks of preprocessing of ship trajectory, collection of ship trajectory and identification of abnormal ship trajectory through the Flink framework, and completes the computing tasks through three different clusters respectively. Application coordination services are realized through Zookeeper between clusters and between clusters, providing consistency services for distributed computing of abnormal ship trajectory identification. In each computing cluster, it is composed of a master node, a resource scheduling node and a working node, which jointly complete the calculation of the task module. Offline computing realizes clustering of ship trajectory based on Spark cluster, and uses PySpark API to realize the calculation of clustering tasks. Data storage uses Redis to realize the cache of ship status and intermediate settlement results, the storage of ship trajectory is realized through Hive data warehouse, and the storage of clustering results is realized through MySQL database. The data visualization layer uses the front-end and back-end separation technology to display the ship's trajectory and potential abnormal behavior. The back-end implements the interface based on Spring Boot, and the front-end uses Vue+Mapbox to display the ship's trajectory on the map.

[0051] As attached Figure 2 As shown, a real-time identification method of abnormal ship trajectory based on streaming computing includes:

[0052] Step 1: AIS data preprocessing,

[0053] The first streaming computing Flink cluster collects ship AIS data through the message queue, and then removes duplicates, parses, and filters out erroneous data. Figure 3 As shown, the method traverses all electronic fences. The electronic fence data is loaded from the MySQL database to the memory when the method is started and stored in the Flink cluster in the form of shared variables. The cluster calculates whether the ship is in a certain electronic fence area and its navigation status, and pushes the ship status and position in the electronic fence area to the ship trajectory collection module and the ship trajectory anomaly recognition module;

[0054] Step 2: Ship track collection

[0055] As attached Figure 5As shown in the figure, the second streaming computing Flink cluster collects the ship navigation status and AIS data pushed by the first Flink cluster through the Kafka message queue, and stores the ship navigation tracks in the electronic fence area within a cycle in the offline data warehouse Hive. Hive stores data based on the HDFS (distributed file system) cluster and uses the Parquet column storage architecture to store ship track information.

[0056] Step 3: Ship track clustering

[0057] Since the navigation track of inland ships is affected by seasons, water levels, weather, etc., in order to ensure that the latest ship trajectory analysis results are used for real-time identification of ship trajectories, this module takes out the electronic fence set from the MySQL database at regular intervals every day, traverses the electronic fence numbers, and takes out the latest ship trajectory set of the day from the Hive data warehouse according to the electronic fence numbers. Based on the Spark cluster, the trajectory is converted into a sample set through PySpark, and the converted samples are clustered based on the improved DBScan algorithm. The clustering results are analyzed, and the clustering results and the navigation characteristics of each cluster are stored in the MySQL database according to the electronic fence numbers;

[0058] Step 4: Identification of abnormal ship trajectories

[0059] The third streaming computing Flink cluster collects the ship navigation status and AIS data pushed by the first Flink cluster through the Kafka message queue, converts the trajectory points into sample points, and compares them with the normal navigation trajectory cluster. If it does not belong to the normal trajectory cluster, the ship trajectory is considered abnormal and the abnormal ship trajectory is pushed to the Kafka message queue.

[0060] Step 1a: Use an AIS data cleaning and indicator data analysis method based on stream computing to achieve data deduplication, parsing, and erroneous data filtering. Related technical references: Lv Taizhi, Tang Peiyi, Zhang Juan. A Real-Time AIS Data Cleaning and Indicator Analysis Algorithm Based on Stream Computing [J]. Scientific Programming, 2023, 2023 (1): 8345603; In the implementation, the TTL feature in the Redis cache database is used to filter duplicate data through the Flink filter operator. According to the attached Figure 4 The AIS data is converted from the original message to plain text based on the JSON format through the map operator of Flink, and then the erroneous data is corrected and filtered through the map and filter operators.

[0061] Step 1b: Based on the five states of the current state of the ship (entering the fence, sailing in the fence, leaving the fence, the ship is anchored, and not in the fence area), a method based on a stream processing engine to analyze and calculate the pre-processed ship position data is used to determine whether the ship is in the electronic fence area to determine the current state of the ship. Related technical reference documents: Lv Taizhi, Zhang Juan, Wu Enze, et al. A ship monitoring method and system based on virtual electronic fence [P]. Jiangsu Province: CN202410082289.X, 2024-05-03. If the ship position is not in the electronic fence area and the current state is not sailing in the fence, execute step 1a to continue receiving the next package of AIS data. Otherwise, execute step 1c; the method is divided into two steps, first based on simplified area judgment, and then based on the accurate judgment of the ray method. If the ship position is not in the electronic fence area and the current state is not sailing in the fence, execute step 1a to continue receiving the next package of AIS data. If the previous state of the ship is out of the fence, set the ship state to not in the fence area, and execute step 1a to continue receiving the next package of AIS data and process it. If the last status of the ship is out of the fence, the ship status is set to not in the fence area, and step 1a is executed to continue receiving the next packet of AIS data and processing it. Otherwise, step 1c is executed;

[0062] Step 1c: Update the ship status and ship position information. If the ship is in or out of the electronic fence, calculate the intersection of the ship's navigation track and the electronic fence boundary. Connect the ship's previous coordinate position and the current coordinate position as a line segment, and calculate whether the line segment has an intersection with each line segment of the electronic fence and the location of the intersection. First, convert the longitude and latitude into a rectangular coordinate system. The mathematical expression formula is as follows:

[0063]

[0064] Where lng and lat are the longitude and latitude of the trajectory point, R is the radius of the earth, PI is the pi, x and y are the converted rectangular coordinate values, y m Represents the intermediate variable. Calculate the intersection point based on the rectangular coordinate system, assuming that the coordinate position of the intersection point is x p ,y p , convert it to latitude and longitude lng p ,lat p The mathematical expression is as follows:

[0065]

[0066] Where e is the natural base.

[0067] Step 1d: The current status of the ship, the fence number and the ship trajectory are pushed to the trajectory collection task and the abnormal trajectory identification task through the Kafka message queue.

[0068] Step 2a: The ship trajectory collection Flink cluster retrieves the ship navigation status data and AIS data from the Kafka message queue;

[0069] Step 2b: Determine the navigation status of the ship. If the status is newly entering the fence, call step 2c; if the status is sailing in the fence, call step 2d; if the status is out of the fence or the ship is moored, call step 2e;

[0070] Step 2c: In the Redis cache, the electronic fence number and the ship MMSI are combined as the key value to add a record for the current ship, including the electronic fence number, the ship MMSI code and the ship track. The ship track includes the intersection point of the ship and the electronic fence and the current navigation status of the ship (including latitude and longitude, navigation, speed, etc.);

[0071] Step 2d: Use the electronic fence number and MMSI to search the Redis cache for the current ship track information, add the current ship's location information to the ship track list, and jump to step 2a to continue receiving and processing data;

[0072] Step 2e: Use the electronic fence number and MMSI to search the Redis cache for the current ship track information, add the current position to the track, store the current ship navigation track in the offline data warehouse Hive, and delete the current record from the Redis cache;

[0073] Step 3a: Extract the trajectory set of ships sailing in the electronic fence area from the data warehouse Hive according to the electronic fence number, and take the intersection point of the ship with the fence boundary when entering the electronic fence or the position where the ship changes from the moored state to the sailing state as the first trajectory point, and take the intersection point of the ship with the fence boundary when leaving the electronic fence area or the position of the ship after mooring as the last trajectory point.

[0074] Step 3b: Since the number of points in each ship track is inconsistent, in order to make the length of each track sample consistent and evenly distributed on the track, 10 points are sampled for each track according to the mean value, and the sampling results are stored in the Redis cache with the electronic fence number + ship MMSI as the key value and the sampled track as the value. First, the ship's navigation distance is calculated. The distance between track points i and i+1 is

[0075]

[0076] where lng i and lat iis the coordinate of the ith trajectory point, where lng i+1 and lat i+1 is the coordinate of the i+1th trajectory point, R is the radius of the earth, D i is the distance between two trajectory points.

[0077]

[0078] Where n is the number of track points, D is the total distance the ship travels, 10 points are sampled on the track, and the average length of each track segment AvgD is

[0079]

[0080] Assume that the ship sampling point S i The coordinate position is and The coordinate position of the next sampling point and for

[0081]

[0082] Course i It is the mean of the sum of the azimuths of the two trajectory points close to the sampling point, and DMP is the increasing latitude difference between the two points.

[0083]

[0084] MP is the length of the meridian arc, which increases with the latitude.

[0085]

[0086] Step 3c: According to the electronic fence number, a sampled ship track set is retrieved from the Redis cache, using the following method: Figure 6 The process is based on the improved DBScan to cluster the converted sample trajectory set. When the initial data is collected, all sampled trajectories are marked as unprocessed, and a trajectory object trajectory is randomly selected. Check whether other trajectories are within the scope of the area with trajectory as the center. The distance calculation method of two trajectories is as follows:

[0087] What is

[0088]

[0089] Where D iis the distance difference between the corresponding index coordinate positions of the two ship trajectories on the corresponding index. For example, D1 is the distance difference between the second sampling point of the first ship trajectory and the second sampling point of the second ship trajectory. The distance between the two trajectory points is calculated using formula (7) and formula (8).

[0090] If there are no less than MinPts (density clustering) trajectories in the neighborhood centered on trajectory, a new cluster label is assigned to the trajectory, and the ship trajectory in the neighborhood is marked as processed. In the implementation of this application, the neighborhood range is set to the width of the ship channel in the electronic fence area. MinPts is set to the number of trajectories The trajectories that are not within the domain are marked as noise trajectories. Take out an unprocessed ship trajectory and repeat the above steps until all trajectories are marked as processed. When all trajectories are visited and classified, the algorithm terminates and finally returns the label set of the cluster to which each trajectory belongs. Figure 7a and 7b The results of ship trajectory clustering in two electronic fence areas are shown. The ship trajectories are mainly clustered into upstream ships, downstream ships and outlier trajectories, and the outlier trajectories are marked as abnormal ship trajectories.

[0091] Step 3d: Analyze the navigation characteristics of each cluster, and count the average, maximum and minimum navigation direction and speed of the cluster by grid. Divide the electronic fence area into 0.5km*0.5km grids, draw a vertical and a horizontal line segment with the center point of the grid, if a track segment in a track has an intersection with these two line segments, it means that the ship is sailing in the grid area, and its speed and heading are calculated as follows:

[0092]

[0093] where pos start Indicates the starting point of a certain segment of a trajectory, pos end Indicates the end point of a certain trajectory, pos p is the intersection point of the track segment with the horizontal or vertical line segment at the center of the grid, and distance is the distance function between two position points calculated according to formulas (7) and (8). start 、speed start Indicates the ship's heading and speed at the starting point, course end 、speed end Indicates the ship's course and speed at the destination; course i and speed i Represents the heading and speed of this trajectory in the i-th grid.

[0094] The average heading of the ith grid and average speed The calculation is as follows,

[0095]

[0096] Course i,j and speed i,j is the heading and speed of the jth trajectory of the i-th grid network.

[0097] Step 3e: Store the clustering results and the navigation characteristics of each cluster in the MySQL database. The clustering results are stored in the clustering result ais_cluster table in the MySQL database, including the electronic fence number, cluster number, number of cluster tracks, clustered ship tracks, update time and other field information. If it is the first clustering, the record is inserted. If it is clustered again later, the clustering results are updated according to the electronic fence number. The grid statistics results are stored in the ais_grid table, including the electronic fence number, grid number, grid coordinates, average navigation, average speed, maximum navigation, maximum speed, minimum navigation, minimum speed, update time and other information.

[0098] Furthermore, the above step 4 includes the following steps:

[0099] Step 4a: Receive the ship heading status and navigation data from the Kafka (subscription messaging system) message queue;

[0100] Step 4b: Identify the ship status and determine whether the ship has left the electronic fence or has been moored. If so, execute step 4e;

[0101] Step 4c: According to the electronic fence area and grid where the ship is located, the average, highest, and lowest headings and speeds of the ship in the current grid are retrieved from the Redis (remote dictionary service) cache. When the ship's heading does not exceed the threshold with the heading of the normal cluster of the grid or there are insufficient track points, jump to 4a. Here, the heading and speed thresholds are set to 150% of the maximum value and 70% of the minimum value. The track point is set to 150% of the maximum value and 70% of the minimum value when the ship's sailing distance is less than the average mileage of the ship passing through the electronic fence area. the following;

[0102] Step 4d: Predict the ship's trajectory based on the ship's existing trajectory points. Assume that the ship's trajectory mileage after entering the electronic fence area is distance ship The average distance that ships pass through the electronic fence area is distance avg , the average receiving time interval of AIS message is t interval , the final trajectory point is lng t-1 ,latt-1 , by iteratively predicting the trajectory behind the ship.

[0103] Since the ship is sailing at a relatively constant speed, it is assumed to be a linear system. The current acceleration and steering angle of the ship are predicted by linear regression of the existing trajectory of the ship. pd and turn pd .

[0104] distance left =distance avg -distance ship (20)

[0105] speed t =speed t-1 +acce pd (twenty one)

[0106] course t =course t-1 +turn pd (twenty two)

[0107] S t =speed t-1 ×t interval (twenty three)

[0108] distance left =distance avg -S t (twenty four)

[0109] lat t =lat t-1 +S t ×cos(course t ) (25)

[0110] lng t =long t-1 +DMP×tan(course t ) (26)

[0111] The DMP is calculated according to formula (13), and the iteration cycle is repeated until the ship trajectory exceeds the average mileage of the ship electronic fence area, i.e., distance left Less than 0;

[0112] Step 4e: Convert the predicted ship trajectory into 10 sampling points and compare them with the trajectory in the normal navigation ship trajectory cluster. If it does not belong to the normal trajectory cluster, the ship trajectory is considered abnormal and the alarm information is pushed to the Kafka message queue.

[0113] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A real-time identification method of abnormal ship trajectory based on streaming computing, characterized in that: The following steps are involved: Step 1: AIS data preprocessing: The streaming computing cluster collects ship AIS data through the message queue, and after deduplication, parsing, and error data filtering operations, calculates whether the ship is in the electronic fence area and its navigation status; Step 2: Ship track collection. The streaming computing cluster receives the ship navigation status and AIS data pushed in step 1 through the message queue, and stores the ship navigation track in the electronic fence area within a cycle in the offline data warehouse; Step 3: Ship track clustering: Regularly retrieve the ship track set from the offline data warehouse according to the electronic fence number, convert the track into a sample set, cluster the converted samples based on the improved DBScan, analyze the clustering results, and store the clustering results and the navigation characteristics of each cluster; Step 4: Identification of abnormal ship trajectories. The streaming computing cluster receives the ship navigation status and AIS data pushed in step 1 through the message queue, converts the trajectory points into sample points, and compares them with the normal navigation trajectory cluster. If it does not belong to the normal trajectory cluster, the ship trajectory is considered abnormal and the abnormal ship trajectory is pushed to the message queue.

2. According to claim 1, the method for real-time identification of abnormal ship trajectories based on streaming computing is characterized in that: The step 1 comprises the following steps: Step 1a: Use a streaming-based AIS data cleaning and indicator data analysis method to achieve data deduplication, parsing, and erroneous data filtering; Step 1b: Based on the current state of the ship, a method based on a stream processing engine is used to analyze and calculate the pre-processed ship position data to determine whether the ship is in the electronic fence area, so as to determine the current state of the ship; If the ship is not in the electronic fence area and the current state is not sailing in the fence, execute step 1a to continue receiving the next packet of AIS data; otherwise, execute step 1c; Step 1c: Update the ship status and ship position information. If the ship is in the state of entering or exiting the electronic fence, calculate the intersection of the ship's navigation track and the boundary of the electronic fence; Connect the last coordinate position and the current coordinate position of the ship into a line segment, calculate whether the line segment and each line segment of the electronic fence boundary have an intersection and the location of the intersection, first convert the longitude and latitude into a rectangular coordinate system; Where lng and lat are the longitude and latitude of the trajectory point, R is the radius of the earth, PI is the pi, and x and y are the converted rectangular coordinate values; the intersection point is calculated based on the rectangular coordinate system, assuming that the coordinate position of the intersection point is x p ,y p , convert it to latitude and longitude lng p ,lat p ; Where e is the natural base; Step 1d: The current state of the ship, the fence number and the ship trajectory are pushed to the trajectory collection task and the abnormal trajectory identification task through the message queue.

3. According to claim 1, the method for real-time identification of abnormal ship trajectories based on streaming computing is characterized in that: The step 2 comprises the following steps: Step 2a: Get the ship's navigation status data and AIS data from the message queue; Step 2b: Determine the navigation status of the ship. If it is a newly entered state, call step 2c; if it is a navigation state, call step 2d; if it is a departure state, call step 2e; Step 2c: Add a new record for the current ship in the cache, including the electronic fence number, the ship MMSI code and the ship track. The ship track includes the intersection point between the ship and the electronic fence and the current position of the ship; Step 2d: Add the current ship's position data to the ship track list, and jump to step 2a to continue receiving and processing data; Step 2e: Store the current record in the offline data warehouse and delete the current record from the cache.

4. According to claim 1, the method for real-time identification of abnormal ship trajectories based on streaming computing is characterized in that: The step 3 comprises the following steps: Step 3a: retrieve the trajectory set of the ship sailing in the electronic fence area from the data warehouse according to the electronic fence number, take the intersection point of the ship entering the electronic fence with the fence boundary or the position where the ship changes from the moored state to the sailing state as the first trajectory point, and take the intersection point of the ship leaving the electronic fence area with the fence boundary or the position of the ship after mooring as the last trajectory point; Step 3b: Since the number of points in each ship trajectory is inconsistent, in order to make the length of each trajectory sample consistent and evenly distributed on the trajectory, 10 points are sampled for each trajectory according to the mean value, and the sampling results are stored in the cache with the electronic fence number + ship MMSI as the key value and the sampled trajectory as the value; first calculate the ship's navigation distance, the distance between trajectory points i and i+1 is where lng i and lat i is the coordinate of the ith trajectory point, where lng i+1 and lat i+1 is the coordinate of the i+1th trajectory point, R is the radius of the earth, D i is the distance between two trajectory points; Where n is the number of trajectory points, D is the total distance the ship travels, 10 points are sampled on the trajectory, and the average length of each trajectory AvgD is Assume that the ship sampling point S i The coordinate position is and The coordinate position of the next sampling point and for Course i It is the mean of the sum of the heading angles of two trajectory points close to the sampling point, and DMP is the difference in the gradually increasing latitudes of the two points; Where MP is the length of the meridian arc that increases with latitude as the latitude increases on the ordinate of the Mercator chart; Step 3c: Take all the ship trajectories in the fence area from the cache according to the electronic fence number, and cluster the converted sample trajectory set based on the improved DBScan algorithm; when initializing data, mark all sampled trajectories as unprocessed, randomly select a trajectory object trajectory, and check whether other trajectories are within the area with trajectory as the center. The distance calculation method of two trajectories is as follows: Where D i is the distance difference between the corresponding index coordinate positions of the two ship trajectories on the corresponding index. For example, D1 is the distance difference between the second sampling point of one ship trajectory and the second sampling point of another ship trajectory. The distance between two trajectory points is calculated using formula (7) and formula (8); Step 3d: Analyze the navigation characteristics of the ships in each cluster, and count the average, maximum and minimum navigation direction and navigation speed of the ships in the cluster by grid; divide the electronic fence area into 0.5km*0.5km grids, draw a vertical and a horizontal line segment with the center point of the grid, if a certain track in a certain track has an intersection with these two line segments, it means that the ship is sailing in the grid area, and its speed and heading are calculated as follows: where pos start Indicates the starting point of a certain segment of a trajectory, pos end Indicates the end point of a certain trajectory, pos p is the intersection point of the trajectory with the horizontal or vertical line segment at the center of the grid, and distance is the distance function between two position points calculated according to formulas (7) and (8); start 、speed start Indicates the ship's heading and speed at the starting point, course end 、speed end Indicates the ship's course and speed at the destination; course i and speed i Represents the heading and speed of this trajectory in the i-th grid; The average heading of the ith grid and average speed The calculation is as follows, Course i,j and speed i,j is the heading and speed of the jth trajectory of the i-th grid network; Step 3e: Store the clustering results and the navigation characteristics of each cluster in the database.

5. According to claim 1, the method for identifying and distinguishing abnormal ship trajectories based on streaming computing is characterized in that: The step 4 comprises the following steps: Step 4a: Receive the ship's heading status and navigation data from the message queue; Step 4b: Identify the ship status and determine whether the ship status is out of the fence or anchored. If yes, execute step 4e; Step 4c: According to the electronic fence area and the grid where the ship is located, when the heading of the ship and the heading of the normal cluster of the grid do not exceed the threshold or there are insufficient track points, jump to 4a; Step 4d: Predict the ship's trajectory based on the ship's existing trajectory points; assume that the ship's trajectory mileage after entering the electronic fence area is distance ship The average distance that ships pass through the electronic fence area is distance avg , the average receiving time interval of AIS message is t interval , the final trajectory point is lng t-1 ,lat t-1 , by iteratively predicting the trajectory behind the ship; Since the ship is sailing at a relatively uniform speed, it is assumed to be a linear system; the current acceleration and steering angle of the ship are predicted by linear regression of the existing trajectory of the ship. pd and turn pd ; Its mathematical expression formula is as follows; distance left =distance avg -distance ship (20) speed t =speed t-1 +acce pd (21) course t =course t-1 +turn pd (22) S t =speed t-1 ×t interval (23) distance left =distance avg -S t (24) lat t =lat t-1 +S t ×cos(course t ) (25) lng t =long t-1 +DMP×tan(course t ) (26) The DMP is calculated according to formula (13), and the iteration cycle is repeated until the ship trajectory exceeds the average mileage of the ship electronic fence area, i.e., distance left Less than 0; Step 4e: Convert the predicted ship trajectory into 10 sampling points and compare them with the trajectory in the normal navigation ship trajectory cluster. If it does not belong to the normal trajectory cluster, the ship trajectory is considered abnormal and an alarm information is pushed to the message queue.

Images