Improved ship track data processing method, device, equipment and medium

Abstract

The invention discloses an improved ship track data processing method and device, equipment and a medium. The method comprises the following steps: periodically obtaining AIS data of each ship in a target sea area; the AIS data comprises a timestamp, real-time navigation data and a navigation state code; preprocessing the AIS data to obtain an AIS data sequence of each ship; dividing the AIS data sequence based on the real-time navigation data and the navigation state code to obtain navigation segment data; performing data thinning based on inflection points of each segment in the navigation segment data to generate a thinned track list; and sorting based on the thinned track list to obtain the ship track of each ship. According to the method, the data storage amount is effectively reduced and the key track feature integrity is kept by adopting a sectional thinning strategy; the data processing efficiency is effectively optimized through a pre-segmentation mechanism based on state codes; the track segmentation accuracy is effectively improved through a dual verification mechanism fusing the motion characteristics of the navigation state code and the real-time navigation data.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to an improved method, device, equipment and medium for processing ship track data. Background Art

[0002] Currently, in the field of ship track monitoring technology, ship track analysis based on AIS data faces the following technical bottlenecks: Data redundancy and storage pressure: Traditional methods store and process all original AIS data. When the daily data volume reaches tens of millions of records, there are problems such as high occupancy rate of storage media and large data transmission bandwidth pressure, especially difficult to implement in edge computing scenarios.

[0003] Insufficient real-time processing ability: The time complexity of existing track generation algorithms is generally of the order of O(n²). In the face of a high-density ship group, the single-node processing delay can reach several hours, which cannot meet the real-time requirements of maritime supervision.

[0004] Track segmentation accuracy defect: Conventional methods only rely on speed thresholds for state division, which may lead to about track mis-segmentation phenomena. Summary of the Invention

[0005] The present invention aims to solve at least to some extent the problems of related technical limitations. For this purpose, the present invention proposes an improved method, device, equipment and medium for processing ship track data, which can efficiently process ship track data.

[0006] On the one hand, an embodiment of the present invention provides an improved method for processing ship track data, including the following steps: Periodically obtain AIS data of each ship in the target sea area; the AIS data includes a timestamp, real-time navigation data, and a navigation status code; Preprocess the AIS data to obtain an AIS data sequence of each ship; Segment the navigation segment data from the AIS data sequence based on the real-time navigation data and the navigation status code; Based on the inflection points of each segment in the navigation segment data, perform data thinning to generate a thinned track list; Based on the thinned track list, organize and obtain the ship tracks of each ship.

[0007] Optionally, the AIS data is stored in a time series database; periodically obtaining the AIS data of each ship in the target sea area includes the following steps: In response to the first cycle node, use multiple threads to periodically pull the AIS data of each ship in the target sea area and store it in the temporary buffer area of the corresponding thread; In response to the second periodic node, batch write the AIS data in the temporary buffer of each thread into the time series database periodically.

[0008] Optionally, the AIS data further includes a maritime mobile service identity code; preprocess the AIS data to obtain the AIS data sequence of each ship, including the following steps: Classify and sort the AIS data based on the maritime mobile service identity code and the timestamp to obtain the original AIS data sequence of each ship; Filter out abnormal data in the AIS data in the original AIS data sequence based on the real-time navigation data to obtain the target AIS data sequence of each ship.

[0009] Optionally, the real-time navigation data includes real-time longitude and latitude, real-time speed, and real-time heading; filtering out abnormal data in the AIS data in the original AIS data sequence based on the real-time navigation data includes the following steps: Take the first AIS data in the original AIS data sequence as the first AIS data; When the real-time speed of the first AIS data is greater than the maximum designed ship speed, remove the first AIS data from the original AIS data sequence, and return to execute the step of taking the first AIS data in the original AIS data sequence as the first AIS data until the real-time speed of the first AIS data is less than or equal to the maximum designed ship speed; Take the next AIS data of the first AIS data in the original AIS data sequence as the second AIS data; Calculate the track point distance between the first AIS data and the second AIS data based on the real-time longitude and latitude combined with the average radius of the earth; Calculate the average speed of the track point corresponding to the second AIS data based on the track point distance combined with the timestamp; When the difference ratio between the average speed and the real-time speed of the second AIS data is greater than the first threshold, determine that the second AIS data is abnormal speed data; Calculate the actual course angle between the track points of the first AIS data and the second AIS data based on the real-time longitude and latitude; When the difference between the real-time course angle of the second AIS data and the actual course angle is greater than the second threshold, determine that the second AIS data is abnormal course data; If the second AIS data is abnormal speed data or abnormal course data, remove the second AIS data from the original AIS data sequence and use the next AIS data of the second AIS data in the original AIS data sequence as the second AIS data, and return to execute the step of calculating the track point distance between the first AIS data and the second AIS data based on the real-time latitude and longitude combined with the average radius of the earth. Otherwise, use the second AIS data as the first AIS data, and return to execute the step of using the next AIS data of the first AIS data in the original AIS data sequence as the second AIS data until all AIS data in the original AIS data sequence are traversed.

[0010] Optionally, the real-time navigation data includes real-time latitude and longitude and real-time speed; splitting the navigation segment data from the AIS data sequence based on the real-time navigation data and the navigation status code includes the following steps: Determine the ship status corresponding to each AIS data in the AIS data sequence based on the real-time navigation data and the navigation status code; Among them, the ship status includes a berthing status and a navigation status; when the navigation status code of the AIS data is of the first type, the real-time speed does not exceed the third threshold, and the real-time latitude and longitude are within the berthing area, determine that the ship status corresponding to the AIS data is the berthing status; when the navigation status code of the AIS data is of the second type and the real-time speed is above the fourth threshold, determine that the ship status corresponding to the AIS data is the navigation status; If all AIS data between two AIS data with berthing status in the AIS data sequence are in the navigation status, determine the initial navigation segment data according to the AIS data with the corresponding navigation status; Divide the initial navigation segment data into target navigation segment data based on a preset data quantity window; Among them, when the quantity of AIS data in the initial navigation segment data is less than twice the data quantity window, divide the initial navigation segment data into one segment of target navigation segment data; otherwise, divide the initial navigation segment data into n segments of target navigation segment data, where n is obtained by rounding down the ratio of the quantity of AIS data in the initial navigation segment data to the data quantity window, and the quantity of AIS data in the first n - 1 segments of target navigation segment data is equal to the data quantity window.

[0011] Optionally, the real-time navigation data includes real-time latitude and longitude and real-time speed; thinning the data based on the inflection points of each segment in the navigation segment data to generate a thinned track list includes the following steps: Use the navigation segment data as the target navigation segment; Construct a target line segment according to the starting point and the ending point of the target navigation segment, and use the AIS data corresponding to the track point with the largest vertical distance from the target line segment in the target navigation segment as the candidate inflection point; Among them, the vertical distance is calculated based on the real-time longitude and latitude corresponding to the starting point, ending point, and waypoint using the position geometric relationship; When the vertical distance between the candidate inflection point corresponding to the target navigation segment and the target line segment is less than or equal to the preset dynamic threshold, add the AIS data corresponding to the starting point and ending point of the target navigation segment to the thinned track list; otherwise, Take the candidate inflection point corresponding to the target navigation segment as the target inflection point; Among them, the expression of the dynamic threshold is:

[0012] In the formula, Delt represents the dynamic threshold corresponding to the candidate inflection point; BaseDelt represents the preset minimum thinning distance; and represents the preset correlation coefficient; represents the real-time speed of the candidate inflection point; represents the average speed from the starting point to the ending point; represents the course angle of the candidate inflection point relative to the starting point; represents the course angle of the ending point relative to the starting point; Add the AIS data corresponding to the target inflection point to the thinned track list, and split the target navigation segment into two sub-navigation segments based on the target inflection point; Take the sub-navigation segment as the target navigation segment, and return to execute the step of constructing the target line segment according to the starting point and ending point of the target navigation segment until the vertical distance between the candidate inflection point corresponding to the target navigation segment and the target line segment is less than or equal to the dynamic threshold of the corresponding target navigation segment, and add the AIS data corresponding to the starting point and ending point of the target navigation segment to the thinned track list.

[0013] Optionally, the ship track of each ship is sorted out based on the thinned track list, including the following steps: Sort all the AIS data in the thinned track list based on the timestamp to obtain the time-series track list; Perform equidistant thinning on the AIS data in the time-series track list based on the first interval to obtain the equidistant track list; Among them, the specific operation of equidistant thinning includes: when the interval between two adjacent waypoints in the time-series track list is less than or equal to the first interval, remove the AIS data corresponding to the latter waypoint from the time-series track list; Based on the AIS data of each waypoint in the equidistant track list, use LSTM for track prediction to obtain the target track list; Among them, track prediction includes filling in the missing waypoints between adjacent waypoints with an interval greater than the second interval in the equidistant track list and predicting the next waypoint; The ship track is drawn based on the AIS data of each track point in the target track list.

[0014] On the other hand, an embodiment of the present invention provides an improved ship track data processing device, including: A first module for periodically acquiring the AIS data of each ship in the target sea area; the AIS data includes a timestamp, real-time navigation data, and a navigation status code; A second module for preprocessing the AIS data to obtain the AIS data sequence of each ship; A third module for splitting the navigation segment data from the AIS data sequence based on the real-time navigation data and the navigation status code; A fourth module for thinning the data based on the inflection points of each segment in the navigation segment data to generate a thinned track list; A fifth module for sorting out the ship tracks of each ship based on the thinned track list.

[0015] On the other hand, an embodiment of the present invention provides an electronic device, including: a processor and a memory; the memory is used for storing programs; the processor executes the programs to implement the above-mentioned improved ship track data processing method.

[0016] On the other hand, an embodiment of the present invention provides a computer storage medium, in which a program executable by a processor is stored, and the program executable by the processor is used to implement the above-mentioned improved ship track data processing method when executed by the processor.

[0017] In the embodiment of the present invention, the AIS data of each ship in the target sea area is periodically acquired; the AIS data includes a timestamp, real-time navigation data, and a navigation status code; the AIS data is preprocessed to obtain the AIS data sequence of each ship; the navigation segment data is split from the AIS data sequence based on the real-time navigation data and the navigation status code; the data is thinned based on the inflection points of each segment in the navigation segment data to generate a thinned track list; the ship tracks of each ship are sorted out based on the thinned track list. The present invention adopts a segmented thinning strategy, which can effectively reduce the data storage amount while maintaining the integrity of the key track features; moreover, the present invention effectively optimizes the data processing efficiency through a pre-segmentation mechanism based on the status code; in addition, the present invention effectively improves the track segmentation accuracy through a dual verification mechanism that combines the motion characteristics of the navigation status code and the real-time navigation data. The embodiment of the present invention can efficiently and accurately implement ship track data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings are used to provide a further understanding of the technical solutions of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solutions of the present invention, and do not constitute a limitation to the technical solutions of the present invention.

[0019] Figure 1 It is a schematic diagram of an implementation environment for improving the ship track data processing method provided by an embodiment of the present invention; Figure 2 It is a schematic flowchart of an improved ship track data processing method provided by an embodiment of the present invention; Figure 3 It is an expanded flowchart of step S100 provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of an application example of data pulling and sorting provided by an embodiment of the present invention; Figure 5 It is an expanded flowchart of step S200 provided by an embodiment of the present invention; Figure 6 It is an expanded flowchart of step S300 provided by an embodiment of the present invention; Figure 7 It is an expanded flowchart of step S500 provided by an embodiment of the present invention; Figure 8 It is a schematic diagram of the overall flowchart of the improved ship track data processing method provided by an embodiment of the present invention; Figure 9 It is a schematic diagram of the structure of an improved ship track data processing device provided by an embodiment of the present invention; Figure 10 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0020] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0021] It should be noted that although functional module division is performed in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the system or the order in the flowchart. Terms such as "first / S100", "second / S200", etc. in the description, claims and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence.

[0022] References to "embodiments" in this invention mean that the specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described in this invention can be combined with other embodiments.

[0023] For the convenience of understanding the technical solution of this invention, first, an explanation of the technical terms that may appear in the technical solution of this invention is provided: The Automatic Identification System (AIS) for ships is a new type of navigation aid system and equipment, which has now developed into a Universal Automatic Identification System (UAIS). It is usually composed of a VHF communication machine, a GPS locator, etc., and can automatically exchange important information such as ship position and speed, and achieve automatic response. Its purpose is to enhance maritime life safety, improve navigation safety and efficiency, and protect the marine environment. Its functions include identifying ships, assisting in tracking targets, etc. AIS strengthens ship collision avoidance measures, enhances the functions of ARPA radar, etc., and can also display ship information on an electronic chart and improve maritime communication.

[0024] The data involved in this invention is from the ship AIS data provided by the supplier. The provided data includes two parts: ship static data and dynamic data. Ship static data mainly refers to the parameter information of registered ships (such as ship MMSI (Maritime Mobile Service Identity), length, width, draft, maximum speed, destination, etc. information), which has been registered in advance and generally does not change. The other part is the AIS data of the ship, which is sent to the operator in real time and mainly includes the real-time navigation data of the ship (such as current longitude and latitude, current speed, current heading, etc. real-time information).

[0025] LSTM (Long Short-Term Memory) is a long short-term memory neural network, which is a special type of recurrent neural network (RNN). When the original RNN processes long sequence data, it is prone to problems such as gradient explosion or gradient disappearance, while LSTM effectively solves these problems by introducing a gating mechanism. It includes a forget gate, an input gate, and an output gate. The forget gate determines the information to be forgotten and retained in the state of the memory cell at the previous moment; the input gate generates new information to be updated; the output gate controls the amount of information output from the current memory cell state to the external state. With this ability of selective memory and forgetting, LSTM can better handle long-range dependence problems and is widely used in fields such as speech recognition, machine translation, and time series prediction.

[0026] It can be understood that the improved ship track data processing method provided by the embodiments of the present invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various types of terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smart phone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto.

[0027] For the convenience of understanding the technical solutions of the present invention, first, the technical feature proper nouns that may appear in the embodiments of the present invention are explained: As Figure 1 shown, it is a schematic diagram of an implementation environment provided by the embodiments of the present invention. Referring to Figure 1 , this implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be network-connected through wireless or wired means to complete data transmission and exchange.

[0028] The server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0029] In addition, the server 101 can also be a node server in a blockchain network. Among them, the blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0030] The terminal 102 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 102 and the server 101 can be directly or indirectly connected through wired or wireless communication means, and the embodiments of the present invention do not make any restrictions here.

[0031] Exemplarily based on Figure 1In the implementation environment shown, an embodiment of the present invention provides an improved method for processing ship track data. Taking the application of this improved ship track data processing method to server 101 as an example for illustration, it can be understood that this improved ship track data processing method can also be applied to terminal 102.

[0032] Referring to Figure 2 , Figure 2 is a flowchart of the improved ship track data processing method applied to a server provided by an embodiment of the present invention. The execution subject of this improved ship track data processing method can be any of the aforementioned computer devices (including servers or terminals). Referring to Figure 2 , the method includes the following steps: S100. Periodically obtain the AIS data of each ship in the target sea area; Among them, the AIS data includes a timestamp, real-time navigation data, and a navigation status code; the real-time navigation data includes real-time longitude and latitude, real-time speed, and real-time heading; It should be noted that the AIS data is stored in a time series database; in some embodiments, as Figure 3 shown, step S100 may include the following steps: S101. In response to the first cycle node, use multiple threads to periodically pull the AIS data of each ship in the target sea area and store it in the temporary buffer of the corresponding thread; S102. In response to the second cycle node, periodically write the AIS data in the temporary buffer of each thread into the time series database in batches.

[0033] Exemplarily, in some specific implementation manners, as Figure 4 shown, the multi-threaded data pulling can be implemented as follows: The AIS data is obtained through the HTTP interface provided by the supplier, and usually an account or interface service permission needs to be purchased. For example, in some projects for processing ship track data, there are approximately 9,000 ships, with an average of 1,000 data per second, and the real-time data generated by all ships is at least 86 million per day. Pulling data with a single thread is slower and has poor timeliness, so generally N threads are started to pull different parts of the data. N is determined according to requirements, and the default is 5 threads. Each thread requests to pull data once every first cycle node (for example, every second). There are some duplicate data in the pulled data, and there are also abnormal point data in the data. The pulled data is stored in a temporary buffer area waiting for subsequent processing. In addition, the AIS data can be stored using the time series database influxDB. To improve the data writing efficiency, the ship data collected within a certain period of time is first cached, and after a second cycle node (for example, a 5-minute interval), a batch writing operation is performed, and then these data are written into InfluxDB at one time.

[0034] S200. Preprocess the AIS data to obtain the AIS data sequence of each ship; It should be noted that the AIS data also includes the maritime mobile service identity code; in some embodiments, such as Figure 5 shown, step S200 may include the following steps: S201. Classify and sort the AIS data based on the maritime mobile service identity code and the timestamp to obtain the original AIS data sequence of each ship; S202. Filter out the abnormal data in the original AIS data sequence based on the real-time navigation data to obtain the target AIS data sequence of each ship.

[0035] Among them, in some embodiments, the real-time navigation data includes real-time longitude and latitude, real-time speed, and real-time heading; filtering out the abnormal data in the original AIS data sequence based on the real-time navigation data may include the following steps: Take the first AIS data in the original AIS data sequence as the first AIS data; when the real-time speed of the first AIS data is greater than the maximum designed speed of the ship, remove the first AIS data from the original AIS data sequence, and return to execute the step of taking the first AIS data in the original AIS data sequence as the first AIS data until the real-time speed of the first AIS data is less than or equal to the maximum designed speed of the ship; Take the next AIS data of the first AIS data in the original AIS data sequence as the second AIS data; Calculate the track point distance between the first AIS data and the second AIS data based on the real-time longitude and latitude combined with the average radius of the earth; Calculate the average speed of the track point corresponding to the second AIS data based on the track point distance combined with the timestamp; When the difference ratio between the average speed and the real-time speed of the second AIS data is greater than the first threshold, determine that the second AIS data is abnormal speed data; Calculate the actual heading angle between the track points of the first AIS data and the second AIS data based on the real-time longitude and latitude; When the difference between the real-time heading angle of the second AIS data and the actual heading angle is greater than the second threshold, determine that the second AIS data is abnormal heading data; If the second AIS data is abnormal speed data or abnormal heading data, remove the second AIS data from the original AIS data sequence and take the next AIS data of the second AIS data in the original AIS data sequence as the second AIS data, and return to execute the step of calculating the track point distance between the first AIS data and the second AIS data based on the real-time longitude and latitude combined with the average radius of the earth, otherwise, take the second AIS data as the first AIS data, and return to execute the step of taking the next AIS data of the first AIS data in the original AIS data sequence as the second AIS data until all AIS data in the original AIS data sequence are traversed.

[0036] Exemplarily, in some embodiments, the classification and sorting of data can be implemented as follows: The data of each AIS includes real-time information such as the ship's MMSI, current longitude and latitude, current speed, and current heading. First, classify according to the ship's MMSI, and the data with the same MMSI is saved in a queue. The data in the queue is sorted in ascending order of timestamp. In some alternative embodiments, the data pulled here is basically sorted by timestamp. There may be some data delay, resulting in a small part of the data not being sorted by time. Therefore, after classification by MMSI, the idea of insertion sort can be used for sorting. During the sorting process, if it is found that the timestamps of the same MMSI are the same, the duplicate data is directly discarded. As Figure 4 shown, the ship AIS data can be pulled from the supplier in the form of interface requests by N threads, and then classified and sorted according to the ship's MMSI.

[0037] The filtering of abnormal data can be implemented as follows: ① Filtering of abnormal speed data: First, traverse the data one by one to determine whether there is any abnormal speed. For two consecutive points A1(x1, y1) and A2(x2, y2), where x1 and y1 represent the longitude and latitude of the point, calculate the distance between the two points based on their longitudes and latitudes.

[0038]

[0039] Calculate the distance from A1 to A2 according to the formula (the unit is km), where R is the average radius of the earth. Since the AIS data carries the timestamp and the current speed information, the average speed at A2 can be calculated as:

[0040] If the calculated average speed at point A2 is too different from the speed reported by the AIS ( , where represents the current speed information reported at point A2 in the AIS data), it indicates that there is a problem with the data at point A2, and point A2 is discarded. The specific process is as follows: Suppose the data sequence to be processed is A1, A2, A3, A4,...... First, determine whether the speed of A2 is abnormal based on A1 and A2. If A2 meets the expectation, then continue to determine A2 and A3, and so on. If the speed of A2 is abnormal, then A2 will be discarded. Next, determine A1 and A3, and so on. In addition, if A1 is the first data point, directly compare its reported speed with the maximum designed speed of the ship. If exceeds the maximum designed speed of the ship (this part of information can be obtained from the basic information of the ship), then point A1 is abnormal and will be discarded.

[0041] Until all the data is processed, the AIS data with abnormal speed is filtered out. The abnormal speed may be caused by inaccurate positioning of longitude, latitude and azimuth, or may be due to incorrect reporting.

[0042] ② Filtering of abnormal course data: First, traverse the data one by one to determine whether the course angle is normal. For two consecutive points A1(x1, y1) and A2(x2, y2), calculate the course angle (with north as the reference) between the two points based on their longitudes and latitudes.

[0043] If the calculated course angle of point A2 differs too much from the course angle reported by AIS (the difference exceeds 0.5 degrees or the corresponding radian value), it indicates that there is a problem with the data of point A2, and point A2 is discarded. Then traverse one by one to check for AIS points with abnormal course angles.

[0044] In this way, through the judgment of factors such as speed and direction angle, most of the abnormal point data information will be eliminated. The obtained data is the data after preliminary cleaning.

[0045] It should also be noted that in some alternative embodiments, multiple threads can be started to synchronously perform data filtering processing. Each thread processes the data of one ship. The number of threads is determined according to the machine performance. Each thread processes a fixed duration of data each time (default is 1 minute). After processing is completed, switch to the data of the next ship. Here, taking the data processing of a single ship as an example, 1 minute of AIS data is taken each time. It is expected that there are 100 - 200 AIS data for a single ship in 1 minute.

[0046] S300. Segment the voyage segment data from the AIS data sequence based on the real-time voyage data and the voyage status code; It should be noted that the real-time voyage data includes real-time longitude and latitude and real-time speed; in some embodiments, as Figure 6 shown, step S300 may include the following steps: S301. Determine the ship status corresponding to each AIS data in the AIS data sequence based on the real-time voyage data and the voyage status code; Among them, the ship status includes the berthing status and the sailing status; when the voyage status code of the AIS data is of the first type, the real-time speed does not exceed the third threshold, and the real-time longitude and latitude are in the berthing area, it is determined that the ship status corresponding to the AIS data is the berthing status; when the voyage status code of the AIS data is of the second type and the real-time speed is above the fourth threshold, it is determined that the ship status corresponding to the AIS data is the sailing status; it should be noted that the thresholds (such as the first threshold, the second threshold, the third threshold, etc.) applied in the embodiments of the present invention can be adjusted according to actual needs, and the detailed numerical settings in the specific embodiments are only for illustrative purposes.

[0047] Exemplarily, in some specific embodiments, the determination of the mooring state: If the AIS navigation state code is 1, 4, or 6 (1 for anchoring, 4 for mooring, 6 for stranding), it indicates that the vessel is in the mooring state. If the real-time speed is not more than 0.5 knots and, in combination with the electronic nautical chart, the point is in the mooring area, then the vessel is in the mooring state. The determination of the navigation state: If the AIS navigation state code is 0 (i.e., normal navigation) and the vessel speed is above 1 knot, it is determined that the vessel is in the navigation state. If the speed of the vessel does not exceed half of the designed speed, it is basically considered to be in low-speed navigation; otherwise, it is in high-speed navigation.

[0048] S302. If all the AIS data between two AIS data in the mooring state in the AIS data sequence are in the navigation state, determine the initial navigation segment data according to the AIS data in the corresponding navigation state; Exemplarily, the data is classified into mooring and in-navigation according to the vessel state. Since the mooring state can be shown by one track point, the data that needs to participate in the Douglas data thinning is the data in the navigation state, and the mooring data does not need to participate. Therefore, in actual processing, to avoid an increase in the amount of data processing, only the data of the navigation segment can be extracted to participate in the subsequent thinning processing. It should be noted that only the data between one mooring state and the next mooring state needs to be thinned.

[0049] S303. Divide the initial navigation segment data into target navigation segment data based on a preset data quantity window; Among them, when the quantity of AIS data in the initial navigation segment data is less than twice the data quantity window, the initial navigation segment data is divided into one segment of target navigation segment data; otherwise, the initial navigation segment data is divided into n segments of target navigation segment data, where n is obtained by rounding down the ratio of the quantity of AIS data in the initial navigation segment data to the data quantity window, and the quantity of AIS data in the first n - 1 segments of target navigation segment data is equal to the data quantity window.

[0050] Exemplarily, in some specific embodiments, the window is at least MinWin, and the default is 1000, that is, approximately 30 minutes of navigation data. Here, assume that the data in the intermediate state from mooring to navigation and then to mooring of the vessel is CurCnt. If , then the data will be divided according to one window, otherwise it will be divided according to (rounding down) segments.

[0051] For example, assume that the current navigation data is 900, then there is only one sliding window with a quantity of 900; assume that the current navigation data is 1200, then there is only one sliding window with a quantity of 1200; assume that the current navigation data is 2200, then there are two sliding windows with quantities of 1000 and 1200 respectively.

[0052] S400. Thin the data based on the inflection points of each segment in the voyage segment data to generate a list of thinned track points; It should be noted that the real-time voyage data includes real-time longitude and latitude and real-time speed. In some embodiments, step S400 may include the following steps: S401. Use the voyage segment data as the target voyage segment; S402. Construct a target line segment based on the starting point and the ending point of the target voyage segment, and use the AIS data corresponding to the track point with the maximum vertical distance from the target line segment in the target voyage segment as the candidate inflection point; wherein, the vertical distance is calculated using the position geometric relationship based on the real-time longitude and latitude corresponding to the starting point, the ending point, and the track point; S403. When the vertical distance between the candidate inflection point corresponding to the target voyage segment and the target line segment is less than or equal to a preset dynamic threshold, add the AIS data corresponding to the starting point and the ending point of the target voyage segment to the list of thinned track points; otherwise, execute the subsequent steps; S404. Use the candidate inflection point corresponding to the target voyage segment as the target inflection point; wherein, the expression of the dynamic threshold is:

[0053] In the formula, Delt represents the dynamic threshold corresponding to the candidate inflection point; BaseDelt represents the preset minimum thinning distance; and represents the preset correlation coefficient; represents the real-time speed of the candidate inflection point; represents the average speed from the starting point to the ending point; represents the course angle of the candidate inflection point relative to the starting point; represents the course angle of the ending point relative to the starting point; S405. Add the AIS data corresponding to the target inflection point to the list of thinned track points, and split the target voyage segment into two sub-voyage segments based on the target inflection point; S406. Use the sub-voyage segment as the target voyage segment, and return to execute the step of constructing the target line segment based on the starting point and the ending point of the target voyage segment until the vertical distance between the candidate inflection point corresponding to the target voyage segment and the target line segment is less than or equal to the dynamic threshold of the corresponding target voyage segment, and add the AIS data corresponding to the starting point and the ending point of the target voyage segment to the list of thinned track points.

[0054] Exemplarily, in some specific embodiments, assume that the starting point of the window is and the ending point is . Connect and into a line, traverse one by one, and calculate the vertical distance to the line segment and connected.

[0055] It should be noted that the AIS uses longitude and latitude to represent coordinates. Since the Earth is an ellipsoid and not in a flat plane, calculating the perpendicular distance is relatively complex. Considering that the sliding window has a maximum distance of 20 km, it can be considered to be in a flat plane, ignoring the influence of the Earth's curvature, and the coordinates are longitude and latitude.

[0056] First, the longitude and latitude coordinates of the foot of the perpendicular need to be obtained here, and then the perpendicular distance can be calculated. Suppose there are three points B1(x1, y1), B2(x2, y2), and B3(x3, y3). The straight line formed by B2 and B3 has the following equation:

[0057] Then the coordinates of the intersection point of the perpendicular line from B1 to the straight line (i.e., the foot of the perpendicular) are (x4, y4):

[0058]

[0059] Then the perpendicular distance is the distance from B1(x1, y1) to the foot of the perpendicular (x4, y4). After obtaining the longitude and latitude coordinates in this article, the distance between two points can be calculated based on the longitude and latitude of the two points (for the calculation principle, see the specific implementation of step S200).

[0060] Calculate and The included angle and the average speed (for the calculation principle, see the specific implementation of step S200), and then traverse one by one, calculate the perpendicular distance VerLen to the line segment and Connected, and finally it is found that the perpendicular distance is the largest at the data point (i.e., the candidate inflection point).

[0061] Currently, it is found that in the sliding window (i.e., the target navigation section), the perpendicular distance is the largest at the data point (assumed to be VerLen), that is, is the largest inflection point. Then calculate the dynamic threshold Delt, and the formula is as follows:

[0062] Among them, BaseDelt represents the minimum thinning distance, and the default value can be 0.1, that is, 0.1 km. and represent the correlation coefficients, and the value range is [0, 1]. The default value can be 0.5. represents The ship speed at this point. Indicates The course angle compared to the starting point of the sliding window. The greater the difference in speed and the greater the difference in course angle, the greater the threshold Delt.

[0063] If , first add the inflection point to the thinned track list (mark the inflection point), then divide the sliding window into and two sub-windows, and finally recursively process the sliding window by overloading the inflection point determination and window division steps and .

[0064] If , then and are added to the thinned track list, and the recursive search ends, where the data points will be discarded because these points are not sufficient to reflect the ship's track.

[0065] S500. Organize the ship tracks of each ship based on the thinned track list.

[0066] It should be noted that in some embodiments, as Figure 7 shown, step S500 may include the following steps: S501. Sort all AIS data in the thinned track list based on the timestamp to obtain a chronological track list; S502. Perform equidistant thinning on the AIS data in the chronological track list based on the first spacing to obtain an equidistant track list; wherein, the specific operation of equidistant thinning includes: when the spacing between two adjacent track points in the chronological track list is less than or equal to the first spacing, the AIS data corresponding to the latter track point is removed from the chronological track list; S503. Based on the AIS data of each track point in the equidistant track list, use LSTM for track prediction to obtain a target track list; wherein, track prediction includes filling in the missing track points between adjacent track points with a spacing greater than the second spacing in the equidistant track list and predicting the next track point; S504. Draw the ship track according to the AIS data of each track point in the target track list.

[0067] Exemplarily, in some specific embodiments, the data has completed Douglas data thinning. Its core idea is to find inflection points through a recursive idea, and the obtained data may be out of order, so it can be re-sorted according to the timestamp. Although the thinned data is obtained by marking inflection points, these data still have the following problems: 1) It is easy to have an uneven distribution of data point spacing. For example, the spacing between windows may be small.

[0068] 2) When the ship may experience data packet loss due to poor network, the ship's track appears discontinuous.

[0069] To solve the above problems, the data can be further processed through the following steps: ① Equi-distance thinning: The equi-distance thinning here is to make the ship's track distribute as evenly as possible. The average distance is dynamically changing and is related to the map scale for showing the ship's track. The relationship is as follows: The scale of the map is 1:D, and the distance between data points is also D (unit: m). Suppose the data sequences are , , ..., calculate the distance and one by one. If , and are retained. Otherwise, point is removed, and continue to compare the distance between and , and so on.

[0070] ② Introduce LSTM for data processing: a. Supplement missing data: Traverse the track data. When it is found that the data interval exceeds twice the average distance, it indicates that there may be missing data in the middle. To better show the change of the ship's track, several position data points need to be filled to make the ship's track smoother. Here, the LSTM neural network is used for track prediction. During the equi-distance thinning process, the five input variable information of the time stamp, longitude, latitude, speed, and course of this point will be input into the LSTM neural network, and the output is the position information of the next 5 - 10 points. The number of data used is dynamically determined according to the distance.

[0071] The basic parameters of LSTM are as follows: The input layer has 5 nodes, the output layer has 2 nodes (i.e., longitude and latitude information, etc.), the hidden layer contains multiple LSTM structures, each layer has 150 neurons by default, the Adam optimizer is used to prevent overfitting, and the root mean square error (RMSE) is used to measure the longitude and latitude deviation.

[0072] Generally speaking, based on the LSTM neural network, by traversing the data after equi-distance thinning, the missing data is filled, and the ship's track distribution becomes more continuous.

[0073] b. Predict the next track: Here, the LSTM neural network is used to predict the future ship trajectories. The five input variable information, namely the timestamp, longitude, latitude, speed, and course of the existing data, is input into the LSTM neural network, and 10 to 20 position information is output for the evaluation of the future dynamics of the ship.

[0074] The basic parameters of the LSTM are as follows: the input layer has 5 nodes, the output layer has 2 nodes (i.e., information such as longitude and latitude), the hidden layer contains multiple LSTM structures, with 150 neurons by default in each layer. The Adam optimizer is used to prevent overfitting, and the root mean square error (RMSE) is used to measure the longitude and latitude deviation.

[0075] The existing ship trajectories are drawn with solid lines, and the future trajectories are drawn with dashed lines. Considering that ships are usually in continuous navigation, a large amount of real trajectory data will be obtained as time goes by. By continuously inputting this data into the LSTM neural network, the ship's trajectory prediction ability will become more and more accurate.

[0076] To explain the principle of the technical solution of the present invention in detail, the overall process of the present invention will be described below in conjunction with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0077] First of all, it should be noted that in some ship data processing projects, there are approximately 90,000 ships in the target sea area. The real-time data generated by all ships is at least 86 million per day, with an average of 1,000 data per second. It is necessary to draw the trajectories of all ships based on this data, analyze and mine the ship status from multiple angles, and achieve the detection of abnormal ship behaviors.

[0078] The existing work directly draws ship trajectories based on AIS data, does not process noise data and duplicate data, and there are also cases of missing trajectories. The ship trajectories appear very rough and messy, with insufficient practicality. At the same time, using traditional relational databases (such as MySQL) to store large-scale ship trajectory big data has problems such as low storage efficiency, poor query performance, and difficulty in processing large-scale time-series data.

[0079] In view of this, the present invention proposes an improved method for processing ship trajectory data, as Figure 8 shown. The embodiments of the present invention can be implemented through the following steps: Step 1: Pull data in multiple threads. AIS data is obtained through the HTTP interface provided by the vendor. Usually, an account or interface service permission needs to be purchased. In our scenario, there are approximately 9,000 ships, with 1,000 data records per second on average. The real-time data generated by all ships is at least 86 million records per day. Pulling data with a single thread is slow and has poor timeliness. Therefore, generally, N threads are started to pull different parts of the data. N is determined according to requirements and defaults to 5 threads. Each thread requests to pull data once per second. There are some duplicate data in the pulled data, and there are also abnormal data points. The pulled data is stored in a temporary buffer, waiting for subsequent processing. Proceed to Step 2.

[0080] Step 2: Store and query ship trajectories. AIS data is stored in the time-series database InfluxDB. To improve the data writing efficiency, the ship data collected within a certain period of time is first cached and then batch-written once every 5-minute interval, and then these data are written into InfluxDB at one time. In addition, the following functional operations can be implemented based on the time-series database: Error handling: During the data writing process, problems such as network failures and abnormal database connections may occur. To ensure the integrity of the data, this method has established a corresponding error handling mechanism. When the writing fails, the data is retried, and the maximum number of retries is set to 1,000 times. If it still fails after exceeding the maximum number of retries, the data is stored in a local file and written again after the failure is resolved. Data storage optimization - Data partitioning: Specifically, the data is partitioned and stored every day to reduce the amount of data to be scanned during querying and improve the query performance. Index optimization: Based on the indexes automatically created by InfluxDB for timestamps and tags, according to the actual query requirements, additional index optimization is performed on the tags frequently used in query conditions to further improve the query efficiency. Time range query: Query the navigation trajectories, speed changes, etc. of ships within a specific time period according to the time range. Conditional query: In addition to the time range, query and analyze specific ship behaviors according to other conditions, such as ship MMSI, ship type, etc. Cache mechanism query: For some frequently queried results in this project, a cache mechanism is set. Specifically, mainly query the situation of all ships in the target sea area. Set to traverse all ships in the target sea area every 60 minutes. When a user has a query request, first obtain the result from the cache. If it does not exist in the cache, perform an actual query operation and store the query result in the cache, thus improving the query efficiency for ships in a large area. Paging query display: Since the data query range involved in this project is relatively wide, when the amount of query result data is large, the paging query method is used to return only part of the data each time, reducing the amount of data in a single query and improving the query response speed. Proceed to Step 3.

[0081] Step 3: Data Classification and Sorting. The data of each AIS includes real-time information such as ship MMSI, current longitude and latitude, current speed, current course, etc. First, classify according to the ship's MMSI, and the data with the same MMSI is saved in a queue. The data in the queue is sorted in ascending order of timestamp. It should be noted that the data pulled here is basically sorted by timestamp, and there may be some data delay, resulting in a small number of data not being sorted by time. Therefore, after classifying by MMSI, the idea of insertion sort can be used for sorting. During the sorting process, if it is found that the timestamps of the same MMSI are the same, the duplicate data is directly discarded.

[0082] As Figure 4 shown, pull the ship AIS data from the supplier in the form of interface requests through N threads, and then classify and sort according to the ship MMSI. Go to Step 4.

[0083] Step 4: Filter Data Based on Sliding Window. Start multiple threads, and each thread processes the data of one ship. The number of threads is determined according to the machine performance. Each single thread processes data for a fixed duration each time (default is 1 minute), and after processing, switch to the data of the next ship. Here, taking the data processing of a single ship as an example, take 1 minute of AIS data each time. It is expected that there are 100 - 200 AIS data for a single ship in 1 minute.

[0084] First, traverse the data one by one to determine whether the speed is abnormal. For two consecutive points A1(x1, y1), A2(x2, y2), where x1 and y1 represent the longitude and latitude of the point, calculate the distance between the two points based on the longitude and latitude of the two points.

[0085]

[0086] Calculate the distance from A1 to A2 according to the formula (the unit is km), R is the average radius of the earth. Since the AIS data carries the timestamp and the current speed information, the average speed at A2 can be calculated as

[0087] If the calculated average speed at point A2 is too different from the speed reported by AIS ( , where represents the current speed information reported by point A2 in the AIS data), it indicates that there is a problem with the data at point A2, and point A2 is discarded. The specific process is as follows: Assuming that the data sequence to be processed is A1, A2, A3, A4, ..., first determine whether the speed of A2 is abnormal based on A1 and A2. If A2 meets expectations, then continue to determine A2 and A3, and repeat. If the speed of A2 is abnormal, A2 will be discarded, and then determine A1 and A3, and repeat. In addition, if A1 is the first data point, directly compare its reported speed If the maximum speed of the ship is If the speed exceeds the ship's maximum design speed (this information can be obtained from the ship's basic information), point A1 is abnormal and will be discarded.

[0088] Until all the data is processed, the AIS data with abnormal speed will be filtered out. The abnormal speed may be caused by inaccurate positioning of latitude and longitude, or it may be an erroneous report.

[0089] First, traverse the data one by one to determine whether the heading angle is normal. For two consecutive points A1 (x1, y1) and A2 (x2, y2), calculate the heading angle of the two points based on the longitude and latitude of the two points (based on the north).

[0090] If the heading angle of point A2 is calculated If the heading angle reported by AIS is too different (the difference exceeds 0.5 degrees or the corresponding arc value), it indicates that there is a problem with the A2 point data, and A2 point is discarded. Then traverse one by one to check the AIS points with abnormal heading angles.

[0091] In this way, most of the abnormal data information will be eliminated after judging factors such as speed and direction angle. The data obtained is the data after preliminary cleaning.

[0092] Here, steps 1 to 4 are functionally used to receive native AIS data and filter duplicate and abnormal data. Each time, data of a fixed length (the default is 1 minute) is processed, that is, 1 minute is a sliding time window. After the current window is processed, the next data is processed and the process goes to step 1 to continuously obtain AIS data. The filtered data will be processed in step 5.

[0093] Step 5: Segment the data based on the ship status. Next, determine the ship status (moored or sailing normally) based on the AIS data.

[0094] Mooring status determination: If the AIS navigation status code is 1, 4 or 6 (1 for anchored, 4 for moored, 6 for stranded), it indicates that the ship is in a mooring state. If the real-time speed does not exceed 0.5 knots and the point is in the mooring area according to the electronic nautical chart, the ship is in a mooring state.

[0095] Voyage status determination: If the AIS voyage status code is 0 (i.e., normal voyage) and the ship's speed is above 1 knot, it is determined that the ship is in a voyage state. If the ship's speed does not exceed half of the designed speed, it is basically considered to be sailing at a low speed, otherwise it is sailing at a high speed.

[0096] During the subsequent thinning process, points where the ship's state changes should be preferentially retained because these points often represent the key nodes of the ship's voyage. For example, when the ship changes from a voyage state to a berthing state, this transition point is very important for analyzing the ship's behavior and predicting the track and should be retained. Therefore, the AIS data is traversed one by one here to clarify the ship's state at this point.

[0097] When this step of processing is completed, the AIS data has completed the preliminary cleaning and status determination, and this part of the data will be written into the database. Regarding the storage of AIS data, a database table is established every day, and the data carried by AIS is written into this table. The auto-sequence number is the primary key, and MMSI, timestamp, longitude and latitude are the indexes. Go to step 6.

[0098] Step 6: Process data based on a sliding window. When the ship's voyage track needs to be displayed, the AIS data of the relevant ship is read from the database, and the obtained data is classified according to MMSI, and the data is sorted in ascending order according to the timestamp. Here, an example of processing the track of a ship is taken. The data is classified into berthing and in-voyage according to the ship's state. Since the berthing state can be shown as one track point, the data that needs to participate in the Douglas data thinning is the data in the voyage state, and the berthing data does not need to participate. In addition, it should be noted that only the data between one berthing state and the next berthing state needs to be thinned. The sliding window here is dynamically divided based on the data quantity, and the window is at least MinWin, with the default being 1000, that is, about 30 minutes of voyage data. Here, it is assumed that the data in the state from berthing to voyage and then to berthing is CurCnt. If , then the data will be divided into one window, otherwise it will be divided into (rounded down) divisions.

[0099] Here is an example. Suppose the current voyage data is 900, then there is only one sliding window with a quantity of 900; suppose the current voyage data is 1200, then there is only one sliding window with a quantity of 1200; suppose the current voyage data is 2200, then there are two sliding windows with quantities of 1000 and 1200 respectively.

[0100] If the ship stops and starts, there will be multiple sets of data in the "berthing -> voyage -> berthing" state. For the data in each voyage segment, the same window splitting method is executed.

[0101] Here, we will further explain how to determine the value of MinWin. The AIS data of a normal ship is transmitted at the fastest rate of once every 2 seconds. For 1000 data points, it is approximately 30 minutes. The ship's speed is usually not very fast, and the economic speed is mostly between 10 and 20 knots, which is 18 km / h to 36 km / h (5 m / s to 10 m / s). Therefore, the default value of MinWin is set to 1000, which represents a voyage of approximately half an hour and a travel distance of about 10 km to 20 km. This value not only allows for a more refined thinning of the data but also improves the computational efficiency of the thinning process.

[0102] The starting point of the default sliding window is , and the ending point is . When it reaches this point, the data thinning operation begins. Proceed to Step 7.

[0103] Step 7: Find the inflection point. Assume the starting point of the window is , and the ending point is . Connect and to form a line. Traverse each , and calculate the perpendicular distance to the line segment and .

[0104] It should be noted that the AIS uses longitude and latitude to represent coordinates. Since the Earth is an ellipsoid, it is not on a flat plane. Calculating the perpendicular distance is relatively complex. Considering that the sliding window has a maximum distance of 20 km, it can be considered to be on a flat plane, ignoring the influence of the Earth's curvature, and the coordinates are longitude and latitude.

[0105] Here, first, the longitude and latitude coordinates of the foot of the perpendicular need to be obtained, and then the perpendicular distance can be calculated. Suppose there are three points B1(x1, y1), B2(x2, y2), and B3(x3, y3), and a straight line is formed by connecting B2 and B3. The equation is as follows:

[0106] Then, the coordinate formula of the perpendicular intersection point (i.e., the foot of the perpendicular) of B1 to the line is (x4, y4):

[0107]

[0108] Then, the perpendicular distance is the distance from B1(x1, y1) to the foot of the perpendicular (x4, y4). After obtaining the longitude and latitude coordinates in this article, the distance between two points can be calculated based on the longitude and latitude of the two points (the formula can be found in Step 4).

[0109] Calculate the and included angle and average speed (Refer to step 4 for the formula), and then traverse one by one , calculate to the line segment and connected vertical distance VerLen. Finally, it is found that at the data point , the vertical distance is the largest. Go to step 8.

[0110] Step 8: Meet the end condition. Currently, it is found that in the sliding window , at the data point , the vertical distance is the largest (assumed to be VerLen), that is is the largest inflection point. Then calculate the dynamic threshold Delt, and the formula is as follows:

[0111] where BaseDelt represents the minimum thinning distance, defaulting to 0.1, that is, 0.1 km, and represents the correlation coefficient, with a value range of [0, 1], and the default value is 0.5 for both, represents the ship speed at, represents the course angle compared to the starting point of the sliding window. The greater the speed difference and the greater the course angle difference, the greater the threshold Delt.

[0112] If , first add the inflection point to the thinned track list (mark the inflection point), then divide the sliding window into and two sub-windows, and finally go to step 7 to recursively process the sliding windows and .

[0113] If , then add and to the thinned track list, end the recursive search, where the data points of will be discarded because these points are not sufficient to reflect the ship track. Go to step 8.

[0114] Generally speaking, the core idea of steps 7 - 8 is to continuously recursively divide the sliding window by obtaining the inflection points of the sliding window until the window cannot be further divided or there are no inflection points. At this time, the whole process ends and the thinned data is obtained. Specifically, steps 7 to 8 recursively execute the data thinning process, and step 6 obtains the windows to be processed. There may be multiple windows, and each window executes the recursive process of steps 7 to 8, so that each window completes thinning.

[0115] Step 9: Get the preliminary thinned data. When you reach this step, it means that the data has completed Douglas data thinning. Its core idea is to find the inflection point through recursion. The data obtained is out of order and needs to be reordered according to the timestamp; specifically, the inflection point data obtained during the recursive execution is not sorted by time, so step 9 here needs to reorder all the data. Although the thinned data is obtained by marking the inflection point, these data still have the following problems: 1) The spacing between data points is likely to be uneven. For example, the spacing between windows may be small.

[0116] 2) The ship may experience data packet loss due to poor network conditions, and the ship's track may not be continuous.

[0117] In order to solve the above problem, it is necessary to continue processing the data and go to step 10. In addition, equidistant thinning can solve the problem of uneven distribution of track points caused by recursive thinning, which is convenient for subsequent missing detection and LSTM prediction.

[0118] Step 10: Perform equidistant thinning. The equidistant thinning here is to make the ship tracks as evenly distributed as possible. The average spacing is dynamically changing, and it is related to the scale of the map showing the ship tracks. The relationship is as follows: the scale of the map is 1:D, and the spacing of the data points is also D (in meters). Assume that the data series are , , ..., calculate one by one and Distance ,if , and Keep. Otherwise, remove the point , continue to compare and Repeat this process. Go to step 11.

[0119] Step 11: Supplement missing data. Traverse the track data. When the data interval is found to be more than twice the average interval, it means that there may be data loss in the middle. In order to better show the changes in the ship's track, it is necessary to fill in some point data to make the ship's track smoother. The track prediction here is done by the LSTM neural network. In the process of equidistant thinning, the five input variable information of the point, such as the timestamp, longitude, latitude, speed, and heading, will be input into the LSTM neural network, and the output is the location information of the next 5 to 10. The number of data used is determined according to the dynamic interval.

[0120] The basic parameters of the LSTM are as follows: the input layer has 5 nodes, the output layer has 2 nodes (i.e., information such as longitude and latitude), the hidden layer contains multiple LSTM structures, each layer has 150 neurons by default, the Adam optimizer is used to prevent overfitting, and the root mean square error (RMSE) is used to measure the longitude and latitude deviation.

[0121] Generally speaking, based on the LSTM neural network, by traversing the equidistant thinned data, the missing data is filled, so that the ship's track distribution is more continuous, and then go to step 12.

[0122] Step 12: Predict the next track. Here, the LSTM neural network is used to predict the future ship track. The five input variable information of the time stamp, longitude, latitude, speed, and heading of the existing data is input into the LSTM neural network, and 10 - 20 position information is output for the evaluation of the future dynamics of the ship.

[0123] The basic parameters of the LSTM are as follows: the input layer has 5 nodes, the output layer has 2 nodes (i.e., information such as longitude and latitude), the hidden layer contains multiple LSTM structures, each layer has 150 neurons by default, the Adam optimizer is used to prevent overfitting, and the root mean square error (RMSE) is used to measure the longitude and latitude deviation.

[0124] The existing ship track is drawn with a solid line, and the future track is drawn with a dotted line. Considering that the ship is usually in continuous navigation, a large amount of real track data will be obtained as time goes by. By continuously inputting this data into the LSTM neural network, the ship's track prediction ability will become more and more accurate.

[0125] To sum up, the present invention proposes an improved ship track data processing method. The beneficial effects of the embodiments of the present invention include but are not limited to: 1) The data filtering method of the moving time window, including data re - sorting, duplicate data filtering, and abnormal data removal, simplifies the data application scenario and reduces the data storage cost.

[0126] 2) The Douglas data thinning method is improved. By introducing dynamic thresholds and inflection point calculations, the data simplification work is better completed based on a sliding window.

[0127] 3) The data missing detection process is optimized. Based on LSTM, the missing data is filled, and at the same time, the future track is predicted, improving the data completeness and accuracy.

[0128] 4) The data storage query is optimized. Based on the InfluxDB time - series database, the ship track data is optimized and stored according to the time series, improving the storage efficiency and query performance of the ship track big data.

[0129] On the other hand, as Figure 9As shown in the figure, an embodiment of the present invention provides an improved ship track data processing device 900, which may include: A first module 901, configured to periodically obtain AIS data of each ship in a target sea area; the AIS data includes a timestamp, real-time navigation data, and a navigation status code; A second module 902, configured to preprocess the AIS data to obtain an AIS data sequence of each ship; A third module 903, configured to segment and obtain navigation segment data from the AIS data sequence based on the real-time navigation data and the navigation status code; A fourth module 904, configured to thin out data based on the inflection points of each segment in the navigation segment data to generate a thinned track list; A fifth module 905, configured to organize the ship tracks of each ship based on the thinned track list.

[0130] The content of the method embodiment of the present invention is applicable to the device embodiment. The functions specifically implemented by the device embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those of the above method.

[0131] On the other hand, an embodiment of the present invention further provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above-mentioned improved ship track data processing method is implemented. The electronic device may be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0132] It can be understood that the content in the above method embodiment is applicable to the device embodiment of the present invention. The functions specifically implemented by the device embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those of the above method embodiment.

[0133] As Figure 10 shown, Figure 10 The hardware structure of an electronic device 1000 according to another embodiment is schematically shown. The electronic device 1000 includes: A processor 1001, which may be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention; The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1002 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1002 and are called by the processor 1001 to execute the network node population optimization method of the embodiments of the present invention; The input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to implement communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.); The bus 1005 transmits information between the various components of the device (such as the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004); Among them, the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004 achieve communication connections with each other inside the device through the bus 1005.

[0134] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solutions of this embodiment.

[0135] The content of the method embodiments of the present invention is applicable to the electronic device embodiments of the present invention. The functions specifically implemented by the electronic device embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method.

[0136] Another aspect of the embodiments of the present invention also provides a computer-readable storage medium. The storage medium stores a program, and the program is executed by a processor to implement the previous method.

[0137] It should be noted that the computer-readable medium shown in the embodiments of the present invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0138] The content of the method embodiments of the present invention is applicable to the embodiments of this computer-readable storage medium. The functions specifically implemented by the embodiments of this computer-readable storage medium are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method.

[0139] The embodiments of the present invention also disclose a computer program product or a computer program. This computer program product or computer program includes computer instructions, and these computer instructions are stored in a computer-readable storage medium. The processor of a computer device can read these computer instructions from the computer-readable storage medium, and the processor executes these computer instructions, causing the computer device to execute the above method.

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as combinations of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0141] It should be noted that although several modules of devices for action execution are mentioned in the above detailed description, such a division is not mandatory. In fact, according to the embodiments of the present invention, the features and functions of two or more of the above-described modules or units can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0142] From the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a portable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the methods according to the embodiments of the present invention.

[0143] In some alternative embodiments, the functions / operations mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two consecutive blocks shown may actually be executed substantially simultaneously or the blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of the present invention are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented in the present invention. Alternative embodiments are foreseeable, in which the order of various operations is changed and the sub-operations described as part of a larger operation are executed independently.

[0144] In addition, although the present invention has been described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It should also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. Rather, given the attributes, functions, and internal relationships of the various functional modules in the devices disclosed in the present invention, the actual implementation of the modules will be understood within the ordinary skills of an engineer. Thus, those skilled in the art can implement the present invention as set forth in the claims without undue experimentation. It should also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0145] If a function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0146] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution device, apparatus, or equipment (such as a computer-based device, a device including a processor, or other devices that can fetch instructions from the instruction execution device, apparatus, or equipment and execute the instructions), or in combination with these instruction execution devices, apparatuses, or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in combination with an instruction execution device, apparatus, or equipment.

[0147] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection (electronic device) having one or more wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable media can even be paper or other suitable media on which a program can be printed, as the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then stored in a computer memory.

[0148] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0149] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0150] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

[0151] The above has specifically described the preferred embodiments of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. An improved method for processing ship track data, characterized in that, Including the following steps: Periodically obtain the AIS data of each ship in the target sea area; the AIS data includes a timestamp, real-time navigation data, and a navigation status code; The real-time navigation data includes real-time longitude and latitude, real-time speed, and real-time course; Preprocess the AIS data to obtain an AIS data sequence for each ship; Segment the navigation segment data from the AIS data sequence based on the real-time navigation data and the navigation status code; Based on the inflection points of each segment in the navigation segment data, perform data thinning to generate a thinned track list, including the following steps: Use the navigation segment data as the target navigation segment; Construct a target line segment according to the starting point and the ending point of the target navigation segment, and use the AIS data corresponding to the track point with the maximum vertical distance from the target line segment in the target navigation segment as the candidate inflection point; Wherein, the vertical distance is calculated using the position geometric relationship based on the starting point, the ending point, and the real-time longitude and latitude corresponding to the track point; When the vertical distance between the candidate inflection point corresponding to the target navigation segment and the target line segment is less than or equal to a preset dynamic threshold, add the AIS data corresponding to the starting point and the ending point of the target navigation segment to the thinned track list; otherwise, Use the candidate inflection point corresponding to the target navigation segment as the target inflection point; Add the AIS data corresponding to the target inflection point to the thinned track list, and split the target navigation segment into two sub-navigation segments based on the target inflection point; Use the sub-navigation segment as the target navigation segment, and return to execute the step of constructing the target line segment according to the starting point and the ending point of the target navigation segment until the vertical distance between the candidate inflection point corresponding to the target navigation segment and the target line segment is less than or equal to the dynamic threshold of the corresponding target navigation segment, and add the AIS data corresponding to the starting point and the ending point of the target navigation segment to the thinned track list; Sort out the ship tracks of each ship based on the thinned track list.

2. The improved ship track data processing method according to claim 1, wherein The AIS data is stored in a time series database; the periodic acquisition of the AIS data of each ship in the target sea area includes the following steps: In response to the first periodic node, periodically pull the AIS data of each ship in the target sea area using multiple threads and store it in the temporary buffer area of the corresponding thread; In response to the second periodic node, periodically write the AIS data in the temporary buffer area of each thread into the time series database in batches.

3. The improved ship track data processing method according to claim 1, wherein, The AIS data further includes a maritime mobile service identity code; the preprocessing of the AIS data to obtain an AIS data sequence for each ship includes the following steps: Classify and sort the AIS data based on the maritime mobile service identity code and the timestamp to obtain the original AIS data sequence for each ship; Filter out abnormal data from the AIS data in the original AIS data sequence based on the real-time navigation data to obtain the target AIS data sequence for each ship.

4. The improved ship track data processing method according to claim 3, characterized in that, Filtering abnormal data from the AIS data in the original AIS data sequence based on the real-time navigation data includes the following steps: Take the first AIS data in the original AIS data sequence as the first AIS data; When the real-time speed of the first AIS data is greater than the maximum designed ship speed, remove the first AIS data from the original AIS data sequence, and return to execute the step of taking the first AIS data in the original AIS data sequence as the first AIS data until the real-time speed of the first AIS data is less than or equal to the maximum designed ship speed; Take the next AIS data of the first AIS data in the original AIS data sequence as the second AIS data; Calculate the track point distance between the first AIS data and the second AIS data based on the real-time longitude and latitude combined with the average radius of the earth; Calculate the average speed of the track point corresponding to the second AIS data based on the track point distance combined with the timestamp; When the difference ratio between the average speed and the real-time speed of the second AIS data is greater than the first threshold, determine that the second AIS data is abnormal speed data; Calculate the actual course angle between the track points of the first AIS data and the second AIS data based on the real-time longitude and latitude; When the difference between the real-time course angle of the second AIS data and the actual course angle is greater than the second threshold, determine that the second AIS data is abnormal course data; If the second AIS data is the abnormal speed data or the abnormal course data, remove the second AIS data from the original AIS data sequence and take the next AIS data of the second AIS data in the original AIS data sequence as the second AIS data, and return to execute the step of calculating the track point distance between the first AIS data and the second AIS data based on the real-time longitude and latitude combined with the average radius of the earth. Otherwise, take the second AIS data as the first AIS data, and return to execute the step of taking the next AIS data of the first AIS data in the original AIS data sequence as the second AIS data until all the AIS data in the original AIS data sequence are traversed.

5. The improved ship track data processing method according to claim 1, characterized in that, Segmenting the navigation segment data from the AIS data sequence based on the real-time navigation data and the navigation status code includes the following steps: Determine the ship status corresponding to each AIS data in the AIS data sequence based on the real-time navigation data and the navigation status code; Among them, the ship state includes a berthing state and a sailing state; when the sailing state code of the AIS data is of the first type, the real-time speed does not exceed the third threshold, and the real-time longitude and latitude are in the berthing area, it is determined that the ship state corresponding to the AIS data is the berthing state; when the sailing state code of the AIS data is of the second type and the real-time speed is above the fourth threshold, it is determined that the ship state corresponding to the AIS data is the sailing state; If all the AIS data between two AIS data in the berthing state in the AIS data sequence are in the sailing state, initial sailing segment data is determined according to the AIS data in the corresponding sailing state; Based on a preset data quantity window, the initial sailing segment data is divided into target sailing segment data; Among them, when the data quantity of the AIS data in the initial sailing segment data is less than twice the data quantity window, the initial sailing segment data is divided into one segment of target sailing segment data; otherwise, the initial sailing segment data is divided into n segments of target sailing segment data, where n is obtained by rounding down the ratio of the data quantity of the AIS data in the initial sailing segment data to the data quantity window, and the data quantity of the AIS data in the first n - 1 segments of target sailing segment data is equal to the data quantity window.

6. The improved ship track data processing method according to claim 1, characterized in that The expression of the dynamic threshold is: Wherein, Delt represents the dynamic threshold corresponding to the candidate inflection point; BaseDelt represents the preset minimum thinning distance; and represents the preset correlation coefficient; represents the real-time speed of the candidate inflection point; represents the average speed from the starting point to the ending point; represents the course angle of the candidate inflection point relative to the starting point; represents the course angle of the ending point relative to the starting point.

7. The improved ship track data processing method according to claim 1, wherein The organizing the ship tracks of each ship based on the thinned track list includes the following steps: Sort all the AIS data in the thinned track list based on the timestamp to obtain a time-sequential track list; Perform equidistant thinning on the AIS data in the time-sequential track list based on the first spacing to obtain an equidistant track list; Among them, the specific operation of the equidistant thinning includes: when the spacing between two adjacent track points in the time-sequential track list is less than or equal to the first spacing, the AIS data corresponding to the latter track point is removed from the time-sequential track list; Based on the AIS data of each track point in the equidistant track list, use LSTM for track prediction to obtain a target track list; Among them, the track prediction includes filling in the missing track points between adjacent track points with a spacing greater than the second spacing in the equidistant track list and predicting the next track point; The ship track is drawn according to the AIS data of each track point in the target track list.

8. An improved ship track data processing device, characterized in that, Including: The first module is used to periodically obtain the AIS data of each ship in the target sea area; the AIS data includes a timestamp, real-time navigation data, and a sailing state code; The real-time navigation data includes real-time longitude and latitude, real-time speed, and real-time course; The second module is used to preprocess the AIS data to obtain the AIS data sequence of each ship; The third module is used to segment and obtain the sailing segment data from the AIS data sequence based on the real-time navigation data and the sailing state code; The fourth module is used to perform data thinning based on the inflection points of each segment in the sailing segment data to generate a thinned track list; Among them, thinning the data based on the inflection points of each segment in the voyage segment data to generate a thinned track list includes the following steps: Regarding the voyage segment data as the target voyage segment; Constructing a target line segment according to the starting point and the ending point of the target voyage segment, and regarding the AIS data corresponding to the track point with the largest vertical distance from the target line segment in the target voyage segment as the candidate inflection point; Among them, the vertical distance is calculated using the position geometric relationship based on the starting point, the ending point, and the real-time longitude and latitude corresponding to the track point; When the vertical distance between the candidate inflection point corresponding to the target voyage segment and the target line segment is less than or equal to a preset dynamic threshold, adding the AIS data corresponding to the starting point and the ending point of the target voyage segment to the thinned track list; otherwise, Regarding the candidate inflection point corresponding to the target voyage segment as the target inflection point; Adding the AIS data corresponding to the target inflection point to the thinned track list, and splitting the target voyage segment into two sub-voyage segments based on the target inflection point; Regarding the sub-voyage segment as the target voyage segment, and returning to execute the step of constructing the target line segment according to the starting point and the ending point of the target voyage segment until the vertical distance between the candidate inflection point corresponding to the target voyage segment and the target line segment is less than or equal to the dynamic threshold of the corresponding target voyage segment, and adding the AIS data corresponding to the starting point and the ending point of the target voyage segment to the thinned track list; The fifth module is used to sort out the ship tracks of each ship based on the thinned track list.

9. An electronic device, characterized in that, It includes a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, characterized in that, The program executable by the processor, when executed by the processor, is used to implement the method according to any one of claims 1 to 7.

Images