Improved ship track data processing method, device, equipment and medium

Abstract

The present invention discloses an improved method, device, equipment and medium for processing ship track data. The method includes: periodically acquiring AIS data of each ship in the target sea area; the AIS data includes a timestamp, real-time navigation data and a navigation status code; pre-processing the AIS data to obtain an AIS data sequence of each ship; segmenting the AIS data sequence based on the real-time navigation data and the navigation status code to obtain navigation segment data; performing data thinning based on the inflection points of each segment in the navigation segment data to generate a thinned track list; and arranging the ship track of each ship based on the thinned track list. The present invention adopts a segmented thinning strategy to effectively reduce the amount of data stored while maintaining the integrity of key track features; effectively optimizes data processing efficiency through a pre-segmentation mechanism based on status codes; and effectively improves the accuracy of track segmentation through a dual verification mechanism that integrates the motion characteristics of navigation status codes and real-time navigation data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an improved ship track data processing method, device, equipment and medium. Background Art

[0002] Currently, in the field of ship track monitoring technology, ship track analysis based on AIS data faces the following technical bottlenecks:

[0003] Data redundancy and storage pressure: Traditional methods store and process all raw AIS data. When the daily data volume reaches tens of millions, there are problems such as high storage media occupancy and heavy data transmission bandwidth pressure, which is particularly difficult to implement in edge computing scenarios.

[0004] Insufficient real-time processing capabilities: The time complexity of existing track generation algorithms is generally on the order of O(n²). When faced with high-density ship groups, the processing delay of a single node can reach several hours, which cannot meet the real-time requirements of maritime supervision.

[0005] Track segmentation accuracy defects: Conventional methods only rely on speed thresholds for state division, which may lead to mis-segmentation of approximate tracks. Summary of the Invention

[0006] The present invention aims to at least partially address the limitations of the related art. To this end, the present invention provides an improved method, apparatus, device, and medium for processing ship track data, which can efficiently process ship track data.

[0007] In one aspect, an embodiment of the present invention provides a method for improving ship track data processing, comprising the following steps:

[0008] Periodically obtain AIS data of each ship in the target sea area; AIS data includes timestamp, real-time navigation data and navigation status code;

[0009] Preprocess the AIS data to obtain the AIS data sequence of each ship;

[0010] Segment the navigation segment data from the AIS data sequence based on real-time navigation data and navigation status codes;

[0011] Perform data thinning based on the inflection points of each segment in the flight segment data to generate a thinned track list;

[0012] The track of each ship is obtained based on the rarefied track list.

[0013] Optionally, the AIS data is stored in a time series database; periodically acquiring the AIS data of each ship in the target sea area includes the following steps:

[0014] In response to the first periodic node, multiple threads are used to periodically pull the AIS data of each ship in the target sea area and store it in the temporary buffer area of the corresponding thread;

[0015] In response to the second periodic node, the AIS data in the temporary buffer area of each thread is periodically written in batches into the time series database.

[0016] Optionally, the AIS data further includes a maritime mobile service identification code; preprocessing the AIS data to obtain an AIS data sequence of each ship includes the following steps:

[0017] Classify and sort AIS data based on the maritime mobile service identification code and timestamp to obtain the original AIS data sequence of each ship;

[0018] Based on the real-time navigation data, the AIS data in the original AIS data sequence is filtered for abnormal data to obtain the target AIS data sequence of each ship.

[0019] Optionally, the real-time navigation data includes real-time latitude and longitude, real-time speed, and real-time heading; and filtering abnormal data from the AIS data in the original AIS data sequence based on the real-time navigation data includes the following steps:

[0020] The first AIS data in the original AIS data sequence is used as the first AIS data;

[0021] When the real-time speed of the first AIS data is greater than the maximum designed speed of the ship, the first AIS data is removed from the original AIS data sequence, and the process returns to the step of using the first AIS data in the original AIS data sequence as the first AIS data until the real-time speed of the first AIS data is less than or equal to the maximum designed speed of the ship.

[0022] The next AIS data after the first AIS data in the original AIS data sequence is used as the second AIS data;

[0023] The track point distance between the first AIS data and the second AIS data is obtained based on the real-time longitude and latitude combined with the average radius of the earth;

[0024] The average speed of the track point corresponding to the second AIS data is obtained based on the track point distance and timestamp calculation;

[0025] When the difference ratio between the average speed and the real-time speed of the second AIS data is greater than a first threshold, the second AIS data is determined to be abnormal speed data;

[0026] Obtaining an actual heading angle between the first AIS data track point and the second AIS data track point based on real-time latitude and longitude calculations;

[0027] When the difference between the real-time heading angle of the second AIS data and the actual heading angle is greater than a second threshold, determining that the second AIS data is heading abnormal data;

[0028] If the second AIS data is abnormal speed data or abnormal heading data, the second AIS data is removed from the original AIS data sequence and the next AIS data of the second AIS data in the original AIS data sequence is used as the second AIS data. The process returns to the step of calculating the track point distance between the first AIS data and the second AIS data based on the real-time latitude and longitude combined with the average radius of the earth. Otherwise, the second AIS data is used as the first AIS data, and the process returns to the step of calculating the next AIS data of the first AIS data in the original AIS data sequence as the second AIS data until all AIS data in the original AIS data sequence are traversed.

[0029] Optionally, the real-time navigation data includes real-time latitude and longitude and real-time speed; and segmenting the navigation segment data from the AIS data sequence based on the real-time navigation data and the navigation status code comprises the following steps:

[0030] Determine the ship status corresponding to each AIS data in the AIS data sequence based on the real-time navigation data and the navigation status code;

[0031] Among them, the ship status includes mooring status and sailing status; when the sailing status code of the AIS data is the first type, the real-time speed does not exceed the third threshold and the real-time longitude and latitude are in the mooring area, the ship status corresponding to the AIS data is determined to be mooring status; when the sailing status code of the AIS data is the second type and the real-time speed is above the fourth threshold, the ship status corresponding to the AIS data is determined to be sailing status;

[0032] If all AIS data between two AIS data in the moored state in the AIS data sequence are in the navigation state, the initial navigation segment data is determined according to the AIS data in the corresponding navigation state;

[0033] Dividing the initial voyage segment data into target voyage segment data based on a preset data quantity window;

[0034] Among them, when the data volume of AIS data in the initial voyage segment data is less than twice the data quantity window, the initial voyage segment data is divided into one segment of target voyage segment data; otherwise, the initial voyage segment data is divided into n segments of target voyage segment data, where n is obtained by rounding down the ratio of the data volume of AIS data in the initial voyage segment data to the data quantity window, and the data volume of AIS data in the first n-1 segments of target voyage segment data is equal to the data quantity window.

[0035] Optionally, the real-time navigation data includes real-time latitude and longitude and real-time speed; performing data thinning based on the inflection points of each segment in the navigation segment data to generate a thinned track list includes the following steps:

[0036] Use the voyage segment data as the target voyage segment;

[0037] Construct a target line segment based on the starting point and end point of the target navigation segment, and use the AIS data corresponding to the track point with the largest vertical distance from the target line segment in the target navigation segment as a candidate inflection point;

[0038] The vertical distance is calculated based on the real-time latitude and longitude corresponding to the starting point, end point and track point using the position geometry relationship;

[0039] When the vertical distance between the candidate inflection point corresponding to the target navigation segment and the target line segment is less than or equal to the preset dynamic threshold, the AIS data corresponding to the starting point and the end point of the target navigation segment are added to the thinned track list; otherwise,

[0040] The candidate inflection point corresponding to the target navigation segment is used as the target inflection point;

[0041] The expression of dynamic threshold is:

[0042]

[0043] In the formula, Delt represents the dynamic threshold corresponding to the candidate inflection point; BaseDelt represents the preset minimum thinning distance; and Indicates the preset correlation coefficient; Indicates the real-time speed of the candidate inflection point; Indicates the average speed from the starting point to the end point; Indicates the heading angle of the candidate inflection point compared to the starting point; Indicates the heading angle of the end point compared to the starting point;

[0044] Add the AIS data corresponding to the target inflection point to the thinned track list, and split the target flight segment into two sub-segments based on the target inflection point;

[0045] Take the sub-segment as the target segment and return to the step of constructing the target line segment based on the starting point and end point of the target segment until the vertical distance between the candidate inflection point corresponding to the target segment and the target line segment is less than or equal to the dynamic threshold of the corresponding target segment. Then add the AIS data corresponding to the starting point and end point of the target segment to the thinned track list.

[0046] Optionally, obtaining the ship track of each ship based on the thinned track list includes the following steps:

[0047] Sort all AIS data in the thinned track list based on timestamps to obtain a time-series track list;

[0048] Based on the first spacing, the AIS data in the time series track list are subjected to equal-interval thinning to obtain an equal-interval track list;

[0049] The specific execution operation of equal-distance thinning includes: when the distance between two adjacent track points in the time series track list is less than or equal to the first distance, the AIS data corresponding to the latter track point is removed from the time series track list;

[0050] Based on the AIS data of each track point in the equidistant track list, LSTM is used to perform track prediction to obtain the target track list;

[0051] The track prediction includes filling in the missing track points between adjacent track points with a spacing greater than the second spacing in the equidistant track list and predicting the next track point;

[0052] The ship track is obtained by drawing the AIS data of each track point in the target track list.

[0053] On the other hand, an embodiment of the present invention provides an improved ship track data processing device, comprising:

[0054] The first module is used to periodically obtain AIS data of each ship in the target sea area; AIS data includes timestamp, real-time navigation data and navigation status code;

[0055] The second module is used to pre-process the AIS data to obtain the AIS data sequence of each ship;

[0056] The third module is used to segment the AIS data sequence to obtain the navigation segment data based on the real-time navigation data and the navigation status code;

[0057] The fourth module is used to perform data thinning based on the inflection points of each segment in the flight segment data and generate a thinned track list;

[0058] The fifth module is used to obtain the ship track of each ship based on the rarefied track list.

[0059] On the other hand, an embodiment of the present invention provides an electronic device, including: a processor and a memory; the memory is used to store programs; the processor executes the program to implement the above-mentioned improved ship track data processing method.

[0060] On the other hand, an embodiment of the present invention provides a computer storage medium storing a program executable by a processor. When the program is executed by the processor, it is used to implement the above-mentioned improved ship track data processing method.

[0061] The embodiment of the present invention periodically obtains the AIS data of each ship in the target sea area; the AIS data includes a timestamp, real-time navigation data and a navigation status code; the AIS data is pre-processed to obtain an AIS data sequence of each ship; the navigation segment data is segmented from the AIS data sequence based on the real-time navigation data and the navigation status code; data thinning is performed based on the inflection points of each segment in the navigation segment data to generate a thinned track list; the ship track of each ship is obtained based on the thinned track list. The present invention adopts a segmented thinning strategy, which can effectively reduce the amount of data storage while maintaining the integrity of key track features; and the present invention effectively optimizes data processing efficiency through a pre-segmentation mechanism based on the status code; in addition, the present invention effectively improves the track segmentation accuracy through a dual verification mechanism that integrates the motion characteristics of the navigation status code and the real-time navigation data. The embodiment of the present invention can efficiently and accurately realize ship track data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation to the technical solution of the present invention.

[0063] Figure 1 This is a schematic diagram of an implementation environment for an improved ship track data processing method provided by an embodiment of the present invention;

[0064] Figure 2 1 is a flow chart of an improved method for processing ship track data provided by an embodiment of the present invention;

[0065] Figure 3 A schematic diagram of the expanded flow of step S100 provided in an embodiment of the present invention;

[0066] Figure 4 A schematic diagram of an application example of data pulling and sorting provided by an embodiment of the present invention;

[0067] Figure 5 A schematic diagram of the expanded flow of step S200 provided in an embodiment of the present invention;

[0068] Figure 6 A schematic diagram of the expanded flow of step S300 provided in an embodiment of the present invention;

[0069] Figure 7 A schematic diagram of the expanded flow of step S500 provided in an embodiment of the present invention;

[0070] Figure 8 A schematic diagram of the overall process of an improved ship track data processing method provided by an embodiment of the present invention;

[0071] Figure 9A schematic structural diagram of an improved ship track data processing device provided by an embodiment of the present invention;

[0072] Figure 10 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0074] It should be noted that although the system diagrams illustrate functional module divisions and the flowcharts illustrate a logical sequence, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the system or the sequence in the flowcharts. The terms "first / S100," "second / S200," and the like in the specification, claims, and drawings are used to distinguish similar objects and are not necessarily intended to describe a specific sequence or precedence.

[0075] References to "embodiments" in this disclosure mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the disclosure. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0076] To facilitate understanding of the technical solution of the present invention, the following technical terms that may appear in the technical solution of the present invention are first explained:

[0077] The Automatic Identification System (AIS) is a new aid-to-navigation system and equipment, now evolving into the Universal Automatic Identification System (UAIS). It typically consists of a VHF communicator and a GPS locator, enabling automatic exchange of vital information such as vessel position and speed, and automatic response. Its purpose is to enhance maritime safety, improve navigation safety and efficiency, and protect the marine environment. Its functions include identifying vessels and assisting in target tracking. AIS strengthens collision avoidance measures, enhances ARPA radar capabilities, and can display vessel information on electronic nautical charts, improving maritime communications.

[0078] The data covered in this application comes from ship AIS data provided by suppliers. This data consists of both static and dynamic ship data. Static ship data primarily includes registered ship parameter information (such as the ship's MMSI (Maritime Mobile Service Identity), length, width, draft, maximum speed, and destination). This information is pre-registered and generally does not change. The other component is ship AIS data, which is sent to operators in relatively real time and primarily includes real-time navigation data (such as current latitude and longitude, current speed, and current heading).

[0079] LSTM (Long Short-Term Memory) is a special type of recurrent neural network (RNN). While traditional RNNs are prone to exploding or vanishing gradients when processing long sequences of data, LSTM effectively addresses these challenges by introducing a gating mechanism. It consists of a forget gate, an input gate, and an output gate. The forget gate determines which information in the previous memory cell state should be forgotten or retained; the input gate generates new information to be updated; and the output gate controls the amount of information output from the current memory cell state to the external state. Leveraging this ability to selectively remember and forget, LSTM is better able to handle long-range dependencies and has been widely used in fields such as speech recognition, machine translation, and time series prediction.

[0080] It is understood that the improved ship track data processing method provided in the embodiments of the present invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiments is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smart phone, tablet computer, laptop computer, desktop computer, etc., but is not limited to this.

[0081] To facilitate understanding of the technical solutions of the present invention, the following are first explained regarding the technical features that may appear in the embodiments of the present invention:

[0082] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1, the implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to the network in a wireless or wired manner to complete data transmission and exchange.

[0083] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0084] In addition, server 101 can also be a node server in a blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0085] The terminal 102 may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited thereto. The terminal 102 and the server 101 may be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0086] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides an improved ship track data processing method. The following is an example of the improved ship track data processing method being applied in the server 101. It can be understood that the improved ship track data processing method can also be applied in the terminal 102.

[0087] Reference Figure 2 , Figure 2 This is a flowchart of an improved ship track data processing method applied to a server according to an embodiment of the present invention. The execution subject of the improved ship track data processing method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method comprises the following steps:

[0088] S100, periodically acquiring AIS data of each ship in the target sea area;

[0089] Among them, AIS data includes timestamp, real-time navigation data and navigation status code; real-time navigation data includes real-time latitude and longitude, real-time speed and real-time heading;

[0090] It should be noted that the AIS data is stored in a time series database; in some embodiments, such as Figure 3As shown, step S100 may include the following steps: S101, in response to a first periodic node, using multiple threads to periodically pull the AIS data of each ship in the target sea area and store it in a temporary cache area of the corresponding thread; S102, in response to a second periodic node, periodically writing the AIS data in the temporary cache area of each thread into a time series database in batches.

[0091] For example, in some specific embodiments, Figure 4 As shown, multi-threaded data pulling can be implemented as follows: AIS data is obtained through the HTTP interface provided by the vendor, which typically requires purchasing an account or interface service permissions. For example, in some ship track data processing projects, there are approximately 9,000 ships, with an average of 1,000 data items per second. All ships generate at least 86 million real-time data items per day. Pulling data through a single thread is slow and has poor timeliness. Therefore, N threads are typically started to pull different portions of data. N is determined based on demand, with a default of 5 threads. Each thread requests data once every first period (for example, every second). The pulled data may contain some duplicate data and may also contain outliers. The pulled data is stored in a temporary buffer pending subsequent processing. Alternatively, AIS data can be stored using the time series database InfluxDB. To improve data writing efficiency, ship data collected within a certain period is first cached and then batch-written at intervals of the second period (for example, every 5 minutes). This data is then written to InfluxDB all at once.

[0092] S200, pre-processing the AIS data to obtain an AIS data sequence of each ship;

[0093] It should be noted that the AIS data also includes a maritime mobile service identification code; in some embodiments, such as Figure 5 As shown, step S200 may include the following steps: S201, classifying and sorting AIS data based on the maritime mobile service identification code and the timestamp to obtain the original AIS data sequence of each ship; S202, filtering the AIS data in the original AIS data sequence for abnormal data based on the real-time navigation data to obtain the target AIS data sequence of each ship.

[0094] In some embodiments, the real-time navigation data includes real-time latitude and longitude, real-time speed and real-time heading; the AIS data in the original AIS data sequence is filtered for abnormal data based on the real-time navigation data, which may include the following steps: taking the first AIS data in the original AIS data sequence as the first AIS data; when the real-time speed of the first AIS data is greater than the maximum design speed of the ship, the first AIS data is removed from the original AIS data sequence, and the step of taking the first AIS data in the original AIS data sequence as the first AIS data is returned to be executed until the real-time speed of the first AIS data is less than or equal to the maximum design speed of the ship; taking the next AIS data after the first AIS data in the original AIS data sequence as the second AIS data; obtaining the track point distance between the first AIS data and the second AIS data based on the real-time latitude and longitude combined with the average radius of the earth; obtaining the average speed of the track point corresponding to the second AIS data based on the track point distance combined with the timestamp; when the average speed is less than the second AIS If the difference ratio of the real-time speed of the data is greater than a first threshold, the second AIS data is determined to be speed anomaly data; the actual heading angle between the track points of the first AIS data and the second AIS data is obtained based on the real-time latitude and longitude; when the difference between the real-time heading angle and the actual heading angle of the second AIS data is greater than a second threshold, the second AIS data is determined to be heading anomaly data; if the second AIS data is speed anomaly data or heading anomaly data, the second AIS data is removed from the original AIS data sequence and the next AIS data of the second AIS data in the original AIS data sequence is used as the second AIS data, and the process returns to the step of obtaining the track point distance between the first AIS data and the second AIS data based on the real-time latitude and longitude combined with the average radius of the earth; otherwise, the second AIS data is used as the first AIS data, and the process returns to the step of using the next AIS data of the first AIS data in the original AIS data sequence as the second AIS data, until all AIS data in the original AIS data sequence are traversed.

[0095] For example, in some embodiments, the classification and sorting of data can be implemented as follows: each AIS data includes real-time information such as the ship's MMSI, current longitude and latitude, current speed, current heading, etc., first classified according to the ship's MMSI, the same MMSI data is stored in a queue, and the data in the queue is sorted from small to large according to the timestamp. In some optional implementations, the data pulled here is basically sorted by timestamp. There may be some data delays that cause a small amount of data to not be sorted by time, so here after classification according to MMSI, the idea of insertion sort can be used for sorting. If the timestamp of the same MMSI is found to be the same during the sorting process, the duplicate data is directly discarded. Figure 4As shown, the ship AIS data can be pulled from the supplier in the form of interface requests through N threads, and then classified and sorted according to the ship MMSI.

[0096] Abnormal data filtering can be achieved as follows:

[0097] ① Abnormal speed data filtering:

[0098] First, we traverse the data one by one to determine if there are any anomalies in the speed. For two consecutive points, A1 (x1, y1) and A2 (x2, y2), where x1 and y1 represent the latitude and longitude of the points, we calculate the distance between the two points based on their latitude and longitude.

[0099]

[0100] Calculate the distance from A1 to A2 according to the formula (unit is km), R is the average radius of the earth. Since AIS data carries timestamp and current speed information, the average speed at A2 can be calculated as:

[0101]

[0102] If the calculated average speed at point A2 is too different from the speed reported by AIS ( ,in Indicates the current speed information reported by point A2 in the AIS data), it indicates that there is a problem with the data of point A2 and point A2 is discarded. The specific process is as follows:

[0103] Assuming the data sequence to be processed is A1, A2, A3, A4, ..., first determine whether the speed of A2 is abnormal based on A1 and A2. If A2 meets expectations, then continue to determine A2 and A3, and so on. If the speed of A2 is abnormal, A2 will be discarded, and then determine A1 and A3, and so on. In addition, if A1 is the first data point, then directly compare its reported speed If the maximum speed of the ship is If the speed exceeds the ship's maximum design speed (this information can be obtained from the ship's basic information), point A1 is abnormal and will be discarded.

[0104] Until all the data is processed, the AIS data with abnormal speed will be filtered out. The abnormal speed may be caused by inaccurate latitude and longitude positioning, or it may be an erroneous report.

[0105] ② Filtering of abnormal heading data:

[0106] First, we traverse the data one by one to determine whether the heading angle is normal. For two consecutive points A1 (x1, y1) and A2 (x2, y2), we calculate the heading angle of the two points based on their latitude and longitude (with north as the reference).

[0107] If the heading angle of point A2 is calculated If the heading angle reported by the AIS differs significantly (by more than 0.5 degrees or the corresponding arc value), there is a problem with the A2 point data and A2 point should be discarded. Then, check each AIS point for abnormal heading angles.

[0108] In this way, most of the abnormal data will be eliminated by judging factors such as speed and direction angle. The data obtained is the data after preliminary cleaning.

[0109] It should also be noted that in some optional implementations, multiple threads can be started to perform data filtering simultaneously, with each thread processing the data of a single vessel. The number of threads is determined by machine performance, and each thread processes data for a fixed duration (one minute by default). After processing, it switches to the next vessel's data. Here, we use the data processing of a single vessel as an example, extracting one minute of AIS data at a time. For a single vessel, we estimate 100-200 AIS data items per minute.

[0110] S300, segmenting the AIS data sequence to obtain navigation segment data based on the real-time navigation data and the navigation status code;

[0111] It should be noted that the real-time navigation data includes real-time latitude and longitude and real-time speed; in some embodiments, such as Figure 6 As shown, step S300 may include the following steps:

[0112] S301, determining the ship status corresponding to each AIS data in the AIS data sequence based on the real-time navigation data and the navigation status code;

[0113] Among them, the ship status includes mooring status and sailing status; when the navigation status code of the AIS data is the first type, the real-time speed does not exceed the third threshold and the real-time longitude and latitude are in the mooring area, the ship status corresponding to the AIS data is determined to be mooring status; when the navigation status code of the AIS data is the second type and the real-time speed is above the fourth threshold, the ship status corresponding to the AIS data is determined to be sailing status; it should be noted that the thresholds applied in the embodiments of the present invention (such as the first threshold, the second threshold, the third threshold, etc.) can be adjusted according to actual needs, and the detailed numerical settings in the specific implementation method are only for example illustration.

[0114] For example, in some specific implementations, anchoring status determination involves: If the AIS navigation status code is 1, 4, or 6 (1 for anchored, 4 for moored, and 6 for stranded), the vessel is anchored. If the real-time speed does not exceed 0.5 knots and the electronic nautical chart indicates that the point is within the anchoring area, the vessel is anchored. Navigation status determination involves: If the AIS navigation status code is 0 (i.e., normal navigation) and the vessel speed is above 1 knot, the vessel is deemed to be underway. If the vessel speed does not exceed half of the design speed, it is generally considered to be sailing at a low speed; otherwise, it is considered to be sailing at a high speed.

[0115] S302: If all AIS data between two AIS data in the moored state in the AIS data sequence are in the sailing state, determine the initial sailing segment data according to the AIS data in the corresponding sailing state;

[0116] For example, data is categorized by vessel status as either moored or sailing. Since moored status can be represented by a track point, Douglas data thinning only requires data from the sailing state, while moored data does not. Therefore, in practice, to minimize data processing overhead, only the sailing segment data can be extracted for subsequent thinning. It's important to note that thinning only requires data between one moored state and the next.

[0117] S303, dividing the initial flight segment data into target flight segment data based on a preset data quantity window;

[0118] Among them, when the data volume of AIS data in the initial voyage segment data is less than twice the data quantity window, the initial voyage segment data is divided into one segment of target voyage segment data; otherwise, the initial voyage segment data is divided into n segments of target voyage segment data, where n is obtained by rounding down the ratio of the data volume of AIS data in the initial voyage segment data to the data quantity window, and the data volume of AIS data in the first n-1 segments of target voyage segment data is equal to the data quantity window.

[0119] For example, in some specific implementations, the window is at least MinWin, which is 1000 by default, i.e., about 30 minutes of navigation data. Here, it is assumed that the data of the ship from berthing to sailing and then to the intermediate state of berthing is CurCnt. If , then the data will be divided according to a window, otherwise it will be divided according to (Round down) divisions.

[0120] For example, assuming that the current navigation data is 900, then there is only one sliding window with a number of 900; assuming that the current navigation data is 1200, then there is only one sliding window with a number of 1200; assuming that the current navigation data is 2200, then there are two sliding windows with a number of 1000 and 1200 respectively.

[0121] S400, performing data thinning based on the inflection points of each segment in the flight segment data to generate a thinned track list;

[0122] It should be noted that the real-time navigation data includes real-time latitude and longitude and real-time speed; in some embodiments, step S400 may include the following steps: S401, using the navigation segment data as the target navigation segment; S402, constructing a target line segment based on the starting point and end point of the target navigation segment, and using the AIS data corresponding to the track point with the largest vertical distance from the target line segment in the target navigation segment as a candidate inflection point; wherein the vertical distance is calculated based on the real-time latitude and longitude corresponding to the starting point, end point and track point using the position geometry relationship; S403, when the vertical distance between the candidate inflection point corresponding to the target navigation segment and the target line segment is less than or equal to a preset dynamic threshold, adding the AIS data corresponding to the starting point and end point of the target navigation segment to the thinned track list; otherwise, executing subsequent steps; S404 using the candidate inflection point corresponding to the target navigation segment as the target inflection point; wherein the expression of the dynamic threshold is:

[0123]

[0124] In the formula, Delt represents the dynamic threshold corresponding to the candidate inflection point; BaseDelt represents the preset minimum thinning distance; and Indicates the preset correlation coefficient; Indicates the real-time speed of the candidate inflection point; Indicates the average speed from the starting point to the end point; Indicates the heading angle of the candidate inflection point compared to the starting point; Indicates the heading angle of the end point compared to the starting point;

[0125] S405. Add the AIS data corresponding to the target inflection point to the thinned track list, and split the target navigation segment into two sub-segments based on the target inflection point. S406. Use the sub-segment as the target navigation segment, and return to the step of constructing a target line segment based on the start and end points of the target navigation segment until the vertical distance between the candidate inflection point corresponding to the target navigation segment and the target line segment is less than or equal to the dynamic threshold of the corresponding target navigation segment. Then, add the AIS data corresponding to the start and end points of the target navigation segment to the thinned track list.

[0126] For example, in some specific implementations, it is assumed that the starting point of the window is The end point is .Will and Connect them into a line and traverse them one by one , calculate to line segment and The vertical distance of the connection.

[0127] It should be noted that AIS uses longitude and latitude to represent coordinates because the Earth is an ellipsoid, not a flat surface. Calculating perpendicular distances is complex. Considering the maximum sliding window distance of 20 km, we can assume it is on a flat surface, ignoring the effect of Earth curvature, and the coordinates are simply longitude and latitude.

[0128] Here we first need to find the longitude and latitude coordinates of the foot of the perpendicular, and then we can calculate the perpendicular distance. Assume there are three points B1 (x1, y1), B2 (x2, y2), and B3 (x3, y3), and B2 and B3 are connected by a straight line. The equation is as follows:

[0129]

[0130] Then B1 to the straight line The coordinate formula of the intersection point (also known as the foot of the perpendicular) of the perpendicular is (x4, y4):

[0131]

[0132]

[0133] Then the perpendicular distance is the distance from B1 (x1, y1) to the foot of the perpendicular (x4, y4). After obtaining the longitude and latitude coordinates in this article, the distance between the two points can be calculated based on the longitude and latitude of the two points (for the calculation principle, see the specific implementation of step S200).

[0134] calculate and Angle and average speed (For the calculation principle, see the specific implementation of step S200), and then traverse one by one , calculate to line segment and The vertical distance VerLen is finally found at the data point (i.e., the candidate inflection point) where the vertical distance is the largest.

[0135] Currently found in the sliding window (i.e. target flight segment), data point The vertical distance is the largest (assuming it is VerLen), that is, It is the largest inflection point. Then calculate the dynamic threshold value Delt, the formula is as follows:

[0136]

[0137] BaseDelt represents the minimum thinning distance, which can be 0.1 by default, i.e. 0.1km. and Indicates the correlation coefficient, the value range is [0,1], and the default value can be 0.5. express The speed at express Compared to the heading angle at the starting point of the sliding window, the greater the speed difference and the heading angle difference, the larger the threshold value Delt.

[0138] if , first the inflection point Add to the list of thinned tracks (mark the turning point), and then divide the sliding window into and Two sub-windows, and finally recursively process the sliding window by overloading the inflection point determination and window division steps and .

[0139] if , then and Add to the list of rarefied tracks and end the recursive search, where The data points are discarded because they are not sufficient to reflect the ship's track.

[0140] S500: Obtain the ship track of each ship based on the thinned track list.

[0141] It should be noted that, in some embodiments, Figure 7 As shown, step S500 may include the following steps: S501, sorting all AIS data in the thinned track list based on timestamps to obtain a time-series track list; S502, performing equidistant thinning on the AIS data in the time-series track list based on a first spacing to obtain an equidistant track list; wherein, the specific execution operation of the equidistant thinning includes: when the spacing between two adjacent track points in the time-series track list is less than or equal to the first spacing, removing the AIS data corresponding to the latter track point from the time-series track list; S503, performing track prediction using LSTM based on the AIS data of each track point in the equidistant track list to obtain a target track list; wherein, the track prediction includes filling in the missing track points between adjacent track points in the equidistant track list whose spacing is greater than the second spacing and predicting the next track point; S504, drawing the ship track according to the AIS data of each track point in the target track list.

[0142] For example, in some specific implementations, Douglas data thinning is performed on the data. Its core idea is to find inflection points through recursion. The data obtained may be out of order, so it can be reordered according to timestamps. Although the thinned data is obtained by marking the inflection points, the data still has the following problems:

[0143] 1) The data points are likely to be unevenly distributed. For example, the spacing between windows may be small.

[0144] 2) The ship may experience data packet loss due to poor network conditions, and the ship's track may not be continuous.

[0145] To solve the above problem, you can continue to process the data through the following steps:

[0146] ① Equal spacing: The equal spacing here is to make the ship tracks as evenly distributed as possible. The average spacing is dynamically changing, and it is related to the scale of the map showing the ship tracks. The relationship is as follows: the scale of the map is 1:D, and the spacing of the data points is also D (in meters). Assume that the data series are 、 、 ..., calculate one by one and distance ,if , and Keep. Otherwise, remove the point , continue to compare and distance, and so on.

[0147] ②Introducing LSTM for data processing:

[0148] a. Supplement missing data:

[0149] When traversing track data, if the data interval exceeds twice the average spacing, this indicates possible data loss. To better display the ship's track changes, it is necessary to fill in some point data to make the ship's track smoother. Here, an LSTM neural network is used for track prediction. During the equidistant thinning process, five input variables (timestamp, longitude, latitude, speed, and heading) are fed into the LSTM neural network. The output is the position information for the next 5 to 10 positions, with the number of data points used determined by the dynamic spacing.

[0150] The basic parameters of LSTM are: the input layer has 5 nodes, the output layer has 2 nodes (that is, information such as longitude and latitude), the hidden layer contains a multi-layer LSTM structure, and each layer has 150 neurons by default. The Adam optimizer is used to prevent overfitting, and the root mean square error (RMSE) is used to measure the longitude and latitude deviation.

[0151] In general, based on the LSTM neural network, by traversing the equidistantly thinned data, the missing data is filled in, so that the ship's track distribution is more continuous.

[0152] b. Predict the next flight path:

[0153] Here, an LSTM neural network is used to predict the future ship track. The five input variables, including the timestamp, longitude, latitude, speed, and heading of the existing data, are input into the LSTM neural network, and the output predicts 10 to 20 positions as an assessment of the ship's future dynamics.

[0154] The basic parameters of LSTM are: the input layer has 5 nodes, the output layer has 2 nodes (that is, information such as longitude and latitude), the hidden layer contains a multi-layer LSTM structure, and each layer has 150 neurons by default. The Adam optimizer is used to prevent overfitting, and the root mean square error (RMSE) is used to measure the longitude and latitude deviation.

[0155] The ship's existing track is drawn with a solid line, and the future track is drawn with a dotted line. Considering that ships are usually in constant sailing, a large amount of real track data will be obtained as time goes by. By continuously inputting this data into the LSTM neural network, the ship's track prediction ability will become more and more accurate.

[0156] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0157] First of all, it should be noted that in some ship data processing projects, the number of ships involved in the target sea area is about 90,000, and the real-time data generated by all ships is at least 86 million per day, an average of 1,000 data per second. Based on this data, it is necessary to draw the tracks of all ships, analyze and explore the ship status from multiple angles, and realize the detection of abnormal ship behavior.

[0158] Existing work directly draws ship tracks based on AIS data, but does not process noise and duplicate data. There are also cases of missing tracks, and ship tracks appear rough and messy, which is not practical enough. At the same time, using traditional relational databases (such as MySQL) to store large-scale ship trajectory big data has problems such as low storage efficiency, poor query performance, and difficulty in processing large-scale time series data.

[0159] In view of this, the present invention proposes an improved method for processing ship track data, such as Figure 8 As shown, the embodiment of the present invention can be implemented through the following steps:

[0160] Step 1: Pull data using multiple threads. AIS data is obtained through the HTTP interface provided by the vendor, which typically requires purchasing an account or interface service permissions. In our case, there are approximately 9,000 ships, with an average of 1,000 data items per second. All ships generate at least 86 million pieces of real-time data per day. Pulling data through a single thread is slow and time-sensitive, so N threads are typically started to pull different parts of the data. N is determined based on demand, with a default of 5 threads, and each thread requests data once per second. The pulled data may contain some duplicates and outliers. The pulled data is stored in a temporary buffer pending subsequent processing. Go to Step 2.

[0161] Step 2: Store and query ship trajectories. AIS data is stored using the time-series database InfluxDB. To improve data writing efficiency, ship data collected within a certain period is first cached, then batch-written at 5-minute intervals, and then written all at once to InfluxDB. Furthermore, the time-series database allows the following operations: Error handling: During the data writing process, network failures and database connection anomalies may occur. To ensure data integrity, this method incorporates a corresponding error handling mechanism. If a write fails, the data is retried, with a maximum retry count of 1000. If the maximum number of retries is exceeded and the data still fails, it is stored in a local file and written again after the fault is resolved. Data storage optimization: Data partitioning is implemented by partitioning data daily to reduce the amount of data scanned during queries, improving query performance. Index optimization: Based on InfluxDB's automatic indexing of timestamps and tags, additional index optimization is performed on tags frequently used in query conditions, according to actual query requirements, to further improve query efficiency. Time range query: Query a ship's navigation trajectory, speed change, and other information within a specific time period using a time range. Conditional query: In addition to the time range, query and analyze specific ship behaviors based on other conditions, such as ship MMSI, ship type, etc. Cache mechanism query: This project uses a cache mechanism for the results of some frequent queries. Specifically, it mainly queries the status of all ships in the target sea area, and sets a traversal of the entire target sea area every 60 minutes. When the user has a query request, the result is first obtained from the cache. If it does not exist in the cache, the actual query operation is performed, and the query result is stored in the cache, which improves the efficiency of ship queries in a large area. Paginated query display: Because this project involves a relatively wide range of data queries, when the query result data volume is large, a paginated query method is used, and only part of the data is returned each time. This reduces the amount of data for a single query and improves the query response speed. Go to step 3.

[0162] Step 3: Data classification and sorting. Each AIS data includes real-time information such as the ship's MMSI, current latitude and longitude, current speed, and current heading. First, the data is classified according to the ship's MMSI. The same MMSI data is stored in a queue, and the data in the queue is sorted from smallest to largest by timestamp. It should be noted that the data pulled here is basically sorted by timestamp. There may be some data delays that cause a small amount of data to be not sorted by time. Therefore, after sorting by MMSI, the insertion sort principle can be used for sorting. During the sorting process, if the timestamp of the same MMSI is found to be the same, the duplicate data is directly discarded.

[0163] like Figure 4 As shown, N threads are used to pull ship AIS data from the supplier through interface requests, and then the data is classified and sorted according to the ship's MMSI. Go to step 4.

[0164] Step 4: Filter data based on a sliding window. Start multiple threads, each processing data for one vessel. The number of threads depends on machine performance. Each thread processes data for a fixed duration (one minute by default). After processing, it switches to the next vessel's data. Here, we use a single vessel as an example, extracting one minute of AIS data each time. For a single vessel, we estimate 100-200 AIS data items per minute.

[0165] First, we traverse the data one by one to determine if there are any anomalies in the speed. For two consecutive points, A1 (x1, y1) and A2 (x2, y2), where x1 and y1 represent the latitude and longitude of the points, we calculate the distance between the two points based on their latitude and longitude.

[0166]

[0167] Calculate the distance from A1 to A2 according to the formula (Unit is km), R is the average radius of the earth. Since AIS data carries timestamp and current speed information, the average speed at A2 can be calculated as

[0168]

[0169] If the calculated average speed at point A2 is too different from the speed reported by AIS ( ,in Indicates the current speed information reported by point A2 in the AIS data), it indicates that there is a problem with the data of point A2 and point A2 is discarded. The specific process is as follows:

[0170] Assuming the data sequence to be processed is A1, A2, A3, A4, ..., first determine whether the speed of A2 is abnormal based on A1 and A2. If A2 meets expectations, then continue to determine A2 and A3, and so on. If the speed of A2 is abnormal, A2 will be discarded, and then determine A1 and A3, and so on. In addition, if A1 is the first data point, then directly compare its reported speed If the maximum speed of the ship is If the speed exceeds the ship's maximum design speed (this information can be obtained from the ship's basic information), point A1 is abnormal and will be discarded.

[0171] Until all the data is processed, the AIS data with abnormal speed will be filtered out. The abnormal speed may be caused by inaccurate latitude and longitude positioning, or it may be an erroneous report.

[0172] First, we traverse the data one by one to determine whether the heading angle is normal. For two consecutive points A1 (x1, y1) and A2 (x2, y2), we calculate the heading angle of the two points based on their latitude and longitude (with north as the reference).

[0173] If the heading angle of point A2 is calculated If the heading angle reported by the AIS differs significantly (by more than 0.5 degrees or the corresponding arc value), there is a problem with the A2 point data and A2 point should be discarded. Then, check each AIS point for abnormal heading angles.

[0174] In this way, most of the abnormal data will be eliminated by judging factors such as speed and direction angle. The data obtained is the data after preliminary cleaning.

[0175] Here, steps 1 through 4 receive raw AIS data and filter out duplicate and abnormal data. Each time, data is processed for a fixed duration (the default is 1 minute), forming a sliding window. After processing the current window, the next data is processed, and the process continues with step 1 to continuously acquire AIS data. The filtered data is then processed in step 5.

[0176] Step 5: Segment the data based on the ship's status. Next, the ship's status (berthed or sailing normally) is determined based on the AIS data.

[0177] Mooring status determination: If the AIS navigation status code is 1, 4 or 6 (1 for anchored, 4 for moored, 6 for stranded), it indicates that the ship is in a mooring state. If the real-time speed does not exceed 0.5 knots and the electronic nautical chart shows that the point is in the mooring area, the ship is in a mooring state.

[0178] Navigation status determination: If the AIS navigation status code is 0 (normal navigation) and the ship's speed is above 1 knot, the ship is considered to be in navigation status. If the ship's speed does not exceed half of the design speed, it is generally considered to be sailing at a low speed; otherwise, it is sailing at a high speed.

[0179] In the subsequent thinning process, we prioritize retaining points where the ship's status changes, as these often represent key milestones in a ship's navigation. For example, when a ship transitions from sailing to anchored, this transition point is crucial for analyzing ship behavior and predicting its trajectory, and should therefore be retained. Therefore, we traverse the AIS data one by one to identify the ship's status at that point.

[0180] After this processing step is complete, the AIS data undergoes preliminary cleaning and status determination, and is then written to the database. For AIS data storage, a database table is created daily, and all AIS data is written to this table. The automatic serial number is the primary key, and the MMSI, timestamp, and latitude and longitude are indexes. Go to step 6.

[0181] Step 6: Process data based on a sliding window. When it is necessary to display the navigation track of a ship, the AIS data of the relevant ship is read from the database. The obtained data is classified according to MMSI, and the data is sorted from small to large according to the timestamp. Here we take the processing of a ship's track as an example. The data is divided into moored and sailing categories according to the ship status. Because the moored status can be displayed according to a track point, the data that needs to participate in Douglas data thinning is the data in the sailing state, and the moored data does not need to participate. In addition, it should be noted that only the data between one moored state and the next moored state needs to be thinned. The sliding window here is dynamically divided based on the number of data. The window is at least MinWin, and the default is 1000, which is about 30 minutes of navigation data. It is assumed here that the data of the ship from mooring to sailing and then to the intermediate state of mooring is CurCnt. If , then the data will be divided according to a window, otherwise it will be divided according to (Round down) divisions.

[0182] For example, if the current number of navigation data is 900, then there is only one sliding window with 900 items; if the current number of navigation data is 1200, then there is only one sliding window with 1200 items; if the current number of navigation data is 2200, then there are two sliding windows with 1000 and 1200 items respectively.

[0183] If the ship stops and starts frequently, there will be multiple "moored->sailing->moored" status data. The same window segmentation method is executed for the data of each sailing segment.

[0184] Here's an explanation of how MinWin is determined. A normal ship's AIS data is transmitted at most every 2 seconds, so 1000 data points translates to approximately 30 minutes. Ships typically travel at moderate speeds, with economic speeds typically between 10 and 20 knots, or 18 to 36 km / h (5 to 10 m / s). Therefore, the default MinWin value is 1000. For a roughly half-hour voyage, the travel distance is approximately 10 to 20 km. This allows for more precise data thinning and improves computational efficiency.

[0185] The default starting point of the sliding window is The end point is At this point, the data thinning operation begins, so go to step 7.

[0186] Step 7: Find the inflection point. Assume that the starting point of the window is The end point is .Will and Connect them into a line and traverse them one by one , calculate to line segment and The vertical distance of the connection.

[0187] It should be noted that AIS uses longitude and latitude to represent coordinates because the Earth is an ellipsoid, not a flat surface. Calculating perpendicular distances is complex. Considering the maximum sliding window distance of 20 km, we can assume it is on a flat surface, ignoring the effect of Earth curvature, and the coordinates are simply longitude and latitude.

[0188] Here we first need to find the longitude and latitude coordinates of the foot of the perpendicular, and then we can calculate the perpendicular distance. Assume there are three points B1 (x1, y1), B2 (x2, y2), and B3 (x3, y3), and B2 and B3 are connected by a straight line. The equation is as follows:

[0189]

[0190] Then B1 to the straight line The coordinate formula of the intersection point (also known as the foot of the perpendicular) of the perpendicular is (x4, y4):

[0191]

[0192]

[0193] Then the perpendicular distance is the distance from B1 (x1, y1) to the foot of the perpendicular (x4, y4). After obtaining the longitude and latitude coordinates in this article, the distance between the two points can be calculated based on the longitude and latitude of the two points (see step 4 for the formula).

[0194] calculate and Angle and average speed (Refer to step 4 for the formula), and then traverse one by one , calculate to line segment and The vertical distance VerLen is finally found at the data point The vertical distance is the largest. Go to step 8.

[0195] Step 8: Satisfy the end condition. In the data point The vertical distance is the largest (assuming it is VerLen), that is, It is the largest inflection point. Then calculate the dynamic threshold value Delt, the formula is as follows:

[0196]

[0197] BaseDelt represents the minimum thinning distance, which defaults to 0.1, i.e. 0.1km. and Indicates the correlation coefficient, the value range is [0,1], and the default value is 0.5. express The speed at express Compared to the heading angle at the starting point of the sliding window, the greater the speed difference and the heading angle difference, the larger the threshold value Delt.

[0198] if , first the inflection point Add to the list of thinned tracks (mark the turning point), and then divide the sliding window into and Two sub-windows, finally go to step 7 to recursively process the sliding windows and .

[0199] if , then and Add to the list of rarefied tracks and end the recursive search, where The data points will be discarded because they are not sufficient to reflect the ship's track, and go to step 8.

[0200] In general, the core idea of steps 7-8 is to obtain the inflection point of the sliding window and then recursively divide the sliding window until the window cannot be further divided or there are no more inflection points. At this point, the entire process ends and the thinned data is obtained. Specifically, steps 7 and 8 are the recursive execution of the data thinning process. Step 6 is to obtain the window to be processed. There may be multiple windows. Each window recursively executes the process of steps 7 and 8, so that each window is thinned.

[0201] Step 9: Obtain preliminary thinned data. This step indicates that Douglas data thinning has been completed. The core idea is to find inflection points through recursion. The resulting data is out of order and needs to be reordered by timestamp. Specifically, the inflection point data obtained during the recursive execution is not sorted by time, so step 9 requires reordering all the data. Although thinned data is obtained by marking inflection points, the following problems still exist with this data:

[0202] 1) The data points are likely to be unevenly distributed. For example, the spacing between windows may be small.

[0203] 2) The ship may experience data packet loss due to poor network conditions, and the ship's track may not be continuous.

[0204] To solve the above problem, we need to continue processing the data and go to step 10. In addition, equidistant thinning can solve the uneven distribution of track points caused by recursive thinning, facilitating subsequent missing point detection and LSTM prediction.

[0205] Step 10: Perform equal-distance thinning. The equal-distance thinning here is to make the ship tracks as evenly distributed as possible. The average spacing is dynamically changing and is related to the scale of the map showing the ship tracks. The relationship is as follows: the scale of the map is 1:D, and the spacing of the data points is also D (in meters). Assume that the data series are 、 、 ..., calculate one by one and distance ,if , and Keep. Otherwise, remove the point , continue to compare and Repeat this process. Go to step 11.

[0206] Step 11: Supplement missing data. When traversing the track data, if the data interval exceeds twice the average spacing, this indicates possible data loss. To better visualize the ship's track changes, it is necessary to supplement the data at certain points to make the ship's track smoother. Track prediction is performed using an LSTM neural network. During the equidistant thinning process, five input variables—timestamp, longitude, latitude, speed, and heading—are fed into the LSTM neural network. The output is the position information for the next 5 to 10 locations, with the number of data points used determined by the dynamic spacing.

[0207] The basic parameters of LSTM are: the input layer has 5 nodes, the output layer has 2 nodes (that is, information such as longitude and latitude), the hidden layer contains a multi-layer LSTM structure, and each layer has 150 neurons by default. The Adam optimizer is used to prevent overfitting, and the root mean square error (RMSE) is used to measure the longitude and latitude deviation.

[0208] In general, based on the LSTM neural network, by traversing the equidistantly thinned data, the missing data is filled in, so that the ship's track distribution is more continuous, and go to step 12.

[0209] Step 12: Predict the next track. Here, we use an LSTM neural network to predict the future ship's track. Five input variables—timestamp, longitude, latitude, speed, and heading—are fed into the LSTM neural network. The network then outputs 10 to 20 predicted locations, which serve as an assessment of the ship's future dynamics.

[0210] The basic parameters of LSTM are: the input layer has 5 nodes, the output layer has 2 nodes (that is, information such as longitude and latitude), the hidden layer contains a multi-layer LSTM structure, and each layer has 150 neurons by default. The Adam optimizer is used to prevent overfitting, and the root mean square error (RMSE) is used to measure the longitude and latitude deviation.

[0211] The ship's existing track is drawn with a solid line, and the future track is drawn with a dotted line. Considering that ships are usually in constant sailing, a large amount of real track data will be obtained as time goes by. By continuously inputting this data into the LSTM neural network, the ship's track prediction ability will become more and more accurate.

[0212] In summary, the present invention proposes an improved method for processing ship track data. The beneficial effects of the embodiments of the present invention include but are not limited to:

[0213] 1) The data filtering method of the moving time window, including data reordering, duplicate data filtering and abnormal data removal, simplifies the data application scenario and reduces the data storage cost.

[0214] 2) The Douglas data thinning method is improved by introducing dynamic threshold and inflection point calculation to better complete data simplification based on sliding windows.

[0215] 3) The missing data detection process was optimized, and missing data was supplemented based on LSTM, while future flight paths were predicted, improving the completeness and accuracy of the data.

[0216] 4) Optimized data storage and query. Ship trajectory data is stored in an optimized time series based on the InfluxDB time series database, which improves the storage efficiency and query performance of ship trajectory big data.

[0217] On the other hand, Figure 9 As shown, an embodiment of the present invention provides an improved ship track data processing device 900, which may include:

[0218] The first module 901 is used to periodically obtain AIS data of each ship in the target sea area; the AIS data includes a timestamp, real-time navigation data and navigation status code;

[0219] The second module 902 is used to pre-process the AIS data to obtain the AIS data sequence of each ship;

[0220] The third module 903 is configured to segment the AIS data sequence to obtain the navigation segment data based on the real-time navigation data and the navigation status code;

[0221] The fourth module 904 is configured to perform data thinning based on the inflection points of each segment in the flight segment data to generate a thinned track list;

[0222] The fifth module 905 is used to obtain the ship track of each ship based on the thinned track list.

[0223] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0224] In another aspect, an embodiment of the present invention further provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor implements the improved ship track data processing method described above when executing the computer program. The electronic device can be any smart terminal, including a tablet computer and an in-vehicle computer.

[0225] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0226] like Figure 10 As shown, Figure 10 The hardware structure of an electronic device 1000 according to another embodiment is shown. The electronic device 1000 includes:

[0227] The processor 1001 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0228] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called by the processor 1001 to execute the network node population optimization method of the embodiment of the present invention.

[0229] Input / output interface 1003, used to implement information input and output;

[0230] Communication interface 1004, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0231] Bus 1005 , which transmits information between various components of the device (e.g., processor 1001 , memory 1002 , input / output interface 1003 , and communication interface 1004 );

[0232] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0233] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one location or distributed across multiple network units. Some or all of these modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0234] The contents of the method embodiments of the present invention are all applicable to the electronic device embodiments. The functions specifically implemented by the electronic device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0235] Another aspect of an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the above method.

[0236] It should be noted that the computer-readable medium described in the embodiments of the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0237] The contents of the method embodiments of the present invention are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0238] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0239] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0240] It should be noted that although several modules of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0241] Through the above description of the embodiments, it will be readily understood by those skilled in the art that the example embodiments described herein can be implemented via software or via a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present invention can be embodied in the form of a software product, which can be stored on a non-volatile storage medium (such as a CD-ROM, USB flash drive, or mobile hard drive) or on a network and includes instructions for causing a computing device (such as a personal computer, server, touch terminal, or network device) to execute the methods according to the embodiments of the present invention.

[0242] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented by the present invention. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0243] Furthermore, while the present invention has been described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed in the present invention, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art will be able to implement the present invention as set forth in the claims using ordinary skill without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0244] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0245] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution apparatus, device, or apparatus (e.g., a computer-based apparatus, a processor-included apparatus, or other apparatus that can fetch and execute instructions from, an instruction execution apparatus, device, or apparatus). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution apparatus, device, or apparatus.

[0246] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a computer memory.

[0247] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0248] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0249] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0250] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. An improved ship track data processing method, characterized in that: The following steps are involved: Periodically obtain AIS data of each ship in the target sea area; the AIS data includes timestamp, real-time navigation data and navigation status code; The real-time navigation data includes real-time latitude and longitude, real-time speed and real-time heading; Preprocessing the AIS data to obtain an AIS data sequence of each ship; Segmenting the AIS data sequence to obtain navigation segment data based on the real-time navigation data and the navigation status code; Performing data thinning based on the inflection points of each segment in the flight segment data to generate a thinned track list includes the following steps: Using the flight segment data as the target flight segment; Constructing a target line segment according to the starting point and the end point of the target navigation segment, and taking the AIS data corresponding to the track point with the largest vertical distance from the target line segment in the target navigation segment as a candidate inflection point; The vertical distance is calculated based on the real-time longitude and latitude corresponding to the starting point, the end point, and the track point using a positional geometric relationship; When the vertical distance between the candidate inflection point corresponding to the target navigation segment and the target line segment is less than or equal to a preset dynamic threshold, the AIS data corresponding to the starting point and the end point of the target navigation segment are added to the thinned track list; otherwise, Taking the candidate inflection point corresponding to the target navigation segment as the target inflection point; adding the AIS data corresponding to the target inflection point to the thinned track list, and splitting the target flight segment into two sub-flight segments based on the target inflection point; Taking the sub-segment as the target segment, returning to the step of constructing a target segment based on the starting point and the end point of the target segment, until the vertical distance between the candidate inflection point corresponding to the target segment and the target segment is less than or equal to the dynamic threshold of the target segment, and adding the AIS data corresponding to the starting point and the end point of the target segment to the thinned track list; The ship track of each ship is obtained based on the thinned track list.

2. The improved ship track data processing method according to claim 1, characterized in that: The AIS data is stored in a time series database; and the periodic acquisition of the AIS data of each ship in the target sea area includes the following steps: In response to a first periodic node, using multiple threads to periodically pull the AIS data of each ship in the target sea area and store it in a temporary buffer area of the corresponding thread; In response to the second periodic node, the AIS data in the temporary buffer area of each thread is periodically written in batches into the time series database.

3. The improved ship track data processing method according to claim 1, characterized in that: The AIS data also includes a maritime mobile service identification code; and preprocessing the AIS data to obtain an AIS data sequence of each ship includes the following steps: Classify and sort the AIS data based on the maritime mobile service identification code and the timestamp to obtain an original AIS data sequence of each ship; Abnormal data is filtered out of the AIS data in the original AIS data sequence based on the real-time navigation data to obtain a target AIS data sequence for each ship.

4. The improved ship track data processing method according to claim 3, characterized in that: The filtering of abnormal data on the AIS data in the original AIS data sequence based on the real-time navigation data comprises the following steps: taking the first AIS data in the original AIS data sequence as the first AIS data; When the real-time speed of the first AIS data is greater than the designed maximum speed of the ship, the first AIS data is removed from the original AIS data sequence, and the process returns to the step of using the first AIS data in the original AIS data sequence as the first AIS data, until the real-time speed of the first AIS data is less than or equal to the designed maximum speed of the ship; taking the next AIS data of the first AIS data in the original AIS data sequence as the second AIS data; Obtaining a track point distance between the first AIS data and the second AIS data based on the real-time longitude and latitude combined with the average radius of the earth; Obtaining an average speed of the track point corresponding to the second AIS data based on the track point distance and the timestamp; When the difference ratio between the average speed and the real-time speed of the second AIS data is greater than a first threshold, determining that the second AIS data is abnormal speed data; Obtaining an actual heading angle between the first AIS data track point and the second AIS data track point based on the real-time latitude and longitude calculation; When a difference between the real-time heading angle of the second AIS data and the actual heading angle is greater than a second threshold, determining that the second AIS data is abnormal heading data; If the second AIS data is the abnormal speed data or the abnormal heading data, the second AIS data is removed from the original AIS data sequence and the next AIS data of the second AIS data in the original AIS data sequence is used as the second AIS data. The process returns to executing the step of calculating the track point distance between the first AIS data and the second AIS data based on the real-time latitude and longitude combined with the average radius of the earth. Otherwise, the second AIS data is used as the first AIS data, and the process returns to executing the step of calculating the next AIS data of the first AIS data in the original AIS data sequence as the second AIS data, until all the AIS data in the original AIS data sequence are traversed.

5. The improved ship track data processing method according to claim 1, characterized in that: The step of segmenting the AIS data sequence based on the real-time navigation data and the navigation status code to obtain the navigation segment data comprises the following steps: Determine the ship status corresponding to each AIS data in the AIS data sequence based on the real-time navigation data and the navigation status code; Wherein, the ship status includes a mooring status and a sailing status; when the sailing status code of the AIS data is of the first type, the real-time speed does not exceed the third threshold value, and the real-time longitude and latitude are in the mooring area, the ship status corresponding to the AIS data is determined to be the mooring status; when the sailing status code of the AIS data is of the second type and the real-time speed is above the fourth threshold value, the ship status corresponding to the AIS data is determined to be the sailing status; If all the AIS data between two AIS data in the moored state in the AIS data sequence are in the sailing state, determining initial sailing segment data according to the AIS data corresponding to the sailing state; dividing the initial flight segment data into target flight segment data based on a preset data quantity window; When the amount of the AIS data in the initial flight segment data is less than twice the data quantity window, the initial flight segment data is divided into one segment of the target flight segment data; otherwise, the initial flight segment data is divided into n segments of the target flight segment data, where n is obtained by rounding down the ratio of the amount of the AIS data in the initial flight segment data to the data quantity window, and the amount of the AIS data in the first n-1 segments of the target flight segment data is equal to the data quantity window.

6. The improved ship track data processing method according to claim 1, characterized in that: The expression of the dynamic threshold is: In the formula, Delt represents the dynamic threshold corresponding to the candidate inflection point; BaseDelt represents the preset minimum thinning distance; and Indicates the preset correlation coefficient; Indicates the real-time speed of the candidate inflection point; Indicates the average speed from the starting point to the end point; Indicates the heading angle of the candidate inflection point compared to the starting point; Indicates the heading angle of the end point compared to the starting point.

7. The improved ship track data processing method according to claim 1, characterized in that: The process of obtaining the track of each ship based on the thinned track list comprises the following steps: sorting all the AIS data in the thinned track list based on the timestamp to obtain a time-series track list; performing equidistant thinning on the AIS data in the time-series track list based on a first spacing to obtain an equidistant track list; The specific execution operation of the equidistant thinning includes: when the distance between two adjacent track points in the time-series track list is less than or equal to the first distance, removing the AIS data corresponding to the latter track point from the time-series track list; Based on the AIS data of each track point in the equidistant track list, track prediction is performed using LSTM to obtain a target track list; The track prediction includes filling in the missing track points between adjacent track points with a spacing greater than a second spacing in the equidistant track list and predicting the next track point; The ship track is obtained by drawing the AIS data of each track point in the target track list.

8. An improved ship track data processing device, characterized in that: include: The first module is used to periodically obtain AIS data of each ship in the target sea area; the AIS data includes a timestamp, real-time navigation data and navigation status code; The real-time navigation data includes real-time latitude and longitude, real-time speed and real-time heading; The second module is used to pre-process the AIS data to obtain the AIS data sequence of each ship; A third module is configured to segment the AIS data sequence to obtain navigation segment data based on the real-time navigation data and the navigation status code; A fourth module is configured to perform data thinning based on the inflection points of each segment in the flight segment data to generate a thinned track list; The step of performing data thinning based on the inflection points of each segment in the flight segment data to generate a thinned track list includes the following steps: Using the flight segment data as the target flight segment; Constructing a target line segment according to the starting point and the end point of the target navigation segment, and taking the AIS data corresponding to the track point with the largest vertical distance from the target line segment in the target navigation segment as a candidate inflection point; The vertical distance is calculated based on the real-time longitude and latitude corresponding to the starting point, the end point, and the track point using a positional geometric relationship; When the vertical distance between the candidate inflection point corresponding to the target navigation segment and the target line segment is less than or equal to a preset dynamic threshold, the AIS data corresponding to the starting point and the end point of the target navigation segment are added to the thinned track list; otherwise, Taking the candidate inflection point corresponding to the target navigation segment as the target inflection point; adding the AIS data corresponding to the target inflection point to the thinned track list, and splitting the target flight segment into two sub-flight segments based on the target inflection point; Taking the sub-segment as the target segment, returning to the step of constructing a target segment based on the starting point and the end point of the target segment, until the vertical distance between the candidate inflection point corresponding to the target segment and the target segment is less than or equal to the dynamic threshold of the target segment, and adding the AIS data corresponding to the starting point and the end point of the target segment to the thinned track list; The fifth module is used to obtain the ship track of each ship based on the thinned track list.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images