Multi-level efficient ship trajectory compression method

Abstract

The invention discloses a ship multi-level high-efficiency track compression method, which comprises the following steps of: firstly connecting a kafka data source, dynamically generating a track compression algorithm with a plurality of levels connected in series according to program configuration, default to generate four levels, namely day, week, month and quarter, and then, processing and reserving data on each layer through the compression algorithm, and sending the data to the kafka for storage. According to the technical scheme, track compression is calculated in real time, compression is continuous, new storage points can be checked and displayed immediately, the problem of off-line calculation window boundary data redundancy does not exist, flexibility, high performance and high adaptability are achieved, the requirement for reserving track detailed characteristics in short query time can be met, and the method is suitable for large-scale popularization and application. The method can also guarantee a rough track with long query time and high-performance quick response, can meet the query requirements of different tracks from several minutes to several years, can flexibly configure the layer number of track compression algorithms, is in father-son series connection in each layer, is high in performance, and can be flexibly combined with a complex algorithm and a simple algorithm in each layer. And most track storage and query scenes are met.

Description

Technical Field

[0001] The present invention relates to the technical field of ship trajectory compression, in particular to a multi-level and efficient ship trajectory compression method. Background Art

[0002] The marine ship monitoring system can continuously obtain the location information of ships through RADAR, AIS, BDS and other equipment. These data form the trajectory of each ship. The amount of data generated every day is very large, reaching 500-1 billion records, and the amount of data in a few years can even reach trillions.

[0003] The track query function needs to display different track details according to the length of the query time period. The track of the ship can be queried for a few hours, a few days, a few months, or even a few years. For queries in a shorter time period, the system should provide as complete a track as possible, showing the ship's position, speed, and navigation status at each moment, and retaining behavioral details such as turning, turning, accelerating, and decelerating. For queries in a longer time period, the system should simplify the track and only show the overall track trend of the ship, such as the daily berthing position or driving path. At the same time, whether it is a short or long time period query, the system needs to return results quickly and maintain high performance.

[0004] The existing multi-level efficient ship trajectory compression method has the following disadvantages when used:

[0005] 1. There are too many ship position track points, which occupy a large amount of storage resources. There is only one level of track compression, and the track compression rate is low. When querying long-term tracks, it is slow to retrieve data from the database. It is also necessary to compress the data twice in the interface to ensure that the data volume is small when it is transmitted to the front end. At the same time, when the interface service is compressed twice, the load is relatively large, and the track points displayed on the chart are not too dense;

[0006] 2. The trajectory compression algorithm is single and cannot simultaneously meet the requirements of trajectory topology and data volume for various trajectory time period queries. It is suitable for scenarios where the query time period is short and the complete trajectory topology can be retained, but it is not suitable for scenarios where the query time period is long and only the trajectory trend needs to be displayed.

[0007] 3. The trajectory compression algorithm cannot compress the MMSI forged data normally. Because the trajectory points of the MMSI forged data frequently jump back and forth over long distances, and the distance change exceeds the algorithm threshold, most of the points are retained, the compression rate is low, and the trajectory query is slow;

[0008] In order to solve the above problems in the prior art, the present invention proposes a multi-level efficient ship trajectory compression method. Summary of the invention

[0009] The technical problem solved by the present invention is to overcome the defects of the prior art and provide a multi-level efficient ship trajectory compression method.

[0010] To achieve the above object, the present invention provides the following technical solution: a multi-level efficient ship trajectory compression method, comprising the following steps:

[0011] S1: First, connect to the Kafka data source and dynamically generate a trajectory compression algorithm with multiple levels in series according to the program configuration. By default, four levels are generated: day, week, month, and quarter.

[0012] S2: The data retained after being processed by the compression algorithm at each layer is sent to Kafka for storage and sent to the next layer of compression algorithm as the data source for the next layer of compression algorithm.

[0013] S3: As data enters each layer of the compression algorithm, it first decides whether to process only track points containing AIS or BDS based on the configuration. By default, the day layer processes all data, and the other layers only process data containing AIS or BDS.

[0014] S4: Then, the data is grouped according to the configuration keyBy id or mmsi to facilitate the processing of the trajectory, such as Figure 2 ;

[0015] S5: Then determine the first and last points of the processing trajectory, store them directly without compression, and ensure the integrity of the trajectory;

[0016] S6: Again, based on the configuration of each layer, decide whether to sample the trajectory according to the time interval in order to improve performance;

[0017] S7: Then temporarily save the trajectory points and apply the trajectory compression algorithm. Each layer can configure whether to enable the complex trajectory compression algorithm. The simple trajectory compression algorithm is forced to be enabled by default. If the current round of complex trajectory compression algorithm retains new points, the subsequent simple trajectory compression algorithm is skipped;

[0018] S8: Then in the simple trajectory compression algorithm, different algorithms will be used to compress the trajectory according to the different starting keyBy modes;

[0019] S9: After multiple layers of compression, the amount of data and the number of data items change;

[0020] S10: Then query the one-day trajectory example;

[0021] S11: query one month trajectory example again;

[0022] S12: Then query the three-month trajectory example;

[0023] S13: The final query of one-year trajectory and time consumption example.

[0024] Preferably, in step S3, it is determined according to the configuration whether to process only AIS and BDS data (when compressed by day, all data is processed including pure RADAR data without AIS and BDS information).

[0025] Preferably, in step S4, the keyBY operation is performed according to the configuration mode (pure AIS data is based on mmsikeyBy and other data is based on id).

[0026] Preferably, in step S5, if it is the first track point, it is directly retained without compression, and a timer is registered, and the position is not updated when the timeout is triggered, and the last track point is saved without compression.

[0027] Preferably, the step S6 determines whether to perform data sampling according to the time interval according to the configuration (reducing the number of data volume calculations to improve performance).

[0028] Preferably, in step S7, the trajectory points are temporarily saved to form a trajectory for compression calculation, and the multi-level compression in series can meet different application scenarios and has high performance. A combination of multiple different algorithms is optional and has priority.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. This technical solution calculates trajectory compression in real time and continuously, and new stored points can be checked and displayed immediately, without the problem of data redundancy at the boundary of offline calculation window;

[0031] 2. This technical solution is flexible, high-performance and highly adaptable. It can meet the needs of retaining detailed features of trajectories in short query time, and can also ensure the approximate trajectory and high-performance rapid response in long query time, and can meet the different trajectory query needs ranging from a few minutes to several years;

[0032] 3. This technical solution can flexibly configure the number of layers of trajectory compression algorithms, and each layer is a parent-child series connection, with high performance. Complex algorithms and simple algorithms can be flexibly combined in each layer to meet most trajectory storage and query scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is the main process of the trajectory compression program of the present invention;

[0034] Figure 2 It is a sub-process of each layer of trajectory compression algorithm of the present invention;

[0035] Figure 3 The amount of data and the number of data items in each layer of the present invention change;

[0036] Figure 4 An example of a one-day trajectory query for the present invention;

[0037] Figure 5 Query one month trajectory example for the present invention;

[0038] Figure 6 Query three-month trajectory examples for the present invention;

[0039] Figure 7 Query one year trajectory and time-consuming example for the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Embodiment 1

[0042] See also Figure 1-7 The present invention provides a technical solution: a multi-level efficient ship trajectory compression method, comprising the following steps:

[0043] S1: First, connect to the Kafka data source and dynamically generate a trajectory compression algorithm with multiple levels in series according to the program configuration. By default, four levels are generated: day, week, month, and quarter. Figure 1 ;

[0044] S2: Then the data retained by each layer after being processed by the compression algorithm is sent to Kafka for storage, and is also sent to the next layer of compression algorithm as the data source for the next layer of compression algorithm, such as Figure 1 ;

[0045] S3: As data enters each layer of the compression algorithm, it first decides whether to process only track points containing AIS or BDS based on the configuration. By default, the day layer processes all data, and the other layers only process data containing AIS or BDS, such as Figure 2 ;

[0046] S4: Then, the data is grouped according to the configuration keyBy id or mmsi to facilitate the processing of the trajectory, such as Figure 2 ;

[0047] S5: Then determine the first and last points of the processing trajectory, do not compress them, and store them directly to ensure the integrity of the trajectory, such as Figure 2 ;

[0048] S6: Again, based on the configuration of each layer, decide whether to sample the trajectory according to the time interval. In order to improve the performance, e.g. Figure 2 ;

[0049] S7: Then temporarily save the trajectory points and apply the trajectory compression algorithm. Each layer can configure whether to enable the complex trajectory compression algorithm. The simple trajectory compression algorithm is forced to be enabled by default. If the current round of complex trajectory compression algorithm retains new points, the subsequent simple trajectory compression algorithm is skipped. Figure 2 ;

[0050] S8: Then in the simple trajectory compression algorithm, different algorithms will be used to compress the trajectory according to the different starting keyBy modes, such as Figure 2 ;

[0051] S9: After multiple layers of compression, the amount of data and the number of data items change, such as Figure 3 ;

[0052] S10: Then query the one-day trajectory example, such as Figure 4 ;

[0053] S11: Query the trajectory example for one month, such as Figure 5 ;

[0054] S12: Then query the three-month trajectory example, such as Figure 6 ;

[0055] S13: Finally, query the trajectory and time consumption for one year, such as Figure 7 .

[0056] In the present invention, an online compression method is adopted, which has better real-time performance. New trajectory points are compressed and retained in the database for immediate retrieval. The compression is continuous. Offline compression has the problem of discontinuous compression at the boundary of each compression time interval, which will cause the data at the boundary to be denser. Different compression strategies are applied to MMSI forged and non-forged data.

[0057] Embodiment 2

[0058] See also Figure 1-7 The present invention provides a technical solution: a multi-level efficient ship trajectory compression method, comprising the following steps:

[0059] S1: First, connect to the Kafka data source and dynamically generate a trajectory compression algorithm with multiple levels in series according to the program configuration. By default, four levels are generated: day, week, month, and quarter. Figure 1 ;

[0060] S2: Then the data retained by each layer after being processed by the compression algorithm is sent to Kafka for storage, and is also sent to the next layer of compression algorithm as the data source for the next layer of compression algorithm, such as Figure 1 ;

[0061] S3: As data enters each layer of the compression algorithm, it first decides whether to process only track points containing AIS or BDS based on the configuration. By default, the day layer processes all data, and the other layers only process data containing AIS or BDS, such as Figure 2 ;

[0062] S4: Then, the data is grouped according to the configuration keyBy id or mmsi to facilitate the processing of the trajectory, such as Figure 2 ;

[0063] S5: Then determine the first and last points of the processing trajectory, do not compress them, and store them directly to ensure the integrity of the trajectory, such as Figure 2 ;

[0064] S6: Again, based on the configuration of each layer, decide whether to sample the trajectory according to the time interval. In order to improve the performance, e.g. Figure 2 ;

[0065] S7: Then temporarily save the trajectory points and apply the trajectory compression algorithm. Each layer can configure whether to enable the complex trajectory compression algorithm. The simple trajectory compression algorithm is forced to be enabled by default. If the current round of complex trajectory compression algorithm retains new points, the subsequent simple trajectory compression algorithm is skipped. Figure 2 ;

[0066] S8: Then in the simple trajectory compression algorithm, different algorithms will be used to compress the trajectory according to the different starting keyBy modes, such as Figure 2 ;

[0067] S9: After multiple layers of compression, the amount of data and the number of data items change, such as Figure 3 ;

[0068] S10: Then query the one-day trajectory example, such as Figure 4 ;

[0069] S11: Query the trajectory example for one month, such as Figure 5 ;

[0070] S12: Then query the three-month trajectory example, such as Figure 6 ;

[0071] S13: Finally, query the trajectory and time consumption for one year, such as Figure 7 .

[0072] In the present invention, in step S3, it is determined whether to process only AIS and BDS data according to the configuration (when compressed by day, all data including pure RADAR data without AIS and BDS information are processed), and in step S4, keyBY operation is performed according to the configuration and mode (pure AIS data is keyBy based on mmsi, other data is keyBy based on id).

[0073] In the present invention, if it is the first trajectory point in step S5, it is directly retained without compression, and a timer is registered. The position is not updated when the timeout triggers, and the last trajectory point is saved without compression. The step S6 determines whether to perform data sampling according to the time interval according to the configuration (reducing the number of data calculations to improve performance). In step S7, the trajectory point is temporarily saved to form a trajectory for compression calculation, and the multi-level compression in series can meet different application scenarios and has high performance. A combination of multiple different algorithms is optional, and there is priority.

[0074] In the present invention, trajectory compression is calculated in real time, compression is continuous, new stored points can be checked and displayed immediately, and there is no problem of data redundancy at the boundary of the offline calculation window;

[0075] The present invention is flexible, high-performance and highly adaptable, which can satisfy the requirement of retaining detailed features of trajectories in short query time, and can also ensure the approximate trajectory and high-performance rapid response in long query time, and can meet the different trajectory query requirements of querying from a few minutes to several years;

[0076] In the present invention, the trajectory compression algorithm of the number of layers can be flexibly configured, and each layer is a parent-child series connection, with high performance. Complex algorithms and simple algorithms can be flexibly combined in each layer to meet most trajectory storage and query scenarios.

[0077] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-level efficient ship trajectory compression method, characterized in that: The following steps are involved: S1: First, connect to the Kafka data source and dynamically generate a trajectory compression algorithm with multiple levels in series according to the program configuration. By default, four levels are generated: day, week, month, and quarter. S2: The data retained after being processed by the compression algorithm at each layer is sent to Kafka for storage and is also sent to the next layer of compression algorithm as the data source for the next layer of compression algorithm. S3: As data enters each layer of the compression algorithm, it first decides whether to process only track points containing AIS or BDS based on the configuration. By default, the day layer processes all data, and the other layers only process data containing AIS or BDS. S4: Furthermore, the data is grouped according to the configuration keyBy id or mmsi to facilitate the processing of the trajectory; S5: Then determine the first and last points of the processing trajectory, store them directly without compression, and ensure the integrity of the trajectory; S6: Again, based on the configuration of each layer, decide whether to sample the trajectory according to the time interval in order to improve performance; S7: Then temporarily save the trajectory points and apply the trajectory compression algorithm. Each layer can configure whether to enable the complex trajectory compression algorithm. The simple trajectory compression algorithm is forced to be enabled by default. If the current round of complex trajectory compression algorithm retains new points, the subsequent simple trajectory compression algorithm is skipped; S8: Then in the simple trajectory compression algorithm, different algorithms will be used to compress the trajectory according to the different starting keyBy modes; S9: After multiple layers of compression, the amount of data and the number of data items change; S10: Then query the one-day trajectory example; S11: query one month trajectory example again; S12: Then query the three-month trajectory example; S13: The final query of one-year trajectory and time consumption example.

2. The method for multi-level and efficient ship trajectory compression according to claim 1, characterized in that: In step S3, it is determined whether to process only AIS and BDS data according to the configuration (when compressed by day, all data including pure RADAR data without AIS and BDS information is processed).

3. The method for multi-level and efficient ship trajectory compression according to claim 1, characterized in that: In step S4, the keyBY operation is performed according to the configuration mode (pure AIS data is based on mmsi keyBy and other data is based on id).

4. The method for multi-level and efficient ship trajectory compression according to claim 1, characterized in that: In step S5, if it is the first track point, it is directly retained without compression, and a timer is registered. If the position is not updated and the timeout is triggered, the last track point is saved without compression.

5. The method for multi-level and efficient ship trajectory compression according to claim 1, characterized in that: The step S6 determines whether to perform data sampling according to the time interval according to the configuration (reducing the number of data volume calculations to improve performance).

6. The method for multi-level and efficient ship trajectory compression according to claim 1, characterized in that: In step S7, the trajectory points are temporarily saved to form a trajectory for compression calculation, and the multi-level compression in series can meet different application scenarios and has high performance. A combination of multiple different algorithms is optional and has priority.

Images