Multi-level efficient ship trajectory compression method

Abstract

This invention discloses a multi-level, high-efficiency trajectory compression method for ships. First, it connects to a Kafka data source and dynamically generates multiple concatenated trajectory compression algorithms based on program configuration. Four levels are generated by default: daily, weekly, monthly, and quarterly. The data retained after processing by each layer's compression algorithm is then sent to Kafka for storage. This technical solution features real-time trajectory compression calculation, continuous compression, and immediate query and display of new stored points. It avoids the problem of redundant data at offline calculation window boundaries, offering flexibility, high performance, and strong adaptability. It can meet the need to retain detailed trajectory features in short query times while also ensuring approximate trajectory data and high-performance, fast response in long query times. It can satisfy different trajectory query needs, ranging from a few minutes to several years. The trajectory compression algorithm allows for flexible configuration of the number of layers, with each layer being a parent-child concatenation. High-performance, complex, and simple algorithms can be flexibly combined at each layer to meet most trajectory storage and query scenarios.

Description

Technical Field

[0001] This invention relates to the field of ship trajectory compression technology, specifically a multi-level, high-efficiency ship trajectory compression method. Background Technology

[0002] Maritime vessel monitoring systems continuously acquire vessel location information using devices such as RADAR, AIS, and BDS, forming the trajectory of each vessel. The amount of data generated daily is enormous, reaching 500 million to 1 billion records, and over several years, the data volume can even reach trillions of records.

[0003] The trajectory query function needs to display different trajectory details depending on the length of the query period. It should be able to query a ship's trajectory for several hours, days, months, or even years. For shorter time periods, the system should provide the most complete trajectory possible, displaying the ship's position, speed, and navigation status at each moment, retaining details of the ship's behavior such as turning, speeding, and deceleration. For longer time periods, the system should simplify the trajectory, displaying only the overall trend of the ship's trajectory, such as daily berthing positions or routes. Simultaneously, regardless of the query period, the system needs to return results quickly to maintain high performance.

[0004] Existing ship multi-level high-efficiency trajectory compression methods have the following drawbacks when used:

[0005] 1. There are too many ship position trajectory points, resulting in a large storage resource consumption. The trajectory compression only has one level, and the trajectory compression rate is low. When querying the trajectory over a long period of time, it is relatively slow to retrieve data from the database. It is also necessary to compress the data again in the interface to ensure that the data volume is small when the data is transmitted to the front end. At the same time, the interface service has a large load when performing secondary compression, and the trajectory is not displayed on the nautical chart with too many trajectory points.

[0006] 2. The trajectory compression algorithm is singular and cannot simultaneously meet the requirements of various trajectory time period queries regarding trajectory topology shape and data volume. It is suitable for scenarios where the query time period is short and the complete trajectory topology shape can be preserved, but it is not suitable for scenarios where the query time period is long and only the trajectory trend needs to be displayed.

[0007] 3. The trajectory compression algorithm cannot compress MMSI forged data properly. Because the trajectory points of MMSI forged data frequently jump back and forth over long distances, the distance changes exceed the algorithm threshold, resulting in most points being retained, low compression rate, and slow trajectory query.

[0008] To address the aforementioned problems in the prior art, this invention proposes a multi-level, high-efficiency trajectory compression method for ships. Summary of the Invention

[0009] The technical problem solved by this invention is to overcome the defects of the prior art and provide a multi-level, efficient trajectory compression method for ships.

[0010] To achieve the above objectives, the present invention provides the following technical solution: a multi-level, high-efficiency trajectory compression method for ships, comprising the following steps:

[0011] S1: First, connect to the Kafka data source. Based on the program configuration, dynamically generate multiple levels of concatenated trajectory compression algorithms. By default, four levels are generated: daily, weekly, monthly, and quarterly.

[0012] S2: Then, the data retained after each layer of compression algorithm is sent to Kafka for storage, and at the same time sent to the next layer of compression algorithm as the data source for the next layer of compression algorithm;

[0013] S3: As the data enters the compression algorithm of each layer, it first determines whether to process only trajectory points containing AIS or BDS based on the configuration. By default, the day layer processes all data, while the other layers only process data containing AIS or BDS.

[0014] S4: Furthermore, the data is grouped according to the configured keyBy id or mmsi to facilitate trajectory processing, such as Figure 2 ;

[0015] S5: Then determine the first and last points of the trajectory, do not compress them, and store them directly to ensure the integrity of the trajectory;

[0016] S6: Again, based on the configuration of each layer, decide whether to sample the trajectory according to the time interval in order to improve performance;

[0017] S7: Then temporarily save the trajectory points and apply the trajectory compression algorithm. Each layer can be configured to enable or disable the complex trajectory compression algorithm. The simple trajectory compression algorithm is enabled by default. If the complex trajectory compression algorithm retains new points in this round, the subsequent simple trajectory compression algorithm is skipped.

[0018] S8: Next, within the simple trajectory compression algorithm, different algorithms will be used to compress the trajectory depending on the initial keyBy mode;

[0019] S9: Following multi-layer compression, the data volume and number of data entries change;

[0020] S10: Then query a day's trajectory example;

[0021] S11: Another example of querying a month's trajectory;

[0022] S12: Then query a three-month trajectory example;

[0023] S13: Example of querying the trajectory and time taken for the last year.

[0024] Preferably, in step S3, it is determined whether to process only AIS and BDS data based on the configuration (when compressing by day, all data is processed, including pure RADAR data without AIS and BDS information).

[0025] Preferably, in step S4, the keyBY operation is performed according to the configuration and mode (pure AIS data is keyBy based on mmsi, other data is keyBy based on id).

[0026] Preferably, in step S5, if it is the first trajectory point, it is directly retained without compression, and a timer is registered. The timer is triggered when the position is not updated. The last trajectory point is saved without compression.

[0027] Preferably, step S6 determines whether to perform data sampling based on time intervals according to the configuration (reducing the amount of data and the number of calculations to improve performance).

[0028] Preferably, in step S7, the trajectory points are temporarily saved to form a trajectory for compression calculation. The multi-level compression in series can meet different application scenarios and has high performance. A combination of different algorithms can be selected and has priority.

[0029] Compared with the prior art, the beneficial effects of the present invention are:

[0030] 1. This technical solution performs real-time trajectory compression calculation, ensuring continuous compression. Newly stored points can be immediately retrieved and displayed, eliminating the problem of redundant data at the boundary of offline calculation windows.

[0031] 2. This technical solution is flexible, high-performance, and highly adaptable. It can meet the need to retain detailed trajectory features in short query times, and also ensure the approximate trajectory and high-performance fast response in long query times. It can meet different trajectory query needs from a few minutes to several years.

[0032] 3. This technical solution features a flexible and configurable trajectory compression algorithm with multiple layers. Each layer is a parent-child chain, offering high performance. Complex and simple algorithms can be flexibly combined at each layer to meet most trajectory storage and query scenarios. Attached Figure Description

[0033] Figure 1 This is the main flow of the trajectory compression program of the present invention;

[0034] Figure 2 This is a sub-process of the trajectory compression algorithm for each layer of this invention;

[0035] Figure 3 This invention relates to the variation in data volume and number of data entries at each layer;

[0036] Figure 4 This is an example of querying a day's trajectory according to the present invention;

[0037] Figure 5 This is an example of querying a one-month trajectory according to the present invention;

[0038] Figure 6 This is an example of querying a three-month trajectory according to the present invention;

[0039] Figure 7 This is an example of how the present invention queries a year's trajectory and the time taken. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Example 1

[0042] Please see Figure 1-7 This invention provides a technical solution: a multi-level, high-efficiency trajectory compression method for ships, comprising the following steps:

[0043] S1: First, connect to the Kafka data source. Based on the program configuration, dynamically generate multiple levels of concatenated trajectory compression algorithms. By default, four levels are generated: daily, weekly, monthly, and quarterly. Figure 1 ;

[0044] S2: Then, the data retained after each layer's compression algorithm is sent to Kafka for storage, and simultaneously sent to the next layer's compression algorithm as its data source, such as... Figure 1 ;

[0045] S3: As data enters the compression algorithm of each layer, it first determines whether to process only trajectory points containing AIS or BDS based on the configuration. By default, the day layer processes all data, while the other layers only process data containing AIS or BDS. Figure 2 ;

[0046] S4: Furthermore, the data is grouped according to the configured keyBy id or mmsi to facilitate trajectory processing, such as Figure 2 ;

[0047] S5: Then determine the first and last points of the trajectory, do not compress them, and store them directly to ensure the integrity of the trajectory. Figure 2 ;

[0048] S6: Again, based on the configuration of each layer, decide whether to sample the trajectory according to the time interval. To improve performance, such as... Figure 2 ;

[0049] S7: The trajectory points are then temporarily saved, and a trajectory compression algorithm is applied. Each layer can be configured to enable or disable the complex trajectory compression algorithm; the simple trajectory compression algorithm is enabled by default. If the complex trajectory compression algorithm retains new points in this round, subsequent simple trajectory compression algorithms are skipped. Figure 2 ;

[0050] S8: Following the simple trajectory compression algorithm, different algorithms will be used to compress the trajectory depending on the initial keyBy mode, such as... Figure 2 ;

[0051] S9: Following multi-layer compression, the data volume and number of data entries change, such as... Figure 3 ;

[0052] S10: Then query a day's trajectory example, such as Figure 4 ;

[0053] S11: Another example of querying a month's trajectory, such as... Figure 5 ;

[0054] S12: Then query the three-month trajectory example, such as Figure 6 ;

[0055] S13: Example of querying the trajectory and time taken for the last year, such as... Figure 7 .

[0056] In this invention, online compression is used, which has better real-time performance. New trajectory points are compressed and stored in the database and can be queried immediately. The compression is continuous. Offline compression has the problem of compression discontinuity at the boundary of each compression time interval, which will result in denser data at the boundary. Different compression strategies are applied for MMSI forged and non-forged data.

[0057] Example 2

[0058] Please see Figure 1-7 This invention provides a technical solution: a multi-level, high-efficiency trajectory compression method for ships, comprising the following steps:

[0059] S1: First, connect to the Kafka data source. Based on the program configuration, dynamically generate multiple levels of concatenated trajectory compression algorithms. By default, four levels are generated: daily, weekly, monthly, and quarterly. Figure 1 ;

[0060] S2: Then, the data retained after each layer's compression algorithm is sent to Kafka for storage, and simultaneously sent to the next layer's compression algorithm as its data source, such as... Figure 1 ;

[0061] S3: As data enters the compression algorithm of each layer, it first determines whether to process only trajectory points containing AIS or BDS based on the configuration. By default, the day layer processes all data, while the other layers only process data containing AIS or BDS. Figure 2 ;

[0062] S4: Furthermore, the data is grouped according to the configured keyBy id or mmsi to facilitate trajectory processing, such as Figure 2 ;

[0063] S5: Then determine the first and last points of the trajectory, do not compress them, and store them directly to ensure the integrity of the trajectory. Figure 2 ;

[0064] S6: Again, based on the configuration of each layer, decide whether to sample the trajectory according to the time interval. To improve performance, such as... Figure 2 ;

[0065] S7: The trajectory points are then temporarily saved, and a trajectory compression algorithm is applied. Each layer can be configured to enable or disable the complex trajectory compression algorithm; the simple trajectory compression algorithm is enabled by default. If the complex trajectory compression algorithm retains new points in this round, subsequent simple trajectory compression algorithms are skipped. Figure 2 ;

[0066] S8: Following the simple trajectory compression algorithm, different algorithms will be used to compress the trajectory depending on the initial keyBy mode, such as... Figure 2 ;

[0067] S9: Following multi-layer compression, the data volume and number of data entries change, such as... Figure 3 ;

[0068] S10: Then query a day's trajectory example, such as Figure 4 ;

[0069] S11: Another example of querying a month's trajectory, such as... Figure 5 ;

[0070] S12: Then query the three-month trajectory example, such as Figure 6 ;

[0071] S13: Example of querying the trajectory and time taken for the last year, such as... Figure 7 .

[0072] In this invention, step S3 determines whether to process only AIS and BDS data based on the configuration (when compressing by day, all data is processed, including pure RADAR data without AIS and BDS information). In step S4, keyBY operation is performed according to the configuration by mode (pure AIS data is keyBy based on mmsi, other data is keyBy based on id).

[0073] In this invention, if the first trajectory point is in step S5, it is directly retained without compression, and a timer is registered. The position is not updated and the timeout is triggered. The last trajectory point is saved without compression. Step S6 determines whether to perform data sampling according to the time interval based on the configuration (reducing the amount of data and the number of calculations to improve performance). In step S7, the trajectory points are temporarily saved to form a trajectory for compression calculation. The multi-level compression in series can meet different application scenarios and has high performance. A combination of different algorithms is available and has priority.

[0074] In this invention, trajectory compression is calculated in real time, compression is continuous, and new stored points can be checked and displayed immediately, without the problem of redundant data at the boundary of the offline calculation window;

[0075] This invention is flexible, high-performance, and highly adaptable. It can meet the need to retain detailed trajectory features in short query times, while also ensuring the approximate trajectory and high-performance fast response in long query times. It can meet different trajectory query needs ranging from a few minutes to several years.

[0076] The present invention features a trajectory compression algorithm with flexibly configurable layers, and each layer is a parent-child chain. High performance, complex algorithms and simple algorithms can be flexibly combined in each layer to meet most trajectory storage and query scenarios.

[0077] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for multi-level, high-efficiency trajectory compression of ships, characterized in that, Includes the following steps: S1: First, connect to the Kafka data source. Based on the program configuration, dynamically generate multiple levels of concatenated trajectory compression algorithms. By default, four levels are generated: daily, weekly, monthly, and quarterly. S2: Then, the data retained after each layer of compression algorithm is sent to Kafka for storage, and at the same time sent to the next layer of compression algorithm as the data source for the next layer of compression algorithm; S3: As the data enters the compression algorithm of each layer, it first determines whether to process only trajectory points containing AIS or BDS based on the configuration. By default, the day layer processes all data, while the other layers only process data containing AIS or BDS. S4: Furthermore, the data is grouped according to the configured keyBy id or mmsi to facilitate trajectory processing; S5: Then determine the first and last points of the trajectory, do not compress them, and store them directly to ensure the integrity of the trajectory; S6: Again, based on the configuration of each layer, decide whether to sample the trajectory according to the time interval in order to improve performance; S7: Then temporarily save the trajectory points and apply the trajectory compression algorithm. Each layer can be configured to enable or disable the complex trajectory compression algorithm. S8: Next, within the simple trajectory compression algorithm, different algorithms will be used to compress the trajectory depending on the initial keyBy mode; S9: Following multi-layer compression, the data volume and number of data entries change; S10: Then query a day's trajectory example; S11: Another example of querying a month's trajectory; S12: Then query a three-month trajectory example; S13: Example of querying the trajectory and time taken for the last year.

2. The ship multi-level high-efficiency trajectory compression method according to claim 1, characterized in that: In step S3, it is determined whether to process only AIS and BDS data based on the configuration.

3. The ship multi-level high-efficiency trajectory compression method according to claim 1, characterized in that: In step S4, the keyBY operation is performed according to the configuration and mode.

4. The ship multi-level high-efficiency trajectory compression method according to claim 1, characterized in that: In step S5, a timer is registered, which is triggered when the position is not updated, and the last trajectory point is saved without compression.

5. The ship multi-level high-efficiency trajectory compression method according to claim 1, characterized in that: Step S6 determines whether to sample data based on time intervals according to the configuration.

6. The ship multi-level high-efficiency trajectory compression method according to claim 1, characterized in that: In step S7, the trajectory points are temporarily saved to form a trajectory for compression calculation.

Images