A Global Ship AIS Big Data Processing Method and System

Abstract

The present invention provides a method and system for processing global ship AIS big data. First, real-time ship AIS messages are received and published to a message queue for caching. Then, the AIS messages in the message queue are received and preprocessed. Next, they are stored separately according to different message types. The AIS messages with the message type of dynamic data type are published to the message queue, and the AIS messages with the message type of static data type are saved to the Redis in-memory database. Then, the AIS messages of the dynamic data type stored in the message queue and the AIS messages of the static data type stored in the Redis in-memory database are merged to form a complete AIS data record, and the AIS data record is published to the message queue to form an AIS data stream. Finally, the AIS data stream is processed for business to generate a data processing record, and the generated data processing record is saved to the target location. The present invention is based on a processing framework integrating big data stream and batch, and adopts the idea of distributed computing, meeting the real-time requirements of the business for AIS data.

Description

Technical Field

[0001] The present invention relates to the technical field of massive data processing, and particularly to a method and system for processing global ship AIS big data. Background Art

[0002] The Automatic Identification System (AIS) is a broadcast automatic identification system installed on ships and shore stations. Currently, the vast majority of ships have installed AIS devices. AIS data is periodically sent by the device in the form of broadcast messages, and the content includes the real-time dynamic information of the ship (such as ship position, speed, course, etc.) and the static information of the ship (such as ship call sign, ship name, Maritime Mobile Service Identity MMSI, ship type, ship size, and voyage, etc.).

[0003] AIS messages are transmitted in WGS84 encoding and ASCII message format. When applying the data, it is necessary to decode and convert the messages, split out each field for processing and storage, and some AIS messages also need to merge and decode multiple messages.

[0004] Currently, there are more than 100,000 ships globally. According to different sending frequencies and ship ranges, the data volume of AIS messages is very high. Applications such as AIS-based monitoring and early warning have relatively high requirements for the timeliness of AIS data, which requires the data processing system to decode and process the AIS message data in a timely manner after receiving it.

[0005] AIS message information is divided into two categories: dynamic information and static information. Generally, combined processing is required when applying the data, but AIS will send through different messages. How to timely associate the static data with the latest dynamic data is an important problem that needs to be solved urgently. Summary of the Invention

[0006] To solve the above problems, the present invention provides a method for processing global ship AIS big data. By adopting big data streaming real-time processing technology, based on a big data processing architecture integrating streaming and batch processing, and adopting the idea of distributed computing, the AIS data is decoded and processed, meeting the real-time requirements of the business for AIS data. The present invention also relates to a system for processing global ship AIS big data.

[0007] The technical solution of the present invention is as follows:

[0008] A method for processing global ship AIS big data, characterized by comprising the following steps:

[0009] Data receiving step: Receive real-time AIS messages of ships from the AIS system and publish the AIS messages to a message queue for caching;

[0010] Data classification step: Consume AIS messages in the message queue, preprocess the AIS messages, and store the preprocessed AIS messages separately according to different message types. Publish the AIS messages with the dynamic data type to the message queue, and save the AIS messages with the static data type to the Redis in-memory database with mmsi as the primary key;

[0011] Data merging step: Adopt big data streaming processing technology to merge the AIS messages with the dynamic data type stored in the message queue and the AIS messages with the static data type stored in the Redis in-memory database with mmsi as the primary key to form a complete AIS data record, and publish the AIS data record to the message queue to form an AIS data stream;

[0012] Data application step: Based on the big data processing architecture integrating streaming and batch processing, perform business processing on the AIS data stream to generate data processing records, and save the generated data processing records to the target location.

[0013] Preferably, in the data receiving step, after receiving the real-time AIS messages of the ship, save the AIS messages to the internal queue, consume the AIS messages in the internal queue in a multi-threaded manner, and publish the consumed data to the message queue to ensure the data processing speed.

[0014] Preferably, the preprocessing in the data classification step includes AIS message merging processing, decoding processing, and downsampling processing. The AIS message merging processing is to merge multiple AIS messages constituting the same AIS data. The decoding processing decodes the merged AIS messages to obtain the data of each field. The downsampling processing is used to reduce the message density of the decoded AIS messages.

[0015] Preferably, in the data classification step, before the downsampling processing, the preprocessing further includes checking the data format of the AIS messages, excluding illegal data formats, retaining qualified data formats, and performing downsampling on the qualified data formats using a session window.

[0016] Preferably, in the data application step, the business processing includes data quality detection, ship status conversion judgment, and area crossing judgment. The target locations include the Redis in-memory database, message queue, relational database, etc.

[0017] A global ship AIS big data processing system, characterized by including a data receiving module, a data classification module, a data merging module, and a data application module connected in sequence,

[0018] The data receiving module: receives real-time AIS messages of ships from the AIS system and publishes the AIS messages to a message queue for caching;

[0019] The data classification module: consumes the AIS messages in the message queue, preprocesses the AIS messages, stores the preprocessed AIS messages separately according to different message types, publishes the AIS messages with the message type of dynamic data type to the message queue, and saves the AIS messages with the message type of static data type to the Redis in-memory database with mmsi as the primary key;

[0020] The data merging module: adopts big data streaming processing technology to merge the AIS messages of dynamic data type stored in the message queue and the AIS messages of static data type stored in the Redis in-memory database with mmsi as the primary key to form a complete AIS data record, and publishes the AIS data record to the message queue to form an AIS data stream;

[0021] The data application module: based on the big data processing architecture integrating stream and batch, conducts business processing on the AIS data stream to generate data processing records, and saves the generated data processing records to the target location.

[0022] Preferably, in the data receiving module, after receiving the real-time AIS messages of ships, the AIS messages are saved to an internal queue, the AIS messages in the internal queue are consumed in a multi-threaded manner, and the consumed data is published to the message queue to ensure the data processing speed.

[0023] Preferably, the preprocessing includes AIS message merging processing, decoding processing, and downsampling processing.

[0024] Preferably, before the downsampling processing, the preprocessing further includes checking the data format of the AIS messages, excluding illegal data formats, retaining qualified data formats, and performing downsampling on the qualified data formats using a session window.

[0025] Preferably, the business processing includes data quality detection, ship status conversion judgment, and area crossing judgment, and the target locations include the Redis in-memory database, message queue, relational database, etc.

[0026] The beneficial effects of the present invention are:

[0027] A global ship AIS big data processing method provided by the present invention sequentially sets a data receiving step, a data classification step, a data merging step, and a data application step. Each step cooperates with each other and works together. First, it receives real-time AIS messages of ships from the AIS system and publishes the AIS messages to a message queue for caching. Then, it consumes the AIS messages in the message queue, preprocesses the AIS messages to reduce the pressure of subsequent data processing, and stores the preprocessed AIS messages separately according to different message types. Using the idea of distributed computing, it decodes and processes the AIS data, publishes the AIS messages with the message type of dynamic data type to the message queue, and saves the AIS messages with the message type of static data type to the Redis in-memory database with mmsi as the primary key, which is convenient for quick search and association according to mmsi during subsequent processing, and at the same time realizes data coverage of the same ship, ensuring the real-time nature of the data. Then, using the big data streaming processing technology, it merges the dynamic AIS messages stored in the message queue and the static AIS messages stored in the Redis in-memory database with mmsi as the primary key to form a complete AIS data record, and publishes the AIS data record to the message queue to form an AIS data stream. Finally, based on the big data processing architecture integrating streaming and batch processing, it performs business processing on the AIS data stream to generate data processing records, that is, big data streaming processing and batch data processing are adopted during data application, and the generated data processing records are saved to the target location. In this method, data caching is carried out between each data processing component through the message queue, effectively improving the operation efficiency of the entire processing chain, reducing the impact of temporary failures and other problems of some components on the entire processing process, and at the same time reducing the coupling degree between components. In addition, relying on the big data platform, each processing step can perform parallel computing, greatly improving the data processing speed.

[0028] The present invention also relates to a global ship AIS big data processing system, which corresponds to the above-mentioned global ship AIS big data processing method and can be understood as a system for implementing the above-mentioned global ship AIS big data processing method. It includes a data receiving module, a data classification module, a data merging module, and a data application module connected in sequence. Each module cooperates with each other and works together. By using the big data streaming processing technology and based on the processing framework integrating big data streaming and batch processing, and adopting the idea of distributed computing, it decodes and processes the AIS data, meeting the real-time requirements of the business for AIS data, and at the same time supporting various real-time business processing of AIS data. In view of the characteristics of massive AIS data, it makes full use of the advantages of big data distribution and has targeted optimizations in aspects such as data access, calculation, and storage, and can realize the efficient processing, conversion, and storage of massive ship AIS data. Description of the Drawings

[0029] Figure 1This is the flowchart of the global ship AIS big data processing method of the present invention.

[0030] Figure 2 This is the working principle diagram of the global ship AIS big data processing system of the present invention. Specific embodiments

[0031] The present invention will be described below with reference to the accompanying drawings.

[0032] The present invention relates to a global ship AIS big data processing method, which includes a data receiving step, a data classifying step, a data merging step, and a data applying step. The data receiving step receives real-time ship AIS messages from the AIS system and publishes the AIS messages to a message queue for caching; the data classifying step consumes the AIS messages in the message queue, preprocesses the AIS messages, and stores the preprocessed AIS messages separately according to different message types. The AIS messages with the dynamic data type are published to the message queue, and the AIS messages with the static data type are saved to the Redis in-memory database with the mmsi as the primary key; the data merging step uses big data streaming processing technology to merge the AIS messages of the dynamic data type stored in the message queue and the AIS messages of the static data type stored in the Redis in-memory database with the mmsi as the primary key to form a complete AIS data record, and publishes the AIS data record to the message queue to form an AIS data stream; the data applying step is based on a big data processing architecture integrating streaming and batch processing, performs business processing on the AIS data stream to generate data processing records, and saves the generated data processing records to the target location. By adopting big data streaming processing technology, based on a big data processing framework integrating streaming and batch processing, and using the idea of distributed computing, this method decodes and processes AIS data, meets the real-time requirements of the business for AIS data, and at the same time supports various real-time business processing of AIS data. In view of the characteristics of massive AIS data, it makes full use of the advantages of big data distribution, and has targeted optimizations in aspects such as data access, calculation, and storage, and can realize the efficient processing, conversion, and storage of massive ship AIS data. In addition, relying on the big data platform, each processing step can perform parallel computing, greatly improving the data processing speed.

[0033] The present invention will be further described in detail below with reference to the accompanying drawings.

[0034] Figure 1 This is the flowchart of a global ship AIS big data processing method of the present invention. As shown in the figure, the method sequentially includes the following steps:

[0035] Data receiving step: Receive real-time AIS messages of ships from the AIS system, save the AIS messages to the internal queue, consume the AIS messages in the internal queue in a multi-threaded manner, and publish the consumed data to the message queue for caching to ensure the data processing speed.

[0036] In this step, the number of threads required can be freely selected according to actual needs. In this embodiment, for the data volume of AIS messages, 2 threads are started for data sending, and two Shards are opened in the message queue to ensure the data writing speed. It can be understood that multiple threads such as 3 or 4 can be started for data sending according to the data writing speed. To ensure the efficiency of data transmission, it is recommended that the number of shards in the message queue be consistent with the number of threads.

[0037] Data classification step: Consume the AIS messages in the message queue, preprocess the AIS messages, and store the preprocessed AIS messages separately according to different message types. Publish the AIS messages with the dynamic data type to the message queue, and save the AIS messages with the static data type to the Redis in-memory database with mmsi as the primary key;

[0038] The real-time stream processing program consumes the message queue, obtains the AIS messages, processes the data according to certain rules such as downsampling and message merging, decodes the merged messages, and adopts different storage strategies for different message types. Specifically, after receiving the AIS messages in the message queue, consume the AIS messages in the message queue, obtain the original AIS messages, check the data format of the original AIS messages (such as the length of the mmsi field, the range of longitude and latitude, etc.), exclude illegal data formats, retain qualified data formats, then merge multiple AIS messages constituting the same AIS data, and decode the merged AIS messages. After decoding, use the session window to downsample the qualified AIS message data according to certain rules (for example, retain the latest AIS message of the same ship within 1m), and finally store them separately according to different message types. Publish the AIS messages with the dynamic data type to the message queue for subsequent processing; save the AIS messages with the static data type to the Redis in-memory database with mmsi as the primary key, which facilitates quick search and association according to mmsi during subsequent processing, and also realizes name coverage, ensuring the real-time nature of the data.

[0039] Data merging step: Using big data streaming processing technology, merge the AIS messages of dynamic data types stored in the message queue and the AIS messages of static data types stored in the Redis in-memory database with mmsi as the primary key to form complete AIS data records. That is to say, consume the message queue storing dynamic AIS messages, associate the static AIS messages in the external cache for each dynamic AIS message to form a complete AIS record, and publish the AIS data record to the message queue to form an AIS data stream.

[0040] Data application step: Based on the big data processing architecture integrating streaming and batch processing, perform business processing on the AIS data stream (such as data quality detection, ship status conversion judgment, and area crossing judgment, etc.) to generate data processing records, and save the generated data processing records to target locations such as the Redis in-memory database, message queue, and relational database.

[0041] It should be noted that as an example of data application, the ship status fields in the AIS messages can be summarized, and the number of ships in each status can be output every 10 minutes. The data is written into the relational database PostgreSQL to form a ship status report table.

[0042] The present invention also relates to a global ship AIS big data processing system, which corresponds to the above-mentioned global ship AIS big data processing method and can be understood as a system for implementing the above method. The system includes a data receiving module, a data classification module, a data merging module, and a data application module connected in sequence. As Figure 2 shown in the schematic diagram, perform AIS data reception, AIS data decoding and classification, AIS data merging, and AIS data application respectively. Data caching is performed between each data processing module through the message queue, effectively improving the operation efficiency of the entire processing chain, reducing the impact of temporary failures and other problems of some modules on the entire processing process, and reducing the coupling degree between modules. Relying on the big data platform, each module is organically combined and can perform parallel computing to improve the data processing speed. Specifically,

[0043] The data receiving module receives real-time ship AIS messages from the AIS system and publishes the AIS messages to the message queue for caching;

[0044] The data classification module consumes the AIS messages in the message queue, preprocesses the AIS messages, and stores the preprocessed AIS messages separately according to different message types. Publish the AIS messages of dynamic data types to the message queue, and save the AIS messages of static data types to the Redis in-memory database with mmsi as the primary key;

[0045] The data merging module uses big data streaming processing technology to merge the AIS messages of dynamic data types stored in the message queue and the AIS messages of static data types stored in the Redis in-memory database with mmsi as the primary key to form complete AIS data records, and publishes the AIS data records to the message queue to form an AIS data stream;

[0046] The data application module, based on the big data processing architecture of stream-batch integration, performs business processing on the AIS data stream to generate data processing records, and saves the generated data processing records to the target location.

[0047] Preferably, in the data receiving module, after receiving the real-time AIS messages of the ship, the AIS messages are saved to the internal queue, the AIS messages in the internal queue are consumed in a multi-threaded manner, and the consumed data is published to the message queue to ensure the data processing speed.

[0048] Preferably, the preprocessing includes AIS message merging processing, decoding processing, and downsampling processing.

[0049] Further preferably, before the downsampling processing, the preprocessing further includes checking the data format of the AIS message, excluding illegal data formats, retaining qualified data formats, and performing downsampling on the qualified data formats using a session window.

[0050] Preferably, the business processing includes data quality detection, ship status conversion judgment, and area crossing judgment, and the target locations include the Redis in-memory database, the message queue, and the relational database, etc.

[0051] Embodiment:

[0052] As Figure 2 shown, it is the structure diagram of the global ship AIS big data processing system of the present invention. First, the real-time AIS messages are received in the Transmission Control Protocol (TCP) manner, and the AIS messages are published to the message queue Datahub for caching, and the topic is named "ais_raw"; for the data volume of AIS, two threads are started for data sending, and two Shards are opened in Datahub to ensure the data writing speed;

[0053] Then develop a data processing job in the Blink computing platform to consume the "ais_raw" topic in Datahub, obtain AIS messages, check the data format of the AIS messages, exclude illegal data, retain qualified data, then merge multiple AIS messages that constitute the same AIS data, and decode the merged AIS messages. After decoding, use a session window to downsample the AIS data with a window size of 1 minute, and then store it separately according to different message types. For the AIS message data of the dynamic data type, save it to Datahub, and the topic is "ais_dynamic"; for the AIS message data of the static data type, save it to the Redis in-memory database with mmsi as the primary key;

[0054] Then develop a job in the Blink computing platform to consume the "ais_dynamic" topic in Datahub. For each AIS message of the dynamic data type, associate the AIS message of the static data type in Redis with mmsi as the primary key to form a complete AIS data record, and write the AIS data record into Datahub, and the topic is "ais_payload";

[0055] Finally, develop a job in the Blink computing platform to consume the "ais_payload" topic in Datahub, and write the generated data processing result data after consumption into the relational database PostgreSQL to form an AIS record table. As an example of data application, the ship status fields in the ais messages can be summarized, and the number of ships in each status can be output every 10 minutes, and the data is written into the relational database PostgreSQL to form a ship status report table.

[0056] The present invention provides an objective and scientific global ship AIS big data processing method and system. By adopting big data streaming processing technology, based on a processing framework that integrates big data stream and batch, and using the idea of distributed computing, the AIS data is decoded and processed, meeting the real-time requirements of the business for AIS data, and at the same time supporting various real-time business processing of AIS data. In view of the characteristics of massive AIS data, the advantages of big data distribution are fully utilized, and targeted optimizations are made in aspects such as data access, calculation, and storage, enabling efficient processing, conversion, and storage of massive ship AIS data.

[0057] It should be noted that the above-described specific embodiments can enable those skilled in the art to more comprehensively understand the present invention-creation, but do not limit the present invention-creation in any way. Therefore, although this specification has described the present invention-creation in detail with reference to the accompanying drawings and embodiments, those skilled in the art should understand that the present invention-creation can still be modified or equivalently replaced. In short, all technical solutions and their improvements that do not depart from the spirit and scope of the present invention-creation should be covered by the protection scope of the patent for the present invention-creation.

Claims

1. A method for processing global ship AIS big data, characterized in that, It includes the following steps: Data receiving step: Receive real-time AIS messages of ships from the AIS system, consume the AIS messages in a multi-threaded manner, and publish the AIS messages to a message queue for caching; Adjust the number of threads according to the data volume of the AIS messages, and open shards in the message queue that are consistent with the number of threads; Data classification step: Consume the AIS messages in the message queue, preprocess the AIS messages, and adopt the idea of distributed computing to perform different storage strategies on the preprocessed AIS messages according to different message types. Publish the AIS messages with the message type of dynamic data type to the message queue, and save the AIS messages with the message type of static data type to the Redis in-memory database with mmsi as the primary key; Data merging step: Adopt big data streaming processing technology to merge the AIS messages of dynamic data type stored in the message queue and the AIS messages of static data type stored in the Redis in-memory database with mmsi as the primary key to form a complete AIS data record, and publish the AIS data record to the message queue to form an AIS data stream; Data application step: Based on the big data processing architecture that combines streaming and batch processing, perform business processing on the AIS data stream to generate data processing records, and save the generated data processing records to the target location.

2. The global ship AIS big data processing method according to claim 1, characterized in that, In the data receiving step, after receiving the real-time AIS messages of ships, save the AIS messages to the internal queue, consume the AIS messages in the internal queue in a multi-threaded manner, and publish the consumed data to the message queue to ensure the data processing speed.

3. The global ship AIS big data processing method according to claim 1, characterized in that The preprocessing in the data classification step includes AIS message merging processing, decoding processing, and downsampling processing. The AIS message merging processing is to merge multiple AIS messages that constitute the same AIS data. The decoding processing decodes the merged AIS messages to obtain the data of each field. The downsampling processing is used to reduce the message density of the decoded AIS messages.

4. The global ship AIS big data processing method according to claim 3, characterized in that In the data classification step, before the downsampling processing, the preprocessing also includes checking the data format of the AIS messages, excluding illegal data formats, retaining qualified data formats, and performing downsampling on the qualified data formats using a session window.

5. The global ship AIS big data processing method according to claim 1, wherein In the data application step, the business processing includes data quality detection, ship status conversion judgment, and area crossing judgment. The target location includes the Redis in-memory database, message queue, relational database, etc.

6. A global ship AIS big data processing system, characterized in that, It includes a data receiving module, a data classification module, a data merging module, and a data application module that are connected in sequence, The data receiving module: Receive real-time AIS messages of ships from the AIS system, consume the AIS messages in a multi-threaded manner, and publish the AIS messages to the message queue for caching; Adjust the number of threads according to the data volume of the AIS messages, and open shards in the message queue that are consistent with the number of threads; The data classification module: processes the AIS messages in the consumption message queue, preprocesses the AIS messages, and adopts the idea of distributed computing to perform different storage strategies on the preprocessed AIS messages according to different message types. It publishes the AIS messages with the message type of dynamic data type to the message queue, and saves the AIS messages with the message type of static data type to the Redis in-memory database with mmsi as the primary key; The data merging module: uses big data streaming processing technology to merge the AIS messages of dynamic data type stored in the message queue and the AIS messages of static data type stored in the Redis in-memory database with mmsi as the primary key to form complete AIS data records, and publishes the AIS data records to the message queue to form an AIS data stream; The data application module: based on the big data processing architecture of stream-batch integration, performs business processing on the AIS data stream to generate data processing records, and saves the generated data processing records to the target location.

7. The global ship AIS big data processing system according to claim 6, wherein, In the data receiving module, after receiving the real-time AIS messages of the ship, the AIS messages are saved to the internal queue, the AIS messages in the internal queue are consumed in a multi-threaded manner, and the consumed data is published to the message queue to ensure the data processing speed.

8. The global ship AIS big data processing system according to claim 6, characterized in that, The preprocessing includes AIS message merging processing, decoding processing, and downsampling processing.

9. The global ship AIS big data processing system according to claim 8, wherein, Before the downsampling processing, the preprocessing also includes checking the data format of the AIS messages, excluding illegal data formats, retaining qualified data formats, and performing downsampling on the qualified data formats using a session window.

10. The global ship AIS big data processing system according to claim 6, characterized in that, The business processing includes data quality detection, ship status conversion judgment, and area crossing judgment, and the target location includes the Redis in-memory database, message queue, relational database, etc.

Images