TD-based inland river AIS data storage model design and query method

Abstract

The invention relates to a TD-based inland river AIS data storage model design and query method, and belongs to the technical field of ship space-time trajectory data processing, and the method comprises the following steps: building an independent MMSI sub-table for each ship stored in the same super-table by using a sub-table building mechanism of a TD engine time sequence database, and storing the uploading time as a timestamp; according to given MMSI-time range query conditions, sub-query is carried out on the MMSI-time super table one by one, and a comprehensive spatio-temporal trajectory data set meeting all the query conditions is obtained through merging; decomposing and mapping an effective navigation area of the river channel into an approximate space grid area by utilizing a space partition mapping function of an S2 algorithm, so as to establish a sub-table; according to a given space-time query condition, a space-time super table is obtained, and a space range is decomposed and mapped to realize parallel query.

Description

Technical Field

[0001] The invention belongs to the technical field of ship space-time trajectory data processing, and relates to a TD-based inland river AIS data storage model design and query method. Background Art

[0002] The Automatic Identification System (AIS) is a widely used digital navigation aid system and equipment. Ship dynamic information such as ship position, speed, rate of change of heading and heading, as well as ship static information such as ship name, call sign, draft, dangerous goods, etc. can be broadcast and communicated between AIS stations and AIS base stations via VHF radio waves to ensure maritime traffic safety. At present, there are hundreds of thousands of ships installed with AIS in the world, and the global monthly AIS data volume can reach hundreds of GB. Among them, each AIS dynamic message can be regarded as a time and space trajectory point of the ship. Therefore, these accumulated AIS data contain a large amount of historical ship trajectory data. Through the analysis and mining of historical ship trajectory data, it can provide good solutions for route optimization, abnormal behavior monitoring, false target identification, port throughput statistical analysis, etc.

[0003] An efficient spatiotemporal trajectory data storage model is the key to effectively analyzing and mining massive historical ship trajectory data. For the storage of AIS data, there is currently a solution that uses NoSQL databases, which is known for its high flexibility, supports high-performance read and write operations and horizontal expansion, and adapts to the storage needs of large-scale data. However, NoSQL databases are insufficient in the accuracy and analysis capabilities of processing time series data. In particular, for inland waterways with strip characteristics that are significantly different from the surface distribution of ocean AIS, the distribution of AIS base stations and AIS data has the characteristics of strip distribution. The NoSQL data storage model cannot support the rapid query requirements of massive historical ship trajectory data.

[0004] The AIS data collection network for inland waterways can essentially be seen as an Internet of Things, in which the ship-borne AIS terminals act as sensors to continuously send data, and the AIS base stations deployed along the waterways act as data collection terminals. The data from all terminals are aggregated to the data center. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a TD-based inland AIS data storage model design and query method, utilizing the unique sub-table construction mechanism and time-indexed column storage characteristics of the TDengine time series database to meet the needs of querying with ships and spatial scopes as the center, while providing efficient data writing and query performance.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A TD-based inland river AIS data storage model design and query method includes the following steps:

[0008] S1: Using the sub-table construction mechanism of the TDengine time series database, an independent MMSI sub-table is established for each ship stored in the same super table, and then the upload time is stored as the timestamp to build an MMSI-time storage model;

[0009] S2: According to the given MMSI-time range query condition, the MMSI-time super table is subqueried one by one, and finally merged to obtain a comprehensive spatiotemporal trajectory dataset that meets all query conditions;

[0010] S3: Using the spatial partition mapping function of the S2 algorithm, the effective navigation area of ​​the river is decomposed and mapped into an approximate spatial grid area, and a sub-table is established to construct a space-time storage model;

[0011] S4: According to the given space-time query conditions, the space-time super table is obtained, the spatial range is decomposed and mapped, and parallel query is realized.

[0012] Further, the steps of constructing the MMSI-time storage model in step S1 are as follows:

[0013] S11: Create a database, configure the number of its virtual groups to be x, and set the buffer size to y;

[0014] S12: Designing an MMSI-time super table, wherein the first field of the MMSI-time super table is set to a receiving timestamp of the spatiotemporal trajectory data point, and the timestamp is set as a primary key;

[0015] S13: For each ship, name the subtable according to its MMSI, and make the subtable inherit the format of the corresponding MMSI-time supertable by automatically creating the table;

[0016] S14: The spatiotemporal trajectory data received by each AIS base station is uploaded to the data center. The data center deduplicates the data received by adjacent AIS base stations, classifies them by MMSI, and stores them in the MMSI-time super table subtables.

[0017] Further, step S2 specifically includes the following steps:

[0018] S21: Given MMSI-time range query conditions, including query object m and query time range T rang =[T start ,T end ], where T start and T endare the start time and end time of the query respectively;

[0019] S22: Obtain MMSI-time super table;

[0020] S23: Query all the rang The query object m in

[0021] S24: Merge the result set through MMSI to obtain a comprehensive spatiotemporal trajectory data set Tras={tra1,tra2,…,tra n}.

[0022] Further, the step S3 of constructing the space-time storage model specifically includes the following steps:

[0023] S31: Create a database, configure the number of virtual groups to be x, and set the buffer size to y;

[0024] S32: Design a space-time hypertable, where the first field of each hypertable is set to the receiving timestamp of the space-time trajectory data point, and set it as the primary key;

[0025] S33: Mapping the input spatiotemporal trajectory data into spatial grid codes using the S2 algorithm;

[0026] S34: For each spatiotemporal trajectory data, the spatial grid coding is used to automatically classify the data and store them into corresponding named sub-tables. Through the automatic table building mechanism, each sub-table will inherit the structure and format of the super-table to which it belongs.

[0027] Further, step S4 specifically includes the following steps:

[0028] S41: Given a space-time query condition, including a query space range s and a query time range T rang =[T start ,T end ], where T start and T end are the start time and end time of the query respectively;

[0029] S42: Obtaining a space-time super table;

[0030] S43: Use the Polygon decomposition function in the S2 algorithm to decompose and map the s spatial range into n spatial grid coding cells;

[0031] S44: perform parallel query on the n cell objects obtained by decomposition.

[0032] The beneficial effects of the present invention are as follows: the present invention aims at the problems of backward storage model and low query efficiency when using the existing ship AIS track data storage device and query method, and completes the orderly organization and storage of ship AIS track data according to the characteristics of the ship track spatiotemporal data structure.

[0033] For ship-centric MMSI-time queries, the sub-table creation mechanism of the TDengine time series database is used to create an independent MMSI sub-table for each ship stored in the same super table, that is, "one sub-table for each ship". At the same time, the upload time is stored as a timestamp by utilizing the columnar storage feature of TDengine with time as the index. This design not only takes advantage of the convenience brought by the super table template, but also ensures the speed and traceability of data query.

[0034] For each spatiotemporal trajectory data, the system will use its spatial grid code to automatically classify and store the data into corresponding named subtables. Through the automatic table building mechanism, each subtable will inherit the structure and format of its supertable to maintain the consistency and efficiency of data organization. By using hierarchical indexing and setting query thresholds to adjust query accuracy, spatial partitioning can be optimized to improve overall query performance.

[0035] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0037] Figure 1 It is a flow chart of the MMSI-time query method;

[0038] Figure 2 This is a schematic diagram of the spatial division of the effective navigation area;

[0039] Figure 3 This is a flow chart of the space-time query method. DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0041] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0042] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0043] The invention provides a TD-based inland river AIS data storage model design and query method, including two steps: a water mobile service identification code MMSI-time storage model and query method and a space-time storage model and query method.

[0044] In inland waterway AIS data query, ship-centered spatiotemporal trajectory query is a common requirement. This type of query can be used for navigation history tracing, navigation behavior analysis, logistics tracking and optimization, etc. The query is based on the ship's MMSI and a certain time range, and the goal is to extract the ship's spatiotemporal trajectory data within a specific time range.

[0045] AIS receiving devices are deployed along inland waterways, and the receiving ranges of adjacent AIS devices overlap to ensure full AIS coverage of the waterway. Each device decodes the AIS message immediately after receiving it, and publishes it through the MQTT protocol according to dynamic information topics, static information topics, heartbeat information topics, etc. The data center subscribes to each topic and uses the TDengine time series database to establish an AIS receiving device super table and an MMSI super table to store AIS information. An independent AIS device subtable is established for each AIS receiving device stored in the AIS receiving device super table, that is, "one device, one subtable". At the same time, each AIS receiving device is also given receiving range information, such as the longitude and latitude of the receiving range, or the range of the waterway mileage line. An independent MMSI subtable is established for each ship stored in the ship super table, that is, "one ship, one subtable". Developed by Taosi Data Technology Co., Ltd., the TDengine time series database is an open source time series big data processing engine for the rapidly developing Internet of Things big data market.

[0046] The MMSI-time query can be described as:

[0047] In a given spatiotemporal trajectory dataset F, for a specific ship query object m and a certain time range T rang =[T start ,T end ], where T start and T end are the start time and end time of the query respectively. The goal of this query is to find all the spatiotemporal trajectory data segments of the query object m within this time period. Tras = {tra1, tra2, ..., tra n}.

[0048] Ship-centric queries hope that the query can reflect the movement trajectory of each ship individually. In view of the fact that the spatiotemporal trajectory data of inland AIS is closely related to the ship object, the sub-table construction mechanism of the TDengine time series database is used to establish an independent MMSI sub-table for each ship stored in the same super table, that is, "one sub-table for one ship". At the same time, using the columnar storage feature of TDengine with time as the index, the upload time is stored as a timestamp. This design not only takes advantage of the convenience brought by the super table template, but also ensures the rapidity and traceability of data query. The specific design steps are as follows:

[0049] ① Create a database named MMSI_TS, configure the number of its virtual groups (vgroup) to 12, and set the buffer size to 10MB. This design takes into account the database's data sharding and memory management to support concurrent data writing and query performance optimization;

[0050] ② Design the MMSI-time supertable and create a supertable named MS. The first field of the supertable is set to the receiving timestamp of the spatiotemporal trajectory data point and set it as the primary key. This step ensures that the data can be effectively sorted and queried according to the time sequence. Other specific information designs are shown in Table 1;

[0051] ③For each ship, the system will name the sub-table according to its MMSI, and automatically create a table so that the sub-table inherits the format of the corresponding super-table;

[0052] ④ Based on the collected spatiotemporal trajectory data, the data are classified according to their MMSI and stored in the supertable subtables respectively. The MMSI-time supertable is shown in Table 1.

[0053] Table 1

[0054]

[0055] The MMSI-time query method is as follows:

[0056] According to the given MMSI-time range query conditions (query object m and query time range T rang ), first, obtain the super table MS. Then, the system will process these sub-queries one by one and merge the result sets through MMSI to obtain a comprehensive spatiotemporal trajectory dataset that meets all query conditions. This method not only considers the scope and accuracy of the query, but also optimizes the data acquisition process to ensure query efficiency and accuracy. The query execution process is as follows Figure 1 shown.

[0057] A space-time query can be described as:

[0058] In a given spatiotemporal trajectory dataset F, for a specific spatial range s and a certain time range T rang =[T start ,T end ], where T start and T end are the start time and end time of the query respectively. The goal of this query is to find all the spatiotemporal trajectory data segments within the time period and the spatial range s. Tras = {tra1, tra2, ..., tra n}.

[0059] The spatial range in the space-time query is polygonal, including the longitude range and the latitude range, which greatly increases the complexity of the query. Using the spatial partition mapping function of the S2 algorithm, the effective navigation area of ​​the river is decomposed and mapped into an approximate spatial grid area to establish a subtable, such as Figure 2As shown in the figure, these partitions cover the entire controlled river section in an approximate spatial grid shape, and each area represents a different spatial range. The S2 algorithm refers to a geocoding processing technology developed by Google, which is mainly used to optimize the storage and query of large-scale geographic data. Its core is to divide the earth's surface into a series of hierarchical cells through mapping, so as to achieve effective encoding and fast access to spatial data. Hierarchical indexing and setting query thresholds are used to adjust query accuracy, thereby optimizing spatial partitions and improving overall query performance. The specific design steps of the database table are as follows:

[0060] ① Create a database named S2_TS, configure the number of vgroups to 12, and set the buffer size to 10MB.

[0061] ② Design a space-time hypertable and create a hypertable named S2_partition. The first field of each hypertable is set to the receiving timestamp of the spatiotemporal trajectory data point and set it as the primary key. The space-time hypertable information is shown in Table 2.

[0062] Table 2

[0063]

[0064] ③ Use the S2 algorithm to map the input spatiotemporal trajectory data into spatial grid codes;

[0065] ④ For each spatiotemporal trajectory data, the spatial grid code is used to automatically classify and store the data into corresponding named subtables. Through the automatic table building mechanism, each subtable will inherit the structure and format of the supertable to which it belongs, thereby maintaining the consistency and efficiency of data organization.

[0066] The spatial-temporal query method is as follows:

[0067] Given a spatial-temporal query condition (query spatial range s and query time range Trange), first obtain the S2_partition super table, then use the Polygon decomposition function in the S2 algorithm to decompose the spatial range s into spatial grid coding cells, and perform parallel query on the decomposed cell objects, which can greatly reduce the query time. The query process is as follows: Figure 3 shown.

[0068] In the above embodiments, the description's reference to "this embodiment" indicates that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily all refer to the same embodiment.

[0069] In the above-described embodiments, although the invention has been described in conjunction with specific embodiments of the invention, many substitutions, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other storage structures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed. Embodiments of the invention are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims.

[0070] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in this embodiment is implemented.

[0071] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0072] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods in this embodiment.

[0073] The computer-readable storage medium in this embodiment can be understood by ordinary technicians in this field: all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0074] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes each step of the above method.

[0075] In this embodiment, the memory may include a random access memory (Random Access Memory, RAM for short), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0076] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0077] The present invention can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0078] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A TD-based inland river AIS data storage model design and query method, characterized by: The following steps are involved: S1: Using the sub-table construction mechanism of the TDengine time series database, an independent MMSI sub-table is established for each ship stored in the same super table, and then the upload time is stored as the timestamp to build an MMSI-time storage model; S2: According to the given MMSI-time range query condition, the MMSI-time super table is subqueried one by one, and finally merged to obtain a comprehensive spatiotemporal trajectory dataset that meets all query conditions; S3: Using the spatial partition mapping function of the S2 algorithm, the effective navigation area of ​​the river is decomposed and mapped into an approximate spatial grid area, and a sub-table is established to construct a space-time storage model; S4: According to the given space-time query conditions, the space-time super table is obtained, the spatial range is decomposed and mapped, and parallel query is realized.

2. The TD-based inland river AIS data storage model design and query method according to claim 1 is characterized by: The steps for constructing the MMSI-time storage model in step S1 are as follows: S11: Create a database, configure the number of its virtual groups to be x, and set the buffer size to y; S12: Designing an MMSI-time super table, wherein the first field of the MMSI-time super table is set to a receiving timestamp of the spatiotemporal trajectory data point, and the timestamp is set as a primary key; S13: For each ship, name the subtable according to its MMSI, and make the subtable inherit the format of the corresponding MMSI-time supertable by automatically creating the table; S14: The spatiotemporal trajectory data received by each AIS base station is uploaded to the data center. The data center deduplicates the data received by adjacent AIS base stations, classifies them by MMSI, and stores them in the MMSI-time super table subtables.

3. The TD-based inland river AIS data storage model design and query method according to claim 1 is characterized by: Step S2 specifically includes the following steps: S21: Given MMSI-time range query conditions, including query object m and query time range T rang =[T start ,T end ], where T start and T end are the start time and end time of the query respectively; S22: Obtain MMSI-time super table; S23: Query all the rang The query object m in S24: Merge the result set through MMSI to obtain a comprehensive spatiotemporal trajectory data set Tras={tra1,tra2,…,tra n }.

4. The TD-based inland river AIS data storage model design and query method according to claim 1 is characterized by: Step S3 of constructing the space-time storage model specifically includes the following steps: S31: Create a database, configure the number of virtual groups to be x, and set the buffer size to y; S32: Design a space-time hypertable, where the first field of each hypertable is set to the receiving timestamp of the space-time trajectory data point, and set it as the primary key; S33: Mapping the input spatiotemporal trajectory data into spatial grid codes using the S2 algorithm; S34: For each spatiotemporal trajectory data, the spatial grid coding is used to automatically classify the data and store them into corresponding named sub-tables. Through the automatic table building mechanism, each sub-table will inherit the structure and format of the super-table to which it belongs.

5. The TD-based inland river AIS data storage model design and query method according to claim 1 is characterized by: Step S4 specifically includes the following steps: S41: Given a space-time query condition, including a query space range s and a query time range T rang =[T start ,T end ], where T start and T end are the start time and end time of the query respectively; S42: Obtaining a space-time super table; S43: Use the Polygon decomposition function in the S2 algorithm to decompose and map the s spatial range into n spatial grid coding cells; S44: perform parallel query on the n cell objects obtained by decomposition.

Images