Partition table implementation method and system of time series database in mass device scenario

Abstract

The application discloses a mass equipment scene time sequence database partition table implementation method and system, belongs to the time sequence database technical field, and wants to solve the technical problem of how to realize the time sequence database mass equipment data management. Including the following steps: defining the partition mode of the time sequence database, so as to realize the time sequence database satisfying the relational database partition table model; defining the partition management mode of the time sequence database; based on the data to be written, the defined partition mode and the partition management mode configure the time sequence database; for the data to be written, based on the writing time, the corresponding partition directory is calculated, and the index position of the specified partition block is inquired from the partition index file, if the corresponding partition block exists, the data is written into the corresponding partition block, if the corresponding partition block does not exist, a new partition block is applied in the data file, the data is written into the new partition block, and the new partition block is updated to the partition index file.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of time series database, in particular to a time series database partition table implementation method and system in a massive device scenario. BACKGROUND

[0002] With the rapid development of Internet of Things technology and application, the data generated by Internet of Things devices is growing explosively. The most widely used time series database for Internet of Things application has a growing number of devices connected. It often happens that a time series database needs to connect to thousands of devices with the same structure. How to better manage the massive number of connected devices becomes a problem that needs to be considered.

[0003] When solving such problems in traditional databases, the general processing method is to divide the database, divide the table, or divide the partition. The database and table division method will have an impact on user business, and the user application needs to be modified, which is not convenient to use. The commonly used method is database partitioning.

[0004] Database partitioning refers to dividing the data of a table into multiple blocks. In logic, it is ultimately only one table, but the underlying is composed of multiple physical blocks. Partitioning the database can greatly reduce system response delay and improve data throughput. There are the following benefits to database partitioning:

[0005] 1. Partitioning makes large tables easier to manage: maintenance operations on data subsets are also more efficient, because these operations only target the data needed and not the entire table. A good partitioning strategy will reduce the amount of data to be scanned by reading only the relevant data that meets the query requirements. When all data is on the same partition, queries, calculations, and other operations on the database are limited to the disk access IO bottleneck;

[0006] 2. Partitioning allows the system to fully utilize all resources: a good partitioning scheme combined with parallel computing and distributed computing can fully utilize all nodes to complete tasks that would normally be completed on a single node. When a task can be divided into several scattered subtasks, each accessing a different partition, the efficiency can be improved;

[0007] 3. Partitioning increases system availability: since the replicas of the partition are usually stored on different physical nodes, if a partition is unavailable, the system can still use other replica partitions to ensure normal operation of the job.

[0008] Currently, mainstream time series databases such as InfluxDB and TDEngine generally only support one-level partitioning according to time, and do not support the partitioning model of ordinary relational databases. When a time series database manages a large number of devices, the following performance problems may occur:

[0009] If all device data is managed by a table: when all devices write concurrently, locking processing needs to be performed on each row of device data, and the efficiency of concurrent reading and writing is reduced; in addition, if the collected data of the same device does not fluctuate greatly, but the data collected by different devices may fluctuate greatly, when all data is randomly placed in the same table, the column change probability of the table is high, thereby causing the compression effect of the data to be poor. The time series database has a high requirement for the compression rate of data due to the large amount of stored data;

[0010] If each device corresponds to a table, TDEngine uses a super table / sub table way to deal with such a scene, each sub table corresponds to a device, so that the lock problem is solved, but the number of tables / sub tables will increase sharply, causing the metadata managed by the time series database to increase, and the number of files also expands by many times, thereby increasing the system load.

[0011] How to realize the data management of the time series database for a large number of devices is a technical problem to be solved. SUMMARY

[0012] The technical task of the present application is to solve the technical problem of how to realize the data management of the time series database for a large number of devices in the scene of a large number of devices by providing a time series database partition table implementation method and system.

[0013] In a first aspect, the present application provides a time series database partition table implementation method in a large number of device scenarios, comprising the following steps:

[0014] Defining a partition mode of the time series database to realize a time series database satisfying a relational database partition table model, wherein the partition mode is: performing one-level partitioning on the time series database based on time, and creating different partition directories according to the time of writing data into the database; performing two-level partitioning on the time series database according to the partition mode of the relational database, and dividing the database table into a plurality of partition blocks, each partition directory corresponding to at least one partition block;

[0015] Defining a partition management mode of the time series database: dividing the data table into a data file and a partition index file, the data file being divided according to the partition block, the data of each partition block being able to be appended and written, a new partition block being applied for in the data file when the current partition block is full, and the partition index file being used to record attribute information of the data stored in the partition block, the attribute information including time information and index position information of writing data into the partition block;

[0016] Configuring the time series database based on the data to be written, according to the defined partition mode and partition management mode: creating different partition directories according to the time of writing data into the time series database, dividing the data table into a data file and a partition retrieval file, and applying for a partition block in the data file according to the specified partition mode;

[0017] Write data: for the data to be written, the corresponding partition directory is calculated based on the write time, and the index position of the specified partition block is queried from the partition index file. If the corresponding partition block exists, the data is written into the corresponding partition block. If the corresponding partition block does not exist, a new partition block is applied in the data file, the data is written into the new partition block, and the new partition block is updated to the partition index file.

[0018] As preferred, when applying a partition block in the data file according to the specified partition mode, each partition block is divided into an order block for storing time-series data and a disorder block for storing disorder data;

[0019] Correspondingly, when writing data into the corresponding partition block, the following operations are performed:

[0020] If the written data is sequential data, the sequential data is written into the sequential block;

[0021] If the written data is disorder data, and the sequential block is not full, the disorder data is written into the sequential block, and the sequential block is sorted and rewritten;

[0022] If the written data is disorder data, and the sequential block is full, if there is a disorder block in the corresponding partition block, the disorder data is written into the disorder block. If there is no disorder block in the corresponding partition block, a disorder block is created in the corresponding partition block, the disorder data is written into the created disorder block, and the newly created disorder block is mounted to the partition index of the corresponding partition.

[0023] As preferred, for the data written into the time-series database, when querying the partition where the data is located, if there is no disorder block in the partition block, the sequential block is queried by bisection. If there is a disorder block in the partition block, the sequential block and the disorder block are combined for sorted query.

[0024] As preferred, the partition mode of the relational database includes four kinds of range partitioning, hash partitioning, value partitioning and list partitioning;

[0025] Range partitioning is used to place all data with values in an interval into a partition block, wherein the value of the partition column is a continuous value;

[0026] Hash partitioning is used to uniformly distribute data into predefined partition blocks through a hash function;

[0027] Value partitioning is used to establish a partition for each value of a specified column;

[0028] List partitioning is used to place all data with values in an interval into a partition block, and the value of the partition column is a discrete value.

[0029] As preferred, for the time series database, the data written is stored by column, and for the data of the same column, the data type and length are the same;

[0030] For the attribute information of the data, the time information of the written partition block includes the minimum value and the maximum value of the time, and the index position information is used to record the offset of the partition block where the data is located.

[0031] In a second aspect, the application discloses a time series database partition table implementation system in a mass equipment scene, which is used for managing data of the mass equipment by the time series database partition table implementation in the mass equipment scene according to any one of the first aspect, and the system comprises a partition mode definition module, a partition management mode configuration module, a time series database configuration module and a data writing module.

[0032] The partition mode definition module is used for defining the partition mode of the time series database, so as to realize the time series database meeting the relational database partition table model, wherein the partition mode is: performing one-level partition on the time series database based on time, creating different partition directories according to the time of writing data into the database; performing two-level partition on the time series database according to the relational database partition mode, dividing the database table into a plurality of partition blocks, and corresponding to at least one partition block under each partition directory;

[0033] The partition management mode definition module is used for defining the partition management mode of the time series database, including: dividing the data table into a data file and a partition index file, the data file is cut according to the partition block, the data of each partition block can be appended and written, when the current partition block is full, a new partition block is applied for in the data file, the partition index file is used for recording attribute information of the data stored in the partition block, and the attribute information includes time information of writing data into the partition block and index position information;

[0034] The time series database configuration module is used for configuring the time series database based on the data to be written, according to the defined partition mode and partition management mode, including: creating different partition directories according to the time of writing data into the time series database, dividing the data table into a data file and a partition retrieval file, and applying for a partition block in the data file according to the specified partition mode;

[0035] The data writing module is used for writing data, including: for the data to be written, calculating the partition directory based on the writing time, and querying the index position of the specified partition block from the partition index file, if the corresponding partition block exists, writing the data into the corresponding partition block, if the corresponding partition block does not exist, applying for a new partition block in the data file, writing the data into the new partition block, and updating the new partition block to the partition index file.

[0036] As preferred, when applying the partition block in the data file according to the specified partition mode, each partition block is divided into a sequential block for storing sequential data and a disordered block for storing disordered data;

[0037] Correspondingly, when writing data into the corresponding partition block, the data writing module is configured to perform the following operations:

[0038] If the written data is sequential data, the sequential data is written into the sequential block;

[0039] If the written data is disordered data, and the sequential block is not full, the disordered data is written into the sequential block, and the sequential block is sorted and rewritten;

[0040] If the written data is disordered data, and the sequential block is full, if there is a disordered block in the corresponding partition block, the disordered data is written into the disordered block, and if there is no disordered block in the corresponding partition block, a disordered block is created in the corresponding partition block, the disordered data is written into the created disordered block, and the newly created disordered block is mounted to the partition index of the corresponding partition.

[0041] As preferred, the system further comprises a data query module, for the data written into the time series database, the data query module is configured to perform the following: when querying the partition where the data is located, if there is no disordered block in the partition block, the sequential block is queried by binary search, and if there is a disordered block in the partition block, the sequential block and the disordered block are combined for sorted query.

[0042] As preferred, the partition mode of the relational database includes four kinds of range partitioning, hash partitioning, value partitioning and list partitioning;

[0043] The range partitioning is used to put all data with values in an interval into a partition block, wherein the value of the partition column is a continuous value;

[0044] The hash partitioning is used to uniformly distribute data into predefined partition blocks through a hash function;

[0045] The value partitioning is used to establish a partition for each value of a specified column;

[0046] The list partitioning is used to put all data with values in an interval into a partition block, and the value of the partition column is a discrete value.

[0047] As preferred, for the time series database, the written data is stored according to columns, and for the data of the same column, the data type and length are the same;

[0048] For the attribute information of the data, the time information for writing the partition block includes the minimum and maximum values of the time, and the index position information is used to record the offset of the data in the partition block.

[0049] The time series database partition table implementation method and system in the mass device scenario of the application has the following advantages:

[0050] 1. For the time series database, the first-level partition is based on time, different partition directories are created according to data writing, for the data table, it is divided into data files and partition index files, the data files are cut according to partition blocks, the data of each partition block can be appended, the partition index file is used to record the attribute information of the data stored in the partition block, the commonly used partition model of the relational database is used on the time series database, which is more in line with the standard SQL syntax, the learning cost of the user is lower, it is easier to apply, and it can meet the partition table model of the ordinary time series database, it is easier to understand and use the extension, at the same time, the concurrent performance of the mass device writing to one table is improved, and the complexity of the system is reduced;

[0051] 2. The first-level partition is based on the data writing time, when writing data, the partition directory is searched according to the data writing time, and the corresponding partition block is searched according to the index position, when querying, the data range is filtered and searched according to time, when the time series data life cycle is cleaned, it is also easier to delete the historical expired data according to the partition directory, the writing in the partition is sorted and disordered

[0052] 3. The partition block is divided into ordered blocks and disordered blocks, which solves the low probability disorder situation in the Internet of Things scenario;

[0053] 4. All devices write to one table, use the same table structure, and it is more convenient to manage, effectively reducing the load of the system. At the same time, the disorder data scenario in the Internet of Things is covered in the design, and the application range is wider. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0055] The application will be further described below in combination with the drawings.

[0056] Figure 1 For the time series data partition management architecture in the time series database partition table implementation method in the mass device scenario of embodiment 1;

[0057] Figure 2 For the time series data partition writing flowchart in the time series database partition table implementation method in the mass device scenario of embodiment 1. DETAILED DESCRIPTION

[0058] The present application will be further described below in conjunction with the drawings and specific embodiments so that those skilled in the art can better understand and implement the present application, but the embodiments are not intended to limit the present application, and the technical features in the embodiments and the embodiments can be combined with each other without conflict.

[0059] The embodiment of the present application provides a time series database partition table implementation method and system in a mass device scenario, and is used for solving the technical problem of how to implement data management of a time series database on mass devices.

[0060] Embodiment 1:

[0061] The time series database partition table implementation method in the mass device scenario comprises the following steps:

[0062] S100, defining a partition mode of a time series database to implement a time series database satisfying a relational database partition table model, wherein the partition mode is: performing one-level partition on the time series database based on time, creating different partition directories according to the time of writing data into the database; performing two-level partition on the time series database according to a relational database partition mode, dividing a database table into a plurality of partition blocks, and each partition directory corresponds to at least one partition block;

[0063] S200, defining a partition management mode of the time series database: dividing a data table into a data file and a partition index file, the data file is cut according to the partition block, the data of each partition block can be appended and written, when the current partition block is full, a new partition block is applied for in the data file, and the partition index file is used for recording attribute information of data stored in the partition block, the attribute information comprises time information and index position information of writing data into the partition block;

[0064] S300, configuring the time series database based on data to be written, according to the defined partition mode and the partition management mode: creating different partition directories according to the time of writing data into the time series database, dividing a data table into a data file and a partition index file, and applying a partition block in the data file according to a specified partition mode;

[0065] S400, writing data: for the data to be written, the corresponding partition directory is calculated based on the writing time, and the index position of the specified partition block is queried from the partition index file, if the corresponding partition block exists, the data is written into the corresponding partition block, if the corresponding partition block does not exist, a new partition block is applied for in the data file, the data is written into the new partition block, and the new partition block is updated to the partition index file.

[0066] The step S100 of the embodiment defines the partition mode. For a time series database, first, a primary partition is made based on time. After data is written according to time, different partition directories are created according to time, so that it is easier to filter the data range according to time when searching. When the life cycle of time series data is cleaned up, it is also easier to delete historical expired data according to the partition directory.

[0067] Then, in combination with the partition mode of the relational database, the four commonly used partition modes are range partition (RANGE), hash partition (HASH), value partition (VALUE) and list partition (LIST).

[0068] The range partition (RANGE) can place all records with values in an interval into a partition. It should be noted that when the inserted data is not in the value defined in a partition, an exception will be thrown.

[0069] The SQL syntax of the range partition is:

[0070]

[0071] Wherein #column corresponds to the column of the range partition; #partition_name is the specified partition name; #value corresponds to the maximum value of the range partition.

[0072] The hash partition (HASH) can uniformly distribute data into each partition defined in advance through a hash function, so that the number of each partition is approximately the same.

[0073] The SQL syntax of the hash partition is:

[0074] CREATE TABLE table_name

[0075] PARTITION BY HASH(#column)

[0076] PARTITIONS#partition_num;

[0077] Wherein #column corresponds to the column of the hash partition; #partition_num is the number of partitions. After setting, the data is hashed according to the column, and the data is placed in the corresponding partition.

[0078] The value partition (VALUE) establishes a partition for each value of the specified column. For example, the Internet of Things scenario can be a device number.

[0079] The SQL syntax of the value partition is:

[0080] CREATE TABLE table_name

[0081] PARTITION BY KEY(#column)

[0082] Where #column corresponds to the column of value partition.

[0083] The value partition can well solve the problem of concurrent read and write of massive devices in the Internet of Things, because each device is assigned a separate partition, and all devices in the corresponding partition are serially written, without concurrent writing problem. In addition, when reading, the data of the corresponding device can be quickly filtered by partition, improving the query speed.

[0084] The LIST partition and the RANGE partition are very similar, except that the values of the partition column are discrete, not continuous. The LIST partition uses VALUES IN, because the values of each partition are discrete, so only values can be defined.

[0085] The SQL syntax of the LIST partition is:

[0086]

[0087]

[0088] Where #column corresponds to the column of value partition; #value_list format is value[,value][,...], such as VALUES IN(1,5,9)

[0089] The hyper table / sub table model in TDEngine is actually a value partition model in nature, but the time series partition model proposed in the embodiment is easier to use, has lower learning cost, and is also easy to extend other partition models.

[0090] In the time series database, the data is generally stored according to the column. The same column has the same data type and length, and the offset can be calculated to quickly locate a certain row, and better data compression effect can be achieved.

[0091] In addition to the traditional time partition, according to the above partition model, in order to facilitate the management of different partitions, step S200 in the embodiment designs a corresponding partition index file (.idx) and data file (.data) for the data table. The time series data partition management architecture is as shown in Figure 1 The data file is divided according to the partition block, each partition block has a fixed number of rows (such as 1000 rows), and the data of each partition is appended. When the partition block is full, a new partition block is applied at the end of the file.

[0092] The partition index records the minimum value and maximum value of the data partition block storage data time, the index position (offset of the data file) of the corresponding data block, and can additionally record the aggregation result of the partition block, facilitating fast positioning and filtering.

[0093] After the timing database library is configured based on the definitions of steps S100 and S200, step S400 of data writing can be performed.

[0094] The timing data partition writing flow is as shown in Figure 2 When data is written, the corresponding partition is calculated according to the partition definition when the table is created, and the position of the corresponding partition in the partition index file is found. If the corresponding partition does not exist, a new partition block (Block) is sequentially applied after the data file, and then written into the corresponding partition. Subsequent writing of data in this partition is all appended to the partition block. Partitioning can reduce the lock granularity of data writing, and can facilitate positioning to the corresponding partition for querying when querying, and can also realize concurrent querying of different partition blocks.

[0095] As an improvement, in the embodiment, when a partition block is applied in the data file according to the specified partitioning method, an order block for storing timing data and a disorder block for storing disorder data are divided in each partition block. Correspondingly, when data is written into the corresponding partition block, the following operations are performed:

[0096] (1) If the written data is sequential data, write the sequential data into the sequential block;

[0097] (2) If the written data is disorder data, and the sequential block is not full, write the disorder data into the sequential block, and reorder and rewrite the sequential block;

[0098] (3) If the written data is disorder data, and the sequential block is full, if there is a disorder block in the corresponding partition block, write the disorder data into the disorder block, if there is no disorder block in the corresponding partition block, create a disorder block in the corresponding partition block, write the disorder data into the created disorder block, and mount the newly created disorder block to the partition index of the corresponding partition.

[0099] The write in the partition block in the embodiment is subjected to corresponding ordering and disordering processing. In order to solve the low-probability disordering situation in the Internet of Things scene, a partition is divided into a sequential block and a disorder block. If it is a sequential write data, the sequential block is directly written into the sequential block. If a disorder data is written, and the sequential block is not full, the sequential block is added and the sequential block is reordered and rewritten. Because each block is not large, and the size is reserved, and the probability of disorder data in the Internet of Things is low, a large amount of time and system resources are not consumed. If the sequential block is full when the disorder data is written, a disorder block needs to be mounted on the corresponding partition index. In this way, when a partition is queried, if there is no disorder block, the sequential block can be quickly divided into two parts for query. If there is a disorder block, the sequential block and the disorder block need to be combined for ordering query.

[0100] The time series database partition table implementation realized by the embodiment can conveniently adapt to the partition model commonly used by the relational database, is more in line with the standard SQL grammar, and has lower learning cost of users. In the scene of the Internet of Things with a large number of devices, a suitable partition mode can be selected to realize efficient concurrent reading and writing. All devices are written into a table, the same table structure is adopted, and the table is more convenient to manage, thereby effectively reducing the load of the system. Meanwhile, the design covers the disorder data scene in the Internet of Things, and the application range is wider.

[0101] Embodiment 2

[0102] The time series database partition table implementation system for a large number of devices in the embodiment of the application comprises a partition mode definition module, a partition management mode configuration module, a time series database configuration module and a data writing module. The system can execute the method disclosed in embodiment 1 to realize data management of a large number of devices.

[0103] The partition mode definition module is used for defining the partition mode of the time series database, so as to realize the time series database that meets the partition table model of the relational database. The partition mode is that the time series database is subjected to one-level partitioning based on time, different partition directories are created according to the time of writing data into the database, the time series database is subjected to two-level partitioning according to the partition mode of the relational database, and the database table is divided into a plurality of partition blocks, and each partition directory corresponds to at least one partition block.

[0104] In the embodiment, when the partition mode definition module defines the partition mode, the time series database is first subjected to one-level partitioning based on time. After data is written according to time, different partition directories are created according to time, so that it is easier to filter and search the data range according to time when querying. When the life cycle of the time series data is cleaned, it is also easier to delete the historical expired data according to the partition directory.

[0105] Then combined with the partitioning mode of the relational database, the four commonly used partitioning modes are range partitioning (RANGE), hash partitioning (HASH), value partitioning (VALUE), and list partitioning (LIST).

[0106] Range partitioning (RANGE) can place all records with values in an interval into a partition. It should be noted that when the inserted data is not in the value defined in a partition, an exception will be thrown.

[0107] The SQL syntax of range partitioning is:

[0108]

[0109]

[0110] Where #column corresponds to the column of range partitioning; #partition_name is the specified partition name; #value corresponds to the maximum value of this range partition.

[0111] Hash partitioning (HASH) can distribute data evenly into predefined partitions through a hash function, ensuring that the number of each partition is roughly the same.

[0112] The SQL syntax of hash partitioning is:

[0113] CREATE TABLE table_name

[0114] PARTITION BY HASH(#column)

[0115] PARTITIONS#partition_num;

[0116] Where #column corresponds to the column of hash partitioning; #partition_num is the number of partitions. After setting, the data will perform hash calculation on the column, and the data will be placed in the corresponding partition.

[0117] Value partitioning (VALUE) will create a partition for each value of the specified column. For example, the Internet of Things scenario can be the device number.

[0118] The SQL syntax of value partitioning is:

[0119] CREATE TABLE table_name

[0120] PARTITION BY KEY(#column)

[0121] Where #column corresponds to the column of value partitioning.

[0122] The value partitioning can well solve the concurrent read-write problem of the Internet of Things mass equipment, because each device is allocated a separate partition, and all devices in the corresponding partition are serially written, without concurrent writing problem. In addition, when reading, the data of the corresponding device can be quickly filtered out according to the partition, improving the query speed.

[0123] The LIST partitioning and the RANGE partitioning are similar, except that the values of the partitioning column are discrete, not continuous. The LIST partitioning uses VALUES IN, because the values of each partition are discrete, so only values can be defined.

[0124] The SQL syntax of the LIST partitioning is as follows:

[0125]

[0126] Where #column corresponds to the column of the list partitioning; the format of #value_list is value[,value][,...], such as VALUES IN(1, 5, 9)

[0127] The hyper table / sub table model in the TDEngine is actually a value partitioning mode, but the time series partitioning model proposed in the embodiment is easier to use, has lower learning cost, and is easy to extend other partitioning modes.

[0128] The partition management mode definition module is used to define the partition management mode of the time series database, including: dividing the data table into a data file and a partition index file, the data file is cut according to the partition block, the data of each partition block can be appended to write, when the current partition block is full, a new partition block is applied in the data file, the partition index file is used to record the attribute information of the data stored in the partition block, the attribute information includes time information and index position information of the data written to the partition block.

[0129] In the time series database, data is generally stored according to columns. The same column has the same data type and length, and the offset can be calculated to quickly locate a certain row, and better data compression effect can be achieved.

[0130] In addition to the traditional time partitioning, according to the above partitioning model, in order to facilitate the management of different partitions, the partition management mode definition module in the embodiment is used to design the corresponding partition index file (.idx) and data file (.data) for the data table. The time series data partition management architecture is as shown in Figure 1 The data file is cut according to the partition block, each partition block has a fixed number of rows (such as 1000 rows), and the data of each partition is appended to write. When the partition block is full, a new partition block is applied at the end of the file.

[0131] The partition index records the minimum value, the maximum value, and the index position (offset of the data file) of the data partition block storing data time. The aggregation result of the partition block can also be recorded additionally to facilitate fast positioning and filtering.

[0132] The time series database configuration module is configured to configure the time series database based on the data to be written, according to the defined partition mode and partition management mode, including: creating different partition directories according to the time of writing data into the time series database, dividing the data table into data files and partition search files, and applying partition blocks in the data files according to the specified partition mode.

[0133] The data writing module is configured to write data, including: for the data to be written, based on the writing time, calculating the corresponding partition directory, and querying the index position of the specified partition block from the partition index file, if the corresponding partition block exists, writing the data into the corresponding partition block, if the corresponding partition block does not exist, applying a new partition block in the data file, writing the data into the new partition block, and updating the new partition block to the partition index file.

[0134] When data is written, the data writing module is configured to perform the following: according to the partition definition when the table is created, the corresponding partition is calculated, and the position of the corresponding partition is found in the partition index file. If the corresponding partition does not exist, a new partition block (Block) will be sequentially applied behind the data file, and then written into the corresponding partition. Subsequent writing of this partition data is all appended to the partition block. Partitioning can reduce the lock granularity of data writing, and when querying, it can also be convenient to locate the corresponding partition for querying, and concurrent querying of different partition blocks can also be achieved.

[0135] As an improvement, in the embodiment, when applying partition blocks in the data file according to the specified partition mode, each partition block is divided into an order block for storing time series data and a disorder block for storing disorder data. Correspondingly, when writing data into the corresponding partition block, the data writing module is configured to perform the following operations:

[0136] (1) If the written data is sequential data, write the sequential data into the sequential block;

[0137] (2) If the written data is disorder data, and the sequential block is not full, write the disorder data into the sequential block and rewrite the sequential block in order;

[0138] (3) If the written data is disorder data, and the sequential block is full, if there is a disorder block in the corresponding partition block, write the disorder data into the disorder block, if there is no disorder block in the corresponding partition block, create a disorder block in the corresponding partition block, write the disorder data into the created disorder block, and mount the newly created disorder block to the partition index of the corresponding partition.

[0139] In this embodiment, the write in the partition block will be processed by corresponding sorting and disordering. In order to solve the low probability disordering situation in the Internet of Things scene, a partition is divided into a sequential block and a disorder block. If it is a sequential write data, it will be directly written into the sequential block. If it is a disorder data, and the sequential block is not full, it will be added to the sequential block and the sequential block will be sorted and rewritten. Because each block is not large, and the size is reserved, and the probability of disorder data in the Internet of Things is low, it will not consume a lot of time and system resources. If the disorder data is written, the sequential block is full, and a disorder block needs to be mounted on the corresponding partition index. In this way, when a partition is queried, if there is no disorder block, the binary query can be quickly performed from the sequential block; if there is a disorder block, the sequential block and the disorder block need to be combined for sorting query.

[0140] The application has been described in detail above through the accompanying drawings and preferred embodiments, however, the application is not limited to these disclosed embodiments, and those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the application, and these embodiments are also within the protection scope of the application.

Claims

1. A method for implementing a partitioned table in a time-series database under a massive device scenario, characterized in that, Includes the following steps: Define the partitioning method for the time series database to achieve a time series database that satisfies the partitioning table model of a relational database. The partitioning method is as follows: perform first-level partitioning of the time series database based on time, creating different partition directories according to the time when data is written to the database; perform second-level partitioning of the time series database according to the partitioning method of a relational database, dividing the database tables into multiple partition blocks, with each partition directory corresponding to at least one partition block. Define the partition management method for time-series databases: divide the data table into data files and partition index files. The data files are split according to partition blocks. Data in each partition block can be appended to. When the current partition block is full, a new partition block is requested in the data file. The partition index file is used to record the attribute information of the data stored in the partition blocks. The attribute information includes the time information of the data being written to the partition block and the index position information. Configure the time-series database based on the data to be written and according to the defined partitioning method and partition management method: create different partition directories according to the time when the data is written to the time-series database, divide the data table into data files and partition retrieval files, and apply for partition blocks in the data files according to the specified partitioning method; Writing data: For the data to be written, the partition directory to which it belongs is calculated based on the write time, and the index position of the specified partition block is queried from the partition index file. If the corresponding partition block exists, the data is written to the corresponding partition block. If the corresponding partition block does not exist, a new partition block is requested in the data file, the data is written to the new partition block, and the new partition block is updated in the partition index file.

2. The method for implementing a time-series database partition table in a massive device scenario according to claim 1, characterized in that, When requesting partition blocks in a data file according to the specified partitioning method, each partition block is divided into sequential blocks for storing time-series data and out-of-order blocks for storing out-of-order data. Correspondingly, when writing data to the corresponding partition block, the following operations are performed: If the data to be written is sequential data, write the sequential data into a sequential block; If the data to be written is out of order and the sequential block is not full, write the out-of-order data into the sequential block and rewrite the sequential block. If the data being written is out of order and the sequential blocks are already full, if an out-of-order block exists in the corresponding partition block, the out-of-order data is written into the out-of-order block. If no out-of-order block exists in the corresponding partition block, an out-of-order block is created in the corresponding partition block, the out-of-order data is written into the created out-of-order block, and the newly created out-of-order block is mounted to the partition index of the corresponding partition.

3. The method for implementing a time-series database partition table in a massive device scenario according to claim 2, characterized in that, When querying the partition where the data is located, if there are no out-of-order blocks in the partition, a binary search is performed from the sequential blocks. If there are out-of-order blocks in the partition, the sequential blocks and out-of-order blocks are merged for a sorted search.

4. The method for implementing a time-series database partition table in a massive device scenario according to any one of claims 1-3, characterized in that, Relational databases can be partitioned in four ways: range partitioning, hash partitioning, value partitioning, and list partitioning. Range partitioning is used to group all data within a range into a single partition block, where the values ​​in the partition column are consecutive. Hash partitioning is used to distribute data evenly into predefined partition blocks using a hash function; Value partitioning is used to create a partition for each value in a specified column; List partitioning is used to group all data within a certain range into a partition block, where the values ​​in the partition column are discrete values.

5. The method for implementing a time-series database partition table in a massive device scenario according to any one of claims 1-3, characterized in that, For time-series databases, the written data is stored according to columns, and the data in the same column has the same data type and length. For data attribute information, the time information for writing to the partition block includes the minimum and maximum times, and the index position information is used to record the offset of the partition block where the data is located.

6. A system for implementing a time-series database partition table in a massive device scenario, characterized in that, The system is used to manage data of massive devices through a time-series database partition table in a massive device scenario as described in any one of claims 1-5. The system includes a partitioning method definition module, a partitioning management method configuration module, a time-series database configuration module, and a data writing module. The partitioning method definition module is used to define the partitioning method of the time series database to realize a time series database that meets the partition table model of a relational database. The partitioning method is as follows: the time series database is partitioned at the first level based on time, and different partition directories are created according to the time when the data is written to the database; the time series database is partitioned at the second level according to the partitioning method of a relational database, and the database table is divided into multiple partition blocks, with at least one partition block corresponding to each partition directory. The partition management method definition module is used to define the partition management method of the time series database, including: dividing the data table into data files and partition index files, splitting the data files according to partition blocks, allowing data to be appended to each partition block, requesting a new partition block in the data file when the current partition block is full, and using the partition index file to record the attribute information of the data stored in the partition blocks, including the time information of data being written to the partition block and the index position information. The time-series database configuration module is used to configure the time-series database based on the data to be written and according to the defined partitioning method and partition management method, including: creating different partition directories according to the time when the data is written to the time-series database, dividing the data table into data files and partition retrieval files, and applying for partition blocks in the data files according to the specified partitioning method; The data writing module is used to write data, including: for the data to be written, calculating the partition directory based on the writing time and querying the index position of the specified partition block from the partition index file; if the corresponding partition block exists, writing the data into the corresponding partition block; if the corresponding partition block does not exist, requesting a new partition block in the data file, writing the data into the new partition block, and updating the new partition block to the partition index file.

7. The system for implementing a time-series database partition table in a massive device scenario according to claim 6, characterized in that, When requesting partition blocks in a data file according to the specified partitioning method, each partition block is divided into sequential blocks for storing time-series data and out-of-order blocks for storing out-of-order data. Correspondingly, when writing data to the corresponding partition block, the data writing module performs the following operations: If the data to be written is sequential data, write the sequential data into a sequential block; If the data to be written is out of order and the sequential block is not full, write the out-of-order data into the sequential block and rewrite the sequential block. If the data being written is out of order and the sequential blocks are already full, if an out-of-order block exists in the corresponding partition block, the out-of-order data is written into the out-of-order block. If no out-of-order block exists in the corresponding partition block, an out-of-order block is created in the corresponding partition block, the out-of-order data is written into the created out-of-order block, and the newly created out-of-order block is mounted to the partition index of the corresponding partition.

8. The system for implementing a time-series database partition table in a massive device scenario according to claim 7, characterized in that, The system also includes a data query module. For data written to the time-series database, the data query module performs the following: when querying the partition where the data is located, if there are no out-of-order blocks in the partition, a binary search is performed from the sequential blocks; if there are out-of-order blocks in the partition, the sequential blocks and out-of-order blocks are merged for a sorting search.

9. The system for implementing a time-series database partition table in a massive device scenario according to any one of claims 6-8, characterized in that, Relational databases can be partitioned in four ways: range partitioning, hash partitioning, value partitioning, and list partitioning. Range partitioning is used to group all data within a range into a single partition block, where the values ​​in the partition column are consecutive. Hash partitioning is used to distribute data evenly into predefined partition blocks using a hash function; Value partitioning is used to create a partition for each value in a specified column; List partitioning is used to group all data within a certain range into a partition block, where the values ​​in the partition column are discrete values.

10. The system for implementing a time-series database partition table in a massive device scenario according to any one of claims 6-8, characterized in that, For time-series databases, the written data is stored according to columns, and the data in the same column has the same data type and length. For data attribute information, the time information for writing to the partition block includes the minimum and maximum times, and the index position information is used to record the offset of the partition block where the data is located.

Images