Time series data storage method, device, equipment, medium and computer program product

Abstract

The present invention relates to the field of storage technology, and specifically discloses a method, apparatus, device, medium and computer program product for storing time series data. The pre-split row key position of the data partition of the target database is determined in advance according to the row key range and the current read-write row key range is determined in the data partition based on this. When writing the time series data to be written, the row key is allocated in the current read-write row key range. When the data partition is split, the sub-data partition including the current read-write row key range is retained in the original storage node, and the sub-data partition migrated to other storage nodes is determined in the row key range not including the current read-write row key range, so that the splitting of the data partition only brings about the migration of metadata but does not include the migration of time series data, so that there will be no disk data input and output storm caused by the migration of a large amount of time series data and the response blocking of the read-write request, and there will be no reduction in the query performance of the time series data under the storage node due to the low localization rate of the storage node.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a method, apparatus, device, medium and computer program product for storing time series data. Background Art

[0002] Time series data is a type of data recorded in chronological order. It is often used to analyze and predict trends and patterns over time and has widespread application in many fields. HBase is a distributed database that stores data in columns and manages metadata through data partitions (regions). Each data partition contains a certain range of row keys. When storing time series data, HBase uses special row key design and tables to evenly write massive amounts of time series data to different data partitions on different storage nodes. In HBase, when the time series data in a data partition reaches its capacity limit or when the cluster is expanded, the data partition is split, for example, splitting a data partition into two sub-data partitions. At the same time, based on the load balancing properties of HBase, the two split sub-data partitions will be distributed to different storage nodes. This process includes migrating the metadata of the sub-data partitions to the new storage nodes. In order to ensure the locality rate (that is, the node where the data partition is located and the time series data it manages should ultimately be distributed on the same storage node), the time series data corresponding to the sub-data partitions also needs to be migrated. This results in a large amount of disk data input and output (I / O), resulting in blocked responses to read and write requests. Before the locality rate of the new storage node reaches 100%, the query performance of the time series data under it will also be severely reduced.

[0003] How to solve the disk data input and output storm problem caused by data migration due to data partition splitting in a time series database is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of the present invention is to provide a time series data storage method, device, equipment, medium and computer program product for solving the disk data input and output storm problem caused by data migration due to data partition splitting in a time series database.

[0005] To solve the above technical problems, the present invention provides a time series data storage method, comprising:

[0006] Obtaining a row key range corresponding to each data partition of the target database, and determining a pre-split row key position of the data partition within the row key range of the data partition;

[0007] Determine the current read / write row key range in the data partition based on the pre-split row key position;

[0008] When writing the time series data to be written into the data partition, assigning a row key to the time series data to be written within the current read-write row key range;

[0009] When the data partition splitting condition is met, the data partition to be split is split into multiple sub-data partitions according to the pre-split row key position of the data partition to be split, so that the sub-data partition including the current read-write row key range is retained in the original storage node, and the sub-data partition within the row key range that does not include the current read-write row key range is determined to be migrated to other storage nodes.

[0010] In one aspect, determining a pre-split row key position of the data partition within a row key range of the data partition includes:

[0011] Determine an upper limit on the number of sub-data partitions that can be split from the data partition based on the size of the row key range of the data partition and the minimum row key splitting granularity of the data partition;

[0012] Determining the number of the sub-data partitions for actual pre-splitting of the data partition within a range not exceeding an upper limit on the number of the sub-data partitions;

[0013] The pre-split row key position is determined within the row key range of the data partition according to the row key range of the data partition and the number of the sub-data partitions actually pre-split of the data partition.

[0014] On the other hand, determining an upper limit on the number of sub-data partitions that can be split from the data partition based on the size of the row key range of the data partition and the minimum row key splitting granularity of the data partition includes:

[0015] Determining the number of optional values ​​corresponding to the time series split flag bit configured in the row key of the data partition according to the data size of the time series split flag bit;

[0016] The number of optional values ​​corresponding to the timing split flag is the upper limit of the number of the sub-data partitions that the data partition can be split into.

[0017] On the other hand, determining a pre-split row key position of the data partition within a row key range of the data partition includes:

[0018] Determining the number of the sub-data partitions to be actually pre-split the data partition within a range that is not less than the number of preset data partitions corresponding to the target database;

[0019] The pre-split row key position is determined within the row key range of the data partition according to the row key range of the data partition and the number of the sub-data partitions actually pre-split of the data partition.

[0020] On the other hand, determining a pre-split row key position of the data partition within a row key range of the data partition includes:

[0021] Determine an upper limit on the number of sub-data partitions that can be split from the data partition based on the size of the row key range of the data partition and the minimum row key splitting granularity of the data partition;

[0022] Determining the number of the sub-data partitions for actual pre-splitting of the data partition within a range that does not exceed the upper limit of the number of sub-data partitions and is not less than the preset number of data partitions corresponding to the target database;

[0023] The pre-split row key position is determined within the row key range of the data partition according to the row key range of the data partition and the number of the sub-data partitions actually pre-split of the data partition.

[0024] On the other hand, determining the current read / write row key range in the data partition with the pre-split row key position as the boundary includes:

[0025] The row key range before the first pre-split row key position of the data partition is used as the current read-write row key range.

[0026] On the other hand, determining the sub-data partition to be migrated to another storage node in a row key range that does not include the current read / write row key range includes:

[0027] The row key ranges that do not include the current read / write row key range are all used as the sub-data partitions to be migrated to other storage nodes.

[0028] On the other hand, when writing the time series data to be written into the data partition, allocating a row key for the time series data to be written within the current read / write row key range includes:

[0029] When writing the time series data to be written into the data partition, the current read-write row key range of the data partition is determined according to the mapping relationship table that records the correspondence between the data partition and the current read-write row key range in the target database, and a row key is allocated to the time series data to be written within the current read-write row key range.

[0030] On the other hand, the mapping relationship table includes a mapping relationship between a row key range of each time series data write in the data partition and a time series split flag bit;

[0031] The time series data write row key range is the interval into which the data partition is divided according to the pre-split row key position, and the current read / write row key range is one of the time series data write row key ranges; the time series split flag is a newly added flag of the row key managed by the data partition, and the time series split flag of the row key in the data partition uniquely corresponds to the time series data write row key range in which it is located.

[0032] On the other hand, determining the current read / write row key range of the data partition according to a mapping relationship table recording the correspondence between the data partition and the current read / write row key range in the target database, and allocating a row key for the time series data to be written within the current read / write row key range includes:

[0033] Query the current read / write row key range corresponding to the data partition and the timing split flag corresponding to the current read / write row key range according to the mapping relationship table;

[0034] A row key is allocated to the time series data to be written in the current read / write row key range, and the time series split flag obtained by the query is written into the allocated row key, so as to allocate a row key to the time series data to be written in the current read / write row key range.

[0035] On the other hand, when writing the time series data to be written into the data partition, allocating a row key for the time series data to be written within the current read / write row key range includes:

[0036] Upon receiving a data write task corresponding to the time series data to be written, calculating a salt value corresponding to the time series data to be written;

[0037] Determine the data partition corresponding to the salt value as the target data partition for writing the time series data;

[0038] A row key is allocated to the time series data to be written in the current read-write row key range of the target data partition, and the salt value is added before the allocated row key.

[0039] On the other hand, allocating a row key for the time series data to be written in the current read / write row key range includes:

[0040] An unassigned row key is randomly selected in the current read-write row key range as the row key of the time series data to be written.

[0041] On the other hand, when writing the time series data to be written into the data partition, allocating a row key for the time series data to be written within the current read / write row key range includes:

[0042] Upon receiving a data write task corresponding to the time series data to be written, calculating a salt value corresponding to the time series data to be written;

[0043] Determine the data partition corresponding to the salt value as the target data partition for writing the time series data;

[0044] An unassigned row key is randomly selected in the current read-write row key range of the target data partition as the row key of the time series data to be written, and the salt value is added before the assigned row key.

[0045] On the other hand, calculating the salt value corresponding to the time series data to be written includes:

[0046] Calculate a hash code based on an array consisting of the indicator name of the time series data to be written and the tag key value of the time series data to be written;

[0047] The hash code is modulo calculated by the number of buckets corresponding to the target database to obtain the salt value corresponding to the time series data to be written.

[0048] On the other hand, the step of determining the number of buckets corresponding to the target database includes:

[0049] The number of buckets is determined according to the load status of the cluster where the target database is located and the number of data partition service nodes in the cluster where the target database is located.

[0050] On the other hand, the data partition splitting condition includes at least one of: the amount of time series data corresponding to the data partition reaches the maximum data capacity corresponding to the data partition, and the cluster expansion of the target database triggers the data partition splitting.

[0051] To solve the above technical problems, the present invention further provides a time series data storage device, comprising:

[0052] A data partition splitting and balancing module is used to obtain the row key range corresponding to each data partition of the target database, determine the pre-split row key position of the data partition within the row key range of the data partition; determine the current read-write row key range in the data partition with the pre-split row key position as the boundary; when the data partition splitting condition is met, split the data partition to be split into multiple sub-data partitions according to the pre-split row key position of the data partition to be split, so that the sub-data partition including the current read-write row key range is retained in the original storage node, and the sub-data partition in the row key range not including the current read-write row key range is determined to be migrated to other storage nodes;

[0053] A mapping management module, configured to manage a mapping relationship table recording the corresponding relationship between the data partition and the current read / write row key range;

[0054] A read-write conversion module is used to query the mapping relationship table through the mapping management module to determine the current read-write row key range of the data partition when writing the time series data to be written into the data partition, and allocate a row key to the time series data to be written within the current read-write row key range.

[0055] To solve the above technical problems, the present invention further provides a time series data storage device, comprising:

[0056] Memory for storing computer programs;

[0057] The processor is configured to execute the computer program, and when the computer program is executed by the processor, the steps of the time series data storage method as described in any one of the above are implemented.

[0058] To solve the above technical problems, the present invention further provides a non-volatile storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the time series data storage method as described in any one of the above items.

[0059] To solve the above technical problems, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned time series data storage methods.

[0060] The time series data storage method provided by the present invention has the beneficial effect of determining the pre-split row key position of the data partition of the target database according to the row key range in advance, and determining the current read-write row key range in the data partition with the pre-split row key position as the boundary, so that when the time series data to be written is written into the data partition, the row key is allocated to the time series data to be written within the current read-write row key range; and when the data partition splitting condition is met, the data partition to be split is split into multiple sub-data partitions according to the pre-split row key position of the data partition to be split, so that the sub-data partition including the current read-write row key range is retained in the original storage node, and the sub-data partition not including the current read-write row key range is retained in the original storage node. The row key range determines the sub-data partitions that are migrated to other storage nodes, so that the sub-data partitions that need to be migrated to other storage nodes do not correspond to time series data, and will not cause the localization rate of other storage nodes to decrease. Therefore, the splitting of data partitions only brings about the migration of metadata but not the migration of time series data. As a result, there will be no disk data input and output storms and response blockages to read and write requests caused by the migration of a large amount of time series data, nor will the query performance of the time series data under the storage node be reduced due to the low localization rate of the storage node. This realizes transparent data partition splitting and cluster expansion, and improves the user experience on the business side.

[0061] The present invention also provides a time series data storage device, equipment, medium and computer program product, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1 A flowchart of a time series data storage method provided by an embodiment of the present invention;

[0064] Figure 2 A schematic diagram of data partition splitting provided by an embodiment of the present invention;

[0065] Figure 3 An architectural diagram of a time series data storage device provided by an embodiment of the present invention;

[0066] Figure 4 A schematic structural diagram of a time series data storage device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The core of the present invention is to provide a time series data storage method, device, equipment, medium and computer program product for solving the disk data input and output storm problem caused by data migration due to data partition splitting in the time series database.

[0068] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0069] HBase is a highly reliable, high-performance, column-oriented distributed storage system. Its architecture consists of components such as the master server (HMaster), region servers (HRegionServers), and coordination service (Zookeeper). The HMaster manages tables and regions, while the HRegionServers respond to user input and output (I / O) requests. Zookeeper provides coordination and failover mechanisms.

[0070] HBase is a column-oriented storage system. In HBase, a table consists of rows and columns, and data is sorted by row. Each row has a unique row key, which is the primary method for accessing data.

[0071] In HBase, a data table has multiple data partitions, which serve as the basic unit of data management. Data partitions are evenly distributed across all storage nodes in the database cluster to achieve load balancing for read and write requests. Each data partition maintains a start row key (startRowKey) and an end row key (endRowKey). If a data entry's row key falls within the range of row keys maintained by a data partition, that data entry is managed and maintained by that partition.

[0072] In the HBase cluster, a region server is usually deployed on each storage node. Each region server can manage one or more data partitions.

[0073] When storing time series data in HBase, it is stored continuously according to the actual time. One implementation method is to format the actual time of the time series data as a timestamp in hourly units, that is, the time at which the time series data occurred. The specific time (offset) of the time series data within the hour is written as the column name. For example, the 1st second of the hour is in column 1, and the 3600th second is in column 3600. When the data time precision is milliseconds, the columns are from column 1 to column 3600000. However, if only this continuous writing method is used, when there is a large amount of time series data at a specific hour, the continuously stored locations will be subject to a large number of read and write accesses, causing overload of access to a data partition. Therefore, in HBase, multiple data partitions are created in advance. Through a special row key design, time series data is evenly distributed across different data partitions on different storage nodes while being stored continuously in chronological order.

[0074] However, the data size supported by a single data partition has an upper limit (such as 10GB). As time series data is continuously written, the data in the data partition will inevitably split when the upper limit is reached. The single data partition will be split into two sub-data partitions, and then the two sub-data partitions will be distributed to the storage nodes where different regional servers (RegionServers) are located according to the specified strategy. This will cause the localization rate of the storage node to which the sub-data partition is migrated to be reduced, and disk I / O storms caused by the migration of underlying files between different storage nodes will occur.

[0075] It should be noted that when the storage node where a data partition resides and the data it manages are distributed on the same storage node, the locality ratio of the data partition is 100%. The smallest unit of the locality ratio indicator is the data partition, and the minimum value of the locality ratio is 0 and the maximum value is 1.

[0076] In order to solve the disk I / O storm problem caused by data migration due to data partition splitting in a time series database, an embodiment of the present invention provides a time series data storage solution. By dividing the current read-write row key range in the data partition, the row key range corresponding to the entire data partition is no longer used as the range for assigning row keys to the time series data, but only the current read-write row key range in the data partition is used as the range for assigning row keys to the time series data. When the data partition splitting condition is met, the sub-data partition including the current read-write row key range is retained in the original storage node where the data partition is located, and the sub-data partition migrated to other storage nodes is determined within the row key range that does not include the current read-write row key range. Therefore, the splitting of the data partition will only trigger the migration of metadata, and will not trigger the migration of time series data. When the sub-data partition is just migrated to the new storage node, it will not cause the localization rate of the new storage node to decrease, and will not cause the problem of reduced query performance for querying time series data.

[0077] Based on the above architecture, the time series data storage method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0078] Figure 1 A flowchart of a time series data storage method provided by an embodiment of the present invention.

[0079] like Figure 1 As shown, the time series data storage method provided by the embodiment of the present invention includes:

[0080] S101: Obtain a row key range corresponding to each data partition of the target database, and determine a pre-split row key position of the data partition within the row key range of the data partition;

[0081] S102: Determine the current read / write row key range in the data partition using the pre-split row key position as a boundary;

[0082] S103: When writing the time series data to be written into the data partition, a row key is allocated to the time series data to be written in the current read / write row key range;

[0083] S104: When the data partition splitting condition is met, the data partition to be split is split into multiple sub-data partitions according to the pre-split row key position of the data partition to be split, so that the sub-data partitions including the current read-write row key range are retained in the original storage node, and the sub-data partitions within the row key range that do not include the current read-write row key range are determined to be migrated to other storage nodes.

[0084] It should be noted that the type of the "target database" in each embodiment of the present invention is HBase, and the cluster on which it is based is a distributed storage system, including multiple storage nodes. A region server (RegionServer) is usually deployed on each storage node. Each region server is used to manage one or more data partitions (Region), and each data partition corresponds to a row key (Row Key).

[0085] The time series data storage method provided by the embodiment of the present invention can be applied to a management node in a distributed storage system, and can be implemented in software by modifying the management program of the target database.

[0086] When storing time series data in HBase, multiple data partitions are pre-deployed on multiple storage nodes to balance reads and writes. Row key design is then used to evenly distribute the time series data across different data partitions and across different row key positions within the data partitions. As the time series data corresponding to a data partition reaches the upper capacity limit of the data partition, or as the distributed storage system's cluster expands, the data partition is triggered to split. This splitting process is automatically controlled by HBase. The time series data storage method provided in this embodiment of the present invention builds on the existing HBase solution for storing time series data by adding a mechanism for pre-determining the pre-split row key positions for data partitions.

[0087] It should be noted that the pre-splitting row key position in the embodiment of the present invention is different from the current HBase pre-splitting mechanism. The latter means that when a data table is first created in HBase, HBase will assign a data partition to the data table by default. However, this will cause all read and write requests to access the same data partition in a regional server, which will not achieve the effect of load balancing. Other regional servers in the cluster may be in a relatively idle state. Therefore, through the pre-partitioning method, multiple data partitions are generated and configured when the data table is created.

[0088] The determination of the pre-split row key position in the embodiment of the present invention is a further improvement on the current HBase pre-partitioning mechanism. That is, it is considered in advance that the data partition may be split in the future due to the time series data capacity reaching the upper limit or the cluster expansion, so the row key position of the data partition is determined when the data partition is split in the future. It can be understood that the current data partition has not been split.

[0089] On this basis, the row key range corresponding to the data partition is segmented according to the pre-split row key position, and one of them is selected as the current read-write row key range. That is, when the data partition has not been split, data is not read and written for the entire row key range corresponding to the data partition, but only for the current read-write row key range. Then, when the data partition splitting conditions are met, the sub-data partitions including the current read-write row key range are retained in the original storage node where the data partition is located, and the sub-data partitions within the row key range that does not include the current read-write row key range are determined to be migrated to other storage nodes. Therefore, the splitting of the data partition will only trigger the migration of metadata, and will not trigger the relocation of time series data. When the sub-data partition is just migrated to the new storage node, it will not cause the localization rate of the new storage node to decrease, and will not cause the problem of reduced query performance for querying time series data.

[0090] In some optional implementations of the embodiments of the present invention, determining the current read / write row key range in the data partition using the pre-split row key position as a boundary in S102 may include: using the row key range before the first pre-split row key position of the data partition as the current read / write row key range. That is, for each data partition, after the row keys are divided into multiple read / write row key ranges in ascending order using the pre-split row key position as a boundary, the first read / write row key range is used as the current read / write row key range. Before the data partition is split, reads and writes to the data partition are limited to the current read / write row key range.

[0091] In some other optional implementations of the embodiments of the present invention, S102 determines the current read-write row key range in the data partition with the pre-split row key position as the boundary, and may also include: dividing the row key range of the data partition into multiple read-write row key ranges with the pre-split row key position as the boundary, and then randomly selecting a read-write row key range as the current read-write row key range of the data partition.

[0092] Regarding S103, when writing the time series data to be written into the data partition, assigning a row key to the time series data to be written within the current read / write row key range may include: when writing the time series data to be written into the data partition, determining the current read / write row key range of the data partition based on a mapping relationship table that records the correspondence between the data partitions in the target database and the current read / write row key range, and assigning a row key to the time series data to be written within the current read / write row key range. As described in the above embodiments of the present invention, a mapping relationship table can be created and maintained to determine the actually accessible location, i.e., the current read / write row key range of each data partition, during the reading and writing process of the target database.

[0093] For S104, the data partition splitting condition includes at least one of the following: the amount of time series data corresponding to the data partition reaches the maximum data capacity corresponding to the data partition, or the cluster expansion of the target database triggers the data partition splitting. When the data partition splitting condition is met, the data partition to be split is split into multiple sub-data partitions based on the pre-split row key position of the data partition to be split, so that the sub-data partitions that include the current read / write row key range remain on the original storage node, and the sub-data partitions within the row key range that do not include the current read / write row key range are determined to be migrated to other storage nodes.

[0094] In some optional implementations of the embodiments of the present invention, determining sub-data partitions to migrate to other storage nodes within a row key range that does not include the current read / write row key range may include: treating all row key ranges that do not include the current read / write row key range as sub-data partitions to migrate to other storage nodes. In other words, the sub-data partitions corresponding to the current read / write row key range are retained on the original storage node, while the row key ranges outside the current read / write row key range are divided into other sub-data partitions and migrated to other storage nodes.

[0095] When applying the time series data storage method provided by an embodiment of the present invention, if the data partition splitting condition is that the number of time series data corresponding to the data partition reaches the maximum data capacity corresponding to the data partition, then after the data partition splitting is completed, the sub-data partition retained in the original storage node is the time series data capacity that reaches the maximum data capacity, and no new data will be written, and only read data requests will be received.

[0096] When the data partition splitting condition is triggered by the cluster expansion of the target database, after the data partition split is completed, if the sub-data partition retained on the original storage node does not reach the maximum data capacity corresponding to the original data partition, new time series data can continue to be written.

[0097] The time series data storage method provided by the embodiment of the present invention determines the pre-split row key position of the data partition of the target database according to the row key range in advance, and determines the current read-write row key range in the data partition with the pre-split row key position as the boundary, so that when the time series data to be written is written into the data partition, the row key is allocated to the time series data to be written in the current read-write row key range; and when the data partition splitting condition is met, the data partition to be split is split into multiple sub-data partitions according to the pre-split row key position of the data partition to be split, so that the sub-data partition including the current read-write row key range is retained in the original storage node, and the sub-data partition not including the row key range of the current read-write row key range is retained in the original storage node. The scope determines the sub-data partitions to be migrated to other storage nodes, so that the sub-data partitions that need to be migrated to other storage nodes do not correspond to time series data, and will not cause the problem of reduced locality rate of other storage nodes to which they are migrated. Therefore, the splitting of data partitions only brings about the migration of metadata but not the migration of time series data. As a result, there will be no disk data input and output storms and response blockages to read and write requests caused by the migration of a large amount of time series data, nor will the query performance of the time series data under the storage node be reduced due to the low locality rate of the storage node. This realizes transparent data partition splitting and cluster expansion, and improves the user experience on the business side.

[0098] Based on the above embodiment, the embodiment of the present invention further introduces a method for determining a pre-split row key position.

[0099] In some optional implementations of the embodiments of the present invention, determining the pre-split row key position of the data partition within the row key range of the data partition in S101 may include: determining the upper limit of the number of sub-data partitions that the data partition can be split into based on the size of the row key range of the data partition and the minimum row key splitting granularity of the data partition; determining the number of sub-data partitions that are actually pre-split for the data partition within a range that does not exceed the upper limit of the number of sub-data partitions; and determining the pre-split row key position within the row key range of the data partition based on the row key range of the data partition and the number of sub-data partitions that are actually pre-split for the data partition.

[0100] That is, first determine the minimum row key splitting granularity of the data partition, that is, the minimum allowable read-write row key range or the maximum number of sub-data partitions that a single data partition can be split into. Based on this, determine the upper limit of the number of sub-data partitions. Under the premise of not exceeding the upper limit of the number of sub-data partitions, select the actual number of sub-data partitions for the data partition to be pre-split, so as to evenly divide the row key range of the data partition into multiple read-write row key ranges.

[0101] In some optional implementations of the embodiments of the present invention, determining the upper limit of the number of sub-data partitions that a data partition can be split into based on the size of the row key range of the data partition and the minimum row key splitting granularity of the data partition can include: determining the number of optional numerical values ​​corresponding to the timing split flag bit according to the data size of the timing split flag bit configured in the row key of the data partition; and taking the number of optional numerical values ​​corresponding to the timing split flag bit as the upper limit of the number of sub-data partitions that the data partition can be split into.

[0102] In an embodiment of the present invention, a time series split flag can be added to the row key to indicate the relationship between the current data partition and the read / write row key range and the pre-split row key position. The time series split flag indicates the correspondence between the row key and the read / write row key range after the data partition is split. At the same time, based on the time series split flag, the time series data within a certain time range is assigned row keys within the current read / write row key range, making it easier for the data partition to specify the pre-split row key position and complete the data partition split without causing data migration issues. As described in the above embodiment of the present invention, the mapping relationship can be maintained by a mapping relationship table, which records the correspondence between the data partition and the current read / write row key range.

[0103] For example, a byte of data (8 bits) can be used to represent the timing split flag. The timing split flag has a total of An optional value, indicating that a single data partition can trigger a maximum of 255 split operations.

[0104] In some other optional implementations of the embodiments of the present invention, determining the pre-split row key position of the data partition within the row key range of the data partition in S101 may also include: determining the number of actual pre-split sub-data partitions of the data partition within a range that is not less than the preset number of data partitions corresponding to the target database; determining the pre-split row key position within the row key range of the data partition based on the row key range of the data partition and the number of actual pre-split sub-data partitions of the data partition. In actual applications, some HBases may pre-determine the number of appropriate data partitions due to operation and maintenance experience or user needs, so that the data partitions are split to continuously approach this number and eventually remain at this number during the operation of the target database. Therefore, the preset number of data partitions corresponding to the target database can be used as the lower limit of the number of sub-data partitions that the data partition can be split, and a value greater than the lower limit of the number of sub-data partitions can be selected as the actual pre-split sub-data partitions of the data partition, and thereby the row key range of the data partition can be evenly divided into multiple read-write row key ranges.

[0105] In combination with the above-mentioned method for determining the lower limit of the number of sub-data partitions that can be split and the upper limit of the number of sub-data partitions that can be split, in other optional implementations of the embodiments of the present invention, determining the pre-split row key position of the data partition within the row key range of the data partition in S101 also includes: determining the upper limit of the number of sub-data partitions that can be split according to the size of the row key range of the data partition and the minimum row key splitting granularity of the data partition; determining the actual pre-split number of sub-data partitions of the data partition within a range that does not exceed the upper limit of the number of sub-data partitions and is not less than the preset number of data partitions corresponding to the target database; and determining the pre-split row key position within the row key range of the data partition according to the row key range of the data partition and the actual pre-split number of sub-data partitions of the data partition.

[0106] The method for determining the lower limit and upper limit of the number of sub-data partitions that a data partition can be split into can be found in the above description. Within the range from the lower limit to the upper limit, the actual number of sub-data partitions to be split into is selected, and the row key range of the data partition is evenly divided into multiple read-write row key ranges.

[0107] Based on the above embodiments, the embodiments of the present invention further introduce a data reading and writing method in a time series data storage method using a new data partitioning and splitting method.

[0108] According to the records of the above embodiments of the present invention, in S103, when writing the time series data to be written into the data partition, a row key is allocated to the time series data to be written in the current read-write row key range, which may include: when writing the time series data to be written into the data partition, according to a mapping relationship table that records the correspondence between the data partition and the current read-write row key range in the target database, determining the current read-write row key range of the data partition, and allocating a row key to the time series data to be written in the current read-write row key range.

[0109] That is, a mapping relationship table can be created and maintained to determine the actual accessible location during the reading and writing process of the target database, that is, the current read and write row key range of each data partition.

[0110] In some optional implementations of the embodiments of the present invention, the mapping relationship table may include a mapping relationship between each time series data write row key range and a time series split flag in a data partition; the time series data write row key range is an interval that divides the data partition according to the pre-split row key position, and the current read and write row key range is one of the time series data write row key ranges; the time series split flag is a newly added flag of the row key managed by the data partition, and the time series split flag of the row key in the data partition uniquely corresponds to the time series data write row key range in which it is located.

[0111] As described in the above embodiments of the present invention, a time series split flag can be added to the row key to indicate the relationship between the data partition at the current moment and the read-write row key range and the pre-split row key position. The time series split flag indicates the correspondence between the row key and the read-write row key range after the data partition is split. At the same time, according to the time series split flag, the time series data within a certain time range is assigned a row key within the current read-write row key range, which facilitates the data partition to specify the pre-split row key position and complete the data partition split without causing data migration problems.

[0112] On this basis, according to the mapping relationship table that records the correspondence between the data partition and the current read-write row key range in the target database, the current read-write row key range of the data partition is determined, and a row key is allocated to the time series data to be written in the current read-write row key range. This can include: querying the current read-write row key range corresponding to the data partition and the time series split flag corresponding to the current read-write row key range according to the mapping relationship table; allocating a row key to the time series data to be written in the current read-write row key range, and writing the time series split flag obtained by the query into the allocated row key, so as to allocate a row key to the time series data to be written in the current read-write row key range.

[0113] Based on the above embodiments, the embodiments of the present invention introduce a method for balancing reading and writing in a time series data storage method using a new data partitioning and splitting method.

[0114] In the time series data storage method provided in the embodiments of the present invention, a balanced storage method is also required to avoid centrally storing large amounts of time series data with similar timings. In this embodiment of the present invention, a balanced storage method for time series data scenarios, such as that currently used in HBase, can be used to divide continuous time series data into different data partitions by "salting" the row keys.

[0115] In some optional implementations of the embodiments of the present invention, in S103, when writing the time series data to be written into the data partition, a row key is allocated to the time series data to be written in the current read-write row key range, which may include: when receiving the write data task corresponding to the time series data to be written, calculating the salt value corresponding to the time series data to be written; determining the data partition corresponding to the salt value as the target data partition for the time series data to be written; allocating a row key to the time series data to be written in the current read-write row key range of the target data partition, and adding the salt value before the allocated row key.

[0116] Salt is a concept used to optimize storage and querying. It refers to a random prefix added to the row key of an HBase table. This is done to disperse data and prevent data skew, that is, to prevent large amounts of data from being concentrated in a few data partitions, thereby improving the system's write and query performance and achieving load balancing. When salting is enabled, each metric and tag combination is assigned to a different "bucket" based on the configuration. Each bucket corresponds to a salt value, which is prepended to the row key.

[0117] In this embodiment of the present invention, a bucket can correspond to one or more data partitions, or a bucket can correspond to all data partitions under a regional server. By adding a salt value to the row key of the time series data to be written, the time series data is allocated to different "buckets", achieving balanced distribution across different data partitions.

[0118] To further achieve balanced distribution of time series data within a single data partition, in some optional implementations of the present invention, assigning row keys to the time series data to be written within the current read / write row key range in S103 may include: randomly selecting unassigned row keys within the current read / write row key range as the row keys for the time series data to be written. By designing a random algorithm, row keys are randomly and non-repeatedly selected within the current read / write row key range of the data partition as the row keys for the time series data to be written, thereby achieving balanced distribution of time series data within a single data partition.

[0119] Based on this, in the time series data storage method provided by an embodiment of the present invention, in S101, when writing the time series data to be written into the data partition, a row key is allocated to the time series data to be written in the current read-write row key range, and it can also include: when receiving the write data task corresponding to the time series data to be written, calculating the salt value corresponding to the time series data to be written; determining the data partition corresponding to the salt value as the target data partition for the time series data to be written; randomly selecting an unassigned row key in the current read-write row key range of the target data partition as the row key for the time series data to be written, and adding the salt value before the assigned row key.

[0120] In some optional implementations of the embodiments of the present invention, calculating the salt value corresponding to the time series data to be written may include: calculating a hash code based on an array consisting of the metric name (metric) of the time series data to be written and the tag key value (tagk, tagv) of the time series data to be written; and performing a modulo calculation on the hash code by the number of buckets corresponding to the target database to obtain the salt value corresponding to the time series data to be written.

[0121] In some optional implementations of the embodiments of the present invention, the step of determining the number of buckets corresponding to the target database may include: determining the number of buckets based on the load status of the cluster where the target database is located and the number of data partition service nodes in the cluster where the target database is located. In actual applications, the default number of buckets for the target database may be 20. In actual business operations, the number of buckets may be adjusted based on the number of data partition service nodes in the cluster or based on the load status of the cluster.

[0122] Based on the above embodiments, an embodiment of the present invention provides another method for storing time series data in practical applications.

[0123] Figure 2 A schematic diagram of data partition splitting provided by an embodiment of the present invention.

[0124] In the embodiment of the present invention, the time series data scenario managed by the open source time series database OpenTSDB plug-in can be referred to, and its native row key composition is as follows:

[0125] [salt]<metric_uid> <timestamp> <tagk1> <tagv1> [... <tagkn> <tagvn>];

[0126] For example:

[0127]

[0128] The salt value (salt) occupies one byte and represents the salt applied to the row key. For example, the number of buckets is 20 (tsd.storage.salt.buckets, which defaults to 20 and is determined in practice based on the number of regional servers). A hash code is calculated based on the metric name (metric_uid) plus the byte array of all [tagK+tagV]uids. The hash code is then modulo the number of buckets to obtain the salt value. A metric is the name of a metric in time series data. Each metric represents a specific data point to be monitored or recorded, such as temperature, system CPU usage, or network traffic. TagK stands for Tag Key, representing the key of a tag, an element used to categorize and identify data points in time series data. TagV stands for Tag Value, representing the tag value. Together with tagK, they form the label in the time series data, providing an additional dimension of information about the metric. TagV is the value portion of tagK, which specifically describes the attributes or classification of the metric.

[0129] Metrics, tagk, and tagv are mapped to unique identifiers (UIDs) through the mapping relationship maintained in the tsdb-uid table to reduce row key length. In OpenTSDB, UIDs are used to uniquely identify each metric, tagK, and tagV to optimize storage and query performance. The mapping is bidirectional: you can find the corresponding UID based on a tagk, and vice versa.

[0130] Time is the time corresponding to the time series data. The actual time of the time series data is formatted as a timestamp in hours, that is, the hour at which the time series data occurred. The specific time (offset) of the time series data within the hour is written as the column name. For example, the 1st second of the hour is the 1st column, and the 3600th second is the 3600th column. When the data time precision is milliseconds, the columns are from the 1st column to the 3600000th column.

[0131] This special row key design uses salting to calculate the hash code of the salt value bit, evenly distributing all time series data read and write requests to all storage nodes in the cluster.

[0132] For example, the offset at the first second of 11 o'clock is 000111. If salting is used to evenly distribute time series data, assuming the cluster has 20 buckets, a random number (the salt value) between 1 and 20 is generated. This number is combined with the managed row key range to randomly distribute the time series data to different data partitions. Without salting, the starting bit of the row key remains the same, meaning that all time series data at 11 o'clock will be concentrated in one data partition, becoming hot data.

[0133] However, the data size supported by a single data partition has an upper limit (the default is 10GB). As time series data is continuously generated, the data in the data partition will inevitably split when the data limit is reached. The single parent data partition is split into two child data partitions, and then the two child data partitions are distributed to the storage nodes where servers in different regions are located according to the specified strategy. This will cause problems such as reduced localization rate and disk I / O storms caused by the relocation of underlying files on different nodes.

[0134] To address this issue, the time series data storage method provided in the embodiments of the present invention improves the original row key design. The improved row key can be expressed as follows:

[0135] [salt][time_split]<metric_uid> <timestamp> <tagk1> <tagv1> [... <tagkn> <tagvn>];

[0136] Among them, time_split is the time series split flag introduced in the above embodiment of the present invention, which indicates the relationship between the data partition and the read-write row key range and the pre-split row key position, and can occupy one byte (8 bits). The time_split bit indicates the one-to-one correspondence between the row key and the read-write row key range after the data partition is split. At the same time, according to this bit, the time series data within a certain time range is written to the specified segment in the data partition, which facilitates the data partition to specify the pre-split row key position to complete the data partition split while ensuring that no data migration problems will occur. It should be noted that for each read-write row key range of the data partition, there is only one current read-write row key range at each moment, so it can be considered that different read-write row key ranges correspond to time series data of different time ranges (time range), so in Figure 2 In the example, the key ranges of each read and write row are tr1, tr2, tr3, tr4, ..., trN.

[0137] Design a mapping relationship table (tsdb-timerange) to record the specific correspondence between the read and write row key ranges and row keys. This mapping relationship table is not originally available in HBase and is a newly added mapping relationship maintenance table. The specific design can be as follows Figure 2 As shown, the left side is a mapping relationship table that records the information of each data partition in the target database. It is expanded to record multiple read-write row key ranges for each data partition according to the pre-split row key position. Assume that at the current moment, the first read-write row key range (read-write row key range 1, tr1) in data partition 2 is the current read-write row key range, storing time series data d1, d2, d3, etc., and the remaining read-write row key ranges do not correspond to time series data. When data partition 2 meets the data partition splitting condition, the data partition split is triggered, and the sub-data partition where the read-write row key range 1 (tr1) is located is data partition 21. The sub-data partitions retained in the original storage node are migrated to other storage nodes with the remaining read-write row key ranges of data partition 2. Figure 2 Data partition 22 in. Since data partition 22 does not correspond to time series data before migration, it will not trigger data relocation or disk I / O storm, and will not cause a decrease in the localization rate of the storage node to which it is migrated. In the new storage node, the first read-write row key range (read-write row key range 2, tr2) of data partition 22 becomes the current read-write row key range of data partition 22 at the current moment, storing time series data d4, d5, d6, etc. The remaining read-write row key ranges do not correspond to time series data. When data partition 22 meets the data partition splitting condition, the data partition splitting is triggered, and the sub-data partition where the read-write row key range 2 (tr1) is located is data partition 221. The sub-data partitions retained in the original storage node are migrated to other storage nodes with the sub-data partitions where the remaining read-write row key ranges of data partition 22 are located. That is, Figure 2 Since data partition 222 does not correspond to time series data before migration, it will not trigger data migration or disk I / O storm, and will not cause the locality rate of the storage node to which it is migrated to to decrease. And so on.

[0138] As described in the above embodiment of the present invention, the number of sub-data partitions that a data partition can be split into can be determined based on the data size of the time series split flag. In the embodiment of the present invention, the time series split flag can be selected to occupy 1 byte, that is, An optional value, indicating that a single data partition can trigger a maximum of 256 split operations.

[0139] In this embodiment of the present invention, the principle that data partition splitting does not result in data relocation is that in read-write row key range 1, the 8-bit time series split flag in the row keys of time series data d1, d2, d3, etc. is all 0, that is, all written to read-write row key range 1 of data partition 2. In actual use, the time series data in read-write row key ranges tr1, tr2, tr3, tr4, ..., trN are all managed by the same data partition. Assuming that the salt bit of data partition 2 is 0, the row key range included in read-write row key range 1 is 00-01, and the row key range included in tr1-trN is 01-0*. When a data partition split occurs in data partition 2, the read and write row key range is *~now, and all the written time series data is in tr1. When the data partition split operation is triggered, the split point of data partition 2 is 01, generating two child data partitions, data partition 21 and data partition 22. Data partition 21 is responsible for all data under tr1 and is retained on the storage node where the original regional server is located. Data partition 22, such as tr2 and tr3, has no data and can be randomly allocated to other regional servers (when the data partition split caused by expansion is allocated to the expanded storage node), thus achieving data partition splitting without causing data relocation and reducing the localization rate.

[0140] In conjunction with the data partition splitting and balancing module 301, mapping management module 302, and read-write conversion module 303 described in the following embodiments of the present invention, the data partition splitting and balancing module 301 is designed to implement fixed-point splitting of data partitions and data partition balancing during expansion in the time series data storage method provided in the embodiments of the present invention. Through a unique row key design and fixed-point splitting scheme, row keys are aligned with read-write row key ranges, avoiding issues such as decreased locality and data relocation during data partition splitting. The mapping management module 302 is used to maintain a mapping relationship between the read-write row key range and the time series split flag (time_split) in the row key through a mapping relationship table (tsdb-timerange) for use by the read-write conversion module 303 and the data partition splitting and balancing module 301. Due to the high frequency of time series data writes, this mapping relationship table can be configured as a resident memory type table to prevent hotspots from blocking business read and write requests when placing requests to this table. The read-write conversion module 303 is responsible for reading or writing time series data. It obtains the value of the corresponding time series split flag through the timestamp of the time series data itself, converts it into the corresponding row key insertion or row key range during query, and completes the time series data read and write business request.

[0141] Therefore, the time series data storage method provided by the embodiment of the present invention designs row keys based on the characteristics of time series data scenarios, aligns row keys with read and write row key ranges, and based on the special design of row keys and the correspondence between data files and time ranges, combined with fixed-point data partition splitting, achieves that after data partition splitting, the data locality rate does not decrease, and data partition balancing does not produce the effect of data relocation. When the time series data of the data partition reaches the upper limit or the cluster is expanded, transparent data partition splitting and cluster load balancing are provided to improve the user experience on the business side.

[0142] It should be noted that in the embodiments of the time series data storage method of the present invention, some of the steps or features may be omitted or not executed. The hardware or software functional modules are divided for the convenience of description and are not the only implementation form of the time series data storage method provided by the embodiments of the present invention.

[0143] The above describes in detail various embodiments corresponding to the time series data storage method. On this basis, the present invention also discloses a time series data storage device, a time series data storage equipment, a non-volatile storage medium and a computer program product corresponding to the above method.

[0144] Figure 3 This is an architectural diagram of a time series data storage device provided by an embodiment of the present invention.

[0145] like Figure 3 As shown, the time series data storage device provided by the embodiment of the present invention includes:

[0146] The data partition splitting and balancing module 301 is used to obtain the row key range corresponding to each data partition of the target database, determine the pre-split row key position of the data partition within the row key range of the data partition; determine the current read-write row key range in the data partition using the pre-split row key position as the boundary; when the data partition splitting condition is met, split the data partition to be split into multiple sub-data partitions based on the pre-split row key position of the data partition to be split, so that the sub-data partitions including the current read-write row key range are retained in the original storage node, and the sub-data partitions in the row key range not including the current read-write row key range are determined to be migrated to other storage nodes;

[0147] A mapping management module 302 is used to manage a mapping relationship table that records the correspondence between data partitions and the current read / write row key range;

[0148] The read-write conversion module 303 is used to query the mapping relationship table through the mapping management module to determine the current read-write row key range of the data partition when writing the time series data to be written into the data partition, and allocate a row key to the time series data to be written within the current read-write row key range.

[0149] like Figure 3 As shown, in this embodiment of the present invention, the data partition splitting and balancing module 301 is used for fixed-point splitting of data partitions and data partition balancing during expansion. Through a special row key design and fixed-point splitting scheme, row keys are aligned with the read and write row key ranges, avoiding problems such as reduced locality and data relocation when data partition splitting occurs.

[0150] The mapping management module 302 is used to maintain the mapping relationship table between read and write row key ranges, row keys and data partitions. Each read and write row key range can contain a certain row key range and a certain number of data partitions, and manage the data writing and query within the range.

[0151] The read-write conversion module 303 processes read and write requests according to the mapping table and sends the requests to the corresponding data partition. The read-write conversion module 303 can convert the write and query conditions in write and query requests, and use the mapping management module 302 to maintain the correspondence between the read and write row key ranges and row keys to load balance the read and write requests across all storage nodes in the cluster.

[0152] like Figure 3 In the figure, RS1, RS2, RS3, ..., and RSM are storage nodes in the cluster where the target database resides, with RSM being an example of an expanded storage node. Assume that initially, the target database contains nine data partitions: Data Partition 1, Data Partition 2, Data Partition 3, Data Partition 4, Data Partition 5, Data Partition 6, Data Partition 7, Data Partition 8, and Data Partition 9. When cluster expansion triggers a data partition split, assuming that the expanded storage node RSM serves as the expansion storage node for storage node 1 (RS1), data partitions 1, 2, and 3 on storage node 1 are split. Data partition 1 is split into data partition 11 and data partition 12, data partition 2 is split into data partition 21 and data partition 22, and data partition 3 is split into data partition 31 and data partition 32. The resulting child data partitions (data partition 11, data partition 21, and data partition 31) remain on storage node 1 (RS1), while the resulting child data partitions (data partition 12, data partition 22, and data partition 32) are migrated to the expanded storage node RSM. By applying the time series data storage solution provided in the embodiment of the present invention, it is ensured that the sub-data partition remaining in storage node 1 (RS1) is a sub-data partition that includes the current read-write row key range, and the sub-data partition migrated to the expanded storage node RS M is a sub-data partition that does not include the current read-write row key range.

[0153] It should be noted that in each implementation of the time series data storage device provided by the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be used. The connection method between different units can adopt electrical, mechanical or other connection methods. Separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, according to actual needs, some or all of the units provided in the embodiment of the present invention can be selected and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0154] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, and they will not be repeated here.

[0155] The time series data storage device provided by the embodiment of the present invention determines the pre-split row key position of the data partition of the target database according to the row key range in advance, and determines the current read-write row key range in the data partition with the pre-split row key position as the boundary, so that when the time series data to be written is written into the data partition, the row key is allocated to the time series data to be written in the current read-write row key range; and when the data partition splitting condition is met, the data partition to be split is split into multiple sub-data partitions according to the pre-split row key position of the data partition to be split, so that the sub-data partition including the current read-write row key range is retained in the original storage node, and the sub-data partition not including the row key range of the current read-write row key range is retained in the original storage node. The scope determines the sub-data partitions to be migrated to other storage nodes, so that the sub-data partitions that need to be migrated to other storage nodes do not correspond to time series data, and will not cause the problem of reduced locality rate of other storage nodes to which they are migrated. Therefore, the splitting of data partitions only brings about the migration of metadata but not the migration of time series data. As a result, there will be no disk data input and output storms and response blockages to read and write requests caused by the migration of a large amount of time series data, nor will the query performance of the time series data under the storage node be reduced due to the low locality rate of the storage node. This realizes transparent data partition splitting and cluster expansion, and improves the user experience on the business side.

[0156] Figure 4 A schematic structural diagram of a time series data storage device provided by an embodiment of the present invention.

[0157] like Figure 4 As shown, the time series data storage device provided by an embodiment of the present invention includes: a memory 410, used to store a computer program 411; a processor 420, used to execute the computer program 411, and when the computer program 411 is executed by the processor 420, the steps of the time series data storage method provided in any of the above embodiments are implemented.

[0158] Processor 420 may include one or more processing cores, such as a 3-core processor or an octa-core processor. Processor 420 may be implemented in at least one of the following hardware forms: digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). Processor 420 may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 420 may integrate a graphics processing unit (GPU), which is responsible for rendering and drawing content required to be displayed on the display screen. In some embodiments, processor 420 may also include an artificial intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0159] The memory 410 may include one or more non-volatile storage media, which may be non-transitory. The memory 410 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 410 is at least used to store the following computer program 411, wherein, after the computer program 411 is loaded and executed by the processor 420, it can implement the relevant steps in the time series data storage method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 410 may also include an operating system 412 and data 413, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 412 may be Windows or other types of operating systems. The data 413 may include but is not limited to the data involved in the above method.

[0160] In some embodiments, the time series data storage device may further include a display screen 430 , a power supply 440 , a communication interface 450 , an input / output interface 460 , a sensor 470 , and a communication bus 480 .

[0161] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation to the time-series data storage device, and may include more or fewer components than shown in the figure.

[0162] The time series data storage device provided in an embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the time series data storage method provided in the above embodiment, and the effect is the same as above.

[0163] An embodiment of the present invention provides a non-volatile storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the time series data storage method provided in any one of the above embodiments can be implemented.

[0164] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program code.

[0165] For an introduction to the non-volatile storage medium provided in an embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the time series data storage method provided in an embodiment of the present invention, and the present invention will not elaborate on it here.

[0166] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the time series data storage method provided in any one of the above embodiments.

[0167] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the time series data storage method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0168] The above is a detailed introduction to the time series data storage method, device, equipment, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0169] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprise," "include," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.< / tagvn> < / tagkn> < / tagv1> < / tagk1> < / timestamp> < / tagvn> < / tagkn> < / tagv1> < / tagk1> < / timestamp>

Claims

1. A method for storing time series data. It is characterized in that include: Obtaining a row key range corresponding to each data partition of the target database, and determining a pre-split row key position of the data partition within the row key range of the data partition; Determine the current read / write row key range in the data partition based on the pre-split row key position; When writing the time series data to be written into the data partition, a row key is allocated to the time series data to be written within the current read-write row key range; When the data partition splitting condition is met, the data partition to be split is split into multiple sub-data partitions according to the pre-split row key position of the data partition to be split, so that the sub-data partition including the current read-write row key range is retained in the original storage node, and the sub-data partition to be migrated to other storage nodes is determined in the row key range that does not include the current read-write row key range; The pre-split row key position is a predetermined row key position for splitting when the data partition is split due to the time series data capacity reaching an upper limit or cluster expansion triggering data partition splitting; Taking the pre-split row key position as a boundary, determining the current read-write row key range in the data partition includes: segmenting the row key range corresponding to the data partition according to the pre-split row key position, and selecting one of them as the current read-write row key range.

2. The time series data storage method according to claim 1, It is characterized in that Determining a pre-split row key position of the data partition within a row key range of the data partition includes: Determine an upper limit on the number of the sub-data partitions that the data partition can be split into according to the size of the row key range of the data partition and the minimum row key splitting granularity of the data partition; Determine the number of the sub-data partitions for actual pre-splitting of the data partition within a range not exceeding an upper limit of the number of the sub-data partitions; The pre-split row key position is determined within the row key range of the data partition according to the row key range of the data partition and the number of the sub-data partitions actually pre-split of the data partition.

3. The time series data storage method according to claim 2, It is characterized in that Determining an upper limit on the number of the sub-data partitions that the data partition can be split into according to the size of the row key range of the data partition and the minimum row key splitting granularity of the data partition includes: Determine the number of optional values ​​corresponding to the timing split flag bit according to the data size of the timing split flag bit configured in the row key of the data partition; The number of optional numerical values ​​corresponding to the timing split flag is the upper limit of the number of sub-data partitions that the data partition can be split into.

4. The time series data storage method according to claim 1, It is characterized in that Determining a pre-split row key position of the data partition within a row key range of the data partition includes: Determine the number of the sub-data partitions of the actual pre-splitting of the data partition within a range that is not less than the preset number of data partitions corresponding to the target database; The pre-split row key position is determined within the row key range of the data partition according to the row key range of the data partition and the number of the sub-data partitions actually pre-split of the data partition.

5. The time series data storage method according to claim 1, It is characterized in that Determining a pre-split row key position of the data partition within a row key range of the data partition includes: Determine an upper limit on the number of the sub-data partitions that the data partition can be split into according to the size of the row key range of the data partition and the minimum row key splitting granularity of the data partition; Determine the number of the sub-data partitions of the actual pre-splitting of the data partition within a range that does not exceed the upper limit of the number of the sub-data partitions and is not less than the preset number of data partitions corresponding to the target database; The pre-split row key position is determined within the row key range of the data partition according to the row key range of the data partition and the number of the sub-data partitions actually pre-split of the data partition.

6. The time series data storage method according to claim 1, It is characterized in that Determining the current read / write row key range in the data partition with the pre-split row key position as the boundary includes: The row key range before the first pre-split row key position of the data partition is taken as the current read-write row key range.

7. The time series data storage method according to claim 1, It is characterized in that Determining the sub-data partition to be migrated to another storage node in a row key range that does not include the current read-write row key range includes: The row key ranges that do not include the current read-write row key range are all used as the sub-data partitions migrated to other storage nodes.

8. The time series data storage method according to claim 1, It is characterized in that When writing the time series data to be written into the data partition, allocating a row key for the time series data to be written in the current read-write row key range includes: When writing the time series data to be written into the data partition, the current read-write row key range of the data partition is determined according to a mapping relationship table recording the correspondence between the data partition in the target database and the current read-write row key range, and a row key is allocated to the time series data to be written within the current read-write row key range.

9. The time series data storage method according to claim 8, It is characterized in that The mapping relationship table includes a mapping relationship between a row key range written into each time series data in the data partition and a time series split flag bit; The time series data write row key range is the interval into which the data partition is divided according to the pre-split row key position, and the current read-write row key range is one of the time series data write row key ranges; the time series split flag is a newly added flag of the row key managed by the data partition, and the time series split flag of the row key in the data partition uniquely corresponds to the time series data write row key range in which it is located.

10. The time series data storage method according to claim 9, It is characterized in that Determining the current read / write row key range of the data partition according to a mapping relationship table recording the corresponding relationship between the data partition in the target database and the current read / write row key range, and allocating a row key for the time series data to be written in the current read / write row key range, including: Query the current read / write row key range corresponding to the data partition and the timing split flag corresponding to the current read / write row key range according to the mapping relationship table; A row key is allocated to the time series data to be written in the current read-write row key range, and the time series split flag obtained by query is written into the allocated row key, so as to allocate a row key to the time series data to be written in the current read-write row key range.

11. The time series data storage method according to claim 1, It is characterized in that When writing the time series data to be written into the data partition, allocating a row key for the time series data to be written in the current read-write row key range includes: When receiving the data writing task corresponding to the time series data to be written, calculating and obtaining the salt value corresponding to the time series data to be written; Determine that the data partition corresponding to the salt value is the target data partition for writing the time series data; A row key is allocated to the time series data to be written in the current read-write row key range of the target data partition, and the salt value is added before the allocated row key.

12. The time series data storage method according to claim 1, It is characterized in that Allocating a row key for the time series data to be written in the current read-write row key range includes: An unassigned row key is randomly selected in the current read-write row key range as the row key of the time series data to be written.

13. The time series data storage method according to claim 1, It is characterized in that When writing the time series data to be written into the data partition, allocating a row key for the time series data to be written in the current read-write row key range includes: When receiving the data writing task corresponding to the time series data to be written, calculating and obtaining the salt value corresponding to the time series data to be written; Determine that the data partition corresponding to the salt value is the target data partition for writing the time series data; An unassigned row key is randomly selected in the current read-write row key range of the target data partition as the row key of the time series data to be written, and the salt value is added before the assigned row key.

14. The time series data storage method according to claim 11 or 13, It is characterized in that Calculating the salt value corresponding to the time series data to be written includes: Calculate a hash code according to an array consisting of the indicator name of the time series data to be written and the tag key value of the time series data to be written; The hash code is modulo-calculated by the number of buckets corresponding to the target database to obtain the salt value corresponding to the time series data to be written.

15. The time series data storage method according to claim 14, It is characterized in that The step of determining the number of buckets corresponding to the target database includes: The number of buckets is determined according to the load status of the cluster where the target database is located and the number of data partition service nodes in the cluster where the target database is located.

16. The time series data storage method according to claim 1, It is characterized in that The data partition splitting condition includes at least one of: the amount of time series data corresponding to the data partition reaches the maximum data capacity corresponding to the data partition, and the cluster expansion of the target database triggers the data partition splitting.

17. A time series data storage device, It is characterized in that include: A data partition splitting and balancing module is used to obtain the row key range corresponding to each data partition of the target database, determine the pre-split row key position of the data partition within the row key range of the data partition; determine the current read-write row key range in the data partition with the pre-split row key position as the boundary; when the data partition splitting condition is met, split the data partition to be split into multiple sub-data partitions according to the pre-split row key position of the data partition to be split, so that the sub-data partition including the current read-write row key range is retained in the original storage node, and the sub-data partition to be migrated to other storage nodes is determined in the row key range that does not include the current read-write row key range; A mapping management module, used to manage a mapping relationship table recording the corresponding relationship between the data partition and the current read / write row key range; a read-write conversion module, configured to query the mapping relationship table through the mapping management module to determine the current read-write row key range of the data partition when writing the time series data to be written into the data partition, and to allocate a row key to the time series data to be written within the current read-write row key range; The pre-split row key position is a predetermined row key position for splitting when the data partition is split due to the time series data capacity reaching an upper limit or cluster expansion triggering data partition splitting; Taking the pre-split row key position as a boundary, determining the current read-write row key range in the data partition includes: segmenting the row key range corresponding to the data partition according to the pre-split row key position, and selecting one of them as the current read-write row key range.

18. A time series data storage device, It is characterized in that include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the time series data storage method as described in any one of claims 1 to 16 are implemented.

19. A non-volatile storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the time series data storage method according to any one of claims 1 to 16 are implemented.

20. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the time series data storage method according to any one of claims 1 to 16 are implemented.

Images