Data management method, device, equipment and storage medium for time series database cluster

Abstract

The present invention relates to the technical field of data processing, and discloses a data management method for a time series database cluster, comprising: obtaining old sharding rules of a historical cluster to be processed, and new sharding rules of a new cluster after updating the historical cluster; obtaining a to-be-migrated list corresponding to data entities of the new cluster based on the new sharding rules and the old sharding rules; the to-be-migrated list includes data entities and old inflow nodes and new inflow nodes corresponding to the data entities, and the old inflow nodes and the new inflow nodes are used to indicate the data flow direction of the data entities; based on a preset migration strategy, the data entities in the to-be-migrated list are migrated from the old inflow nodes to the new inflow nodes; the data entities corresponding to the old inflow nodes in the to-be-migrated list are deleted, and the state information of the new cluster is modified accordingly. The present invention can improve the flexibility of data management of a time series database cluster.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, electronic device and computer-readable storage medium for data management of a time series database cluster. Background Art

[0002] Existing time series databases can also be called time series databases, which are mainly used to process data with time tags (changing in time order, i.e. time serialization). Time series data are mainly collected and generated by various types of real-time monitoring, inspection and analysis equipment in the power industry, chemical industry, meteorological industry, geographic information, etc. The typical characteristics of these industrial data are: fast generation frequency (each monitoring point can generate multiple data within one second), heavy dependence on collection time (each data is required to correspond to a unique time), large number of measuring points and large amount of information (conventional real-time monitoring systems have thousands of monitoring points, which generate data every second and generate tens of GB of data every day).

[0003] At present, time series databases that process time series data have the characteristics of high performance, data compression, automatic cleanup of expired data, and support for multiple aggregation functions. Time series databases are gradually becoming mainstream in the fields of monitoring and the Internet of Things. However, in large-scale application scenarios, clusters composed of multiple time series database components still need to consider the issue of capacity expansion. For example, the influxdb database officially provides a cluster version and a stand-alone version. The cluster version is expensive and not open source; although the stand-alone version is free and open source, it only supports one influxdb instance.

[0004] In addition, the existing cluster invention based on the stand-alone version of InfluxDB can realize the routing of read and write requests through sharding rules, but it cannot take into account online, historical data migration, old data cleaning, seamless switching of sharding rules, progress management and other issues in terms of cluster expansion. It also requires more manual operations, resulting in poor flexibility and limited scope of application, which affects the user experience. Summary of the invention

[0005] The present invention provides a data management method, device, electronic device and computer-readable storage medium for a time series database cluster, the main purpose of which is to improve the flexibility and efficiency of data management of a time series database cluster.

[0006] To achieve the above object, the present invention provides a data management method for a time series database cluster, comprising:

[0007] Obtaining an old sharding rule of a historical cluster to be processed and a new sharding rule of a new cluster after updating the historical cluster;

[0008] Based on the new sharding rule and the old sharding rule, a to-be-migrated list corresponding to the data entity of the new cluster is obtained; the to-be-migrated list includes the data entity and the old inflow node and the new inflow node corresponding to the data entity, the old inflow node and the new inflow node are used to indicate the data flow direction of the data entity;

[0009] Based on a preset migration strategy, migrate the data entity in the to-be-migrated list from the old inflow node to the new inflow node;

[0010] The data entity corresponding to the old incoming node in the to-be-migrated list is deleted, and the state information of the new cluster is modified accordingly.

[0011] In addition, an optional technical invention is that, based on the new sharding rule and the old sharding rule, obtaining a list to be migrated corresponding to the data entity of the new cluster includes:

[0012] Marking the state of the new cluster as ready;

[0013] Traverse the new cluster in the prepared state based on the new sharding rule and the old sharding rule, and query the data entity of the new cluster;

[0014] Based on the data entity, a to-be-migrated list corresponding to the data entity is obtained.

[0015] In addition, an optional technical invention is that traversing the new cluster in the prepared state based on the new sharding rule and the old sharding rule and querying the data entity of the new cluster includes:

[0016] Based on the processing type of the new cluster, the new sharding rule and the old sharding rule, respectively obtain query nodes of the new cluster;

[0017] Send a query request to the query node to obtain all data entities of the new cluster.

[0018] In addition, an optional technical invention is that the preset migration strategy includes a time strategy and an entity strategy; wherein,

[0019] Migrating the data entity in the to-be-migrated list from the old inflow node to the new inflow node based on the time strategy includes:

[0020] Based on a preset configuration file, obtain the data retention time, migration interval and concurrency number of the data entity;

[0021] When the end time of the migration list, the migration interval, and the data retention time meet the preset conditions, the list to be migrated is traversed and a migration request is sent to the old inflow node of the data entity in the list to be migrated according to the concurrency number;

[0022] Acquire data entities within a time range corresponding to the preset condition based on the migration request, and send the data entities to the corresponding new inflow node;

[0023] Determine whether the end time, the migration interval, and the data retention time meet preset conditions, and repeat the above steps.

[0024] In addition, an optional technical invention is that migrating the data entity in the to-be-migrated list from the old inflow node to the new inflow node based on the entity policy includes:

[0025] Based on a preset configuration file, obtain the data retention time, migration interval and concurrency number of the data entity;

[0026] Traversing the list to be migrated, and acquiring data entities in the list to be migrated according to the concurrency number;

[0027] Sending a migration request to the old inflow node corresponding to the data entity in the to-be-migrated list within a preset time range;

[0028] Acquire the data entity within the preset time range based on the migration request, and send it to the corresponding new inflow node;

[0029] The unique identifier of the migrated data entity is used as progress and recorded in the database;

[0030] The above steps are repeated until all data entities in the to-be-migrated list are migrated.

[0031] In addition, an optional technical invention is that the deleting of the data entity corresponding to the old inflow node in the to-be-migrated list and correspondingly modifying the state information of the new cluster includes:

[0032] Mark the status of the new cluster as being cleaned up;

[0033] Traversing the to-be-migrated list, and deleting the data entity in the to-be-migrated list corresponding to the old inflow node;

[0034] The state of the new cluster is marked as idle.

[0035] In order to solve the above problem, the present invention further provides a data management device for a time series database cluster, the device comprising:

[0036] A new sharding rule acquisition unit, used to acquire old sharding rules of a historical cluster to be processed, and new sharding rules of a new cluster after updating the historical cluster;

[0037] a to-be-migrated list acquisition unit, configured to acquire a to-be-migrated list corresponding to a data entity of the new cluster based on the new sharding rule and the old sharding rule; the to-be-migrated list includes the data entity and an old inflow node and a new inflow node corresponding to the data entity, the old inflow node and the new inflow node being used to indicate a data flow direction of the data entity;

[0038] A data migration unit, configured to migrate the data entities in the to-be-migrated list from the old inflow node to the new inflow node based on a preset migration strategy;

[0039] The state modification unit is used to delete the data entity corresponding to the old inflow node in the list to be migrated, and modify the state information of the new cluster accordingly.

[0040] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0041] a memory storing at least one instruction; and

[0042] The processor executes the instructions stored in the memory to implement the data management method for the time series database cluster described above.

[0043] In order to solve the above problem, the present invention also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the data management method of the time series database cluster described above.

[0044] The embodiment of the present invention obtains the old sharding rules and the new sharding rules of the historical cluster to be processed; based on the new sharding rules and the old sharding rules, obtains the to-be-migrated list corresponding to the data entities of the new cluster, and then migrates the data entities in the to-be-migrated list from the old inflow node to the new inflow node based on the preset migration strategy; finally, deletes the data entities in the to-be-migrated list corresponding to the old inflow node, and modifies the status information of the new cluster accordingly. When the cluster adds a new node set or reduces a node set, it can achieve online expansion or reduction. Without affecting the use of the cluster, it realizes functions such as data migration and cleaning, seamless switching of sharding rules, and progress management, which greatly reduces manual operations and improves the flexibility and efficiency of data management of the cluster database. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1A schematic diagram of a flow chart of a data management method for a time series database cluster provided by an embodiment of the present invention;

[0046] Figure 2 A logical block diagram of a data management method for a time series database cluster provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of a module of a data management device for a time series database cluster provided by an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of the internal structure of an electronic device for implementing a data management method for a time series database cluster provided by an embodiment of the present invention;

[0049] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0051] The present invention provides a data management method for a time series database cluster. Specifically, refer to Figure 1 FIG. 1 is a flow chart of a data management method for a time series database cluster provided by an embodiment of the present invention. The method can be executed by a device, and the device can be implemented by software and / or hardware.

[0052] In this embodiment, the data management method of the time series database cluster includes:

[0053] S100: Acquire the old sharding rule of the historical cluster to be processed and the new sharding rule of the new cluster after updating the historical cluster.

[0054] Among them, the historical cluster mainly refers to the original database cluster. The update of the historical cluster mainly refers to the change of the database in the historical cluster, that is, the historical cluster has expanded, shrunk or the sharding rules have changed. When the historical cluster is updated, the sharding rules of the corresponding data will also change, and then the old sharding rules and new sharding rules will appear.

[0055] Specifically, the sharding rule mainly refers to distributing the data stored in the database to multiple databases through certain specific conditions, so as to achieve the effect of distributing the device load. Among them, the sharding rule mainly includes two modes, one is to split according to different tables, which can also be called vertical data segmentation, and the other is to split the data in the same table into multiple databases according to certain conditions based on the logical relationship of the data in the table, which can also be called horizontal data segmentation.

[0056] In other words, the old sharding rules mainly refer to the data processing rules in the database. The new sharding rules will have a certain impact on data processing when the number of databases in the cluster increases or decreases. The data processing rules at this time can be understood as the new sharding rules.

[0057] S200: Based on the new sharding rule and the old sharding rule, obtain a list to be migrated corresponding to the data entity of the new cluster; the list to be migrated includes the data entity and the old inflow node and the new inflow node corresponding to the data entity, and the old inflow node and the new inflow node are used to indicate the data flow direction of the data entity.

[0058] When obtaining the list to be migrated, the inflow nodes can be calculated according to the new sharding rules and the old sharding rules respectively, and the new inflow nodes and the old inflow nodes can be obtained. If the two are the same, they are ignored. Otherwise, the data entity, the new inflow node and the old inflow node are combined into a migration entity and put into the list to be migrated. After all the data entities are calculated, the data retention time period of the database can be queried, and the end time attribute of the list to be migrated is set to the current time, and the start time attribute is the difference between the end time and the data retention time period. The number of concurrent and migration time, etc. are obtained from the preset configuration file for subsequent data migration.

[0059] Specifically, the step S200 may further include:

[0060] S210: Mark the state of the new cluster as a ready state;

[0061] S220: traverse the new cluster in the prepared state based on the new sharding rule and the old sharding rule, and query the data entity of the new cluster;

[0062] S230: Based on the data entity, obtain a to-be-migrated list corresponding to the data entity.

[0063] In step S200, the new sharding rule can be obtained through the sharding rule engine, and the sharding rule engine can calculate the node corresponding to the entity according to the entity characteristics corresponding to the data, and the node can be understood as the new inflow node of the data corresponding to the new sharding rule formed based on the sharding rule engine. In the present invention, the sharding rule engine does not limit the specific algorithm, for example, a consistent hashing algorithm can be used.

[0064] In addition, the list to be migrated is composed of multiple migration entities that need to perform data migration. The determination of the list to be migrated is the determination of the data entities that need to be migrated (i.e., migration entities). The migration entities include data entities, old inflow nodes, and new inflow nodes.

[0065] Wherein, in the above step S220, the step of querying all data entities of the new cluster further includes:

[0066] S221: Based on the processing type of the new cluster, the new sharding rule, and the old sharding rule, respectively obtain query nodes of the new cluster;

[0067] Among them, the processing type may include a read request or a write request. When the processing type is a write request, the inflow node of the new cluster is determined respectively according to the new sharding rule and the old sharding rule. If the new inflow node of the new sharding rule is different from the old inflow node of the old sharding rule, it is determined that the new inflow node and the old inflow node are both query nodes, and write requests are sent to the new inflow node and the old inflow node at the same time; when the processing type is a read request, the corresponding query node is determined directly according to the old sharding rule.

[0068] S222: Send a query request to the query node to obtain all data entities of the new cluster.

[0069] Among them, all databases in the new cluster can be traversed, and corresponding query requests can be sent to query nodes of all databases in the new cluster, and corresponding data entities can be obtained based on the query requests, and the data entity is a unique data entity.

[0070] Specifically, assuming that there were originally 5 databases in the historical cluster to be processed, after the expansion operation, the number of databases in the new cluster may increase to 10. For example, if the old sharding rule corresponding to the old cluster is to read data from database 1, then the new sharding rule of the expanded new cluster may be to read data from database 6. At this time, due to the change in the sharding rule, the data in the original 5 databases need to be redivided into 10 databases according to the new sharding rule.

[0071] In the above process, it is necessary to compare and judge the inflow node corresponding to the new sharding rule (new inflow node) and the inflow node corresponding to the old sharding rule (old inflow node). If they are the same, it means that the data does not need to be migrated and the existing storage location can be maintained. Otherwise, it needs to be migrated and the corresponding write request is sent to the new inflow node and the old inflow node respectively. For read requests, there is no need to consider the new sharding rule. The old inflow node can be directly determined as the query node according to the old sharding rule.

[0072] As a specific example, in the database, data will have one or more features indicating which entity the data belongs to. For example, the indicator of disk B of server A is the reading speed data, and the corresponding entity has three features: server = A, disk = B, indicator = reading speed. The above-mentioned calculation query node (inflow node) is to perform a certain shard calculation on all the characteristics of the entity to obtain which node the data corresponding to the entity should be written to or which node should be read from. Finally, based on the determined query node, the unique data entity that needs to be migrated is obtained, and a list to be migrated is formed.

[0073] S300: Based on a preset migration strategy, migrate the data entities in the to-be-migrated list from the old inflow node to the new inflow node.

[0074] Among them, the preset migration strategies mainly include time strategy and entity strategy. During the application process, these two strategies can be flexibly selected according to the scenario or requirements.

[0075] For the first strategy, the step of migrating the data entities in the to-be-migrated list from the previous inflow stage to the new inflow node based on the time strategy includes:

[0076] S310: Based on a preset configuration file, obtain the data retention time, migration interval and concurrency number of the data entity;

[0077] S320: When the end time of the migration list, the migration interval, and the data retention time meet the preset conditions, traverse the list to be migrated and send a migration request to the old inflow node of the data entity in the list to be migrated according to the concurrency number;

[0078] S330: acquiring data entities within a time range corresponding to the preset condition based on the migration request, and sending the data entities to the corresponding new inflow node;

[0079] S340: Determine whether the end time, the migration interval, and the data retention time meet preset conditions, and repeat the above steps S320 to S340.

[0080] Specifically, let R=end time, L=R-migration interval, and the start time be set to the difference between the end time and the data retention time. The preset condition mainly means that if L is less than the start time, then L=start time, and traverse the list to be migrated to send the corresponding migration request; when step S330 is executed, the L time is used as a progress and recorded in a database, and then step S340 is further set to R=L, L=L-migration time, and the above steps are repeated until R is less than or equal to the end time, indicating that the data entity migration is completed.

[0081] For the second strategy, the step of migrating the data entities in the to-be-migrated list from the previous inflow stage to the new inflow node based on the entity strategy includes:

[0082] S350: Based on a preset configuration file, obtain the data retention time, migration interval and concurrency number of the data entity;

[0083] S360: traverse the list to be migrated, and acquire data entities in the list to be migrated according to the concurrency number;

[0084] S370: Sending a migration request to the old inflow node corresponding to the data entity in the to-be-migrated list within a preset time range;

[0085] S380: Acquire data entities within the preset time range based on the migration request, and send them to the corresponding new inflow node;

[0086] S390: Using the unique identifier of the migrated data entity as progress and recording it in a database;

[0087] S340: Repeat steps S350 to S390 until all data entities in the to-be-migrated list are migrated.

[0088] Among them, the preset time range can be determined according to L and R in the time strategy. The preset time range mainly refers to the time range [L, R), where R = end time (end time when the migration list is completed), and L = R-migration interval.

[0089] As a specific example, assuming that the migration interval is set to 1 day and the data retention time is set to 30 days, then each time the data entity's data volume for one day is migrated, read from the old inflow node, and written to the new inflow node, 30 days of data can be migrated at one time, indicating that the data of this entity has been completed. If there are multiple data entities in the list to be migrated, assuming there are 10 entities, if the concurrency is set to 3, then data migration will be performed on 3 data entities each time, and each data entity will be migrated according to the time period and the data volume for one day, until all 10 data entities in the list have completed the migration, indicating that all migration work is completed.

[0090] S400: Deleting the data entity corresponding to the old incoming node in the list to be migrated, and modifying the state information of the new cluster accordingly.

[0091] Specifically, after all data entities in the list to be migrated are migrated, it is necessary to further delete the data entities of the old inflow nodes in the migration list. This step may further include:

[0092] S410: Mark the status of the new cluster as being cleaned up;

[0093] S420: traverse the list to be migrated, and delete the data entity in the list to be migrated corresponding to the old inflow node;

[0094] S430: Mark the state of the new cluster as idle.

[0095] In a specific embodiment of the present invention, when a data entity is unexpectedly interrupted during the migration process, or the management node interrupts the migration operation for some reason, it can be restored and the new cluster state before the interruption can be checked during the recovery, and then the corresponding operation can be continued according to the new cluster state before the abnormal interruption.

[0096] Specifically, if the state before the abnormal interruption is in preparation, data management can be restarted from the preparation stage after the abnormal interruption is restored; data management may include data management in scenarios such as cluster expansion, reduction, sharding rule adjustment, and sharding algorithm change;

[0097] If the status before the abnormal interruption is data migration in progress, obtain the migration progress and continue the migration of data entities;

[0098] If the status before the abnormal interruption is cleaning, the data cleaning operation continues.

[0099] Figure 2 A schematic block diagram of a data management method for a time series database cluster according to an embodiment of the present invention is shown.

[0100] As a specific example, Figure 2 As shown in the figure, the client can calculate the time series database node to which the request needs to be routed through routing, and read and write the time series database through the database connection pool. The database connection pool is mainly used to manage the connection of each time series database node; the sharding rule is mainly based on certain algorithms and rules to convert the input parameters into the identifier of the managed time series database. The expansion module is the main module for performing expansion operations, including functions such as adjusting sharding rules, progress management, recovery, and cluster state transfer; of course, in addition to expansion, it can also be applied to scenarios such as shrinking, adjusting sharding rules, and changing sharding algorithms.

[0101] According to the data management method based on the time series database cluster of the present invention, it is possible to achieve online expansion or reduction in capacity when the cluster adds a new node set or reduces a node set. Without affecting the use of the cluster, it realizes functions such as data migration and cleaning, seamless switching of sharding rules, and progress management, which greatly reduces manual operations and improves the flexibility and efficiency of data management of the cluster database.

[0102] like Figure 3, which is a functional module diagram of the data management device of the time series database cluster of the present invention.

[0103] The data management device 100 of the time series database cluster of the present invention can be installed in an electronic device. According to the functions to be implemented, the data management device of the time series database cluster may include a new sharding rule acquisition unit 101, a to-be-migrated list acquisition unit 102, a data migration unit 103 and a state modification unit 104. The unit described in the present invention can also be called a module, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0104] In this embodiment, the functions of each module / unit are as follows:

[0105] The new sharding rule acquisition unit 101 is used to acquire the old sharding rule of the historical cluster to be processed and the new sharding rule of the new cluster after updating the historical cluster.

[0106] Among them, the historical cluster mainly refers to the original database cluster. The update of the historical cluster mainly refers to the change of the database in the historical cluster, that is, the historical cluster has expanded, shrunk or the sharding rules have changed. When the historical cluster is updated, the sharding rules of the corresponding data will also change, and then the old sharding rules and new sharding rules will appear.

[0107] Specifically, the sharding rule mainly refers to distributing the data stored in the database to multiple databases through certain specific conditions, so as to achieve the effect of distributing the device load. Among them, the sharding rule mainly includes two modes, one is to split according to different tables, which can also be called vertical data segmentation, and the other is to split the data in the same table into multiple databases according to certain conditions based on the logical relationship of the data in the table, which can also be called horizontal data segmentation.

[0108] In other words, the old sharding rules mainly refer to the data processing rules in the database. The new sharding rules will have a certain impact on data processing when the number of databases in the cluster increases or decreases. The data processing rules at this time can be understood as the new sharding rules.

[0109] The to-be-migrated list acquisition unit 102 is used to acquire a to-be-migrated list corresponding to the data entity of the new cluster based on the new sharding rule and the old sharding rule; the to-be-migrated list includes the data entity and the old inflow node and the new inflow node corresponding to the data entity, and the old inflow node and the new inflow node are used to indicate the data flow direction of the data entity.

[0110] When obtaining the list to be migrated, the inflow nodes can be calculated according to the new sharding rules and the old sharding rules respectively, and the new inflow nodes and the old inflow nodes can be obtained. If the two are the same, they are ignored. Otherwise, the data entity, the new inflow node and the old inflow node are combined into a migration entity and put into the list to be migrated. After all the data entities are calculated, the data retention time period of the database can be queried, and the end time attribute of the list to be migrated is set to the current time, and the start time attribute is the difference between the end time and the data retention time period. The number of concurrent and migration time, etc. are obtained from the preset configuration file for subsequent data migration.

[0111] Specifically, the to-be-migrated list obtaining unit 102 may further include:

[0112] A state marking module 1021 is used to mark the state of the new cluster as a ready state;

[0113] A data entity query module 2022, configured to traverse the new cluster in the prepared state based on the new sharding rule and the old sharding rule, and query the data entity of the new cluster;

[0114] The to-be-migrated list acquisition module 2023 is configured to acquire, based on the data entity, a to-be-migrated list corresponding to the data entity.

[0115] In this unit, the new sharding rule can be obtained through the sharding rule engine, and the sharding rule engine can calculate the node corresponding to the entity according to the entity characteristics corresponding to the data, and the node can be understood as the new inflow node of the data corresponding to the new sharding rule formed based on the sharding rule engine. In the present invention, the sharding rule engine does not limit the specific algorithm, for example, a consistent hashing algorithm can be used.

[0116] In addition, the list to be migrated is composed of multiple migration entities that need to perform data migration. The determination of the list to be migrated is the determination of the data entities that need to be migrated (i.e., migration entities). The migration entities include data entities, old inflow nodes, and new inflow nodes.

[0117] The step of querying all data entities of the new cluster in the data entity query module 2022 further includes:

[0118] A query node acquisition module, used to respectively acquire query nodes of the new cluster based on the processing type of the new cluster, the new sharding rule and the old sharding rule;

[0119] Among them, the processing type may include a read request or a write request. When the processing type is a write request, the inflow node of the new cluster is determined respectively according to the new sharding rule and the old sharding rule. If the new inflow node of the new sharding rule is different from the old inflow node of the old sharding rule, it is determined that the new inflow node and the old inflow node are both query nodes, and write requests are sent to the new inflow node and the old inflow node at the same time; when the processing type is a read request, the corresponding query node is determined directly according to the old sharding rule.

[0120] The data entity acquisition module is used to send a query request to the query node to acquire all data entities of the new cluster.

[0121] Among them, all databases in the new cluster can be traversed, and corresponding query requests can be sent to query nodes of all databases in the new cluster, and corresponding data entities can be obtained based on the query requests, and the data entity is a unique data entity.

[0122] Specifically, assuming that there were originally 5 databases in the historical cluster to be processed, after the expansion operation, the number of databases in the new cluster may increase to 10. For example, if the old sharding rule corresponding to the old cluster is to read data from database 1, then the new sharding rule of the expanded new cluster may be to read data from database 6. At this time, due to the change in the sharding rule, the data in the original 5 databases need to be redivided into 10 databases according to the new sharding rule.

[0123] In the above process, it is necessary to compare and judge the inflow node corresponding to the new sharding rule (new inflow node) and the inflow node corresponding to the old sharding rule (old inflow node). If they are the same, it means that the data does not need to be migrated and the existing storage location can be maintained. Otherwise, it needs to be migrated and the corresponding write request is sent to the new inflow node and the old inflow node respectively. For read requests, there is no need to consider the new sharding rule. The old inflow node can be directly determined as the query node according to the old sharding rule.

[0124] As a specific example, in the database, data will have one or more features indicating which entity the data belongs to. For example, the indicator of disk B of server A is the reading speed data, and the corresponding entity has three features: server = A, disk = B, indicator = reading speed. The above-mentioned calculation query node (inflow node) is to perform a certain shard calculation on all the characteristics of the entity to obtain which node the data corresponding to the entity should be written to or which node should be read from. Finally, based on the determined query node, the unique data entity that needs to be migrated is obtained, and a list to be migrated is formed.

[0125] The data migration unit 103 is configured to migrate the data entities in the to-be-migrated list from the old inflow node to the new inflow node based on a preset migration strategy.

[0126] Among them, the preset migration strategies mainly include time strategy and entity strategy. During the application process, these two strategies can be flexibly selected according to the scenario or requirements.

[0127] For the first strategy, migrating the data entity in the to-be-migrated list from the previous inflow stage to the new inflow node based on the time strategy further includes:

[0128] The first module is used to obtain the data retention time, migration interval and concurrency number of the data entity based on a preset configuration file;

[0129] The second module is used for traversing the list to be migrated and sending a migration request to the old inflow node of the data entity in the list to be migrated according to the concurrency number when the end time of the migration list, the migration interval, and the data retention time meet the preset conditions;

[0130] A third module is used to obtain data entities within a time range corresponding to the preset condition based on the migration request, and send the data entities to the corresponding new inflow node;

[0131] The fourth module is used to determine whether the end time, the migration interval, and the data retention time meet preset conditions, and repeatedly execute the above steps S320 to S340.

[0132] Specifically, let R=end time, L=R-migration interval, and the start time be set to the difference between the end time and the data retention time. The preset condition mainly means that if L is less than the start time, then L=start time, and traverse the list to be migrated to send the corresponding migration request; when step S330 is executed, the L time is used as a progress and recorded in a database, and then step S340 is further set to R=L, L=L-migration time, and the above steps are repeated until R is less than or equal to the end time, indicating that the data entity migration is completed.

[0133] For the second strategy, the step of migrating the data entities in the to-be-migrated list from the previous inflow stage to the new inflow node based on the entity strategy includes:

[0134] A fifth module is used to obtain the data retention time, migration interval and concurrency number of the data entity based on a preset configuration file;

[0135] A sixth module is used to traverse the list to be migrated and obtain data entities in the list to be migrated according to the concurrency number;

[0136] The seventh module is used to send a migration request to the corresponding old inflow node of the data entity in the to-be-migrated list within a preset time range;

[0137] An eighth module is used to obtain data entities within the preset time range based on the migration request, and send the data entities to the corresponding new inflow node;

[0138] A ninth module is used to use the unique identifier of the migrated data entity as progress and record it in a database;

[0139] Module 10: Repeat the execution of the modules until all data entities in the to-be-migrated list are migrated.

[0140] Among them, the preset time range can be determined according to L and R in the time strategy. The preset time range mainly refers to the time range [L, R), where R = end time (end time when the migration list is completed), and L = R-migration interval.

[0141] As a specific example, assuming that the migration interval is set to 1 day and the data retention time is set to 30 days, then each time the data entity's data volume for one day is migrated, read from the old inflow node, and written to the new inflow node, 30 days of data can be migrated at one time, indicating that the data of this entity has been completed. If there are multiple data entities in the list to be migrated, assuming there are 10 entities, if the concurrency is set to 3, then data migration will be performed on 3 data entities each time, and each data entity will be migrated according to the time period and the data volume for one day, until all 10 data entities in the list have completed the migration, indicating that all migration work is completed.

[0142] The state modifying unit 104 is configured to delete the data entity corresponding to the old incoming node in the to-be-migrated list, and modify the state information of the new cluster accordingly.

[0143] Specifically, the state modification unit 104 may further include:

[0144] A status marking module, used for marking the status of the new cluster as being cleaned;

[0145] A data deletion module, used for traversing the list to be migrated, and deleting the data entity corresponding to the old inflow node in the list to be migrated;

[0146] The state changing module is used to mark the state of the new cluster as idle.

[0147] In a specific embodiment of the present invention, when a data entity is unexpectedly interrupted during the migration process, or the management node interrupts the migration operation for some reason, it can be restored and the new cluster state before the interruption can be checked during the recovery, and then the corresponding operation can be continued according to the new cluster state before the abnormal interruption.

[0148] Specifically, if the state before the abnormal interruption is in preparation, data management can be restarted from the preparation stage after the abnormal interruption is restored; data management may include data management in scenarios such as cluster expansion, reduction, sharding rule adjustment, and sharding algorithm change;

[0149] If the status before the abnormal interruption is data migration in progress, obtain the migration progress and continue the migration of data entities;

[0150] If the status before the abnormal interruption is cleaning, the data cleaning operation continues.

[0151] As a specific example, Figure 2 As shown in the figure, the client can calculate the time series database node to which the request needs to be routed through routing, and read and write the time series database through the database connection pool. The database connection pool is mainly used to manage the connection of each time series database node; the sharding rule is mainly based on certain algorithms and rules to convert the input parameters into the identifier of the managed time series database. The expansion module is the main module for performing expansion operations, including functions such as adjusting sharding rules, progress management, recovery, and cluster state transfer; of course, in addition to expansion, it can also be applied to scenarios such as shrinking, adjusting sharding rules, and changing sharding algorithms.

[0152] like Figure 4 , which is a schematic diagram of the structure of an electronic device for implementing the data management method of a time series database cluster according to the present invention.

[0153] The electronic device 1 may include a processor 10, a memory 11 and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a data management program 12 for a time series database cluster.

[0154] Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 11 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the data management program of the time series database cluster, but also can be used to temporarily store data that has been output or is to be output.

[0155] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (such as data management programs of time series database clusters, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device 1 and process data.

[0156] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0157] Figure 4 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 4The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0158] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0159] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0160] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0161] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0162] The data management program 12 of the time series database cluster stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:

[0163] Obtaining an old sharding rule of a historical cluster to be processed and a new sharding rule of a new cluster after updating the historical cluster;

[0164] Based on the new sharding rule and the old sharding rule, a to-be-migrated list corresponding to the data entity of the new cluster is obtained; the to-be-migrated list includes the data entity and the old inflow node and the new inflow node corresponding to the data entity, the old inflow node and the new inflow node are used to indicate the data flow direction of the data entity;

[0165] Based on a preset migration strategy, migrate the data entity in the to-be-migrated list from the old inflow node to the new inflow node;

[0166] The data entity corresponding to the old incoming node in the to-be-migrated list is deleted, and the state information of the new cluster is modified accordingly.

[0167] In addition, an optional technical invention is that, based on the new sharding rule and the old sharding rule, obtaining a list to be migrated corresponding to the data entity of the new cluster includes:

[0168] Marking the state of the new cluster as ready;

[0169] Traverse the new cluster in the prepared state based on the new sharding rule and the old sharding rule, and query the data entity of the new cluster;

[0170] Based on the data entity, a to-be-migrated list corresponding to the data entity is obtained.

[0171] In addition, an optional technical invention is that traversing the new cluster in the prepared state based on the new sharding rule and the old sharding rule and querying the data entity of the new cluster includes:

[0172] Based on the processing type of the new cluster, the new sharding rule and the old sharding rule, respectively obtain query nodes of the new cluster;

[0173] Send a query request to the query node to obtain all data entities of the new cluster.

[0174] In addition, an optional technical invention is that the preset migration strategy includes a time strategy and an entity strategy; wherein,

[0175] Migrating the data entity in the to-be-migrated list from the old inflow node to the new inflow node based on the time strategy includes:

[0176] Based on a preset configuration file, obtain the data retention time, migration interval and concurrency number of the data entity;

[0177] When the end time of the migration list, the migration interval, and the data retention time meet the preset conditions, the list to be migrated is traversed and a migration request is sent to the old inflow node of the data entity in the list to be migrated according to the concurrency number;

[0178] Acquire data entities within a time range corresponding to the preset condition based on the migration request, and send the data entities to the corresponding new inflow node;

[0179] Determine whether the end time, the migration interval, and the data retention time meet preset conditions, and repeat the above steps.

[0180] In addition, an optional technical invention is that migrating the data entity in the to-be-migrated list from the old inflow node to the new inflow node based on the entity policy includes:

[0181] Based on a preset configuration file, obtain the data retention time, migration interval and concurrency number of the data entity;

[0182] Traversing the list to be migrated, and acquiring data entities in the list to be migrated according to the concurrency number;

[0183] Sending a migration request to the old inflow node corresponding to the data entity in the to-be-migrated list within a preset time range;

[0184] Acquire the data entity within the preset time range based on the migration request, and send it to the corresponding new inflow node;

[0185] The unique identifier of the migrated data entity is used as progress and recorded in the database;

[0186] The above steps are repeated until all data entities in the to-be-migrated list are migrated.

[0187] In addition, an optional technical invention is that the deleting of the data entity corresponding to the old inflow node in the to-be-migrated list and correspondingly modifying the state information of the new cluster includes:

[0188] Mark the status of the new cluster as being cleaned up;

[0189] Traversing the to-be-migrated list, and deleting the data entity in the to-be-migrated list corresponding to the old inflow node;

[0190] The state of the new cluster is marked as idle.

[0191] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0192] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0193] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0194] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0195] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0196] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0197] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0198] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical invention of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical invention of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical invention of the present invention.

Claims

1. A data management method for a time series database cluster, characterized in that: The method comprises: Obtaining an old sharding rule of a historical cluster to be processed and a new sharding rule of a new cluster after updating the historical cluster; Based on the new sharding rule and the old sharding rule, a to-be-migrated list corresponding to the data entity of the new cluster is obtained; the to-be-migrated list includes the data entity and the old inflow node and the new inflow node corresponding to the data entity, the old inflow node and the new inflow node are used to indicate the data flow direction of the data entity; Based on a preset migration strategy, migrate the data entity in the to-be-migrated list from the old inflow node to the new inflow node; Deleting the data entity corresponding to the old incoming node in the list to be migrated, and modifying the state information of the new cluster accordingly; The preset migration strategy includes a time strategy and an entity strategy; wherein, Migrating the data entity in the to-be-migrated list from the old inflow node to the new inflow node based on the time strategy includes: S310: Based on a preset configuration file, obtain the data retention time, migration interval and concurrency number of the data entity; S320: When the end time of the list to be migrated, the migration interval, and the data retention time meet preset conditions, traverse the list to be migrated and send a migration request to the old inflow node of the data entity in the list to be migrated according to the concurrency number; S330: acquiring data entities within a time range corresponding to the preset condition based on the migration request, and sending the data entities to the corresponding new inflow node; S340: Determine whether the end time, the migration interval, and the data retention time meet preset conditions, and repeat the above steps S320 to S340; Migrating the data entity in the to-be-migrated list from the old inflow node to the new inflow node based on the entity policy includes: S350: Based on a preset configuration file, obtain the data retention time, migration interval and concurrency number of the data entity; S360: traverse the list to be migrated, and acquire data entities in the list to be migrated according to the concurrency number; S370: Sending a migration request to the old inflow node corresponding to the data entity in the to-be-migrated list within a preset time range; S380: Acquire data entities within the preset time range based on the migration request, and send them to the corresponding new inflow node; S390: Using the unique identifier of the migrated data entity as progress and recording it in a database; S3100: Repeat the above steps S350 to S390 until the migration of all data entities in the to-be-migrated list is completed.

2. The data management method for a time series database cluster according to claim 1, characterized in that: The acquiring, based on the new sharding rule and the old sharding rule, a list to be migrated corresponding to the data entities of the new cluster comprises: Marking the state of the new cluster as ready; Traverse the new cluster in the prepared state based on the new sharding rule and the old sharding rule, and query the data entity of the new cluster; Based on the data entity, a to-be-migrated list corresponding to the data entity is obtained.

3. The data management method for a time series database cluster according to claim 2, characterized in that: The traversing the new cluster in the prepared state based on the new sharding rule and the old sharding rule and querying the data entity of the new cluster includes: Based on the processing type of the new cluster, the new sharding rule and the old sharding rule, respectively obtain query nodes of the new cluster; Send a query request to the query node to obtain all data entities of the new cluster.

4. The data management method for a time series database cluster according to claim 1, characterized in that: The deleting of the data entity corresponding to the old inflow node in the list to be migrated and correspondingly modifying the state information of the new cluster includes: Mark the status of the new cluster as being cleaned up; Traversing the to-be-migrated list, and deleting the data entity in the to-be-migrated list corresponding to the old inflow node; The state of the new cluster is marked as idle.

5. A data management device for a time series database cluster, characterized in that: The device comprises: A new sharding rule acquisition unit, used to acquire old sharding rules of a historical cluster to be processed, and new sharding rules of a new cluster after updating the historical cluster; a to-be-migrated list acquisition unit, configured to acquire a to-be-migrated list corresponding to a data entity of the new cluster based on the new sharding rule and the old sharding rule; the to-be-migrated list includes the data entity and an old inflow node and a new inflow node corresponding to the data entity, the old inflow node and the new inflow node being used to indicate a data flow direction of the data entity; A data migration unit, configured to migrate the data entities in the to-be-migrated list from the old inflow node to the new inflow node based on a preset migration strategy; A state modification unit, configured to delete the data entity corresponding to the old inflow node in the to-be-migrated list, and modify the state information of the new cluster accordingly; The preset migration strategy includes a time strategy and an entity strategy; wherein, Migrating the data entity in the to-be-migrated list from the old inflow node to the new inflow node based on the time strategy includes: S310: Based on a preset configuration file, obtain the data retention time, migration interval and concurrency number of the data entity; S320: When the end time of the list to be migrated, the migration interval, and the data retention time meet preset conditions, traverse the list to be migrated and send a migration request to the old inflow node of the data entity in the list to be migrated according to the concurrency number; S330: acquiring data entities within a time range corresponding to the preset condition based on the migration request, and sending the data entities to the corresponding new inflow node; S340: Determine whether the end time, the migration interval, and the data retention time meet preset conditions, and repeat the above steps S320 to S340; Migrating the data entity in the to-be-migrated list from the old inflow node to the new inflow node based on the entity policy includes: S350: Based on a preset configuration file, obtain the data retention time, migration interval and concurrency number of the data entity; S360: traverse the list to be migrated, and acquire data entities in the list to be migrated according to the concurrency number; S370: Sending a migration request to the old inflow node corresponding to the data entity in the to-be-migrated list within a preset time range; S380: Acquire data entities within the preset time range based on the migration request, and send them to the corresponding new inflow node; S390: Using the unique identifier of the migrated data entity as progress and recording it in a database; S3100: Repeat the above steps S350 to S390 until the migration of all data entities in the to-be-migrated list is completed.

6. The data management device for a time series database cluster according to claim 5, characterized in that: The acquiring, based on the new sharding rule and the old sharding rule, a list to be migrated corresponding to the data entities of the new cluster comprises: Marking the state of the new cluster as ready; Traverse the new cluster in the prepared state based on the new sharding rule and the old sharding rule, and query the data entity of the new cluster; Based on the data entity, a to-be-migrated list corresponding to the data entity is obtained.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps in the data management method for a time series database cluster as described in any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the data management method for a time series database cluster as described in any one of claims 1 to 4 are implemented.

Images