Time series data fragment copy placement method considering storage balance and disaster recovery capability

Abstract

The invention provides a time series data fragment copy placement method considering both storage balance and disaster recovery capability, which comprises the following steps: modeling according to cluster nodes in a time series database cluster and placed time series data fragment copies to obtain a weighted copy adjacency graph; the weighted copy adjacency graph comprises a vertex set and an edge set, the vertex set is obtained by mapping cluster nodes, and the edge set is obtained by enumerating the cluster nodes contained in the time sequence data fragment copy; based on the weighted copy adjacency graph and a given valuation function, determining a fragment copy placement strategy of the next time sequence data fragment; and the valuation function considers a storage equilibrium constraint index and a disaster recovery capability optimization index of the time sequence database cluster at the same time. According to the method, sufficient disaster recovery capability can be provided for the time sequence database cluster in various industrial Internet of Things production environments, and meanwhile, the storage load balance of the time sequence database cluster is ensured. In addition, the method fits the characteristics of time sequence data and can be realized in any time sequence database.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data management, and in particular to a method for placing time series data shard replicas taking into account both storage balance and disaster recovery capabilities. Background Art

[0002] With the rapid development of the Industrial Internet of Things, time series database clusters have become a key solution for processing the resulting massive time series data. In the Industrial Internet of Things environment, the number of sampling devices is large and growing rapidly. For example, in China, the number of IoT devices will be close to 7.4 billion by the end of 2020. This rapid growth places extremely high demands on the storage and processing capabilities of database systems. Therefore, rationally planning the scale of the time series database cluster has become an important means to control costs and improve resource utilization efficiency. Enterprises usually gradually expand their database clusters according to the number of sampling devices in the current production environment to meet the growing data processing needs.

[0003] In order to cope with the challenges brought by large-scale data processing, aggregating time series data into shards is an effective strategy that can not only significantly improve write throughput but also effectively reduce management costs. However, in the early stages of a business, due to budget constraints or insufficient forecasting, clusters often contain only a limited number of nodes, which poses a huge challenge to achieving efficient and disaster-tolerant data replica placement. Although existing research attempts to solve this problem by combining optimization models and probabilistic methods, these methods often fail to achieve ideal storage balance when applied to small-scale clusters.

[0004] Therefore, how to solve the problem that the existing time series data shard replica placement algorithm cannot provide storage balance and fault tolerance for the time series database cluster that is gradually expanding in the industrial Internet of Things production environment is an important issue that needs to be urgently solved in the field of computer data management. Summary of the invention

[0005] The present invention provides a method for placing replicas of time series data shards that takes into account both storage balance and disaster tolerance, so as to overcome the defect that the existing time series data shard replica placement algorithm cannot provide storage balance and fault tolerance for a time series database cluster that is gradually expanded in an industrial Internet of Things production environment. The method fully considers the demand for gradual expansion of the time series database cluster, can provide sufficient disaster tolerance for the time series database cluster in a variety of industrial Internet of Things production environments, and ensures the storage load balance of the time series database cluster.

[0006] On the one hand, the present invention provides a method for placing shard replicas of time-series data that takes into account both storage balance and disaster tolerance capabilities, including: modeling a weighted replica adjacency graph based on the cluster nodes in the time-series database cluster and the placed shard replicas of time-series data; wherein, the weighted replica adjacency graph includes a vertex set and an edge set, the vertex set is mapped through the cluster nodes, and the edge set is obtained by enumerating the cluster nodes included in the shard replicas of time-series data; determining a placement strategy for the shard replicas of the next time-series data based on the weighted replica adjacency graph and a given valuation function; wherein, the valuation function simultaneously considers the storage balance constraint index and the disaster tolerance capability optimization index of the time-series database cluster.

[0007] Further, the modeling of the weighted replica adjacency graph based on the cluster nodes in the time-series database cluster and the placed shard replicas of time-series data includes: mapping the cluster nodes to the vertex set of the weighted replica adjacency graph; determining the edge set of the weighted replica adjacency graph by enumerating all the cluster nodes included in the shard replicas of time-series data; obtaining the weighted replica adjacency graph according to the vertex set and the edge set.

[0008] Further, the determining of the placement strategy for the shard replicas of the next time-series data based on the weighted replica adjacency graph and a given valuation function includes: dividing the vertex set in the weighted replica adjacency graph into a set number of vertex subsets; wherein, each of the vertex subsets includes the same number of vertices; selecting target vertices from the vertex subsets based on the given valuation function; determining the placement strategy for the shard replicas according to the target vertices.

[0009] Further, the target vertices include first target vertices; correspondingly, the selecting of the target vertices from the vertex subsets based on the given valuation function includes: selecting a set number of vertices from each of the vertex subsets to obtain corresponding minimum replica number vertex sets; wherein, the number of shard replicas of time-series data placed in the set number of vertices is the least; for each of the vertex subsets, determining a first valuation minimum vertex according to the minimum replica number vertex set based on the given valuation function to obtain a candidate vertex set; determining the vertex set with the minimum valuation function in the candidate vertex sets corresponding to all the vertex subsets as the first target vertices.

[0010] Further, the target vertices include second target vertices; correspondingly, the selecting of the target vertices from the vertex subsets based on the given valuation function includes: for any vertex subset other than the vertex subset where the first target vertex is located, determining the vertex with the minimum valuation function therein as the second valuation minimum vertex to obtain the second target vertex.

[0011] Further, before splitting the vertex set in the weighted replica adjacency graph into a set number of vertex subsets, it includes: determining the replica load quantity of cluster nodes in the time series database cluster; screening out vertices to be removed from the vertex set, and removing the vertices to be removed from the vertex set; wherein, the number of time series data shard replicas placed in the vertices to be removed is the replica load quantity.

[0012] In a second aspect, the present invention further provides a time series data shard replica placement device that takes into account storage balance and disaster tolerance capabilities, including: a weighted replica adjacency graph modeling module, configured to model a weighted replica adjacency graph according to cluster nodes in a time series database cluster and time series data shard replicas that have been placed; wherein, the weighted replica adjacency graph includes a vertex set and an edge set, the vertex set is mapped through the cluster nodes, and the edge set is obtained by enumerating the cluster nodes included in the time series data shard replicas; a shard replica placement strategy determination module, configured to determine the shard replica placement strategy for the next time series data shard based on the weighted replica adjacency graph and a given evaluation function; wherein, the evaluation function simultaneously considers the storage balance constraint index and the disaster tolerance capability optimization index of the time series database cluster.

[0013] In a third aspect, the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, it implements the time series data shard replica placement method as described in any one of the above.

[0014] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the time series data shard replica placement method as described in any one of the above.

[0015] In a fifth aspect, the present invention further provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the time series data shard replica placement method as described in any one of the above.

[0016] The method for placing shard replicas of time-series data that takes into account both storage balance and disaster tolerance provided by the present invention models a weighted replica adjacency graph based on the cluster nodes in the time-series database cluster and the already placed shard replicas of time-series data; wherein, the weighted replica adjacency graph includes a vertex set and an edge set, the vertex set is obtained by mapping the cluster nodes, and the edge set is obtained by enumerating the cluster nodes included in the shard replicas of time-series data; based on the weighted replica adjacency graph and a given evaluation function, determine the placement strategy of the shard replicas of the next time-series data shard; wherein, the evaluation function simultaneously considers the storage balance constraint index and the disaster tolerance optimization index of the time-series database cluster. This method models a weighted replica adjacency graph and uses it as an input to solve the placement strategy of the shard replicas of the next time-series data shard in the cluster, fully considering the gradually expanding requirements of the time-series database cluster, and can provide sufficient disaster tolerance for the time-series database cluster in a variety of industrial Internet of Things production environments, while ensuring the storage load balance of the time-series database cluster. In addition, this method conforms to the characteristics of time-series data and can be implemented in any time-series database. Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] Figure 1 It is a schematic flowchart of the method for placing shard replicas of time-series data that takes into account both storage balance and disaster tolerance provided by an embodiment of the present invention.

[0019] Figure 2 It is a schematic diagram of modeling the weighted replica adjacency graph provided by an embodiment of the present invention.

[0020] Figure 3 It is a schematic diagram of the current state of the time-series database cluster and the corresponding weighted replica adjacency graph provided by an embodiment of the present invention.

[0021] Figure 4 It is a schematic diagram of the determination process of the shard replica placement strategy provided by an embodiment of the present invention.

[0022] Figure 5 It is a schematic diagram of determining the placement strategy of the shard replicas of the next time-series data shard provided by an embodiment of the present invention.

[0023] Figure 6 It is a schematic structural diagram of the device for placing shard replicas of time-series data that takes into account both storage balance and disaster tolerance provided by an embodiment of the present invention.

[0024] Figure 7It is a schematic diagram of the physical structure of the electronic device provided by the embodiment of the present invention. Detailed implementation manners

[0025] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0026] It should be noted that aggregating time series data into shards is an effective solution to improve write throughput and reduce management costs. However, since the cluster usually only has a limited number of nodes at the beginning of the business, it greatly increases the difficulty of generating a shard replica placement strategy with disaster tolerance capabilities.

[0027] For example, assume that the cluster consists of 4 nodes ( , , , ), and the replica factor, that is, the number of replicas contained in each shard, , needs to place replicas of shards , , , . A simple round-robin placement algorithm: (1) Place two replicas of shard on nodes and respectively; (2) Place two replicas of shard on nodes and respectively; (3) Place two replicas of shard on nodes and respectively; (4) Place two replicas of shard on nodes and .

[0028] Although the round-robin algorithm can achieve storage balance, it will affect the disaster tolerance ability of the cluster. For example, if node fails, the load originally processed by node can only be transferred to node , which may cause node to be overloaded because the round-robin algorithm will inevitably form fixed pairings of the nodes in the cluster.

[0029] Although related research has modeled this problem based on combinatorial optimization models and probability theory, and proposed replica placement algorithms that take into account both cluster storage balance and disaster tolerance capabilities, these existing works cannot achieve storage balance when the cluster scale is small.

[0030] Considering this, the present invention proposes a new time-series data sharding replica placement method, that is, a time-series data sharding replica placement method that takes into account both storage balance and disaster tolerance capabilities. Specifically, Figure 1 FIG. shows a schematic flowchart of the time-series data sharding replica placement method provided by an embodiment of the present invention that takes into account both storage balance and disaster tolerance capabilities.

[0031] As Figure 1 shown, the method includes steps S110-S120, and the following will elaborate on steps S110-S120 and related steps in detail.

[0032] S110, according to the cluster nodes in the time-series database cluster and the already placed time-series data sharding replicas, model to obtain a weighted replica adjacency graph; wherein, the weighted replica adjacency graph includes a vertex set and an edge set, the vertex set is mapped through the cluster nodes, and the edge set is obtained by enumerating the cluster nodes included in the time-series data sharding replicas.

[0033] A time-series database is a database system specifically designed for storing and processing time-series data. Time-series data refers to a set of data points arranged in chronological order, usually used for monitoring, measuring, and analyzing phenomena or events that change over time. Such data widely exists in fields such as industrial Internet of Things, financial transactions, and environmental monitoring.

[0034] A cluster is a network composed of multiple computers (nodes) that work together to provide higher performance, reliability, and scalability. In the field of databases, clusters manage large-scale data sets through a distributed architecture, improve data processing speed, and enhance the fault tolerance of the system.

[0035] A time-series database cluster is an optimized solution for the characteristics of time-series data. It can not only meet the needs of high-speed data acquisition and real-time analysis, but also provide strong expansion capabilities and reliability guarantees. It is one of the indispensable technical components in the modern big data environment.

[0036] In a time-series database cluster, a cluster node refers to each computer or server instance that makes up the cluster. Each node is an independently operating entity responsible for processing a part of the data storage and computing tasks. By distributing data to multiple nodes, the cluster can provide higher performance, better fault tolerance, and stronger scalability.

[0037] In a time-series database cluster, the placement of time-series data shard replicas is one of the key mechanisms to ensure the high availability and fault tolerance of the system. Time-series data sharding is a technique that divides a large dataset into smaller, more manageable parts. Each part is called a time-series data shard (shard), which is usually divided based on certain rules (such as time range, device ID, etc.). To enhance the fault tolerance and high availability of the cluster, multiple replicas are usually created for each time-series data shard and distributed across different nodes.

[0038] It is easy to understand that in this embodiment, all the cluster nodes in the time-series database cluster, as well as the time-series data shard replicas already placed in the cluster nodes, are modeled into a weighted replica adjacency graph.

[0039] In the weighted replica adjacency graph, it includes multiple vertices (vertex set ). And multiple edges (edge set). Among them, the vertex set can be mapped through the current cluster nodes, represents the total number of cluster nodes. Let represent the cluster node numbered in the cluster. The edge set can be represented by the adjacency matrix .

[0040] In , , , if , then ; conversely, . Among them, represents the time-series data shard numbered in the cluster, , represents the replica factor of the cluster, that is, the number of replicas contained in each time-series data shard. has the same value as , which will not be elaborated here. It is easy to understand that in this formula, represents the number of time-series data shards whose replicas are simultaneously placed on node and node .

[0041] In , if , it means that there is no edge connection between vertices and ; if , it means that there is an undirected edge with a weight of and between vertices .

[0042] It is worth mentioning that the weighted replica adjacency graph in this embodiment effectively visualizes the storage balance and disaster tolerance capabilities between cluster nodes and the placed time-series data shard replicas. This is because: (1) The adjacency matrix in the weighted replica adjacency graph implies the storage load of the cluster nodes, because . Among them, represents the number of time-series data shard replicas held by the cluster node , and represents the replica factor of the cluster, that is, the number of replicas contained in each time-series data shard. (2) The adjacency matrix in the weighted replica adjacency graph implies the disaster tolerance capabilities of the cluster nodes, because the degree of the cluster node is equivalent to "the number of other cluster nodes that can share its load when the cluster node goes down".

[0043] For the degree of the cluster node , this embodiment defines an indicator function , which has a value of 1 when the conditional expression is true; otherwise it is 0. This indicator function is used to determine the degree of each vertex.

[0044] On the basis of modeling the weighted replica adjacency graph according to the cluster nodes and the placed time-series data shard replicas in the time-series database cluster in step S110, further, step S120 is executed.

[0045] S120. Based on the weighted replica adjacency graph and a given valuation function, determine the placement strategy of the shard replicas of the next time-series data shard; wherein, the valuation function simultaneously considers the storage balance constraint index and the disaster tolerance capability optimization index of the time-series database cluster.

[0046] The valuation function refers to a mathematical model or index used to evaluate the pros and cons of different placement schemes of time-series data shard replicas, which can help determine which shard replica placement strategy can achieve the optimal effect while meeting specific indexes (such as load balance, disaster tolerance capability, etc.).

[0047] In this embodiment, the valuation function is . Among them, the input of the valuation function is a vertex set and an adjacency matrix , which is equivalent to the modeled weighted replica adjacency graph. The output of the valuation function is a tuple , represents twice the number of edges covered by the vertex set , while represents the node / vertex set The number of shard replicas of the held time-series data.

[0048] Since each edge covered by the vertex set indicates that the corresponding disaster recovery metric has contributed to this pair of cluster nodes, it is desired to be as small as possible. At the same time, to ensure that each cluster node has an equal number of shard replicas of the time-series data, is also desired to be as small as possible. Therefore, it is defined that when then . That is to say, when or (logical or " "), when and (logical and " ") then the tuple output by the evaluation function.

[0049] It is easy to understand that in this embodiment, the subgraph placement algorithm is adopted to use the vertex set and the adjacency matrix as inputs. First, the vertices in the weighted replica adjacency graph that have been filled with shard replicas of the time-series data are removed, and then the vertex set after removing the vertices is divided into several equal-sized vertex subsets , and an appropriate number of vertices that make the evaluation function take the optimal value are selected in each vertex subset to form the shard replica placement strategy for the next time-series data shard. Among them, represents the vertex subset, represents the smallest integer greater than or equal to .

[0050] Among them, the shard replica placement strategy includes multiple target vertices / nodes, which determines how to distribute the next time-series data shard and its replicas to each node in the cluster.

[0051] It is worth mentioning that, on the one hand, the subgraph placement algorithm meets the requirements of cluster storage balance (considering the storage balance constraint metric), because the algorithm no longer allocates shard replicas of the time-series data to nodes / vertices that already hold sufficient shard replicas of the time-series data. Therefore, finally holds. Among them, represents the load factor of the cluster, that is, the number of shard replicas of the time-series data expected to be held by each cluster node, represents the cluster node holds the number of shard replicas of the time-series data.

[0052] On the other hand, the subgraph placement algorithm provides sufficient disaster recovery capabilities for the cluster (considering the disaster recovery capability optimization metric). Define as the ratio of the actual solution to the theoretical optimal solution. Then it can be proved that: (1) When the replica factor , There is a lower bound ; (2) When the replication factor , There is a lower bound . Among them, represents the degree of the cluster node after all time-series data shard replicas are placed, represents the degree of the cluster node that the cluster node may reach after all time-series data shard replicas are placed.

[0053] It can be seen that in the scenario where the replication factor , each node has sufficient disaster tolerance ability, while in the scenario where the replication factor , the existence of the lower bound indicates the universality of this algorithm.

[0054] In this embodiment, by modeling a weighted replica adjacency graph based on the cluster nodes and the placed time-series data shard replicas in the time-series database cluster; among them, the weighted replica adjacency graph includes a vertex set and an edge set, the vertex set is obtained by mapping the cluster nodes, and the edge set is obtained by enumerating the cluster nodes included in the time-series data shard replicas; based on the weighted replica adjacency graph and a given valuation function, determine the shard replica placement strategy for the next time-series data shard; among them, the valuation function simultaneously considers the storage balance constraint index and the disaster tolerance ability optimization index of the time-series database cluster. This method models a weighted replica adjacency graph and uses it as input to solve the placement strategy of the next time-series data shard replica in the cluster, fully considering the gradually expanding needs of the time-series database cluster, and can provide sufficient disaster tolerance ability for the time-series database cluster in a variety of industrial Internet of Things production environments, while ensuring the storage load balance of the time-series database cluster. In addition, this method conforms to the characteristics of time-series data and can be implemented in any time-series database.

[0055] On the basis of the above embodiment, further, the modeling process of the weighted replica adjacency graph will be described in detail below.

[0056] Modeling a weighted replica adjacency graph according to the cluster nodes and the placed time-series data shard replicas in the time-series database cluster includes: mapping the cluster nodes to the vertex set of the weighted replica adjacency graph; determining the edge set of the weighted replica adjacency graph by enumerating all cluster nodes included in the time-series data shard; obtaining the weighted replica adjacency graph according to the vertex set and the edge set.

[0057] Specifically, Figure 2 shows the modeling schematic diagram of the weighted replica adjacency graph provided by the embodiment of the present invention. As Figure 2 shown, first, the cluster node set The vertex set mapped to the weighted replica adjacency graph Then, enumerate each time-series data shard in order That is, for each time-series data shard Enumerate all the cluster nodes it contains And increase the corresponding edge weight The edge set of the weighted replica adjacency graph can be obtained and can be represented by an adjacency matrix as .

[0058] In , . . If , then ; conversely . Among them Represents the set of nodes where the replicas of the time-series data shards have been placed in the cluster , Represents the time-series data shard numbered in the cluster , Represents the replica factor of the cluster, that is, the number of replicas contained in each time-series data shard has the same value as and will not be elaborated one by one here

[0059] In , if , it means that there is no edge connection between vertices and ; if , it means that there is an undirected edge with a weight of and between vertices .

[0060] Furthermore, based on the vertex set and edge set of the weighted replica adjacency graph, the complete weighted replica adjacency graph obtained by modeling can be obtained

[0061] In a specific embodiment, taking Figure 3 as an example, the modeling process of the weighted replica adjacency graph is further elaborated Figure 3 Shows the current state of the time-series database cluster provided by the embodiment of the present invention and the schematic diagram of the corresponding weighted replica adjacency graph

[0062] In Figure 3 , two replicas of shard are respectively placed on nodes and , two replicas of shard are respectively placed on nodes and , shard Two copies are respectively placed at nodes and , and two copies of the shard are respectively placed at nodes and .

[0063] When modeling, first map the cluster nodes to the vertex set of the weighted replica adjacency graph . Then, the set of nodes in the cluster where the time-series data shard replicas have been placed is . Enumerate each time-series data shard in order, and the adjacency matrix can be obtained.

[0064] Thus, a weighted replica adjacency graph can be modeled. Specifically, refer to the right side of Figure 3 . Using the cluster node as the vertex of the weighted replica adjacency graph, there is an undirected edge with a weight of 2 between nodes and . There is also an undirected edge with a weight of 2 between nodes and . There are no edges connected between nodes and , and between nodes and .

[0065] It is worth mentioning that the weighted replica adjacency graph in this embodiment effectively embodies the storage balance and disaster tolerance capabilities between the cluster nodes and the placed time-series data shard replicas. This is because: (1) The adjacency matrix in the weighted replica adjacency graph contains the storage load of the cluster nodes, because . Among them, represents the number of time-series data shard replicas held by the cluster node , and represents the replica factor of the cluster, that is, the number of replicas contained in each time-series data shard. (2) The adjacency matrix in the weighted replica adjacency graph contains the disaster tolerance capabilities of the cluster nodes, because the degree of the cluster node is equivalent to "the number of other cluster nodes that can share its load when the cluster node fails".

[0066] In this embodiment, by mapping the cluster nodes to the vertex set of the weighted replica adjacency graph, and by enumerating all the cluster nodes included in the time-series data shard, the edge set of the weighted replica adjacency graph is determined. Furthermore, based on the vertex set and the edge set, the weighted replica adjacency graph is obtained. Thus, based on the weighted replica adjacency graph and the given valuation function, the shard replica placement strategy for the next time-series data shard is determined. This method models the weighted replica adjacency graph and uses it as the input to solve the placement strategy of the next time-series data shard replica in the cluster, fully considering the gradually expanding requirements of the time-series database cluster, and can provide sufficient disaster tolerance capabilities for the time-series database cluster in various industrial Internet of Things production environments, while ensuring the storage load balance of the time-series database cluster. In addition, this method conforms to the characteristics of time-series data and can be implemented in any time-series database.

[0067] Based on the above embodiments, further, the determination process of the shard replica placement strategy for the next time-series data shard will be described in detail below.

[0068] Based on the weighted replica adjacency graph and the given valuation function, determining the shard replica placement strategy for the next time-series data shard includes: dividing the vertex set in the weighted replica adjacency graph into a set number of vertex subsets; where each vertex subset includes the same number of vertices; based on the given valuation function, selecting target vertices from the vertex subsets; and determining the shard replica placement strategy according to the target vertices.

[0069] Figure 4 FIG. shows the schematic diagram of the determination process of the shard replica placement strategy provided by the embodiment of the present invention. As Figure 4 shown, first, remove the vertices in the weighted replica adjacency graph that already hold sufficient time-series data shard replicas. Specifically, determine the load factor of the time-series database cluster as the replica load quantity of the cluster nodes in the time-series database cluster, and filter out the vertices (i.e., the vertices to be removed) that hold the number of time-series data shard replicas from the initial vertex set. Considering that these vertices already hold a sufficient number of time-series data shard replicas, in order to meet the storage balance of the cluster, remove the vertices to be removed from the initial vertex set. Determine the load factor of the time-series database cluster as the replica load quantity of the cluster nodes in the time-series database cluster, and filter out the vertices that hold the number of time-series data shard replicas from the initial vertex set (i.e., the vertices to be removed). Considering that these vertices already hold a sufficient number of time-series data shard replicas, in order to meet the storage balance of the cluster, remove the vertices to be removed from the initial vertex set.

[0070] Then, evenly divide the vertex set after removing the vertices to be removed into a set number of vertex subsets , and these vertex subsets hold the same number of vertices. Among them, the set number is , , represents the replica factor of the cluster, that is, the number of replicas included in each time-series data shard.

[0071] Next, based on the given valuation function, the target vertex is selected from the partitioned vertex subsets by sequential enumeration.

[0072] In this embodiment, the target vertex includes the first target vertex and the second target vertex, and the final shard replica placement strategy is composed of the first target vertex and the second target vertex.

[0073] In a specific embodiment, the determination process of the first target vertex is described in detail.

[0074] From each vertex subset, a set number of vertices are selected to obtain the corresponding minimum replica number vertex set; among them, the set number of vertices have the fewest time series data shard replicas placed; for each vertex subset, based on the given valuation function, the first valuation minimum vertex is determined according to the minimum replica number vertex set to obtain the candidate vertex set; the vertex with the minimum valuation function in all the candidate vertex sets corresponding to the vertex subsets is determined as the first target vertex.

[0075] Specifically, for each vertex subset obtained by partitioning , first, a set number (such as ) of vertices are selected. These selected vertices hold the fewest time series data shard replicas, so they are denoted as the minimum replica number vertex set . Here, there is a one-to-one correspondence between the vertex subset and the minimum replica number vertex set. Denotes the largest integer less than or equal to .

[0076] Then, for each vertex subset, a vertex is selected such that the valuation function takes the minimum value. The vertex is determined as the first valuation minimum vertex, and the first valuation minimum vertex is merged into the corresponding minimum replica number vertex set to obtain the candidate vertex set . Here, there is a one-to-one correspondence between the candidate vertex set and the vertex subset.

[0077] Finally, for all the candidate vertex sets corresponding to the vertex subsets , find the vertex set that makes the valuation function minimum, and is determined as the first target vertex. The number of first target vertices may be one or more, depending on the number of vertex subsets, and is not specifically limited here.

[0078] In another specific embodiment, the determination process of the second target vertex is described in detail.

[0079] For any vertex subset other than the vertex subset where the first target vertex is located, determine the vertex with the minimum value of the valuation function therein as the second vertex with the minimum valuation, and obtain the second target vertex.

[0080] Specifically, for the vertex subset where the first target vertex is located other than any other vertex subset , select a vertex from the vertex subset such that the value of the valuation function is the minimum, and determine the vertex as the second vertex with the minimum valuation. Here, there is a one-to-one correspondence between the second vertex with the minimum valuation and the vertex subset . Combine the second vertices with the minimum valuation corresponding to all vertex subsets , and the second target vertex can be obtained. The number of second target vertices may be one or multiple, depending on the specific situation, and no specific limitation is made here.

[0081] It should be noted that the valuation function mentioned in these two specific embodiments is . Among them, the input of the valuation function is a vertex set and an adjacency matrix . The output of the valuation function is a tuple , represents twice the number of edges covered by the vertex set , while represents the number of time series data shard replicas held by the node / vertex set .

[0082] Since each edge covered by the vertex set indicates that the corresponding disaster recovery index has contributed to this pair of cluster nodes, so is expected to be as small as possible. At the same time, to ensure that each cluster node has an equal number of time series data shard replicas, is also expected to be as small as possible. Therefore, it is defined that when , . That is to say, when , or (logical or " "), when and (logical and " ") , the tuple output by the valuation function. In this way, it can be known how to compare the sizes of the valuation functions corresponding to different vertices or vertex subsets.

[0083] According to the above, the obtained first target vertex and the second target vertex By merging, the placement strategy / scheme of the shard copy of the next time-series data shard can be obtained. .

[0084] Ideally, the degree of each vertex in increases by at least each vertex in obtains from and obtains from while each vertex in obtains This algorithm has high running efficiency because the time complexity of sorting is and the time complexity of calling the evaluation function is .

[0085] In some other embodiments, taking Figure 5 as an example, the determination process of the placement strategy of the shard copy of the next time-series data shard is further expanded. Figure 5 FIG. shows a schematic diagram for determining the placement strategy of the shard copy of the next time-series data shard provided by the embodiment of the present invention.

[0086] Regarding the determination of the first target vertex.

[0087] In Figure 5 , the replica factor , so the vertex subset can be divided into . Among them, there are a set number of vertex subsets, that is, : .

[0088] Furthermore, for the vertex subset , since the number of time-series data shard replicas held by vertex is , which is the smallest, the vertex set with the minimum number of replicas can be selected. For the vertex subset , since the number of time-series data shard replicas held by vertex is , the vertex set with the minimum number of replicas can be selected.

[0089] Immediately afterwards, find a vertex from the vertex subset such that the evaluation function takes the minimum value. In this embodiment, since is the smallest tuple, the vertex is the first vertex with the minimum evaluation value, and thus the candidate vertex set 。Vertex subset The corresponding candidate vertex set can be obtained through similar steps.

[0090] Finally, for all vertex subsets and the corresponding candidate vertex sets respectively, find the set that makes the smallest as the first target vertex . In this embodiment, since has the smallest valuation function, therefore, the first target vertex .

[0091] Determination of the second target vertex.

[0092] For any other vertex subset other than the vertex subset where the first target vertex is located , find a vertex such that the valuation function has the smallest value. In this embodiment, since is the smallest tuple, the vertex is selected as the second vertex with the smallest valuation, that is, the second target vertex .

[0093] Finally, according to the first target vertex and the second target vertex, the target vertex can be obtained as , that is, . This means that the three replica shards of the next-time-series data shard will be placed on the cluster nodes respectively.

[0094] It is worth mentioning that the method for placing replica shards of time-series data that takes into account both storage balance and disaster tolerance provided by the present invention first models the cluster nodes and the already placed replica shards of time-series data into a weighted replica adjacency graph, and then uses this weighted replica adjacency graph as the input of the subgraph placement algorithm to solve the placement scheme of the next time-series data shard in the cluster. During this process, the subgraph placement algorithm cuts the weighted replica adjacency graph into several subgraphs with an equal number of nodes, and searches for a feasible replica placement strategy in each subgraph based on a given valuation function.

[0095] Since the heuristic search process enumerates each node fewer times, the subgraph placement algorithm can calculate the placement strategy of the next shard replica with a lower time complexity. Moreover, the valuation function used in the heuristic search takes into account both the storage balance constraint of the cluster and the disaster tolerance optimization index, which enables the generated shard replica placement strategy to well meet the various requirements of the time-series database cluster that is gradually expanding in the industrial Internet of Things production environment.

[0096] Corresponding to the method for placing time-series data shard replicas that takes into account both storage balance and disaster tolerance described in the above embodiments, the present invention also proposes a device for placing time-series data shard replicas that takes into account both storage balance and disaster tolerance.

[0097] Specifically, Figure 6 FIG. shows a schematic structural diagram of a device for placing time-series data shard replicas that takes into account both storage balance and disaster tolerance provided by an embodiment of the present invention.

[0098] As Figure 6 shown, the device includes: a weighted replica adjacency graph modeling module 610, configured to model a weighted replica adjacency graph according to cluster nodes in a time-series database cluster and placed time-series data shard replicas; wherein, the weighted replica adjacency graph includes a vertex set and an edge set, the vertex set is mapped through the cluster nodes, and the edge set is obtained by enumerating the cluster nodes included in the time-series data shard replicas; a shard replica placement strategy determination module 620, configured to determine a shard replica placement strategy for the next time-series data shard based on the weighted replica adjacency graph and a given evaluation function; wherein, the evaluation function simultaneously considers a storage balance constraint index and a disaster tolerance capability optimization index of the time-series database cluster.

[0099] In this embodiment, the weighted replica adjacency graph modeling module 610 models a weighted replica adjacency graph according to cluster nodes in a time-series database cluster and placed time-series data shard replicas; wherein, the weighted replica adjacency graph includes a vertex set and an edge set, the vertex set is mapped through the cluster nodes, and the edge set is obtained by enumerating the cluster nodes included in the time-series data shard replicas; the shard replica placement strategy determination module 620 determines a shard replica placement strategy for the next time-series data shard based on the weighted replica adjacency graph and a given evaluation function; wherein, the evaluation function simultaneously considers a storage balance constraint index and a disaster tolerance capability optimization index of the time-series database cluster. The device models a weighted replica adjacency graph and uses it as an input to solve the placement strategy of the next time-series data shard replica in the cluster, fully considering the gradually expanding requirements of the time-series database cluster, and can provide sufficient disaster tolerance capabilities for the time-series database cluster in a variety of industrial Internet of Things production environments, while ensuring the storage load balance of the time-series database cluster. In addition, this method conforms to the characteristics of time-series data and can be implemented in any time-series database.

[0100] It should be noted that the device for placing time-series data shard replicas that takes into account both storage balance and disaster tolerance provided by the embodiments of the present invention can be correspondingly referred to the method for placing time-series data shard replicas that takes into account both storage balance and disaster tolerance described in the above embodiments, and will not be elaborated here.

[0101] Figure 7Illustrates a schematic diagram of the physical structure of an electronic device, as Figure 7 shown. The electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call the logical instructions in the memory 730 to execute a time-series data shard replica placement method that takes into account both storage balance and disaster tolerance capabilities. The method includes: modeling a weighted replica adjacency graph based on the cluster nodes in the time-series database cluster and the placed time-series data shard replicas; wherein, the weighted replica adjacency graph includes a vertex set and an edge set, the vertex set is mapped through the cluster nodes, and the edge set is obtained by enumerating the cluster nodes included in the time-series data shard replicas; determining the shard replica placement strategy for the next time-series data shard based on the weighted replica adjacency graph and a given evaluation function; wherein, the evaluation function simultaneously considers the storage balance constraint index and the disaster tolerance capability optimization index of the time-series database cluster.

[0102] In addition, when the logical instructions in the above-mentioned memory 730 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. And the aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0103] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the time-series data shard replica placement method that takes into account both storage balance and disaster tolerance capabilities provided by the above-mentioned various methods. The method includes: modeling a weighted replica adjacency graph based on the cluster nodes in the time-series database cluster and the already placed time-series data shard replicas; wherein, the weighted replica adjacency graph includes a vertex set and an edge set, the vertex set is mapped through the cluster nodes, and the edge set is obtained by enumerating the cluster nodes included in the time-series data shard replicas; based on the weighted replica adjacency graph and a given valuation function, determining the shard replica placement strategy for the next time-series data shard; wherein, the valuation function simultaneously considers the storage balance constraint index and the disaster tolerance capability optimization index of the time-series database cluster.

[0104] In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it realizes the time-series data shard replica placement method that takes into account both storage balance and disaster tolerance capabilities provided by the above-mentioned various methods. The method includes: modeling a weighted replica adjacency graph based on the cluster nodes in the time-series database cluster and the already placed time-series data shard replicas; wherein, the weighted replica adjacency graph includes a vertex set and an edge set, the vertex set is mapped through the cluster nodes, and the edge set is obtained by enumerating the cluster nodes included in the time-series data shard replicas; based on the weighted replica adjacency graph and a given valuation function, determining the shard replica placement strategy for the next time-series data shard; wherein, the valuation function simultaneously considers the storage balance constraint index and the disaster tolerance capability optimization index of the time-series database cluster.

[0105] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0106] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solutions, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A time-series data sharding replica placement method that takes into account both storage balance and disaster tolerance capabilities, characterized in that, Including: Based on the cluster nodes in the time series database cluster and the placed replicas of time series data shards, a weighted replica adjacency graph is modeled; wherein, the weighted replica adjacency graph includes a vertex set and an edge set, the vertex set is mapped by the cluster nodes, and the edge set is obtained by enumerating the cluster nodes included in the replicas of the time series data shards; Based on the weighted replica adjacency graph and a given valuation function, a placement strategy for replicas of the next time series data shard is determined; wherein, the valuation function simultaneously considers the storage balance constraint index and the disaster tolerance capacity optimization index of the time series database cluster.

2. The method for placing time-series data shard replicas that takes into account storage balance and disaster tolerance according to claim 1, characterized in that The step of modeling a weighted replica adjacency graph based on the cluster nodes in the time series database cluster and the placed replicas of time series data shards includes: Mapping the cluster nodes to the vertex set of the weighted replica adjacency graph; By enumerating all the cluster nodes included in the replicas of the time series data shards, determining the edge set of the weighted replica adjacency graph; Based on the vertex set and the edge set, obtaining the weighted replica adjacency graph.

3. The method for placing time-series data shard replicas that takes into account storage balance and disaster tolerance according to claim 1, characterized in that, The step of determining a placement strategy for replicas of the next time series data shard based on the weighted replica adjacency graph and a given valuation function includes: Dividing the vertex set in the weighted replica adjacency graph into a set number of vertex subsets; wherein, each of the vertex subsets includes the same number of vertices; Based on the given valuation function, selecting target vertices from the vertex subsets; Based on the target vertices, determining the placement strategy for the replicas of the shard.

4. The method for placing time-series data shard replicas that takes into account storage balance and disaster tolerance according to claim 1, wherein The target vertices include first target vertices; Correspondingly, the step of selecting target vertices from the vertex subsets based on the given valuation function includes: Selecting a set number of vertices from each of the vertex subsets to obtain a corresponding vertex set with the minimum number of replicas; wherein, the number of replicas of the time series data shards placed in the set number of vertices is the least; For each of the vertex subsets, based on the given valuation function, determining a first vertex with the minimum valuation according to the vertex set with the minimum number of replicas to obtain a candidate vertex set; Determining the vertex set with the minimum valuation function in the candidate vertex sets corresponding to all the vertex subsets as the first target vertices.

5. The method for placing time-series data shard replicas that takes into account storage balance and disaster tolerance according to claim 4, characterized in that The target vertices include second target vertices; Correspondingly, the step of selecting target vertices from the vertex subsets based on the given valuation function includes: For any vertex subset other than the vertex subset where the first target vertices are located, determining the vertex with the minimum valuation function therein as a second vertex with the minimum valuation to obtain the second target vertices.

6. The sequential data sharding replica placement method that takes into account both storage balance and disaster tolerance capabilities according to claim 3, characterized in that, Before the step of dividing the vertex set in the weighted replica adjacency graph into a set number of vertex subsets, it includes: Determining the number of replica loads of the cluster nodes in the time series database cluster; Filtering out vertices to be removed from the vertex set, and removing the vertices to be removed from the vertex set; Wherein, the number of replicas of the time series data shards placed in the vertices to be removed is the number of replica loads.

7. A time-series data sharding replica placement device that takes into account both storage balance and disaster tolerance capabilities, characterized in that Including: The weighted replica adjacency graph modeling module is used to model a weighted replica adjacency graph according to the cluster nodes in the time series database cluster and the placed time series data shard replicas; wherein, the weighted replica adjacency graph includes a vertex set and an edge set, the vertex set is mapped through the cluster nodes, and the edge set is obtained by enumerating the cluster nodes included in the time series data shard replicas; The shard replica placement strategy determination module is used to determine the shard replica placement strategy of the next time series data shard based on the weighted replica adjacency graph and a given valuation function; wherein, the valuation function simultaneously considers the storage balance constraint index and the disaster tolerance capability optimization index of the time series database cluster.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the time series data shard replica placement method that takes into account both storage balance and disaster tolerance capability as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the time series data shard replica placement method that takes into account both storage balance and disaster tolerance capability as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the time series data shard replica placement method that takes into account both storage balance and disaster tolerance capability as described in any one of claims 1 to 6.

Images