Multi-region data migration method based on association perception and storage system

Abstract

The invention relates to a multi-area data migration method based on association perception and a storage system, and the method comprises the following steps: each storage node records data access information, and periodically uploads the data access information to a central control node, the data access information comprises a source area, time and target data of each access request, the heat of the nodes and the data volume of the nodes; hotness data processing is carried out at the central processing node to obtain hotspot data, and correlation analysis is carried out to obtain the correlation between the hotspot data and the area and the correlation between the hotspot data; and performing a data migration decision based on the data access information and the relevance, the data migration decision including determining nodes migrating data and migrated data in the nodes, and determining an area receiving the data and nodes receiving the data in the area. Compared with the prior art, the method has the advantages of reducing cross-regional access times, relieving generation of hotspots and the like.

Description

Technical Field

[0001] This invention relates to the field of data migration decision technology, and in particular to a multi-region data migration method and storage system based on correlation perception. Background Technology

[0002] Current distributed storage systems face two common challenges: (1) High overhead for cross-region data access: In distributed storage systems, data and access points are distributed across different regions. Access points may need to access data in other regions, but the network latency between regions is high, resulting in high overhead for cross-region access. How can the number of cross-region accesses be reduced? (2) Uneven distribution of hot data: There is a 2 / 8 rule in data access; not all data is accessed frequently, and hot data may only be accessed for a few data points. How can load balancing be achieved through data migration to alleviate hot data issues?

[0003] In a multi-region distributed storage system, automatic data migration may occur due to various reasons, such as load balancing, scaling up or down.

[0004] Based on the perceived factors during data migration decisions, existing multi-region distributed storage systems can be mainly divided into the following three categories.

[0005] 1. Data migration decisions are made based solely on the amount of data on the storage nodes. The advantage of this approach is its simplicity, requiring only the recording of the amount of data on the storage nodes. The disadvantages of this approach are: (1) it cannot reduce the number of cross-Region accesses; (2) it does not consider heat or correlation during load balancing.

[0006] 2. The data migration decision takes into account the data volume and popularity of the storage nodes. The advantage of this approach is that it takes into account the popularity factor, which can better achieve load balancing of each node. The disadvantages are: (1) it cannot reduce the number of cross-Region accesses; (2) it does not take into account the correlation during load balancing.

[0007] 3. Data migration decisions take into account data volume, popularity, and the relationships between data. The advantage of this approach is that it recognizes the relationships between data. The disadvantages are: (1) it cannot reduce the number of cross-Region accesses; (2) the relationships between data are specified by the user and are based on tables, which limits its applicability.

[0008] None of the three existing automatic data migration solutions considers reducing the number of cross-regional accesses. While some systems allow users to specify the data storage location to reduce cross-regional access, this effect is not achieved through automatic system identification and data migration. Summary of the Invention

[0009] The purpose of this invention is to provide a multi-region data migration method and storage system based on correlation awareness. In the data migration decision, correlation awareness is used to perceive the correlation between data and the correlation between data and regions, thereby reducing the number of cross-region accesses. Furthermore, during load balancing, the method senses heat and correlation factors to more effectively alleviate hotspots.

[0010] The objective of this invention can be achieved through the following technical solutions:

[0011] A multi-region data migration method based on correlation awareness includes the following steps:

[0012] S1. Information Collection: Each storage node records data access information and periodically uploads it to the central control node. The data access information includes the source region, time, and target data of each access request, as well as the node's popularity and the amount of data in the node.

[0013] S2. Information Processing: At the central processing node, heat data processing and correlation analysis are performed sequentially. Heat data processing involves processing the heat data of the nodes to obtain hotspot data. Correlation analysis specifically involves obtaining the correlation between hotspot data and regions, as well as the correlation between hotspot data and other hotspot data, based on principal component analysis and association rule mining techniques, respectively.

[0014] S3. Data migration decision: Based on data access information and the correlation determined in S2, a data migration decision is made. The data migration decision includes determining the nodes from which data is migrated and the data to be migrated in the nodes based on the data volume and popularity of the nodes, and determining the regions that receive data and the nodes that receive data in the regions based on the correlation.

[0015] The popularity of a node is represented by the consumption of CPU time slices or the frequency of read / write requests.

[0016] In step S2, the hot data processing specifically involves: determining whether the access frequency of the data is greater than a preset threshold based on the source region, time, and target data of each access request; if so, it is determined to be hot data.

[0017] In step S2, the method for determining the correlation between hotspot data and regions based on principal component analysis is to dynamically determine the regional correlation degree between the calculated data and each region using an exponentially weighted moving average. Specifically, the data access information uploaded by the storage nodes is traversed from the beginning, divided into multiple time periods for processing, and the regional correlation degree R of the data is calculated using the request access time, the target data accessed, and the source region in the data access information. tx :

[0018] R tx =αg(X) tx )+(1-α)R (t-1)x

[0019] Among them, R tx This represents the regional correlation between time period t and region x; α is the attenuation coefficient, ranging from 0 to 1; X tx Let g(X) be the number of access requests from region x during time period t. tx The correlation is determined by the number of access requests; the more access requests, the greater the correlation.

[0020] In step S2, the method for obtaining the correlation between hot data based on association rule mining technology is as follows:

[0021] To specify reasonable minimum support and minimum confidence thresholds for the association rule mining algorithm, visits within a similar time period are considered transactions, and hot data is considered commodities. The algorithm obtains frequent itemsets and association rules. Frequent itemsets are data combinations frequently accessed within a similar time period. The association rule between data X and Y with confidence 'a' is denoted as the association strength A between X and Y. XY A XY =a, that is, if data X is accessed, then the probability that data Y will be accessed within a similar time frame is a.

[0022] In step S3, when determining the area for receiving data, not only is the amount and popularity of the data perceived, but also the correlation between visitors in the area and the accessed hot data is perceived. The hot data is preferentially placed on nodes in the associated areas to reduce cross-regional access of hot data.

[0023] In step S3, when determining the nodes that receive data in the region, not only is the amount and popularity of the data perceived, but also the correlation between the hot data. The system tends to place related hot data on different nodes in order to alleviate the hotspots.

[0024] In step S3, the specific method for determining the area for receiving data is as follows:

[0025] Based on the principal component analysis in step S2, the regional correlation R between the hotspot data and each region is calculated. n ;

[0026] Based on regional correlation R n Each region is scored based on its node data volume and node popularity, using the following scoring formula:

[0027] S n =f1(C n )+f2(H n )+f3(R n )

[0028] Among them, S n C represents the final score for the nth region. n H represents the total amount of data across all storage nodes in the nth region. n R is the sum of the heat of all storage nodes in the nth region. n Let f1, f2, and f3 represent the correlation between the nth region and the current data, respectively. The regions with less data, lower popularity, and higher correlation score higher.

[0029] The final scores for each region are calculated and compared, and the region with the highest score is regarded as the target region for data migration.

[0030] In step S3, the specific method for determining the nodes receiving data in the region is as follows:

[0031] Each node is scored using the following formula:

[0032]

[0033] Among them, S m c represents the final score of the m-th node. m h represents the amount of data on the m-th node. m K represents the heat value at the m-th node. m Let A be the set of data associated with the data to be migrated at the m-th node; kx The association strength between data k and data x is calculated based on the association rules in step S2; g1, g2, and g3 are the scoring functions for data volume, popularity, and data association strength, respectively; nodes with less data volume, lower popularity, fewer associated data, and lower association strength of the associated data have higher scores;

[0034] The scores of each node are calculated and compared, and the node with the highest score is regarded as the target node for data migration.

[0035] A storage system that performs data migration using the method described above;

[0036] When the storage system is a distributed storage system, the method described above is used to distribute the identified hot data to nodes in the associated regions.

[0037] When the storage system is a single machine, a small cluster, a single data center, or a single region storage system, the identified hot data will be distributed to different nodes.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] In data migration decision-making, this invention perceives the correlation between data and the correlation between data and regions:

[0040] 1. By learning the correlation between data and regions, the number of cross-regional accesses is reduced: Compared with existing methods, this invention perceives the correlation between data and regions during data migration. This invention tends to place data on nodes in related regions to reduce the number of cross-regional accesses for frequently accessed data.

[0041] 2. By learning the correlation between data, an anti-affinity load balancing scheme is designed: Compared with existing methods, this invention perceives the correlation between data during load balancing and tends to place related hot data on different nodes to disperse hot data, which can greatly alleviate the generation of hot spots. Attached Figure Description

[0042] Figure 1 This is a flowchart of the method of the present invention;

[0043] Figure 2 This is a schematic diagram illustrating the data migration decision-making process of the present invention.

[0044] Figure 3 This is a schematic diagram of the associated sensing of the present invention;

[0045] Figure 4 A framework diagram of a distributed storage system (centralized metadata management architecture) implementation in one embodiment. Detailed Implementation

[0046] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0047] Example 1

[0048] This embodiment provides a multi-region data migration method based on correlation perception, such as Figure 1As shown, it includes the following steps:

[0049] S1. Information Collection: Each storage node records data access information and periodically uploads it to the central control node. This data access information includes:

[0050] (1) The source region, time and target data of each access request, that is, what data was requested, when it was accessed, and which region (Client) the request was sent by.

[0051] (2) Node heat: can be represented by CPU time slice consumption or the frequency of read and write requests.

[0052] (3) The amount of data in a node.

[0053] S2. Information Processing: Heat data processing and correlation analysis are performed sequentially at the central processing node.

[0054] S21. Popularity data processing: Based on the source region, time, and target data of each access request, calculate the access frequency of each data and determine whether the access frequency of the data is greater than the preset threshold. If so, it is determined to be hot data.

[0055] S22. Correlation Analysis: Based on principal component analysis and association rule mining techniques, the correlation between hot data and regions, as well as the correlation between hot data and other hot data, are obtained.

[0056] S221. Determine the correlation between hotspot data and regions based on principal component analysis techniques.

[0057] The exponentially weighted moving average (EWMA) is used to dynamically determine the regional correlation between the calculated data and each region. EWMA is a technique for smoothing time series data, which gives more weight to newer values ​​in the data, thus more sensitively reflecting recent data changes.

[0058] Specifically, the data access information uploaded by the storage nodes is traversed from the beginning, divided into multiple small time periods for processing, and the regional correlation degree R of the data is calculated using the request access time, the target data accessed, and the source region in the data access information. tx :

[0059] R tx =αg(X) tx )+(1-α)R (t-1)x

[0060] Among them, R txThis represents the regional correlation between time period t and region x; α is the attenuation coefficient, ranging from 0 to 1. α determines the degree of influence of new data values ​​on the smoothing result. A larger α value means that newer data has a greater impact on the average, while a smaller α value means that the average is smoothed more by historical data; X tx Let g(X) be the number of access requests from region x during time period t. tx The correlation is determined by the number of access requests; the more access requests, the greater the correlation.

[0061] In this embodiment, the number of access requests for data from all visitors within each region is counted to determine the correlation between the data and the region, thus identifying the associated regions of the data. In another embodiment, the visitor can be identified as the visitor with the highest number of access requests for the data. This visitor is then identified as the associated visitor, and the region where this visitor is located is determined, thereby establishing the correlation between the data and the region and identifying the associated regions of the data.

[0062] S222. Obtain the correlation between hot data based on association rule mining technology.

[0063] Use association rule mining methods, such as Apriori or FP-Growth, to discover frequently accessed hot data combinations that are accessed simultaneously or within similar timeframes, and treat these data as associated data.

[0064] Specify reasonable minimum support and minimum confidence thresholds for Apriori or FP-Growth, treat accesses within a similar timeframe as transactions, and hot data as commodities. Obtain frequent itemsets and association rules through association rule mining algorithms. The frequent itemsets are data combinations frequently accessed within a similar timeframe. The association rule between data X->Y with confidence 'a' is denoted as the association strength A from X to Y. XY A XY =a, meaning that if data X is accessed, the probability that data Y will be accessed within a similar timeframe is a. It's worth noting that, generally, distributed databases partition data, and data migration is also done within partitions. The cost of simply calculating the correlation between hot partitions is acceptable.

[0065] S3. Data migration decision: Based on the data access information and the correlation determined in S2, make a data migration decision.

[0066] like Figure 2 As shown, the data migration decision includes determining the nodes from which data will be migrated and the data to be migrated within those nodes based on the amount and popularity of the data, and determining the regions from which the data will be received and the nodes within those regions from which the data will be received based on correlation.

[0067] (1) Which node migrated the data?

[0068] (2) Which data of this node were migrated?

[0069] (3) Which area receives the data.

[0070] (4) Which node in this area receives the data?

[0071] The first two decision-making methods of this invention are similar to those of existing technologies, relying on data volume and popularity to make decisions. The key to this invention lies in determining which region receives the data and which node within that region receives the data, taking into account correlation. This invention favors placing data on nodes in related regions to reduce the number of cross-regional accesses for hot data. This invention also favors placing related hot data on different nodes to disperse the hot data. A schematic diagram of correlation sensing is shown below. Figure 3 As shown.

[0072] S31. The correlation between visitors and accessed data within the perception area:

[0073] When determining the area for receiving data, this invention not only senses the amount and popularity of data, but also senses the correlation between visitors in the area and the accessed hot data, and tends to place hot data on nodes in related areas to reduce cross-regional access of hot data.

[0074] Specifically, it includes the following steps:

[0075] S311. Calculate the regional correlation R between hotspot data and each region based on the principal component analysis in step S2. n ;

[0076] S312, Based on Regional Relationship R n Each region is scored based on its node data volume and node popularity, using the following scoring formula:

[0077] S n =f1(C n )+f2(H n )+f3(R n )

[0078] Among them, S n C represents the final score for the nth region. n H represents the total amount of data across all storage nodes in the nth region. n R is the sum of the heat of all storage nodes in the nth region. nLet f1, f2, and f3 represent the correlation between the nth region and the current data, respectively. The regions with less data, lower popularity, and higher correlation score higher.

[0079] S313. Calculate the final score for each region and compare them. The region with the highest score is regarded as the target region for data migration.

[0080] S32. Perceive the correlation between accessed data:

[0081] When determining the nodes that receive data in a region, this invention argues that placing related hot data on the same node may lead to overlapping node loads because related data is accessed at similar times, while unrelated hot data is accessed at off-peak times and can be placed on the same node. Therefore, an anti-affinity migration strategy is proposed, which not only senses the data volume and popularity but also the correlation between hot data, favoring the placement of related hot data on different nodes to alleviate the hotspot problem.

[0082] Specifically, it includes the following steps:

[0083] S321. Score each node using the following formula:

[0084]

[0085] Among them, S m c represents the final score of the m-th node. m h represents the amount of data on the m-th node. m K represents the heat value at the m-th node. m Let A be the set of data associated with the data to be migrated at the m-th node; kx The association strength between data k and data x is calculated based on the association rules in step S2; g1, g2, and g3 are the scoring functions for data volume, popularity, and data association strength, respectively; nodes with less data volume, lower popularity, fewer associated data, and lower association strength of the associated data have higher scores;

[0086] S322. Calculate and compare the scores of each node, and regard the node with the highest score as the target node for data migration.

[0087] In summary, the main improvements of this invention are as follows:

[0088] 1. In data migration decision-making, the correlation between data and the correlation between data and regions were perceived;

[0089] 2. Reduce cross-region visits based on the correlation between data and visitors within the region;

[0090] 3. Anti-affinity load balancing scheme based on the correlation between data.

[0091] Example 2

[0092] This embodiment provides a storage system that performs data migration using the method described in Embodiment 1 above. This storage system can be a distributed storage system (centralized metadata management architecture), a distributed storage system (decentralized metadata management architecture), or a single-machine, small-cluster, single-datacenter, or single-region storage system. The following provides a detailed description of different user scenarios:

[0093] 1. Distributed storage systems (centralized metadata management architecture): e.g., TiDB, MongoDB. Data ownership across storage nodes is managed by a central control node. Before a client can access specific data, it needs to query the central control node to determine which node the data is stored on. It's possible that the central control node will show the data as being on a node in another region, resulting in cross-regional data access. This invention's solution prioritizes placing data on nodes in relevant regions, reducing the number of cross-regional accesses to frequently accessed data.

[0094] 2. Distributed storage systems (decentralized metadata management architecture): such as CRDB and ScyllaDB. Each storage node stores the data ownership information of all nodes in the cluster. When a client accesses data, it directly accesses the nearest node, which may result in the client finding that the required data is located on a node in another Region, leading to cross-Region data access. The solution of this invention tends to place data on nodes in related Regions, reducing the number of cross-Region accesses for frequently accessed data.

[0095] 3. Single-Region Storage Systems: Such as single-machine, small cluster, single-datacenter, or single-Region storage systems, data is accessed within the Region, and some nodes may experience hotspot data. By adopting the solution of this invention, hotspot data is identified, and related accessed hotspot data is distributed to different nodes, greatly alleviating the generation of hotspots.

[0096] A typical scenario, namely the technical solution system framework of a distributed storage system (centralized metadata management architecture), is as follows: Figure 4 As shown in the figure. This embodiment provides a detailed description of some of the components shown in the figure.

[0097] 1. Router: The query layer, responsible for forwarding user requests to the correct nodes.

[0098] 2. Information Collector: Collects necessary information from nodes and uploads it to the central processing node for data migration.

[0099] 3. Model Learner: Based on data access information, principal component analysis and association rule mining techniques are used to obtain the correlation between regions and data, and the correlation between data.

[0100] 4. Migration Decision Maker: Makes data migration decisions based on popularity and correlation information, and generates and sends commands to the corresponding nodes.

[0101] right Figure 4 The two processes involved will be briefly introduced.

[0102] 1. Data access process: Visitor 1, located in region 1, needs to access data. First, it accesses the router. Then, the router looks up which storage nodes the data is located on from the central processing node. Finally, it forwards the data access request to the corresponding storage node.

[0103] 2. Data Migration Process: Each storage node periodically uploads information, including data access information and popularity information, to the central processing node. The data access information is then transferred to the Model Learner, where principal component analysis and association rule mining techniques are used to determine the relationships between data points and between data and regions. Next, the popularity and relationship information are transferred to the migration decision-maker, which issues data migration commands and sends them to the corresponding storage nodes. There are two types of commands: cross-regional data migration and intra-regional popularity dispersion. The scope of this invention lies in the data migration process; the detailed steps are as described in Example 1, and will not be repeated here.

[0104] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A multi-region data migration method based on correlation sensing, characterized in that, Includes the following steps: S1. Information Collection: Each storage node records data access information and periodically uploads it to the central control node. The data access information includes the source region, time, and target data of each access request, as well as the node's popularity and the amount of data in the node. S2. Information Processing: At the central processing node, heat data processing and correlation analysis are performed sequentially. Heat data processing involves processing the heat data of the nodes to obtain hotspot data. Correlation analysis specifically involves obtaining the correlation between hotspot data and regions, as well as the correlation between hotspot data and other hotspot data, based on principal component analysis and association rule mining techniques, respectively. S3. Data migration decision: Based on data access information and the correlation determined in S2, a data migration decision is made. The data migration decision includes determining the nodes from which data is migrated and the data to be migrated in the nodes based on the data volume and popularity of the nodes, and determining the regions that receive data and the nodes that receive data in the regions based on the correlation.

2. The multi-region data migration method based on correlation sensing according to claim 1, characterized in that, The popularity of a node is represented by the consumption of CPU time slices or the frequency of read / write requests.

3. The multi-region data migration method based on correlation sensing according to claim 1, characterized in that, In step S2, the hot data processing specifically involves: determining whether the access frequency of the data is greater than a preset threshold based on the source region, time, and target data of each access request; if so, it is determined to be hot data.

4. The multi-region data migration method based on correlation perception according to claim 1, characterized in that, In step S2, the method for determining the correlation between hotspot data and regions based on principal component analysis is to dynamically determine the regional correlation degree between the calculated data and each region using an exponentially weighted moving average. Specifically, the data access information uploaded by the storage nodes is traversed from the beginning, divided into multiple time periods for processing, and the regional correlation degree R of the calculated data is calculated using the request access time, the target data accessed, and the source region in the data access information. tx : R tx =αg(X tx )+(1-α)R (t-1)x Among them, R tx This represents the regional correlation between time period t and region x; α is the attenuation coefficient, ranging from 0 to 1; X tx Let g(X) be the number of access requests from region x during time period t. tx The correlation is determined by the number of access requests; the more access requests, the greater the correlation.

5. The multi-region data migration method based on correlation sensing according to claim 1, characterized in that, In step S2, the method for obtaining the correlation between hot data based on association rule mining technology is as follows: To specify reasonable minimum support and minimum confidence thresholds for the association rule mining algorithm, visits within a similar time period are considered transactions, and hot data is considered commodities. The algorithm obtains frequent itemsets and association rules. Frequent itemsets are data combinations frequently accessed within a similar time period. The association rule between data X and Y with confidence 'a' is denoted as the association strength A between X and Y. XY A XY =a, that is, if data X is accessed, then the probability that data Y will be accessed within a similar time frame is a.

6. The multi-region data migration method based on correlation sensing according to claim 1, characterized in that, In step S3, when determining the area for receiving data, not only is the amount and popularity of the data perceived, but also the correlation between visitors in the area and the accessed hot data is perceived. The hot data is preferentially placed on nodes in the associated areas to reduce cross-regional access of hot data.

7. The multi-region data migration method based on correlation sensing according to claim 1, characterized in that, In step S3, when determining the nodes that receive data in the region, not only is the amount and popularity of the data perceived, but also the correlation between the hot data. The system tends to place related hot data on different nodes in order to alleviate the hotspots.

8. A multi-region data migration method based on correlation sensing according to claim 6, characterized in that, In step S3, the specific method for determining the area for receiving data is as follows: Based on the principal component analysis in step S2, the regional correlation R between the hotspot data and each region is calculated. n ; Based on regional correlation R n Each region is scored based on its node data volume and node popularity, using the following scoring formula: S n =f1(C n )+f2(H n )+f3(R n ) Among them, S n C represents the final score for the nth region. n H represents the total amount of data across all storage nodes in the nth region. n R is the sum of the heat of all storage nodes in the nth region. n Let f1, f2, and f3 represent the correlation between the nth region and the current data, respectively. The regions with less data, lower popularity, and higher correlation score higher. The final scores for each region are calculated and compared, and the region with the highest score is regarded as the target region for data migration.

9. A multi-region data migration method based on correlation sensing according to claim 7, characterized in that, In step S3, the specific method for determining the nodes receiving data in the region is as follows: Each node is scored using the following formula: Among them, S m c represents the final score of the m-th node. m h represents the amount of data on the m-th node. m K represents the heat value at the m-th node. m Let A be the set of data associated with the data to be migrated at the m-th node; kx The association strength between data k and data x is calculated based on the association rules in step S2; g1, g2, and g3 are the scoring functions for data volume, popularity, and data association strength, respectively; nodes with less data volume, lower popularity, fewer associated data, and lower association strength of the associated data have higher scores; The scores of each node are calculated and compared, and the node with the highest score is regarded as the target node for data migration.

10. A storage system, characterized in that, The storage system performs data migration using the method described in any one of claims 1-9; When the storage system is a distributed storage system, the method described above is used to distribute the identified hot data to nodes in the associated regions. When the storage system is a single machine, a small cluster, a single data center, or a single region storage system, the identified hot data will be distributed to different nodes.

Images