Data storage allocation scheme acquisition method

Abstract

The invention relates to a data storage allocation scheme acquisition method, belongs to the technical field of data storage space allocation, and solves the problem of data skew caused by a partition algorithm in the prior art. Obtaining a to-be-processed data set, and determining a cluster for processing the to-be-processed data set; calculating the number of partitions; calculating an initial target partition corresponding to each piece of data in the to-be-processed data set based on a first partition algorithm, and calculating the data volume of each initial target partition; judging whether the data volume of each initial target partition exceeds the size of the preset partition or not, and sequentially executing initial data distribution operation on the initial target partitions exceeding the size of the preset partition, so that the data volume of each initial target partition is smaller than the size of the preset partition; calculating a balance coefficient under the current distribution scheme, and if the balance coefficient meets a preset threshold range interval, distributing each piece of data in the to-be-processed data set to a corresponding initial target partition according to the current distribution scheme; and if not, executing a redistribution operation. The data storage and distribution method is uniform in data resource distribution, high in data query efficiency and balanced in resource use.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage space allocation, and in particular to a method for acquiring a data storage allocation solution. Background Art

[0002] When facing large-scale data storage, it is necessary to use partitioning algorithms to partition the data according to data characteristics and business needs to ensure that the data is dynamically distributed and balanced. Common partitioning algorithms include: Range Partitioning, List Partitioning, Hash Partitioning, Composite Partitioning, Key Partitioning, and Round Robin Partitioning.

[0003] In the prior art, the original intention of using partitioning algorithms is to evenly distribute data. However, if the execution logic of the distribution algorithm is improperly designed or the data itself has a certain pattern, the amount of data in some partitions may be much larger than that in other partitions, resulting in data skew. When a large amount of data is concentrated in one area, the following problems may occur: (1) The access speed is reduced. This is because the performance of the storage system is usually limited by hardware resources (such as disk I / O, network bandwidth, etc.). If multiple users or applications access the data in this area at the same time, it may cause resource competition, thereby increasing the delay of data access; (2) The storage system in this area is overloaded, while the resources in other areas are not fully utilized. This unbalanced load distribution may cause performance problems in some servers or storage devices, and may even cause system failures.

[0004] Therefore, it is necessary to provide a data storage allocation method that evenly distributes data resources, has high data query efficiency, and balanced resource usage. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a method for obtaining a data storage allocation solution, so as to solve the problem that the existing partitioning algorithm may cause data skew.

[0006] An embodiment of the present invention provides a method for obtaining a data storage allocation solution, comprising:

[0007] S1: Obtain a data set to be processed and determine a cluster for processing the data set to be processed;

[0008] S2: Calculate the number of partitions based on the computing capacity of the cluster, the data volume of the data set to be processed, and the preset partition size;

[0009] S3: Calculate the initial target partition corresponding to each piece of data in the to-be-processed data set based on the first partitioning algorithm, and calculate the data volume of each initial target partition;

[0010] S4: determining whether the data volume of each initial target partition exceeds the preset partition size, and for the initial target partitions that exceed the preset partition size, sequentially performing data initial allocation operations so that the data volume of each initial target partition is smaller than the preset partition size;

[0011] S5: Calculate the balance coefficient under the current allocation scheme. If the balance coefficient meets the preset threshold range, each data in the to-be-processed data set is allocated to the corresponding initial target partition according to the current allocation scheme; if not, perform a redistribution operation.

[0012] Based on the further improvement of the above method, step S1 also includes: preprocessing the data set to be processed, and converting each data into a key-value pair.

[0013] Based on the further improvement of the above method, the data initial allocation operation is sequentially performed for the initial target partition that exceeds the preset partition size, including:

[0014] The initial target partitions exceeding the preset partition size are sorted in descending order according to the data volume to obtain a second sequence variable;

[0015] Select initial target partitions in sequence based on the second sequence variable, and perform the following operations: take the data in the initial target partition that exceeds the preset partition size as the first data to be allocated, and sort the remaining initial target partitions in order of data volume from small to large to obtain a first sequence variable;

[0016] Determine whether the data amount of the first data to be allocated is greater than the initial target partition with the smallest data amount. If so, first add part of the data in the first data to be allocated to the smallest initial target partition. When the data amount of the smallest initial target partition reaches the preset partition size, add the remaining data to the next smallest initial target partition. When the data amount of the next smallest initial target partition reaches the preset partition size, add the remaining data to the next initial target partition, and so on, until all data in the first data to be allocated is effectively placed. If less than, directly add the first data to the smallest initial target partition.

[0017] Based on the further improvement of the above method, the data initial allocation operation is sequentially performed for the initial target partition that exceeds the preset partition size, including:

[0018] The initial target partitions exceeding the preset partition size are sorted in descending order according to the data volume to obtain a second sequence variable;

[0019] Select initial target partitions in sequence based on the second sequence variable, and perform the following operations: take the data in the initial target partition that exceeds the preset partition size as the first data to be allocated, and sort the remaining initial target partitions in order of data volume from small to large to obtain a first sequence variable;

[0020] Determine whether the data amount of the first data to be allocated is greater than the initial target partition with the smallest data amount. If so, first add part of the data in the first data to be allocated to the smallest initial target partition. When the data amount of the smallest initial target partition reaches the preset partition size, re-establish an extended partition of the preset partition size, and then add the remaining data to the extended partition. When the data amount of the extended partition reaches the preset partition size, re-establish an extended partition of the preset partition size, and so on, until all the data in the first data to be allocated are effectively placed; if less than, directly add the first data to the smallest initial target partition.

[0021] A further improvement based on the above method is characterized in that the performing of the reallocation operation comprises:

[0022] S621: using the initial target partition corresponding to the maximum data volume under the current allocation scheme as the high data partition, and using the initial target partition corresponding to the minimum data volume as the low data partition;

[0023] S622: Calculate the occurrence frequency or size distribution of each value value in the high data partition, and sort them from large to small to obtain a third sequence variable;

[0024] S623: Calculate the difference between the data amounts of the high data partition and the low data partition, and transfer data with a higher frequency of occurrence or a larger size distribution in the high data partition to the low data partition according to the order of the third sequence variable, until the transferred data amount reaches half of the difference;

[0025] S624: Calculate the balance coefficient. If the balance coefficient meets the preset threshold range, store each data item in the corresponding initial target partition. If not, return to S621.

[0026] Based on a further improvement of the above method, the calculating the occurrence frequency or size distribution of each value value in the high data partition includes:

[0027] or Among them, f1(k) is the frequency of occurrence, f2(k) is the size distribution, X1 is the number of occurrences of value in the high data partition, Y1 is the total number of data in the high data partition, X2 is the amount of data of value in the high data partition, and Y2 is the total amount of data in the high data partition.

[0028] Based on a further improvement of the above method, the step of calculating the number of partitions based on the computing capability of the cluster, the data volume of the data set to be processed, and the preset partition size includes:

[0029]

[0030] Wherein, a, b, c are constants, p is the number of partitions, C is the computing power of the cluster, D is the data volume of the data set to be processed, and S is the preset partition size.

[0031] Based on the further improvement of the above method, the calculation of the balance coefficient includes:

[0032]

[0033] Among them, size(p n ) is the pth n The data volume of each partition, n is the number of partitions.

[0034] Based on the further improvement of the above method, the preprocessing of the data set to be processed, converting each piece of data into a key-value pair, includes:

[0035] If a piece of data in the data set to be processed is structured data, any unique field in the data is selected as the key;

[0036] If a piece of data in the data set to be processed is unstructured data, a unique key is generated by using a hash algorithm, a combined field or a timestamp.

[0037] Further improvements based on the above method include: the computing power of the cluster refers to the number of CPU cores in the cluster.

[0038] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0039] 1. An embodiment of the present invention provides a method for obtaining a data storage allocation plan. When performing data storage allocation, the frequency of occurrence or size distribution of each value in the high data partition is used as a condition for data transfer, so that high-frequency data is evenly distributed to each partition, and the balance coefficient is used as a criterion for judging the effectiveness of the partition, which can make the final allocation plan data allocation more balanced and resource usage more balanced.

[0040] 2. An embodiment of the present invention provides a method for obtaining a data storage allocation plan. The number of partitions is calculated based on the computing power of the cluster, the amount of data sets to be processed, and the preset partition size. The factors affecting the balanced data distribution are fully considered to avoid subjectively setting the number of partitions. The partition number calculation method proposed in the present invention can further improve the efficiency of balanced resource allocation.

[0041] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0043] Figure 1 This is an example diagram of a method for obtaining a data storage allocation solution in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0045] A specific embodiment of the present invention discloses a method for obtaining a data storage allocation scheme, such as Figure 1 As shown, including:

[0046] S1: Obtain a data set to be processed and determine a cluster that processes the data set to be processed.

[0047] After obtaining the data set to be processed, it is necessary to perform preprocessing operations on it to convert each data into a key-value pair, including: if a data in the data set to be processed is structured data, select any unique field in the data as the key; if a data in the data set to be processed is unstructured data, use a hash algorithm, a combined field or a timestamp to generate a unique key.

[0048] It is understandable that the preprocessing operation may also include data cleaning, data conversion, data dimension reduction, data reconstruction, etc., and through these preprocessing operations, the data quality can be improved. It should be noted that the data preprocessing operation can be determined according to business needs, and the present invention is not limited here, and it is subject to being able to achieve business needs.

[0049] The data sets targeted by the present invention are mainly large-scale data sets, which require the use of clusters with powerful computing and storage capabilities. The cluster for processing the data set is determined according to business needs and goals. For example, if real-time data processing is required, a cluster that supports stream processing can be selected, and an Apache Storm or Apache Kafka cluster can be selected; if data processing can be performed offline, a Hadoop MapReduce or Spark cluster can be selected; if the data set contains sensitive information, a cluster with powerful security functions needs to be selected, such as a cluster that supports data encryption, access control, and auditing. It is understandable that if the hardware parameters of some servers in the cluster are also specified in the business requirements, the cluster needs to be manually configured so that it can meet the business requirements. The present invention configures the cluster according to business needs, which can further improve the effectiveness of data distribution and meet the personalized needs of users.

[0050] S2: Calculating the number of partitions based on the computing power of the cluster, the data volume of the data set to be processed, and the preset partition size, including:

[0051]

[0052] Wherein, a, b, c are constants, p is the number of partitions, C is the computing power of the cluster, D is the data volume of the data set to be processed, and S is the preset partition size.

[0053] a, b, and c can be obtained by trial and error or machine learning, and the present invention does not make any specific limitation here, and can be specifically set according to actual needs. The computing power of a cluster refers to the number of CPU cores in the cluster.

[0054] To ensure the accuracy of the calculation results, the units of data volume and preset partition size need to be unified during calculation. If the units of C, D, and S corresponding to a, b, and c currently obtained by trial and error or machine learning methods are number, GB, and GB respectively, then during calculation, the units of data volume and preset partition size in the data set to be processed need to be converted to GB. For example, the cluster determined for the current data set to be processed includes 16 CPU cores, that is, C=16, the data volume of the data set to be processed is 100GB, that is, D=100GB, and the size of each preset partition is 1GB, that is, S=1GB; constants a=2, b=1, and c=5, then the number of partitions p=2*16+1*100+5*1=137.

[0055] S3: Calculate the initial target partition corresponding to each piece of data in the to-be-processed data set based on the first partitioning algorithm, and calculate the data volume of each initial target partition.

[0056] The first partitioning algorithm may be range partitioning, list partitioning, hash partitioning, composite partitioning, key partitioning, or round robin partitioning. Preferably, the first partitioning algorithm is a hash partitioning algorithm.

[0057] After this step, each piece of data in the data set to be processed corresponds to a partition (ie, the initial target partition). This correspondence means that after the allocation scheme is determined, the data will be stored in the partition.

[0058] S4: Determine whether the data volume of each initial target partition exceeds the preset partition size. For the initial target partitions that exceed the preset partition size, perform the initial data allocation operation in sequence so that the data volume of each initial target partition is smaller than the preset partition size.

[0059] After processing in step S3, the corresponding initial target partition is determined for each data in the data set to be processed. At this time, data skew may occur due to data characteristics, features, etc. or partitioning algorithms, that is, there is an initial target partition that is allocated more data, and its total data volume exceeds the preset partition size. Therefore, it is necessary to verify whether the amount of data stored in each initial target partition exceeds the preset partition size. If it exceeds, the initial data allocation operation is performed.

[0060] For the initial target partition that exceeds the preset partition size, the data initial allocation operation is performed in sequence. The present invention provides two implementation methods. The first method is:

[0061] The initial target partitions exceeding the preset partition size are sorted in descending order according to the data volume to obtain a second sequence variable;

[0062] Select initial target partitions in sequence based on the second sequence variable, and perform the following operations: take the data in the initial target partition that exceeds the preset partition size as the first data to be allocated, and sort the remaining initial target partitions in order of data volume from small to large to obtain a first sequence variable;

[0063] Determine whether the data amount of the first data to be allocated is greater than the initial target partition with the smallest data amount. If so, first add part of the data in the first data to be allocated to the smallest initial target partition. When the data amount of the smallest initial target partition reaches the preset partition size, add the remaining data to the next smallest initial target partition. When the data amount of the next smallest initial target partition reaches the preset partition size, add the remaining data to the next initial target partition, and so on, until all data in the first data to be allocated is effectively placed. If less than, directly add the first data to the smallest initial target partition.

[0064] The second type is:

[0065] The initial target partitions exceeding the preset partition size are sorted in descending order according to the data volume to obtain a second sequence variable;

[0066] Select initial target partitions in sequence based on the second sequence variable, and perform the following operations: take the data in the initial target partition that exceeds the preset partition size as the first data to be allocated, and sort the remaining initial target partitions in order of data volume from small to large to obtain a first sequence variable;

[0067] Determine whether the data amount of the first data to be allocated is greater than the initial target partition with the smallest data amount. If so, first add part of the data in the first data to be allocated to the smallest initial target partition. When the data amount of the smallest initial target partition reaches the preset partition size, re-establish an extended partition of the preset partition size, and then add the remaining data to the extended partition. When the data amount of the extended partition reaches the preset partition size, re-establish an extended partition of the preset partition size, and so on, until all the data in the first data to be allocated are effectively placed; if less than, directly add the first data to the smallest initial target partition.

[0068] The above two methods have different usage scenarios. The first method is suitable for scenarios where the amount of data in the data set to be processed will not increase excessively, that is, the total amount of data in the final data set will not exceed the total storage capacity of all partitions; the second method is suitable for variable data sets, that is, the amount of data will continue to increase and update during the use process, and there is no upper limit.

[0069] S5: Calculate the balance coefficient under the current allocation scheme. If the balance coefficient meets the preset threshold range, each data in the to-be-processed data set is allocated to the corresponding initial target partition according to the current allocation scheme; if not, perform a redistribution operation.

[0070] The performing of the reallocation operation comprises:

[0071] S621: using the initial target partition corresponding to the maximum data volume under the current allocation scheme as the high data partition, and using the initial target partition corresponding to the minimum data volume as the low data partition;

[0072] S622: Calculate the occurrence frequency or size distribution of each value value in the high data partition, and sort them from large to small to obtain a third sequence variable;

[0073] S623: Calculate the difference between the data amounts of the high data partition and the low data partition, and transfer data with a higher frequency of occurrence or a larger size distribution in the high data partition to the low data partition according to the order of the third sequence variable, until the transferred data amount reaches half of the difference;

[0074] S624: Calculate the balance coefficient. If the balance coefficient meets the preset threshold range, store each data item in the corresponding initial target partition. If not, return to S621.

[0075] The calculating the occurrence frequency or size distribution of each value value in the high data partition includes:

[0076] or Among them, f1(k) is the frequency of occurrence, f2(k) is the size distribution, X1 is the number of occurrences of value in the high data partition, Y1 is the total number of data in the high data partition, X2 is the amount of data of value in the high data partition, and Y2 is the total amount of data in the high data partition.

[0077] The calculation of the balance coefficient comprises:

[0078]

[0079] Among them, size(p n ) is the pth n The data volume of each partition, n is the number of partitions.

[0080] The present invention uses the balance coefficient as a criterion for judging the effectiveness of the current data storage allocation scheme. When the balance coefficient meets the preset threshold, it indicates that the current data storage allocation scheme can achieve balanced data distribution, and the data is distributed according to the initial target partition corresponding to each piece of data in the current data storage allocation scheme. If it is not satisfied, a redistribution operation is performed so that the data in the to-be-processed data set can be evenly distributed to each partition.

[0081] Exemplarily, the preset threshold range may be 0.95 to 1.05. Meeting the preset threshold range means that the balance coefficient of the current data storage allocation solution is within the preset threshold range.

[0082] Compared with the prior art, the present embodiment provides a method for obtaining a data storage allocation scheme. When performing data storage allocation, the frequency of occurrence or size distribution of each value value in the high data partition is used as a condition for data transfer, so that high-frequency data is evenly distributed to each partition, and the balance coefficient is used as a judgment criterion for the effectiveness of the partition, which can make the final allocation scheme data allocation more balanced and the resource usage more balanced; the number of partitions is calculated based on the computing power of the cluster, the data volume of the data set to be processed, and the preset partition size, fully considering the factors affecting the balanced data distribution, avoiding the subjective setting of the number of partitions by humans, and the partition number calculation method proposed in the present invention can further improve the efficiency of balanced resource allocation.

[0083] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0084] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for obtaining a data storage allocation plan, characterized in that: include: S1: Obtain a data set to be processed and determine a cluster for processing the data set to be processed; S2: Calculate the number of partitions based on the computing capacity of the cluster, the data volume of the data set to be processed, and the preset partition size; S3: Calculate the initial target partition corresponding to each piece of data in the to-be-processed data set based on the first partitioning algorithm, and calculate the data volume of each initial target partition; S4: determining whether the data volume of each initial target partition exceeds the preset partition size, and for the initial target partitions that exceed the preset partition size, sequentially performing data initial allocation operations so that the data volume of each initial target partition is smaller than the preset partition size; S5: Calculate the balance coefficient under the current allocation scheme. If the balance coefficient meets the preset threshold range, each data in the to-be-processed data set is allocated to the corresponding initial target partition according to the current allocation scheme; if not, perform a redistribution operation.

2. A method for obtaining a data storage allocation scheme according to claim 1, characterized in that: Step S1 also includes: The data set to be processed is preprocessed to convert each piece of data into a key-value pair.

3. A method for obtaining a data storage allocation scheme according to claim 2, characterized in that: The step of sequentially performing the data initial allocation operation on the initial target partition that exceeds the preset partition size includes: The initial target partitions exceeding the preset partition size are sorted in descending order according to the data volume to obtain a second sequence variable; Select initial target partitions in sequence based on the second sequence variable, and perform the following operations: take the data in the initial target partition that exceeds the preset partition size as the first data to be allocated, and sort the remaining initial target partitions in order of data volume from small to large to obtain a first sequence variable; Determine whether the data amount of the first data to be allocated is greater than the initial target partition with the smallest data amount. If so, first add part of the data in the first data to be allocated to the smallest initial target partition. When the data amount of the smallest initial target partition reaches the preset partition size, add the remaining data to the next smallest initial target partition. When the data amount of the next smallest initial target partition reaches the preset partition size, add the remaining data to the next initial target partition, and so on, until all data in the first data to be allocated is effectively placed. If less than, directly add the first data to the smallest initial target partition.

4. A method for obtaining a data storage allocation scheme according to claim 2, characterized in that: The step of sequentially performing the data initial allocation operation on the initial target partition that exceeds the preset partition size includes: The initial target partitions exceeding the preset partition size are sorted in descending order according to the data volume to obtain a second sequence variable; Select initial target partitions in sequence based on the second sequence variable, and perform the following operations: take the data in the initial target partition that exceeds the preset partition size as the first data to be allocated, and sort the remaining initial target partitions in order of data volume from small to large to obtain a first sequence variable; Determine whether the data amount of the first data to be allocated is greater than the initial target partition with the smallest data amount. If so, first add part of the data in the first data to be allocated to the smallest initial target partition. When the data amount of the smallest initial target partition reaches the preset partition size, re-establish an extended partition of the preset partition size, and then add the remaining data to the extended partition. When the data amount of the extended partition reaches the preset partition size, re-establish an extended partition of the preset partition size, and so on, until all the data in the first data to be allocated are effectively placed; if less than, directly add the first data to the smallest initial target partition.

5. A method for acquiring a data storage allocation solution according to claim 3 or claim 4, characterized in that: The performing of the reallocation operation comprises: S621: using the initial target partition corresponding to the maximum data volume under the current allocation scheme as the high data partition, and using the initial target partition corresponding to the minimum data volume as the low data partition; S622: Calculate the occurrence frequency or size distribution of each value value in the high data partition, and sort them from large to small to obtain a third sequence variable; S623: Calculate the difference between the data amounts of the high data partition and the low data partition, and transfer data with a higher frequency of occurrence or a larger size distribution in the high data partition to the low data partition according to the order of the third sequence variable, until the transferred data amount reaches half of the difference; S624: Calculate the balance coefficient. If the balance coefficient meets the preset threshold range, store each data item in the corresponding initial target partition. If not, return to S621.

6. A method for obtaining a data storage allocation scheme according to claim 5, characterized in that: The calculating the occurrence frequency or size distribution of each value value in the high data partition includes: or Among them, f1(k) is the frequency of occurrence, f2(k) is the size distribution, X1 is the number of occurrences of value in the high data partition, Y1 is the total number of data in the high data partition, X2 is the amount of data of value in the high data partition, and Y2 is the total amount of data in the high data partition.

7. A method for obtaining a data storage allocation scheme according to claim 6, characterized in that: The calculating the number of partitions based on the computing capability of the cluster, the data volume of the data set to be processed, and the preset partition size includes: Wherein, a, b, c are constants, p is the number of partitions, C is the computing power of the cluster, D is the data volume of the data set to be processed, and S is the preset partition size.

8. A method for obtaining a data storage allocation scheme according to claim 7, characterized in that: The calculation of the balance coefficient comprises: Among them, size(p n ) is the pth n The data volume of each partition, n is the number of partitions.

9. A method for obtaining a data storage allocation scheme according to claim 8, characterized in that: The preprocessing of the data set to be processed, converting each piece of data into a key-value pair, includes: If a piece of data in the data set to be processed is structured data, any unique field in the data is selected as the key; If a piece of data in the data set to be processed is unstructured data, a unique key is generated by using a hash algorithm, a combined field or a timestamp.

10. A method for acquiring a data storage allocation scheme according to claim 9, characterized in that: include: The computing capacity of the cluster refers to the number of CPU cores in the cluster.

Images