A method for obtaining a data storage allocation solution

Abstract

The present invention relates to a method for obtaining a data storage allocation scheme, which belongs to the technical field of data storage space allocation and solves the problem in the prior art that the partitioning algorithm may cause data skew. The method comprises: obtaining a data set to be processed and determining a cluster for processing the data set to be processed; calculating the number of partitions; calculating the initial target partition corresponding to each piece of data in the data set to be processed based on a first partitioning algorithm, and calculating the data volume of each initial target partition; judging whether the data volume of each initial target partition exceeds the preset partition size, and for the initial target partitions that exceed the preset partition size, performing the initial data allocation operation in sequence, so that the data volume of each initial target partition is less than the preset partition size; calculating the balance coefficient under the current allocation scheme, and if the balance coefficient meets the preset threshold range interval, allocating each piece of data in the data set to be processed to the corresponding initial target partition according to the current allocation scheme; if not, performing a redistribution operation. The method realizes a data storage allocation method with uniform distribution of data resources, high data query efficiency, and balanced resource utilization.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage space allocation, and in particular to a method for obtaining a data storage allocation solution. Background Art

[0002] When storing large amounts of data, partitioning algorithms are needed based on data characteristics and business needs to ensure dynamic and balanced data distribution. Common partitioning algorithms include: Range Partitioning, List Partitioning, Hash Partitioning, Composite Partitioning, Key Partitioning, and Round Robin Partitioning.

[0003] In the existing technology, the original intention of using partitioning algorithms is to evenly distribute data. However, if the execution logic of the distribution algorithm is improperly designed or the data itself has a certain pattern, the amount of data in some partitions may be much larger than that in other partitions, resulting in data skew. When a large amount of data is concentrated in one area, the following problems will arise: (1) Access speed will decrease. This is because the performance of the storage system is usually limited by hardware resources (such as disk I / O, network bandwidth, etc.). If multiple users or applications access the data in this area at the same time, it may cause resource competition, thereby increasing the delay of data access; (2) The storage system in this area will be overloaded, while the resources in other areas will not be fully utilized. This unbalanced load distribution may cause performance problems on some servers or storage devices, and may even cause system failures.

[0004] Therefore, it is necessary to provide a data storage allocation method that evenly distributes data resources, has high data query efficiency, and balanced resource usage. Summary of the Invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a method for obtaining a data storage allocation solution to solve the problem that existing partitioning algorithms may cause data skew.

[0006] An embodiment of the present invention provides a method for obtaining a data storage allocation solution, comprising:

[0007] S1: Obtain the data set to be processed and determine the cluster that processes the data set to be processed;

[0008] S2: Calculating the number of partitions based on the computing capacity of the cluster, the data volume of the data set to be processed, and the preset partition size;

[0009] S3: Calculating the initial target partition corresponding to each piece of data in the to-be-processed data set based on the first partitioning algorithm, and calculating the data volume of each initial target partition;

[0010] S4: determining whether the data volume of each initial target partition exceeds the preset partition size, and sequentially performing a data initial allocation operation for the initial target partitions that exceed the preset partition size, so that the data volume of each initial target partition is smaller than the preset partition size;

[0011] S5: Calculate the balance coefficient under the current allocation scheme. If the balance coefficient meets the preset threshold range, each data in the to-be-processed data set is allocated to the corresponding initial target partition according to the current allocation scheme; if not, perform a redistribution operation.

[0012] Based on a further improvement of the above method, step S1 further includes: preprocessing the data set to be processed, and converting each piece of data into a key-value pair.

[0013] Based on a further improvement of the above method, the data initial allocation operation is sequentially performed for the initial target partition that exceeds the preset partition size, including:

[0014] The initial target partitions that exceed the preset partition size are sorted in descending order according to the data volume to obtain the second order variable;

[0015] Selecting initial target partitions based on the second sequence variable, and performing the following operations: taking data in the initial target partition that exceeds the preset partition size as first data to be allocated, and sorting the remaining initial target partitions in ascending order of data volume to obtain a first sequence variable;

[0016] Determine whether the data amount of the first data to be allocated is greater than the initial target partition with the smallest data amount. If it is greater, first add part of the data in the first data to be allocated to the smallest initial target partition. When the data amount of the smallest initial target partition reaches the preset partition size, add the remaining data to the next smallest initial target partition. When the data amount of the next smallest initial target partition reaches the preset partition size, add the remaining data to the next initial target partition, and so on, until all data in the first data to be allocated are effectively placed; if it is less, directly add the first data to the smallest initial target partition.

[0017] Based on a further improvement of the above method, the data initial allocation operation is sequentially performed for the initial target partition that exceeds the preset partition size, including:

[0018] The initial target partitions that exceed the preset partition size are sorted in descending order according to the data volume to obtain the second order variable;

[0019] Selecting initial target partitions based on the second sequence variable, and performing the following operations: taking data in the initial target partition that exceeds the preset partition size as first data to be allocated, and sorting the remaining initial target partitions in ascending order of data volume to obtain a first sequence variable;

[0020] Determine whether the data volume of the first data to be allocated is greater than the initial target partition with the smallest data volume. If so, first add part of the data in the first data to be allocated to the smallest initial target partition. When the data volume of the smallest initial target partition reaches the preset partition size, re-establish an extended partition of the preset partition size, and then add the remaining data to the extended partition. When the data volume of the extended partition reaches the preset partition size, re-establish an extended partition of the preset partition size, and so on, until all the data in the first data to be allocated are effectively placed; if less than, directly add the first data to the smallest initial target partition.

[0021] A further improvement based on the above method is characterized in that the performing of the reallocation operation includes:

[0022] S621: The initial target partition corresponding to the maximum data volume under the current allocation scheme is used as the high data partition, and the initial target partition corresponding to the minimum data volume is used as the low data partition;

[0023] S622: Calculate the occurrence frequency or size distribution of each value in the high data partition, and sort them from large to small to obtain a third sequence variable;

[0024] S623: Calculate the difference between the data amounts of the high data partition and the low data partition, and transfer data with a higher frequency of occurrence or a larger size distribution in the high data partition to the low data partition according to the order of the third sequence variable, until the transferred data amount reaches half of the difference;

[0025] S624: Calculate the balance coefficient. If the balance coefficient meets the preset threshold range, store each data item in the corresponding initial target partition. If not, return to S621.

[0026] Based on a further improvement of the above method, the calculation of the occurrence frequency or size distribution of each value value in the high data partition includes:

[0027] or Where f1(k) is the frequency of occurrence, f2(k) is the size distribution, X1 is the number of occurrences of the median value in the high data partition, Y1 is the total number of data in the high data partition, X2 is the amount of data in the median value in the high data partition, and Y2 is the total amount of data in the high data partition.

[0028] Based on a further improvement of the above method, the calculating the number of partitions based on the computing capacity of the cluster, the data volume of the data set to be processed, and the preset partition size includes:

[0029]

[0030] Wherein, a, b, and c are constants, p is the number of partitions, C is the computing power of the cluster, D is the data volume of the data set to be processed, and S is the preset partition size.

[0031] Based on a further improvement of the above method, the calculation of the balance coefficient includes:

[0032]

[0033] Among them, size(p n ) is the pth n The data volume of each partition, n is the number of partitions.

[0034] Based on the further improvement of the above method, the data set to be processed is preprocessed to convert each data into a key-value pair, including:

[0035] If a piece of data in the dataset to be processed is structured data, any unique field in the data is selected as the key;

[0036] If a piece of data in the dataset to be processed is unstructured data, a unique key is generated by using a hash algorithm, a combination of fields, or a timestamp.

[0037] Further improvements based on the above method include: the computing power of the cluster refers to the number of CPU cores in the cluster.

[0038] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0039] 1. An embodiment of the present invention provides a method for obtaining a data storage allocation plan. When performing data storage allocation, the frequency of occurrence or size distribution of each value in the high-data partition is used as a condition for data transfer, so that high-frequency data is evenly distributed to each partition, and the balance coefficient is used as a judgment standard for the effectiveness of the partition, which can make the final allocation plan data distribution more balanced and resource usage more balanced.

[0040] 2. An embodiment of the present invention provides a method for obtaining a data storage allocation plan. The number of partitions is calculated based on the computing power of the cluster, the amount of data in the data set to be processed, and the preset partition size. It fully considers the factors affecting the balanced data distribution and avoids the subjective setting of the number of partitions. The partition number calculation method proposed in the present invention can further improve the efficiency of balanced resource allocation.

[0041] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference symbols denote the same components.

[0043] Figure 1 This is an example diagram of a method for obtaining a data storage allocation solution in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0045] A specific embodiment of the present invention discloses a method for obtaining a data storage allocation solution, such as Figure 1 As shown, including:

[0046] S1: Obtain a data set to be processed and determine a cluster to process the data set.

[0047] After obtaining the data set to be processed, it needs to be preprocessed to convert each data into a key-value pair, including: if a data in the data set to be processed is structured data, any unique field in the data is selected as the key; if a data in the data set to be processed is unstructured data, a unique key is generated by using a hash algorithm, a combined field or a timestamp.

[0048] It is understood that preprocessing operations may also include data cleaning, data conversion, data dimensionality reduction, data reconstruction, etc., and these preprocessing operations can improve data quality. It should be noted that data preprocessing operations can be determined based on business needs and are not limited in this invention. The data preprocessing operations are based on the ability to meet business needs.

[0049] The data sets targeted by the present invention are mainly large-scale data sets, which require the use of clusters with strong computing and storage capabilities. The cluster for processing the data set is determined according to business needs and goals. For example, if real-time data processing is required, a cluster that supports stream processing can be selected, such as an Apache Storm or Apache Kafka cluster; if data processing can be performed offline, a Hadoop MapReduce or Spark cluster can be selected; if the data set contains sensitive information, a cluster with strong security functions needs to be selected, such as a cluster that supports data encryption, access control, and auditing. It is understandable that if the business requirements also specify the hardware parameters of some servers in the cluster, it is necessary to manually configure the cluster so that it can meet the business requirements. The present invention configures the cluster according to business needs, which can further improve the effectiveness of data allocation and meet the personalized needs of users.

[0050] S2: Calculating the number of partitions based on the computing power of the cluster, the data size of the data set to be processed, and the preset partition size, including:

[0051]

[0052] Wherein, a, b, and c are constants, p is the number of partitions, C is the computing power of the cluster, D is the data volume of the data set to be processed, and S is the preset partition size.

[0053] a, b, and c can be obtained by trial and error or machine learning, and are not specifically limited in the present invention and can be set according to actual needs. The computing power of a cluster refers to the number of CPU cores in the cluster.

[0054] To ensure accurate calculation results, the units of data volume and preset partition size need to be unified during calculation. If the units of C, D, and S corresponding to a, b, and c currently obtained by trial and error or machine learning methods are number, GB, and GB respectively, then during calculation, the units of data volume and preset partition size in the dataset to be processed need to be converted to GB. For example, the cluster determined for the current dataset to be processed includes 16 CPU cores, that is, C = 16, the data volume of the dataset to be processed is 100GB, that is, D = 100GB, and each preset partition size is 1GB, that is, S = 1GB; constants a = 2, b = 1, and c = 5, then the number of partitions p = 2*16 + 1*100 + 5*1 = 137.

[0055] S3: Calculate the initial target partition corresponding to each piece of data in the to-be-processed data set based on the first partitioning algorithm, and calculate the data volume of each initial target partition.

[0056] The first partitioning algorithm may be range partitioning, list partitioning, hash partitioning, composite partitioning, key partitioning, or round robin partitioning. Preferably, the first partitioning algorithm is a hash partitioning algorithm.

[0057] After this step, each piece of data in the dataset to be processed corresponds to a partition (ie, the initial target partition). This correspondence means that after the allocation plan is determined, the data will be stored in the partition.

[0058] S4: Determine whether the data volume of each initial target partition exceeds the preset partition size. For the initial target partitions that exceed the preset partition size, perform the initial data allocation operation in sequence so that the data volume of each initial target partition is smaller than the preset partition size.

[0059] After processing in step S3, the corresponding initial target partition is determined for each data in the data set to be processed. At this time, data skew may occur due to data characteristics, features, or partitioning algorithms, that is, there is an initial target partition that is allocated more data, and its total data size exceeds the preset partition size. Therefore, it is necessary to verify whether the amount of data stored in each initial target partition exceeds the preset partition size. If it exceeds, the initial data allocation operation is performed.

[0060] For the initial target partition that exceeds the preset partition size, the data initial allocation operation is performed in sequence. The present invention provides two implementation methods. The first method is:

[0061] The initial target partitions that exceed the preset partition size are sorted in descending order according to the data volume to obtain the second order variable;

[0062] Selecting initial target partitions based on the second sequence variable, and performing the following operations: taking data in the initial target partition that exceeds the preset partition size as first data to be allocated, and sorting the remaining initial target partitions in ascending order of data volume to obtain a first sequence variable;

[0063] Determine whether the data amount of the first data to be allocated is greater than the initial target partition with the smallest data amount. If it is greater, first add part of the data in the first data to be allocated to the smallest initial target partition. When the data amount of the smallest initial target partition reaches the preset partition size, add the remaining data to the next smallest initial target partition. When the data amount of the next smallest initial target partition reaches the preset partition size, add the remaining data to the next initial target partition, and so on, until all data in the first data to be allocated are effectively placed; if it is less, directly add the first data to the smallest initial target partition.

[0064] The second type is:

[0065] The initial target partitions that exceed the preset partition size are sorted in descending order according to the data volume to obtain the second order variable;

[0066] Selecting initial target partitions based on the second sequence variable, and performing the following operations: taking data in the initial target partition that exceeds the preset partition size as first data to be allocated, and sorting the remaining initial target partitions in ascending order of data volume to obtain a first sequence variable;

[0067] Determine whether the data volume of the first data to be allocated is greater than the initial target partition with the smallest data volume. If so, first add part of the data in the first data to be allocated to the smallest initial target partition. When the data volume of the smallest initial target partition reaches the preset partition size, re-establish an extended partition of the preset partition size, and then add the remaining data to the extended partition. When the data volume of the extended partition reaches the preset partition size, re-establish an extended partition of the preset partition size, and so on, until all the data in the first data to be allocated are effectively placed; if less than, directly add the first data to the smallest initial target partition.

[0068] The above two methods have different usage scenarios. The first method is suitable for scenarios where the data volume of the dataset to be processed will not increase excessively, that is, the total data volume of the final dataset will not exceed the total storage capacity of all partitions; the second method is suitable for variable datasets, that is, the data volume will continue to increase and be updated during use, with no upper limit.

[0069] S5: Calculate the balance coefficient under the current allocation scheme. If the balance coefficient meets the preset threshold range, each data in the to-be-processed data set is allocated to the corresponding initial target partition according to the current allocation scheme; if not, perform a redistribution operation.

[0070] The performing of the reallocation operation includes:

[0071] S621: The initial target partition corresponding to the maximum data volume under the current allocation scheme is used as the high data partition, and the initial target partition corresponding to the minimum data volume is used as the low data partition;

[0072] S622: Calculate the occurrence frequency or size distribution of each value in the high data partition, and sort them from large to small to obtain a third sequence variable;

[0073] S623: Calculate the difference between the data amounts of the high data partition and the low data partition, and transfer data with a higher frequency of occurrence or a larger size distribution in the high data partition to the low data partition according to the order of the third sequence variable, until the transferred data amount reaches half of the difference;

[0074] S624: Calculate the balance coefficient. If the balance coefficient meets the preset threshold range, store each data item in the corresponding initial target partition. If not, return to S621.

[0075] The calculating the occurrence frequency or size distribution of each value value in the high data partition includes:

[0076] or Where f1(k) is the frequency of occurrence, f2(k) is the size distribution, X1 is the number of occurrences of the median value in the high data partition, Y1 is the total number of data in the high data partition, X2 is the amount of data in the median value in the high data partition, and Y2 is the total amount of data in the high data partition.

[0077] The calculation of the balance coefficient includes:

[0078]

[0079] Among them, size(p n ) is the pth n The data volume of each partition, n is the number of partitions.

[0080] This invention uses the balance coefficient as a criterion for determining the effectiveness of the current data storage allocation scheme. When the balance coefficient meets a preset threshold, it indicates that the current data storage allocation scheme can achieve balanced data distribution. Data is then allocated according to the initial target partition corresponding to each piece of data in the current data storage allocation scheme. If the balance coefficient does not meet the threshold, a redistribution operation is performed to ensure that the data in the dataset to be processed is evenly distributed across the partitions.

[0081] For example, the preset threshold range may be 0.95 to 1.05. Meeting the preset threshold range means that the balance coefficient of the current data storage allocation solution is within the preset threshold range.

[0082] Compared with the prior art, the present embodiment provides a method for obtaining a data storage allocation scheme. When performing data storage allocation, the frequency of occurrence or size distribution of each value in the high-data partition is used as a condition for data transfer, thereby evenly distributing high-frequency data to each partition, and using a balance coefficient as a criterion for judging the effectiveness of the partition, which can make the final allocation scheme data distribution more balanced and resource utilization more balanced; the number of partitions is calculated based on the computing power of the cluster, the data volume of the data set to be processed, and the preset partition size, fully considering the factors affecting the balance of data distribution, avoiding the subjective setting of the number of partitions by humans, and the partition number calculation method proposed in the present invention can further improve the efficiency of balanced resource allocation.

[0083] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0084] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A method for obtaining a data storage allocation plan, characterized in that: include: S1: Obtain the data set to be processed and determine the cluster that processes the data set to be processed; S2: Calculating the number of partitions based on the computing capacity of the cluster, the data volume of the data set to be processed, and the preset partition size; S3: Calculating the initial target partition corresponding to each piece of data in the to-be-processed data set based on the first partitioning algorithm, and calculating the data volume of each initial target partition; S4: determining whether the data volume of each initial target partition exceeds the preset partition size, and sequentially performing a data initial allocation operation for the initial target partitions that exceed the preset partition size, so that the data volume of each initial target partition is smaller than the preset partition size; S5: Calculate the balance coefficient under the current allocation scheme. If the balance coefficient satisfies a preset threshold range, allocate each piece of data in the to-be-processed data set to the corresponding initial target partition according to the current allocation scheme. If not satisfied, then perform the reallocation operation; Step S1 also includes: pre-processing the data set to be processed, converting each piece of data into a key-value pair; The performing of the reallocation operation includes: S621: The initial target partition corresponding to the maximum data volume under the current allocation scheme is used as the high data partition, and the initial target partition corresponding to the minimum data volume is used as the low data partition; S622: Calculate the occurrence frequency or size distribution of each value in the high data partition, and sort them from large to small to obtain a third sequence variable; S623: Calculate the difference between the data amounts of the high data partition and the low data partition, and transfer data with a higher frequency of occurrence or a larger size distribution in the high data partition to the low data partition according to the order of the third sequence variable, until the transferred data amount reaches half of the difference; S624: Calculate the balance coefficient. If the balance coefficient meets the preset threshold range, store each data item in the corresponding initial target partition. If not, return to S621.

2. The method for obtaining a data storage allocation solution according to claim 1, wherein: The method of sequentially performing the data initial allocation operation on the initial target partition that exceeds the preset partition size includes: The initial target partitions that exceed the preset partition size are sorted in descending order according to the data volume to obtain the second order variable; Selecting initial target partitions based on the second sequence variable, and performing the following operations: taking data in the initial target partition that exceeds the preset partition size as first data to be allocated, and sorting the remaining initial target partitions in ascending order of data volume to obtain a first sequence variable; Determine whether the data amount of the first data to be allocated is greater than the initial target partition with the smallest data amount. If it is greater, first add part of the data in the first data to be allocated to the smallest initial target partition. When the data amount of the smallest initial target partition reaches the preset partition size, add the remaining data to the next smallest initial target partition. When the data amount of the next smallest initial target partition reaches the preset partition size, add the remaining data to the next initial target partition, and so on, until all data in the first data to be allocated are effectively placed; if it is less, directly add the first data to the smallest initial target partition.

3. The method for obtaining a data storage allocation solution according to claim 1, wherein: The method of sequentially performing the data initial allocation operation on the initial target partition that exceeds the preset partition size includes: The initial target partitions that exceed the preset partition size are sorted in descending order according to the data volume to obtain the second order variable; Selecting initial target partitions based on the second sequence variable, and performing the following operations: taking data in the initial target partition that exceeds the preset partition size as first data to be allocated, and sorting the remaining initial target partitions in ascending order of data volume to obtain a first sequence variable; Determine whether the data volume of the first data to be allocated is greater than the initial target partition with the smallest data volume. If so, first add part of the data in the first data to be allocated to the smallest initial target partition. When the data volume of the smallest initial target partition reaches the preset partition size, re-establish an extended partition of the preset partition size, and then add the remaining data to the extended partition. When the data volume of the extended partition reaches the preset partition size, re-establish an extended partition of the preset partition size, and so on, until all the data in the first data to be allocated are effectively placed; if less than, directly add the first data to the smallest initial target partition.

4. The method for obtaining a data storage allocation solution according to claim 1, wherein: The calculating the occurrence frequency or size distribution of each value value in the high data partition includes: or Where f1(k) is the frequency of occurrence, f2(k) is the size distribution, X1 is the number of occurrences of the median value in the high data partition, Y1 is the total number of data in the high data partition, X2 is the amount of data in the median value in the high data partition, and Y2 is the total amount of data in the high data partition.

5. The method for obtaining a data storage allocation solution according to claim 4, wherein: The calculating the number of partitions based on the computing capability of the cluster, the data size of the data set to be processed, and the preset partition size includes: Wherein, a, b, and c are constants, p is the number of partitions, C is the computing power of the cluster, D is the data volume of the data set to be processed, and S is the preset partition size.

6. The method for obtaining a data storage allocation solution according to claim 5, wherein: The calculation of the balance coefficient includes: Among them, size(p n ) is the pth n The data volume of each partition, n is the number of partitions.

7. The method for obtaining a data storage allocation solution according to claim 6, wherein: The preprocessing of the data set to be processed, converting each piece of data into a key-value pair, includes: If a piece of data in the dataset to be processed is structured data, any unique field in the data is selected as the key; If a piece of data in the dataset to be processed is unstructured data, a unique key is generated by using a hash algorithm, a combination of fields, or a timestamp.

8. The method for obtaining a data storage allocation solution according to claim 7, wherein: include: The computing power of a cluster refers to the number of CPU cores in the cluster.

Images