Massive data sorting method, device, equipment and storage medium based on Spark

Abstract

The present invention relates to the field of big data, and discloses a method, device, equipment and storage medium for sorting massive data based on Spark. The method comprises: receiving a data sorting request and determining a target elastic distributed data set according to the request, and sampling each partition therein to obtain multiple sample data; calculating the partition boundary of each partition based on a preset number of partitions and each sample data; partitioning the target elastic distributed data set according to the partition boundary, and sorting the data in each partition to obtain an intermediate data sequence; obtaining triple data from an external storage medium, and globally sorting the intermediate data sequence according to the triple data to obtain a target data sequence. The present invention first calculates the partition boundary so that the partitions are ordered and then sorts the data in the groups, so that the data in any group is distributed to each partition with equal probability, thereby avoiding the data skew problem during the global sorting of the groups, and thus improving the performance of the server.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a method, device, equipment and storage medium for sorting massive data based on Spark. Background Art

[0002] As the amount of data grows exponentially, people have higher requirements for the performance of big data processing. Among them, sorting data is a common requirement in big data processing scenarios. When the amount of data reaches a certain scale, the memory-based sorting algorithm can no longer meet production needs. The solutions provided by conventional big data frameworks, such as the Spark distributed computing framework, use the merge sort algorithm at the bottom layer for optimization.

[0003] The existing method for sorting massive data based on Spark is to distribute the data of the same group to the same partition (multiple groups are allowed in the same partition, but the data of a group must be in the same partition) for sorting. If the amount of data in a group is too large, it will inevitably lead to data skew, which will affect the performance of the server. Summary of the invention

[0004] The main purpose of the present invention is to solve the problem that the existing Spark-based massive data sorting method may cause data skew and thus affect server performance.

[0005] The first aspect of the present invention provides a method for sorting massive data based on Spark, comprising:

[0006] Receive a data sorting request sent by a user, and determine a target elastic distributed data set according to the data sorting request;

[0007] Sampling each partition in the target elastic distributed data set to obtain multiple sample data;

[0008] Based on the preset number of partitions and each of the sample data, calculating the partition boundary of each of the partitions;

[0009] Partitioning the target elastic distributed data set according to the partition boundary of each partition;

[0010] Sorting the data in each partition obtained by the data partitioning according to the sorting method in the data sorting request to obtain an intermediate data sequence;

[0011] Acquire triplet data from a preset external storage medium, and group and globally sort the intermediate data sequence according to the triplet data to obtain a target data sequence, wherein the triplet data is used to represent the grouping requirement information corresponding to each partition.

[0012] Optionally, in a first implementation of the first aspect of the present invention, sampling each partition in the target elastic distributed data set to obtain multiple sample data includes:

[0013] Based on a preset calculation rule, calculate the total amount of sampled data of the target elastic distributed data set;

[0014] Calculate the maximum number of partitions of the target elastic distributed data set based on the total amount of sampled data and the preset number of partitions;

[0015] Based on the total amount of sampled data, the maximum number of samples for the partitions, and the pond sampling algorithm, each partition in the target elastic distributed data set is sampled respectively to obtain a plurality of sample data.

[0016] Optionally, in a second implementation of the first aspect of the present invention, based on the total amount of sampled data, the maximum number of samples of the partitions, and the pond sampling algorithm, each partition in the target elastic distributed data set is sampled respectively to obtain multiple sample data including:

[0017] The sketch operator in the Spark operator library is called, and the total amount of sampled data and the maximum number of samples of the partition are used as input parameters of the sketch operator to perform sampling calculations to obtain multiple sample data and output parameters, wherein the output parameters are used to describe information about the sample data and information about the target elastic distributed data set.

[0018] Optionally, in a third implementation of the first aspect of the present invention, the output parameter includes the total data volume of the target elastic distributed data set and the data volume of each partition, and after calling the sketch operator in the Spark operator library and using the total amount of sampled data and the maximum number of samples of the partition as input parameters of the sketch operator for sampling calculation to obtain multiple sample data and output parameters, it also includes:

[0019] Calculate the data sampling rate corresponding to the target elastic distributed data set according to the total data volume of the target elastic distributed data set and the total amount of sampled data;

[0020] Sequentially multiplying the data sampling rate by the data volume of each partition to obtain the actual sampling quantity corresponding to each partition;

[0021] The actual sampling quantity corresponding to each partition is compared with the maximum sampling quantity of the partition, and the sampling calculation is re-performed on the unbalanced partitions where the actual sampling data volume is greater than the maximum sampling quantity of the partition.

[0022] Optionally, in a fourth implementation manner of the first aspect of the present invention, the calculating a partition boundary of each partition based on a preset number of partitions and each of the sample data includes:

[0023] Obtaining a key value and a weight corresponding to each of the sample data, and sorting each of the sample data according to the size of the key value to obtain a sample data sequence;

[0024] Calculate the average weight of each partition based on the preset number of partitions and the weight corresponding to each of the sample data;

[0025] The sample data sequence is iterated multiple times until the partition boundaries of the preset number of partitions are determined, and the partition boundaries of each partition are obtained, wherein one iteration includes traversing the sample data in the sample data sequence, and accumulating the weights of the traversed sample data in turn, until the accumulated result is not less than the average weight of the partition, then determining that the key value corresponding to the current sample data is a partition boundary, and removing the current sample data and all sample data before the current sample data from the sample data sequence.

[0026] Optionally, in a fifth implementation of the first aspect of the present invention, partitioning the target elastic distributed data set according to the partition boundary of each partition includes:

[0027] If the number of the preset partitions is not greater than the preset threshold, sequentially searching the partition where each piece of data in the target elastic distributed data set is located based on the partition boundaries;

[0028] If the preset number of partitions is greater than a preset threshold, the partition where each piece of data in the target elastic distributed data set is located is determined based on the partition boundaries and a binary search algorithm.

[0029] Optionally, in a sixth implementation of the first aspect of the present invention, acquiring triple data from a preset external storage medium, and grouping and globally sorting the intermediate data sequence according to the triple data to obtain the target data sequence includes:

[0030] Acquire triplet data from a preset external storage medium, and transmit the triplet data to a multi-core GPU;

[0031] The GPU is called to perform data analysis on the triple data, and the intermediate data sequence is grouped and globally sorted according to the grouping requirement information obtained from the data analysis to obtain a target data sequence, wherein the grouping requirement information is used to represent the grouping requirement corresponding to each partition in the target elastic distributed data set.

[0032] A second aspect of the present invention provides a mass data sorting device based on Spark, comprising:

[0033] A data set determination module, used to receive a data sorting request sent by a user, and determine a target elastic distributed data set according to the data sorting request;

[0034] A sampling module, used to sample each partition in the target elastic distributed data set to obtain multiple sample data;

[0035] A partition boundary determination module, used for calculating the partition boundary of each partition based on a preset number of partitions and each of the sample data;

[0036] A partition sorting module, used to partition the target elastic distributed data set according to the partition boundary of each partition, and sort the data in each partition obtained by the data partitioning according to the sorting method in the data sorting request to obtain an intermediate data sequence;

[0037] A grouped global sorting module is used to obtain triplet data from a preset external storage medium, and to group and globally sort the intermediate data sequence according to the triplet data to obtain a target data sequence, wherein the triplet data is used to represent the grouping requirement information corresponding to each partition.

[0038] Optionally, in a first implementation of the second aspect of the present invention, the sampling module specifically includes:

[0039] A first calculation unit, configured to calculate the total amount of sampled data of the target elastic distributed data set based on a preset calculation rule;

[0040] A second calculation unit, configured to calculate a maximum number of partition samples of the target elastic distributed data set according to the total amount of sampled data and a preset number of partitions;

[0041] The partition sampling unit is used to sample each partition in the target elastic distributed data set based on the total amount of sampled data, the maximum number of partition samples and a pond sampling algorithm to obtain multiple sample data.

[0042] Optionally, in a second implementation of the second aspect of the present invention, the partition sampling unit is specifically used to:

[0043] The sketch operator in the Spark operator library is called, and the total amount of sampled data and the maximum number of samples of the partition are used as input parameters of the sketch operator to perform sampling calculations to obtain multiple sample data and output parameters, wherein the output parameters are used to describe information about the sample data and information about the target elastic distributed data set.

[0044] Optionally, in a third implementation of the second aspect of the present invention, the partition sampling unit is specifically used to:

[0045] Calling the sketch operator in the Spark operator library, and taking the total amount of sampled data and the maximum number of samples of the partition as input parameters of the sketch operator to perform sampling calculation, to obtain multiple sample data and output parameters, wherein the output parameters are used to describe information of the sample data and information of the target elastic distributed dataset, and the output parameters include the total amount of data of the target elastic distributed dataset and the amount of data of each partition;

[0046] Calculate the data sampling rate corresponding to the target elastic distributed data set according to the total data volume of the target elastic distributed data set and the total amount of sampled data;

[0047] Sequentially multiplying the data sampling rate by the data volume of each partition to obtain the actual sampling quantity corresponding to each partition;

[0048] The actual sampling quantity corresponding to each partition is compared with the maximum sampling quantity of the partition, and the sampling calculation is re-performed on the unbalanced partitions where the actual sampling data volume is greater than the maximum sampling quantity of the partition.

[0049] Optionally, in a fourth implementation manner of the second aspect of the present invention, the partition boundary determination module specifically includes:

[0050] A sorting unit, used to obtain a key value and a weight corresponding to each sample data, and sort each sample data according to the size of the key value to obtain a sample data sequence;

[0051] A weight calculation unit, which calculates the average weight of each partition based on the preset number of partitions and the weight corresponding to each of the sample data;

[0052] An iteration unit is used to iterate the sample data sequence for multiple times until the partition boundaries of the preset number of partitions are determined, and the partition boundaries of each partition are obtained, wherein one iteration includes traversing the sample data in the sample data sequence, and accumulating the weights of the traversed sample data in turn, until the accumulated result is not less than the average weight of the partition, then determining that the key value corresponding to the current sample data is a partition boundary, and removing the current sample data and all sample data before the current sample data from the sample data sequence.

[0053] Optionally, in a fifth implementation of the second aspect of the present invention, the partition sorting module is specifically used to:

[0054] A sequential search unit, configured to sequentially search the partition where each piece of data in the target elastic distributed data set is located based on the partition boundary if the number of the preset partitions is not greater than a preset threshold, and sort the data in each partition obtained by the data partitioning according to the sorting method in the data sorting request to obtain an intermediate data sequence;

[0055] The binary search sorting unit is used to determine the partition where each data in the target elastic distributed data set is located based on the partition boundary and the binary search algorithm if the preset number of partitions is greater than a preset threshold, and sort the data in each partition obtained by the data partition according to the sorting method in the data sorting request to obtain an intermediate data sequence.

[0056] Optionally, in a sixth implementation manner of the second aspect of the present invention, the group global sorting module specifically includes:

[0057] A transmission unit, used for acquiring triplet data from a preset external storage medium, and transmitting the triplet data to a GPU with a multi-core structure;

[0058] A sorting unit is used to call the GPU to perform data analysis on the triple data, and to globally sort the intermediate data sequence into groups according to the grouping requirement information obtained from the data analysis to obtain a target data sequence, wherein the grouping requirement information is used to represent the grouping requirement corresponding to each partition in the target elastic distributed data set.

[0059] The third aspect of the present invention provides a Spark-based massive data sorting device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the Spark-based massive data sorting device executes the above-mentioned Spark-based massive data sorting method.

[0060] A fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned Spark-based massive data sorting method.

[0061] In the technical solution provided by the present invention, the partition boundaries of each data partition in the target elastic distributed data set are determined, and the data in the target elastic distributed data set is distributed to reasonable intervals according to the partition boundaries, thereby ensuring order between partitions; at the same time, on the basis of order between partitions, the data in each partition is grouped and sorted by obtaining the grouping sorting requirement information maintained in the external storage medium, that is, the partition boundaries are first calculated to make the partitions orderly and then the groups are sorted, so that the data in any group is distributed to each partition with equal probability, thereby avoiding the data skew problem during global group sorting, thereby improving the performance of the server. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a schematic diagram of a first embodiment of a method for sorting massive data based on Spark in an embodiment of the present invention;

[0063] Figure 2 Schematic diagram of a second embodiment of a method for sorting massive data based on Spark in an embodiment of the present invention;

[0064] Figure 3 Schematic diagram of a third embodiment of a method for sorting massive data based on Spark in an embodiment of the present invention;

[0065] Figure 4 Schematic diagram of a fourth embodiment of a method for sorting massive data based on Spark in an embodiment of the present invention;

[0066] Figure 5 Schematic diagram of an embodiment of a massive data sorting device based on Spark in an embodiment of the present invention;

[0067] Figure 6 Schematic diagram of another embodiment of a massive data sorting device based on Spark in an embodiment of the present invention;

[0068] Figure 7 The figure is a schematic diagram of an embodiment of a massive data sorting device based on Spark in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The embodiments of the present invention provide a method, apparatus, device and storage medium for sorting massive data based on Spark, which avoids data skew by distributing samples of any grouping to each partition with equal probability and performing global grouping sorting with the assistance of an external storage medium.

[0070] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0071] The server in the present invention can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0072] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of a method for sorting massive data based on Spark in an embodiment of the present invention includes:

[0073] 101. Receive a data sorting request sent by a user, and determine a target elastic distributed data set according to the data sorting request;

[0074] It is understandable that the Resilient Distributed Dataset (RDD) is the most basic abstract dataset in the Spark distributed computing engine. A general data cluster includes multiple RDDs, each of which can be composed of several data partitions, and then the data in different partitions are distributed to multiple servers for parameter operations. It is worth mentioning that only RDDs with key-value data types have partitions.

[0075] 102. Sample each partition in the target elastic distributed data set to obtain multiple sample data;

[0076] It can be understood that the server determines the corresponding RDD and triggers the Action operator of the RDD according to the data sorting request sent by the user, thereby generating the corresponding workflow (job); the server calls the execution process scheduler (DAGScheduler) to divide the job into multiple computing task groups (stages), and the server executes the first stage to trigger sampling of each partition in the target elastic distributed data set, such as random sampling, stratified sampling, etc., which is not limited in this embodiment.

[0077] 103. Calculate the partition boundary of each partition based on the preset number of partitions and each sample data;

[0078] It can be understood that the partition boundary is used to divide the partition range length of each partition in the target elastic distributed data set. Since the sample data in this embodiment are all in the form of key-value pairs, the server uses the key position of the sample data as the partition boundary to determine the range length of each partition. Preferably, the server calls the determineBounds operator to determine the key range corresponding to the sample data contained in each partition. The determineBounds operator calculates the array BoundsArray[K] used to represent the partition boundary through the sampling array, where K is the maximum key value in each partition, for example BoundsArray=[3,5], which means that the boundary between the first partition and the second partition is key=3, and the boundary between the second partition and the third partition is key=5.

[0079] 104. Partition the target elastic distributed data set according to the partition boundary of each partition, and sort the data in each partition obtained by the data partitioning according to the sorting method in the data sorting request to obtain an intermediate data sequence;

[0080] It can be understood that the server outputs the partition number of each data in the target elastic distributed data set according to the determined partition boundary, thereby partitioning all data according to the partition number, thereby mapping data within a certain range to the corresponding partition, wherein the data between partitions is ordered, that is, the key value corresponding to all data in partition A is greater than or less than the key value corresponding to all data in partition B, but the data in each partition is not ordered. In order to achieve global ordering of data, the server further performs internal sorting on each partition according to the sorting method requested in the data sorting request. The sorting method may be, for example, sorting in ascending or descending order according to the size of the data, sorting in chronological order according to the date of data generation, etc. This embodiment does not limit it.

[0081] Optionally, if the number of preset partitions is not greater than the preset threshold, the server sequentially searches for the partition where each data in the target elastic distributed data set is located based on the partition boundary; if the number of preset partitions is greater than the preset threshold, the partition where each data in the target elastic distributed data set is located is determined based on the partition boundary and the binary search algorithm. It can be understood that when the number of partitions set for the target elastic distributed data set is too large, the server uses a binary search algorithm to search the array BoundsArray[K] used to represent the partition boundary to quickly locate the partition boundary corresponding to the data to be searched. Specifically, the server starts with the middle element of the array. If the middle element is exactly the element to be searched, the search process ends; if it is greater than or less than the middle element, it searches in the half of the array that is greater than or less than the middle element, and still starts the comparison from the middle element. This search algorithm reduces the search range by half each comparison, thereby improving computational efficiency.

[0082] 105. Acquire triple data from a preset external storage medium, and group and globally sort the intermediate data sequence according to the triple data to obtain a target data sequence, wherein the triple data is used to represent the grouping requirement information corresponding to each partition.

[0083] It is understandable that the external storage medium can be a memory in the form of solid-state storage, mechanical hard disk, etc., which is not limited in this embodiment. Triple data is a nested data structure, such as Map[Pkey, Map[Gkey, count]], where Pkey is the identifier of the partition (such as the partition number), Gkey is the identifier of the group (such as the group number), and count is the amount of data in the group. The triple data in the external storage medium is called by the business personnel according to the grouping requirements to trigger the operator to split the corresponding computing task group (stage), so as to maintain the triple data. The number of groups in each partition and the samples in each group can be flexibly adjusted, and then the server completes the global sorting of the groups with the assistance of the external storage medium. Optionally, the server can also set a corresponding security storage policy for the external storage medium to ensure the security of its internal triple information. This embodiment does not limit the security storage policy.

[0084] It is understandable that in most cases, the server processes the computing scheduling task process based on the CPU (central processing unit). Optionally, the server can also call a preset GPU (graphics processing unit) to replace the CPU to perform partition global sorting on the intermediate data sequence. Specifically, the server obtains triple data from a preset external storage medium, and transmits the triple data to a multi-core GPU, and then calls the GPU to perform data analysis on the triple data, thereby obtaining the partition group sorting requirement information, such as parsing the triple data Map [1, Map [1, 10]], that is, dividing the first group with the partition number 1 into 10 data, and performing group global sorting on the intermediate data sequence according to the partition group sorting requirement information obtained by the data analysis, to obtain the target data sequence. It is understandable that when the amount of data in the intermediate data sequence reaches a certain amount, it is easy to load or even cause downtime when the server calls the CPU to sort it, and the GPU with a multi-core structure has better computing performance than the CPU. In this embodiment, the server sorts the intermediate data sequence by utilizing the performance advantage of the GPU, thereby improving computing efficiency.

[0085] In this embodiment, partition boundaries are calculated first to order the partitions and then the groups are sorted, so that data in any group is distributed to each partition with equal probability, thereby avoiding the data skew problem during global sorting of groups and improving server performance.

[0086] See also Figure 2 , the second embodiment of the method for sorting massive data based on Spark in the embodiment of the present invention includes:

[0087] 201. Receive a data sorting request sent by a user, and determine a target elastic distributed data set according to the data sorting request;

[0088] Among them, step 201 is similar to the execution steps of the above step 101, and the details are not repeated here.

[0089] 202. Calculate the total amount of sampled data of the target elastic distributed data set based on a preset calculation rule;

[0090] It is understandable that the server collects 20 data for each partition in the target elastic distributed data set by default, and the maximum amount of sample data extracted does not exceed 1e6, that is, the total amount of sampled data sampleSize = min(20*Partition, 1e6), where Partition is the number of partitions.

[0091] 203. Calculate the maximum number of partition samples of the target elastic distributed data set according to the total amount of sampled data and the preset number of partitions;

[0092] It can be understood that the server divides the total amount of sampled data in the target elastic distributed data set by the preset number of partitions to obtain the average sampling number of all partitions. On this basis, the average sampling number is increased by 3 times, which is the maximum sampling number of the partition.

[0093] 204. Based on the total amount of sampled data, the maximum number of samples in the partition, and the pool sampling algorithm, sample each partition in the target elastic distributed data set to obtain multiple sample data;

[0094] It is understandable that since the amount of data contained in each partition is unknown, the server samples each partition in the target elastic distributed data set based on the pond sampling algorithm, so that each data can be extracted with equal probability even if the overall data amount is unknown. Specifically, when the server extracts the nth data in a partition, it generates a random number p of (0, n). If p is less than Then randomly remove any data in the extracted data queue and add the currently extracted data to the extracted data queue; if p is greater than The data in the extracted data queue will continue to be retained until the data flow ends, and the data corresponding to the total amount of extracted data will be returned.

[0095] 205. Calculate a partition boundary of each partition based on the preset number of partitions and each sample data;

[0096] 206. Partition the target elastic distributed data set according to the partition boundary of each partition, and sort the data in each partition obtained by the data partitioning according to the sorting method in the data sorting request to obtain an intermediate data sequence;

[0097] 207. Acquire triple data from a preset external storage medium, and group and globally sort the intermediate data sequence according to the triple data to obtain a target data sequence, wherein the triple data is used to represent the grouping requirement information corresponding to each partition.

[0098] Among them, steps 205-207 are similar to the execution steps of the above steps 103-105, and will not be repeated here.

[0099] In this embodiment, the process of data sampling for the target elastic distributed data set is described in detail. By using the pond sampling algorithm, equal probability sampling is performed from the target elastic distributed data set, thereby ensuring sample balance.

[0100] See also Figure 3 The third embodiment of the method for sorting massive data based on Spark in the embodiment of the present invention includes:

[0101] 301. Receive a data sorting request sent by a user, and determine a target elastic distributed data set according to the data sorting request;

[0102] 302. Calculate the total amount of sampled data of the target elastic distributed data set based on a preset calculation rule;

[0103] 303. Calculate the maximum number of partition samples of the target elastic distributed data set according to the total amount of sampled data and the preset number of partitions;

[0104] Among them, steps 301-303 are similar to the execution steps of the above steps 201-203, and will not be repeated here.

[0105] 304. Call the sketch operator in the Spark operator library, and use the total amount of sampled data and the maximum number of samples of the partition as input parameters of the sketch operator to perform sampling calculation, thereby obtaining a plurality of sample data and output parameters, wherein the output parameters include the total amount of data of the target elastic distributed data set and the amount of data of each partition;

[0106] It can be understood that in this embodiment, the server calls the sketch operator to perform sampling, and the return value of the sketch operator includes, in addition to multiple sample data, a four-tuple for describing the sampling information. <R,<PartitionId,S,Z> >, where R is the total data volume of the target elastic distributed dataset (i.e., the size of the sample space), PartitionId is the identification number of the partition, S is the data volume of the partition, and Z is the number of samples extracted from the partition.

[0107] 305. Calculate a data sampling rate corresponding to the target elastic distributed dataset according to the total data volume of the target elastic distributed dataset and the total amount of sampled data;

[0108] It can be understood that the calculation rule of the data sampling rate corresponding to the target elastic distributed data set is the total amount of sampled data divided by the total amount of data in the target elastic distributed data set.

[0109] 306. Perform product operations on the data sampling rate and the data volume of each partition in turn to obtain the actual sampling quantity corresponding to each partition;

[0110] It can be understood that the server uses the amount of data in each partition as the base and the data sampling rate as the decimation coefficient, and multiplies the base by the decimation coefficient to obtain the actual number of samples in each partition.

[0111] 307. Compare the actual sampling quantity corresponding to each partition with the maximum sampling quantity of the partition, and re-sampling and calculate the unbalanced partitions whose actual sampling data volume is greater than the maximum sampling quantity of the partition;

[0112] It can be understood that when the actual sampling number of each partition is greater than the maximum sampling number of the partition (that is, 3 times the average sampling number of the partition), it means that the partition is an unbalanced partition with data skew problems, and it is then resampled.

[0113] 308. Calculate the partition boundary of each partition based on the preset number of partitions and each sample data;

[0114] 309. Partition the target elastic distributed data set according to the partition boundary of each partition, and sort the data in each partition obtained by the data partitioning according to the sorting method in the data sorting request to obtain an intermediate data sequence;

[0115] 310. Acquire triple data from a preset external storage medium, and group and globally sort the intermediate data sequence according to the triple data to obtain a target data sequence, wherein the triple data is used to represent the grouping requirement information corresponding to each partition.

[0116] Among them, steps 308-310 are similar to the execution steps of the above steps 205-207, and will not be repeated here.

[0117] In this embodiment, the process of determining an unbalanced partition and then rebalancing the partition is described in detail. By determining the unbalanced partition, balanced sampling in the partition is achieved.

[0118] See also Figure 4 The fourth embodiment of the method for sorting massive data based on Spark in the embodiment of the present invention includes:

[0119] 401. Receive a data sorting request sent by a user, and determine a target elastic distributed data set according to the data sorting request;

[0120] 402. Sampling each partition in the target elastic distributed data set to obtain multiple sample data;

[0121] Among them, steps 401-402 are similar to the execution steps of the above steps 101-102, and the details are not repeated here.

[0122] 403. Obtain the key value and weight corresponding to each sample data, and sort each sample data according to the size of the key value to obtain a sample data sequence;

[0123] It can be understood that the sample data in this embodiment are all in the form of key-value pairs, where key is the key value used to identify the data sample, and value is the weight of the data sample. Preferably, the data collection time interval is used as its weight.

[0124] 404. Calculate the average weight of the partitions based on the preset number of partitions and the weight corresponding to each sample data;

[0125] It can be understood that the average weight of a partition is the average weight of the samples contained in each partition. The server adds up the weights corresponding to all sample data and divides it by the number of partitions to obtain the average weight of the partition. For example, the sample data sequence T = [<1,0.2>, <2,0.1>, <3,0.1>, <4,0.3>, <5,0.1>, <6,0.3>] obtained by sorting the 6 sample data obtained by sampling. Now the sample data in the sequence T is distributed to 3 partitions, and the average range length (i.e., average weight) of each partition is: (0.2+0.1+0.1+0.3+0.1+0.3) / 3 = 0.36.

[0126] 405. Iterate the sample data sequence multiple times until the partition boundaries of the preset number of partitions are determined, and the partition boundaries of each partition are obtained, wherein one iteration includes traversing the sample data in the sample data sequence, and accumulating the weights of the traversed sample data in turn, until the accumulated result is not less than the average weight of the partition, then determining that the key value corresponding to the current sample data is a partition boundary, and removing the current sample data and all sample data before the current sample data from the sample data sequence;

[0127] It can be understood that the server first sorts the sample data in order of the size of the key value, then calculates the average weight of the samples contained in each partition, and then uses the average distribution principle to determine the key value range of the sample data corresponding to each partition. For example, the sample data sequence T = [<1,0.2>, <2,0.1>, <3,0.1>, <4,0.3>, <5,0.1>, <6,0.3>], the average partition weight is 0.36, the sample data sequence T is traversed, and the weight of each traversed sample data is added in turn. When traversing to the third data sample, it is calculated that 0.2+0.1+0.1=0.4>0.36 (that is, the cumulative weight is greater than the average partition weight), then the first is determined. A partition boundary is key=3, and the sample data with key<=3 is removed from the sequence T, and a new sequence T1=[<4,0.3>,<5,0.1>,<6,0.3>] is obtained. When T1 is traversed to the second sample data, 0.3+0.1=0.4>0.36 is calculated, and the second partition boundary is determined to be key=5, and the sample data with key<=5 is removed from the sequence T. At this time, the sample data allocated to partitions 1 to 3 is<Partition1,{1,2,3}><Partition2,{4,5}><Partition1,{6},> , the corresponding partition boundaries are key=3 and key=5, thus ensuring that the amount of data in each partition is uniform.

[0128] 406. Partition the target elastic distributed data set according to the partition boundary of each partition, and sort the data in each partition obtained by the data partitioning according to the sorting method in the data sorting request to obtain an intermediate data sequence;

[0129] 407. Acquire triple data from a preset external storage medium, and group and globally sort the intermediate data sequence according to the triple data to obtain a target data sequence, wherein the triple data is used to represent the grouping requirement information corresponding to each partition.

[0130] Among them, steps 406-407 are similar to the execution steps of the above steps 104-105, and will not be repeated here.

[0131] In this embodiment, the process of determining the partition boundary is described in detail, and the total weight of each partition is made to converge to an average value range, so as to achieve balanced data distribution in each partition.

[0132] The above describes the method for sorting massive data based on Spark in the embodiment of the present invention. The following describes the device for sorting massive data based on Spark in the embodiment of the present invention. Figure 5 , an embodiment of a massive data sorting device based on Spark in an embodiment of the present invention includes:

[0133] The data set determination module 501 is used to receive a data sorting request sent by a user, and determine a target elastic distributed data set according to the data sorting request;

[0134] A sampling module 502 is used to sample each partition in the target elastic distributed data set to obtain multiple sample data;

[0135] A partition boundary determination module 503, configured to calculate a partition boundary of each partition based on a preset number of partitions and each of the sample data;

[0136] A partition sorting module 504 is used to partition the target elastic distributed data set according to the partition boundary of each partition, and sort the data in each partition obtained by the data partitioning according to the sorting mode in the data sorting request to obtain an intermediate data sequence;

[0137] The grouped global sorting module 505 is used to obtain triplet data from a preset external storage medium, and group and globally sort the intermediate data sequence according to the triplet data to obtain a target data sequence, wherein the triplet data is used to represent the grouping requirement information corresponding to each partition.

[0138] In this embodiment, samples of any grouping are distributed to each partition with equal probability, and global grouping sorting is performed with the assistance of an external storage medium, thereby avoiding data skew.

[0139] See also Figure 6 Another embodiment of the mass data sorting device based on Spark in the embodiment of the present invention includes:

[0140] The data set determination module 501 is used to receive a data sorting request sent by a user, and determine a target elastic distributed data set according to the data sorting request;

[0141] A sampling module 502 is used to sample each partition in the target elastic distributed data set to obtain multiple sample data;

[0142] A partition boundary determination module 503, configured to calculate a partition boundary of each partition based on a preset number of partitions and each of the sample data;

[0143] A partition sorting module 504 is used to partition the target elastic distributed data set according to the partition boundary of each partition, and sort the data in each partition to obtain an intermediate data sequence;

[0144] The grouped global sorting module 505 is used to obtain triplet data from a preset external storage medium, and group and globally sort the intermediate data sequence according to the triplet data to obtain a target data sequence, wherein the triplet data is used to represent the grouping requirement information corresponding to each partition.

[0145] The sampling module 502 specifically includes:

[0146] The first calculation unit 5021 is used to calculate the total amount of sampled data of the target elastic distributed data set based on a preset calculation rule;

[0147] A second calculation unit 5022 is used to calculate the maximum number of partition samples of the target elastic distributed data set according to the total amount of sampled data and the preset number of partitions;

[0148] The partition sampling unit 5023 is used to sample each partition in the target elastic distributed data set based on the total amount of sampled data, the maximum number of partition samples and the pond sampling algorithm to obtain multiple sample data.

[0149] Wherein, the partition sampling unit is specifically used for:

[0150] The sketch operator in the Spark operator library is called, and the total amount of sampled data and the maximum number of samples of the partition are used as input parameters of the sketch operator to perform sampling calculations to obtain multiple sample data and output parameters, wherein the output parameters are used to describe information about the sample data and information about the target elastic distributed data set.

[0151] The partition boundary determination module 503 specifically includes:

[0152] The sorting unit 5031 is used to obtain the key value and weight corresponding to each sample data, and sort each sample data according to the size of the key value to obtain a sample data sequence;

[0153] The weight calculation unit 5032 calculates the average weight of each partition based on the preset number of partitions and the weight corresponding to each of the sample data;

[0154] The iteration unit 5033 is used to iterate the sample data sequence for multiple times until the partition boundaries of the preset number of partitions are determined, and the partition boundaries of each partition are obtained, wherein one iteration includes traversing the sample data in the sample data sequence, and accumulating the weights of the traversed sample data in turn, until the accumulated result is not less than the average weight of the partition, then determining that the key value corresponding to the current sample data is a partition boundary, and removing the current sample data and all sample data before the current sample data from the sample data sequence.

[0155] The partition sorting module 504 is specifically used for:

[0156] A sequential search unit 5041 is configured to sequentially search the partition where each piece of data in the target elastic distributed data set is located based on the partition boundary if the number of the preset partitions is not greater than a preset threshold, and sort the data in each partition obtained by the data partitioning according to the sorting method in the data sorting request to obtain an intermediate data sequence;

[0157] The binary search sorting unit 5042 is used to determine the partition where each data in the target elastic distributed data set is located based on the partition boundary and the binary search algorithm if the preset number of partitions is greater than the preset threshold, and sort the data in each partition obtained by the data partition according to the sorting method in the data sorting request to obtain an intermediate data sequence.

[0158] The group global sorting module 505 specifically includes:

[0159] The transmission unit 5051 is used to obtain triplet data from a preset external storage medium and transmit the triplet data to the GPU with a multi-core structure;

[0160] The sorting unit 5052 is used to call the GPU to perform data analysis on the triple data, and to perform grouping and global sorting on the intermediate data sequence according to the grouping requirement information obtained from the data analysis to obtain a target data sequence.

[0161] In the embodiment of the present invention, the modular design allows the hardware of each part of the Spark-based massive data sorting device to focus on the realization of a certain function, thereby maximizing the performance of the hardware. At the same time, the modular design also reduces the coupling between the modules of the device, making maintenance more convenient.

[0162] above Figure 5 and Figure 6The massive data sorting device based on Spark in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The massive data sorting device based on Spark in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0163] Figure 7 1 is a schematic diagram of the structure of a massive data sorting device based on Spark provided by an embodiment of the present invention. The massive data sorting device 700 based on Spark may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 710 (for example, one or more processors) and a memory 720, and one or more storage media 730 (for example, one or more massive storage devices) storing application programs 733 or data 732. Among them, the memory 720 and the storage medium 730 may be short-term storage or persistent storage. The program stored in the storage medium 730 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the massive data sorting device 700 based on Spark. Furthermore, the processor 710 may be configured to communicate with the storage medium 730 to execute a series of instruction operations in the storage medium 730 on the massive data sorting device 700 based on Spark.

[0164] The Spark-based massive data sorting device 700 may also include one or more power supplies 740, one or more wired or wireless network interfaces 750, one or more input and output interfaces 760, and / or one or more operating systems 731, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 7 The structure of the Spark-based massive data sorting device shown does not constitute a limitation on the Spark-based massive data sorting device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0165] The present invention also provides a massive data sorting device based on Spark, which includes a memory and a processor, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the massive data sorting method based on Spark in the above-mentioned embodiments.

[0166] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the Spark-based massive data sorting method.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0169] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for sorting massive data based on Spark, characterized in that: The Spark-based massive data sorting method includes: Receive a data sorting request sent by a user, and determine a target elastic distributed data set according to the data sorting request; Sampling each partition in the target elastic distributed data set to obtain multiple sample data; Based on the preset number of partitions and each of the sample data, calculating the partition boundary of each of the partitions; Partitioning the target elastic distributed data set according to the partition boundary of each partition; Sorting the data in each partition obtained by the data partitioning according to the sorting mode in the data sorting request to obtain an intermediate data sequence; Acquire triple data from a preset external storage medium, and group and globally sort the intermediate data sequence according to the triple data to obtain a target data sequence, wherein the triple data is used to represent the grouping requirement information corresponding to each partition; The sampling of each partition in the target elastic distributed data set to obtain a plurality of sample data includes: Based on a preset calculation rule, calculate the total amount of sampled data of the target elastic distributed data set; Calculate the maximum number of partitions of the target elastic distributed data set based on the total amount of sampled data and the preset number of partitions; Based on the total amount of sampled data, the maximum number of samples for the partitions, and the pond sampling algorithm, sampling each partition in the target elastic distributed data set to obtain a plurality of sample data; Based on the total amount of sampled data, the maximum number of samples of the partitions, and the pool sampling algorithm, each partition in the target elastic distributed data set is sampled respectively to obtain a plurality of sample data including: Calling the sketch operator in the Spark operator library, and using the total amount of sampled data and the maximum number of samples of the partition as input parameters of the sketch operator to perform sampling calculations, thereby obtaining a plurality of sample data and output parameters, wherein the output parameters are used to describe information about the sample data and information about the target elastic distributed data set; The output parameters include the total data volume of the target elastic distributed data set and the data volume of each partition. After the sketch operator in the Spark operator library is called and the total amount of sampled data and the maximum number of samples of the partition are used as input parameters of the sketch operator for sampling calculation to obtain multiple sample data and output parameters, the following is also included: Calculate the data sampling rate corresponding to the target elastic distributed data set according to the total data volume of the target elastic distributed data set and the total amount of sampled data; Sequentially multiplying the data sampling rate by the data volume of each partition to obtain the actual sampling quantity corresponding to each partition; The actual sampling quantity corresponding to each partition is compared with the maximum sampling quantity of the partition, and the sampling calculation is re-performed on the unbalanced partitions where the actual sampling data volume is greater than the maximum sampling quantity of the partition.

2. The method for sorting massive data based on Spark according to claim 1, characterized in that: The calculating the partition boundary of each partition based on the preset number of partitions and each of the sample data includes: Obtaining a key value and a weight corresponding to each of the sample data, and sorting each of the sample data according to the size of the key value to obtain a sample data sequence; Calculate the average weight of each partition based on the preset number of partitions and the weight corresponding to each of the sample data; The sample data sequence is iterated multiple times until the partition boundaries of the preset number of partitions are determined, and the partition boundaries of each partition are obtained, wherein one iteration includes traversing the sample data in the sample data sequence, and accumulating the weights of the traversed sample data in turn, until the accumulated result is not less than the average weight of the partition, then determining that the key value corresponding to the current sample data is a partition boundary, and removing the current sample data and all sample data before the current sample data from the sample data sequence.

3. The method for sorting massive data based on Spark according to claim 1, characterized in that: Partitioning the target elastic distributed data set according to the partition boundary of each partition includes: If the number of the preset partitions is not greater than the preset threshold, sequentially searching the partition where each piece of data in the target elastic distributed data set is located based on the partition boundaries; If the preset number of partitions is greater than a preset threshold, the partition where each piece of data in the target elastic distributed data set is located is determined based on the partition boundaries and a binary search algorithm.

4. The method for sorting massive data based on Spark according to any one of claims 1 to 3, characterized in that: The step of acquiring triplet data from a preset external storage medium and grouping and globally sorting the intermediate data sequence according to the triplet data to obtain the target data sequence includes: Acquire triplet data from a preset external storage medium, and transmit the triplet data to a multi-core GPU; The GPU is called to perform data analysis on the triple data, and the intermediate data sequence is grouped and globally sorted according to the grouping requirement information obtained from the data analysis to obtain a target data sequence, wherein the grouping requirement information is used to represent the grouping requirement corresponding to each partition in the target elastic distributed data set.

5. A massive data sorting device based on Spark, characterized in that: The Spark-based massive data sorting device comprises: A data set determination module, used to receive a data sorting request sent by a user, and determine a target elastic distributed data set according to the data sorting request; A sampling module, used to sample each partition in the target elastic distributed data set to obtain multiple sample data; A partition boundary determination module, used for calculating the partition boundary of each partition based on a preset number of partitions and each of the sample data; A partition sorting module, used to partition the target elastic distributed data set according to the partition boundary of each partition, and sort the data in each partition obtained by the data partitioning according to the sorting method in the data sorting request to obtain an intermediate data sequence; A grouping global sorting module, used to obtain triplet data from a preset external storage medium, and group and globally sort the intermediate data sequence according to the triplet data to obtain a target data sequence, wherein the triplet data is used to represent the grouping requirement information corresponding to each partition; The sampling module is further used to calculate the total amount of sampled data of the target elastic distributed data set based on a preset calculation rule; calculate the maximum number of partition sampling of the target elastic distributed data set according to the total amount of sampled data and the preset number of partitions; and sample each partition in the target elastic distributed data set based on the total amount of sampled data, the maximum number of partition sampling and the pond sampling algorithm to obtain multiple sample data; The sampling module is further used to call the sketch operator in the Spark operator library, and use the total amount of sampled data and the maximum number of samples of the partition as input parameters of the sketch operator to perform sampling calculations to obtain multiple sample data and output parameters, wherein the output parameters are used to describe information about the sample data and information about the target elastic distributed data set; The unbalanced partition sampling module is used to calculate the data sampling rate corresponding to the target elastic distributed data set based on the total data volume of the target elastic distributed data set and the total amount of sampled data; multiply the data sampling rate by the data volume of each partition in turn to obtain the actual sampling number corresponding to each partition; compare the actual sampling number corresponding to each partition with the maximum sampling number of the partition, and re-sample the unbalanced partitions whose actual sampling data volume is greater than the maximum sampling number of the partition.

6. A massive data sorting device based on Spark, characterized in that: The Spark-based massive data sorting device includes: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instruction in the memory so that the Spark-based massive data sorting device executes the Spark-based massive data sorting method as described in any one of claims 1-4.

7. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, a Spark-based massive data sorting method as described in any one of claims 1 to 4 is implemented.

Images