Data sorting method, device and computer equipment

Abstract

The present application discloses a method for data sorting, including: obtaining the amount of data of computing resources that can be called in the system when a specified task is executed; determining the number of quantiles according to the amount of data of computing resources and the number of data blocks into which the data set corresponding to the specified task is split; obtaining n quantiles corresponding to each data block corresponding to the specified task in descending or ascending order according to the number of quantiles, where n is a positive integer; splitting the data of each specified task into a specified number of small-order-of-magnitude data sets according to the quantiles, where the specified number is equal to n+1; inputting each small-order-of-magnitude data set into the function ROW_NUMBER() for sorting; splicing the sorted small-order-of-magnitude data sets according to the sorting of the quantiles corresponding to each small-order-of-magnitude data set to form a full sorting corresponding to the specified task. By dynamically adjusting the size of the data block, the utilization rate of the system computing resources and the high efficiency of task execution are improved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method, apparatus and computer equipment for sorting data. Background Art

[0002] When the existing ROW_NUMBER() function is used to sort data, the function of the ROW_NUMBER() function is to number the data, and the parameters involved are partition by and order by. The function of partition by is to partition the same data, and the function of order by is to sort the data in a certain order. During the sorting process, the function needs to put the data in a reduce to execute. When sorting large-scale data, it cannot meet the timeliness requirements, and sometimes even cannot complete the sorting smoothly. Therefore, it is necessary to design a large-scale data sorting method that meets production needs based on existing resources. Summary of the invention

[0003] The main purpose of this application is to provide a method for data sorting, aiming to solve the technical problem that existing sorting methods cannot meet the sorting requirements of large-scale data.

[0004] This application proposes a data sorting method, comprising:

[0005] Get the amount of computing resources available in the system when executing a specified task;

[0006] According to the amount of data of the computing resources, and Calculate the number of data blocks into which the data set corresponding to the specified task is split to determine the number of quantiles, where KBlock is the number of data blocks split into the maximum data set, S is the amount of data in the current computing resources, max_n is the maximum data set that can be sorted within the business-allowed time, and pmod() is modulo;

[0007] According to the number of quantiles, obtain quantiles corresponding to each data block corresponding to the specified task, wherein the quantiles are n in descending order, or the quantiles are n in ascending order, and n is a positive integer;

[0008] Splitting the data of each of the specified tasks into a specified number of small-order-of-magnitude data sets according to the quantiles, wherein the specified number is equal to n+1;

[0009] Input each of the small-order-of-magnitude data sets into the function ROW_NUMBER() for sorting;

[0010] The sorted small-order-of-magnitude data sets are concatenated according to the sorting of the quantiles corresponding to each of the small-order-of-magnitude data sets, to form a full sorting corresponding to the specified task.

[0011] Preferably, the step of obtaining the quantiles corresponding to the data blocks corresponding to the designated task respectively according to the number of quantiles includes:

[0012] Sampling the data of the designated task to obtain a sampling sample corresponding to the data of the designated task, wherein the order of magnitude of the sampling sample is smaller than the data volume of the data block;

[0013] Determine whether the sample meets the preset conditions;

[0014] If yes, then the value corresponding to the specified quantile of the sample is calculated using the score information and dimension information of the sample;

[0015] The value assigned to the specified quantile is used as the value assigned to the quantile corresponding to the data block.

[0016] Preferably, before the step of assigning the value of the designated quantile as the value of the quantile corresponding to the data block, the step includes:

[0017] Calculate the first conversion rate corresponding to the sampled sample according to the percentage of users who meet the business requirements in the sampled sample, and calculate the second conversion rate corresponding to the specified data block according to the percentage of users who meet the business requirements in the number of the specified data blocks;

[0018] Determining whether a difference between the first conversion rate and the second conversion rate is within a preset range;

[0019] If yes, determining that the distribution trend of the quantiles obtained according to the sample is consistent with the distribution trend of the quantiles of the specified data block;

[0020] An instruction is generated to use the value of the designated quantile as the value of the quantile corresponding to the data block.

[0021] Preferably, the data of the designated task is two-dimensional data consisting of a predicted probability score value and number information, and the step of calculating the value of the designated quantile corresponding to the sampled sample through the score information and dimension information of the sampled sample comprises:

[0022] Respectively obtain the values ​​of the fields corresponding to the predicted probability scores in the sampled samples;

[0023] Arrange the values ​​of the fields corresponding to the predicted probability scores in ascending or descending order to form a value sorting table;

[0024] The assignment ranking table is evenly divided according to n quantiles to obtain the assignment corresponding to each quantile.

[0025] Preferably, the sampling samples include specified positive samples and specified negative samples, the data of the specified task include positive samples and negative samples, and the step of judging whether the sampling samples meet the preset conditions includes:

[0026] Calculate a first ratio according to the specified number of positive samples / (specified number of positive samples+specified number of negative samples), and calculate a second ratio according to the number of positive samples / (number of positive samples+number of negative samples);

[0027] Determining whether a difference between the first ratio and the second ratio is within a preset range;

[0028] If so, it is determined that the sample meets the preset conditions, otherwise it does not meet the conditions.

[0029] Preferably, before the step of obtaining the data volume of computing resources that can be called in the system when the specified task is executed, the step includes:

[0030] Determine whether the size of each file in the specified task is greater than the processing capacity of each scoring processor Mapper;

[0031] If the size of each file in the specified task is larger than the processing capacity corresponding to each scoring processor Mapper, the files larger than the processing capacity corresponding to each scoring processor Mapper are cut according to the processing capacity corresponding to the scoring processor Mapper, and a scoring processor Mapper is allocated to each of the cut files.

[0032] Preferably, after the step of determining whether the size of each file in the specified task is greater than the processing volume corresponding to each scoring processor Mapper, the following steps are included:

[0033] If the size of each file in the specified task is not greater than the processing capacity of each scoring processor Mapper, the files smaller than the processing capacity of the scoring processor Mapper are merged to form a merged file;

[0034] The merged file is cut according to the processing volume corresponding to the scoring processor Mapper, and a scoring processor Mapper is respectively allocated to the sub-files of the merged file after cutting.

[0035] The present application also provides a data sorting device, comprising:

[0036] The first acquisition module is used to obtain the data volume of the computing resources that can be called in the system when the specified task is executed;

[0037] A computing module, configured to calculate the amount of data of the computing resources according to the amount of data of the computing resources; and Calculate the number of data blocks into which the data set corresponding to the specified task is split to determine the number of quantiles, where KBlock is the number of data blocks split into the maximum data set, S is the amount of data in the current computing resources, max_n is the maximum data set that can be sorted within the business-allowed time, and pmod() is modulo;

[0038] A second acquisition module is used to acquire the quantiles corresponding to each data block corresponding to the specified task according to the number of quantiles, wherein the quantiles are n in descending order, or the quantiles are n in ascending order, and n is a positive integer;

[0039] A splitting module, used for splitting the data of each of the specified tasks into a specified number of small-order-of-magnitude data sets according to the quantiles, wherein the specified number is equal to n+1;

[0040] A sorting module, used for inputting each of the small-order-of-magnitude data sets into a function ROW_NUMBER() for sorting;

[0041] The splicing module is used to splice the sorted small-order-of-magnitude data sets according to the sorting of the quantiles corresponding to each of the small-order-of-magnitude data sets, so as to form a full sorting corresponding to the specified task.

[0042] The present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0043] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0044] Under the limitation of existing available computing resources, this application can smoothly execute tasks by dynamically adjusting the size of data blocks, without waiting for the full amount of data to meet the task requirements, thereby improving the utilization of system computing resources and the efficiency of task execution. During the execution process, small-scale data sets are sampled, the scores corresponding to the sample quantiles are calculated, mapped to the quantiles of large data sets, and finally orderly full data is formed by scoring in blocks and merging data sets, thus realizing the sorting of large-scale data. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a method for sorting data according to an embodiment of the present application;

[0046] Figure 2 A schematic diagram of a data sorting device structure according to an embodiment of the present application;

[0047] Figure 3 A schematic diagram of the internal structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] Reference Figure 1 , a data sorting method according to an embodiment of the present application includes:

[0050] S1: Get the amount of computing resources available in the system when executing a specified task;

[0051] S2: according to the data volume of the computing resources, and Calculate the number of data blocks into which the data set corresponding to the specified task is split to determine the number of quantiles, where KBlock is the number of data blocks split into the maximum data set, S is the amount of data in the current computing resources, max_n is the maximum data set that can be sorted within the business-allowed time, and pmod() is modulo;

[0052] S3: according to the number of the quantiles, obtaining the quantiles corresponding to the data blocks corresponding to the specified task, wherein the quantiles are n in descending order, or the quantiles are n in ascending order, and n is a positive integer;

[0053] S4: splitting the data of each of the specified tasks into a specified number of small-order-of-magnitude data sets according to the quantiles, wherein the specified number is equal to n+1;

[0054] S5: Input each of the small-order-of-magnitude data sets into the function ROW_NUMBER() for sorting;

[0055] S7: Connecting the sorted small-order-of-magnitude data sets according to the sorting of the quantiles corresponding to the small-order-of-magnitude data sets to form a full sorting corresponding to the specified task.

[0056] In the embodiment of the present application, when the amount of data reaches the order of tens of millions or hundreds of millions, the data sorting cannot be completed by the sorting function ROW_NUMBER(). According to the computing power threshold of the function ROW_NUMBER(), or the CPU computing threshold of the computer, or the current callable amount of system computing resources, the data corresponding to the task is divided into data blocks. For example, the amount of data at the level of 20 million is the maximum amount of data processed at a single time, so as to ensure the smooth progress of the data sorting process, the large amount of data is processed in blocks. When the amount of data to be processed is greater than the maximum amount of data, the smoothness of data sorting is affected, and even the execution result cannot be obtained. The above-mentioned maximum amount of data can be determined according to the computing resources currently available in the system. The present application dynamically obtains the setting parameter of hive.exec.reducers.bytes.per.reducer, S is the data amount size of hive.exec.reducers.bytes.per.reducer, indicating the data amount of the current computing resources of the reducer, and uses G as the unit to determine the currently callable computing resources to adjust the size of the data block, so as to ensure the timeliness of the execution of the task, and also meet the reasonable call of computing resources. According to the size of the data volume of the specified task and the maximum amount of data that the current computing resources can support, a reasonable quantile is calculated to divide the data of the specified task into multiple small-order-magnitude data sets, and then the multiple small-order-magnitude data sets are sorted by the function ROW_NUMBER(), and finally arranged in sequence according to the ascending or descending order pre-formed by the multiple quantiles, so as to achieve orderly arrangement of the large-order-magnitude data. By sorting each small-order-magnitude data set obtained by the split separately, and then merging all the sorted small-order-magnitude data sets according to the sorting of the quantiles, an ordered large-order-magnitude data is finally formed, that is, the full amount of data, so as to achieve the sorting of the large-order-magnitude data. For example, when KBlock=5, the division is as shown in Table 1:

[0057] Table 1

[0058] KBlock Quantile Model scoring range Number of data blocks ROW_NUMBER() sorting KBlock1 0.2 [0,30) max_n Local order KBlock2 0.4 [30,50) max_n Local order KBlock3 0.6 [50,70) max_n Local order KBlock4 0.8 [70,90) max_n Local order KBlock5 - [90,100] n-4*max_n Local order

[0059] Furthermore, the step S3 of obtaining the quantiles corresponding to the data blocks corresponding to the designated task respectively according to the number of quantiles includes:

[0060] S31: sampling the data of the designated task to obtain a sampling sample corresponding to the data of the designated task, wherein the order of magnitude of the sampling sample is smaller than the data volume of the data block;

[0061] S32: Determine whether the sample meets the preset conditions;

[0062] S33: If yes, then calculate the value corresponding to the specified quantile of the sample according to the score information and dimension information of the sample;

[0063] S34: Using the value of the designated quantile as the value of the quantile corresponding to the data block.

[0064] In the embodiment of the present application, in order to ensure that the obtained quantiles are more reasonable, a sampling sample is formed by extracting sample data from a large order of magnitude data, so that the data composition law of the sampling sample is the same or similar to the composition law of the large order of magnitude data, so that the sampling sample becomes the representative data of the large order of magnitude data, and the designated quantiles of the sampling sample are obtained by analyzing and calculating the sampling sample. The number of the above-mentioned designated quantiles is the same as the number of quantiles predetermined by the large order of magnitude data.

[0065] Furthermore, before the step S34 of assigning the value of the designated quantile as the value of the quantile corresponding to the data block, the following steps are included:

[0066] S35: Calculate the first conversion rate corresponding to the sampled sample according to the statistical proportion of users who meet the business requirements in the sampled sample, and calculate the second conversion rate corresponding to the designated data block according to the statistical proportion of users who meet the business requirements in the number of designated data blocks;

[0067] S36: Determine whether the difference between the first conversion rate and the second conversion rate is within a preset range;

[0068] S37: If yes, determine that the distribution trend of the quantiles obtained according to the sample is consistent with the distribution trend of the quantiles of the specified data block;

[0069] S38: Generate an instruction to use the value of the designated quantile as the value of the quantile corresponding to the data block.

[0070] The number of quantiles is determined according to the number of data blocks, and the value of the model score corresponding to each quantile is calculated. The data set of the specified task is cut according to the model scoring scores corresponding to each quantile, and the cut data is then sorted locally, and finally an overall ordered sort is formed, making full use of the system computing resources, and the processing time also meets the business needs. The above quantiles are calculated based on the small sample data contained in the large-scale data, that is, the sampling sample. For example, if the content of the large-scale data is the predicted probability percentage, then each quantile is obtained according to the number of pre-determined quantiles. For example, there are ten quantiles, and the probabilities corresponding to each quantile are 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9 respectively. By collecting large-scale data, the scores corresponding to the quantiles of the split data are calculated. This embodiment is to reversely verify the probability percentage corresponding to the quantile of the sampling sample, which is consistent with the probability percentage corresponding to the quantile in the specified task, that is, the quantile of the small sample can be applied to the quantile of the large-scale data corresponding to the specified task, and verified by the conversion rate in the data. For example, the above conversion rate is the share of purchasing users. If the conversion rates of the two are the same or very close, they are considered to have the same quantile distribution pattern. For example, if the conversion rate in the sample is 80%, the corresponding score is 30 points, and the corresponding probability percentage is 50%, that is, half of the users score 30 points or above. If the conversion rate of the specified task is also 80%, then it is considered that in the data of the specified task, the probability percentage corresponding to the score of 30 points is 50%, and the distribution trend of the quantile is determined based on the same conversion rate.

[0071] Further, the sampling samples include specified positive samples and specified negative samples, the data of the specified task include positive samples and negative samples, and the step S32 of judging whether the sampling samples meet the preset conditions includes:

[0072] S321: Calculate a first ratio according to the specified number of positive samples / (specified number of positive samples+specified number of negative samples), and calculate a second ratio according to the number of positive samples / (number of positive samples+number of negative samples);

[0073] S322: Determine whether the difference between the first ratio and the second ratio is within a preset range;

[0074] S323: If yes, determine that the sample meets the preset conditions, otherwise it does not meet the conditions.

[0075] In the embodiment of the present application, both the large-scale data and the sampling samples include positive samples and negative samples. For example, the sampling samples include data on users purchasing insurance. The samples with purchase data are positive samples, and the samples without purchase data are negative samples. By comparing the two ratios of the number of positive samples in the large-scale data and the sampling samples to see whether they are equal or similar, if the two ratios are equal or similar, it means that the data composition law of the sampling samples is the same or similar to the composition law of the large-scale data, and the sampling samples can become representative data of the large-scale data to improve the accuracy of splitting the large-scale data by quantiles. The above preset range is within the range of 0-5%.

[0076] Furthermore, the data of the designated task is two-dimensional data consisting of a predicted probability score value and number information, and step S23 of calculating the value of the designated quantile corresponding to the sampled sample through the score information and dimension information of the sampled sample includes:

[0077] S231: respectively obtaining the values ​​of the fields corresponding to the predicted probability scores in the sampled samples;

[0078] S232: Arrange the values ​​of the fields corresponding to the predicted probability scores in ascending or descending order to form a value ranking table;

[0079] S233: Divide the value sorting table equally according to n quantiles to obtain the values ​​corresponding to each quantile.

[0080] In this embodiment, n is 10, and the values ​​corresponding to the quantiles are arranged in ascending order, from the values ​​corresponding to the quantile 0.1 to the values ​​corresponding to the quantile 0.9. Then, the data of the large order of magnitude that is less than the values ​​corresponding to the quantile 0.1 is divided into a small order of magnitude data set, and the data that is greater than or equal to the values ​​corresponding to the quantile 0.1 and less than the values ​​corresponding to the quantile 0.2 is divided into a small order of magnitude data set, and so on, the large order of magnitude data is split into 10 small order of magnitude data sets, and then each small order of magnitude data set is input into the function ROW_NUMBER() for sorting, and then each sorted small order of magnitude data set is spliced ​​and combined in turn according to the arrangement rules of the values ​​of each quantile to obtain ordered large order of magnitude data.

[0081] This embodiment takes the sorting of large-scale data with a data volume of 46 million and a dimension of 150 dimensions, one of which is the predicted probability score, as an example to explain the sorting process in detail. First, use "distribute by rand()sort byrand()limit 1000000" to randomly extract 1 million of the 46 million data to form a sampling sample. Then, the positive and negative sample rates are calculated by the value ratio of the predicted probability score dimension data in the data set, and the sample samples are determined to be representative for the large-scale data by the positive and negative sample rates. And through the predicted probability score dimension data, the preset probability scores corresponding to each quantile are calculated. As shown in Table 2 below, the sample set score is the predicted probability score. According to the preset probability scores corresponding to the 9 quantiles divided by the sampled samples, the large-scale data is divided into 10 equal parts to obtain 10 small-scale data sets, as shown in Table 3 below. Use the function ROW_NUMBER() to sort the 10 small-scale data sets respectively to form a locally ordered data set. Then, according to the order of the predicted probability scores, the above 10 locally ordered small-scale data sets are spliced ​​to achieve an overall ordered arrangement, which speeds up the sorting process. The sorting time of this scheme is 1.02 hours, which saves 3.3 hours compared to the previous time of sorting large-scale data directly in the ROW_NUMBER() function. It also avoids the problem that the ROW_NUMBER() function cannot produce a sorting result when the large-scale data level is too high, such as hundreds of millions of data.

[0082] Table 2

[0083]

[0084] Table 3

[0085]

[0086] Furthermore, before step S1 of obtaining the data volume of computing resources that can be called in the system when the specified task is executed, the method includes:

[0087] S101: Determine whether the size of each file in the specified task is greater than the processing capacity of each scoring processor Mapper;

[0088] S102: If the size of each file in the specified task is larger than the processing capacity corresponding to each scoring processor Mapper, the files larger than the processing capacity corresponding to each scoring processor Mapper are cut according to the processing capacity corresponding to the scoring processor Mapper, and a scoring processor Mapper is allocated to each of the cut files.

[0089] Furthermore, after the step S101 of determining whether the size of each file in the specified task is greater than the processing amount corresponding to each scoring processor Mapper, the following steps are included:

[0090] S103: If the size of each file in the specified task is not greater than the processing capacity of each scoring processor Mapper, merge the files smaller than the processing capacity of the scoring processor Mapper to form a merged file;

[0091] S104: cutting and merging the file according to the processing amount corresponding to the scoring processor Mapper, and assigning a scoring processor Mapper to each of the sub-files of the cut merged file.

[0092] In this embodiment, by adjusting the correspondence between the file and the scoring processor Mapper in the map stage, the hardware resources are reasonably adjusted so that the appropriate number of scoring processor Mappers can be used to ensure timeliness while saving computing resources. When the number of small input files is too large, too many scoring processor Mappers will be created, which will consume a large amount of computing resources, resulting in the inability to implement the sorting process under the existing computing resources. On the contrary, if the number of scoring processor Mappers is too small, the concurrency is too small, and the Job execution time is too long, the distributed hardware resources cannot be fully utilized, affecting the execution efficiency. The number of scoring processor Mappers is controlled by file size and file merging, so as to make full use of the existing computing resources.

[0093] Reference Figure 2 , a data sorting device according to an embodiment of the present application includes:

[0094] The first acquisition module 1 is used to obtain the data volume of the computing resources that can be called in the system when the specified task is executed;

[0095] A calculation module 2 is used to calculate the amount of data of the computing resources, and Calculate the number of data blocks into which the data set corresponding to the specified task is split to determine the number of quantiles, where KBlock is the number of data blocks split into the maximum data set, S is the amount of data in the current computing resources, max_n is the maximum data set that can be sorted within the business-allowed time, and pmod() is modulo;

[0096] A second acquisition module 3 is used to acquire the quantiles corresponding to each data block corresponding to the specified task according to the number of quantiles, wherein the quantiles are n in descending order, or the quantiles are n in ascending order, and n is a positive integer;

[0097] A splitting module 4 is used to split the data of each of the specified tasks into a specified number of small-order-of-magnitude data sets according to the quantiles, wherein the specified number is equal to n+1;

[0098] A sorting module 5 is used to input each of the small-order-of-magnitude data sets into a function ROW_NUMBER() for sorting;

[0099] The splicing module 6 is used to splice the sorted small-order-of-magnitude data sets according to the sorting of the quantiles corresponding to each of the small-order-of-magnitude data sets, so as to form a full sorting corresponding to the specified task.

[0100] In the embodiment of the present application, when the amount of data reaches the order of tens of millions or hundreds of millions, the data sorting cannot be completed by the sorting function ROW_NUMBER(). According to the computing power threshold of the function ROW_NUMBER(), or the CPU computing threshold of the computer, or the current callable amount of system computing resources, the data corresponding to the task is divided into data blocks. For example, the amount of data at the level of 20 million is the maximum amount of data processed at a single time, so as to ensure the smooth progress of the data sorting process, the large amount of data is processed in blocks. When the amount of data to be processed is greater than the maximum amount of data, the smoothness of data sorting is affected, and even the execution result cannot be obtained. The above-mentioned maximum amount of data can be determined according to the computing resources currently available in the system. The present application dynamically obtains the setting parameter of hive.exec.reducers.bytes.per.reducer, S is the data amount size of hive.exec.reducers.bytes.per.reducer, indicating the data amount of the current computing resources of the reducer, and uses G as the unit to determine the currently callable computing resources to adjust the size of the data block, so as to ensure the timeliness of the execution of the task, and also meet the reasonable call of computing resources. According to the size of the data volume of the specified task and the maximum amount of data that the current computing resources can support, a reasonable quantile is calculated to divide the data of the specified task into multiple small-order-magnitude data sets, and then the multiple small-order-magnitude data sets are sorted by the function ROW_NUMBER(), and finally arranged in sequence according to the ascending or descending order pre-formed by the multiple quantiles, so as to achieve orderly arrangement of the large-order-magnitude data. By sorting each small-order-magnitude data set obtained by the split separately, and then merging all the sorted small-order-magnitude data sets according to the sorting of the quantiles, an ordered large-order-magnitude data is finally formed, that is, the full amount of data, so as to achieve the sorting of the large-order-magnitude data. For example, when KBlock=5, the division is as shown in Table 1:

[0101] Table 1

[0102] KBlock Quantile Model scoring range Number of data blocks ROW_NUMBER() sorting KBlock1 0.2 [0,30) max_n Local order KBlock2 0.4 [30,50) max_n Local order KBlock3 0.6 [50,70) max_n Local order KBlock4 0.8 [70,90) max_n Local order KBlock5 - [90,100] n-4*max_n Local order

[0103] Furthermore, the second acquisition module 3 includes:

[0104] A sampling unit, used for sampling the data of the specified task to obtain a sampling sample corresponding to the data of the specified task, wherein the order of magnitude of the sampling sample is smaller than the data volume of the data block;

[0105] A first judging unit, used to judge whether the sample meets a preset condition;

[0106] A first calculation unit, configured to calculate a value corresponding to a specified quantile of the sampled sample by using the score information and dimension information of the sampled sample if a preset condition is met;

[0107] As a unit, it is used to assign the value of the specified quantile as the value of the quantile corresponding to the data block.

[0108] In the embodiment of the present application, in order to ensure that the obtained quantiles are more reasonable, a sampling sample is formed by extracting sample data from a large order of magnitude data, so that the data composition law of the sampling sample is the same or similar to the composition law of the large order of magnitude data, so that the sampling sample becomes the representative data of the large order of magnitude data, and the designated quantiles of the sampling sample are obtained by analyzing and calculating the sampling sample. The number of the above-mentioned designated quantiles is the same as the number of quantiles predetermined by the large order of magnitude data.

[0109] Furthermore, the second acquisition module 3 includes:

[0110] A second calculation unit is used to calculate the first conversion rate corresponding to the sampled sample according to the statistical proportion of users who meet the business requirements in the sampled sample, and calculate the second conversion rate corresponding to the specified data block according to the statistical proportion of users who meet the business requirements in the number of the specified data blocks;

[0111] a second determination unit, configured to determine whether a difference between the first conversion rate and the second conversion rate is within a preset range;

[0112] A determination unit, configured to determine that the distribution trend of the quantile points obtained according to the sampled data is consistent with the distribution trend of the quantile points of the designated data block if it is within a preset range;

[0113] A generating unit is used to generate an instruction to use the value of the specified quantile as the value of the quantile corresponding to the data block.

[0114] The number of quantiles is determined according to the number of data blocks, and the value of the model score corresponding to each quantile is calculated. The data set of the specified task is cut according to the model scoring scores corresponding to each quantile, and the cut data is then sorted locally, and finally an overall ordered sort is formed, making full use of the system computing resources, and the processing time also meets the business needs. The above quantiles are calculated based on the small sample data contained in the large-scale data, that is, the sampling sample. For example, if the content of the large-scale data is the predicted probability percentage, then each quantile is obtained according to the number of pre-determined quantiles. For example, there are ten quantiles, and the probabilities corresponding to each quantile are 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9 respectively. By collecting large-scale data, the scores corresponding to the quantiles of the split data are calculated. This embodiment is to reversely verify the probability percentage corresponding to the quantile of the sampling sample, which is consistent with the probability percentage corresponding to the quantile in the specified task, that is, the quantile of the small sample can be applied to the quantile of the large-scale data corresponding to the specified task, and verified by the conversion rate in the data. For example, the above conversion rate is the share of purchasing users. If the conversion rates of the two are the same or very close, they are considered to have the same quantile distribution pattern. For example, if the conversion rate in the sample is 80%, the corresponding score is 30 points, and the corresponding probability percentage is 50%, that is, half of the users score 30 points or above. If the conversion rate of the specified task is also 80%, then it is considered that in the data of the specified task, the probability percentage corresponding to the score of 30 points is 50%, and the distribution trend of the quantile is determined based on the same conversion rate.

[0115] Furthermore, the sampling samples include specified positive samples and specified negative samples, the data of the specified task include positive samples and negative samples, and the first judgment unit includes:

[0116] a calculation subunit, configured to calculate a first ratio according to a specified number of positive samples / (specified number of positive samples+specified number of negative samples), and to calculate a second ratio according to the number of positive samples / (number of positive samples+number of negative samples);

[0117] A judging subunit, used to judge whether the difference between the first ratio and the second ratio is within a preset range;

[0118] The determination subunit is used to determine that the sample meets the preset condition if it is within the preset range, otherwise it does not meet the preset condition.

[0119] In the embodiment of the present application, both the large-scale data and the sampling samples include positive samples and negative samples. For example, the sampling samples include data on users purchasing insurance. The samples with purchase data are positive samples, and the samples without purchase data are negative samples. By comparing the two ratios of the number of positive samples in the large-scale data and the sampling samples to see whether they are equal or similar, if the two ratios are equal or similar, it means that the data composition law of the sampling samples is the same or similar to the composition law of the large-scale data, and the sampling samples can become representative data of the large-scale data to improve the accuracy of splitting the large-scale data by quantiles. The above preset range is within the range of 0-5%.

[0120] Furthermore, the data of the designated task is two-dimensional data consisting of a predicted probability score value and number information, and the first calculation unit includes:

[0121] An acquisition subunit, used to respectively acquire the values ​​of the fields corresponding to the predicted probability scores in the sampled samples;

[0122] An arranging subunit, used to arrange the values ​​of the fields corresponding to the predicted probability scores in ascending or descending order to form a value ranking table;

[0123] A subunit is obtained, which is used to equally divide the assignment ranking table according to n quantiles to obtain the assignments corresponding to each quantile.

[0124] In this embodiment, n is 10, and the values ​​corresponding to the quantiles are arranged in ascending order, from the values ​​corresponding to the quantile 0.1 to the values ​​corresponding to the quantile 0.9. Then, the data of the large order of magnitude that is less than the values ​​corresponding to the quantile 0.1 is divided into a small order of magnitude data set, and the data that is greater than or equal to the values ​​corresponding to the quantile 0.1 and less than the values ​​corresponding to the quantile 0.2 is divided into a small order of magnitude data set, and so on, the large order of magnitude data is split into 10 small order of magnitude data sets, and then each small order of magnitude data set is input into the function ROW_NUMBER() for sorting, and then each sorted small order of magnitude data set is spliced ​​and combined in turn according to the arrangement rules of the values ​​of each quantile to obtain ordered large order of magnitude data.

[0125] This embodiment takes the sorting of large-scale data with a data volume of 46 million and a dimension of 150 dimensions, one of which is the predicted probability score, as an example to explain the sorting process in detail. First, use "distribute by rand()sort byrand()limit 1000000" to randomly extract 1 million of the 46 million data to form a sampling sample. Then, the positive and negative sample rates are calculated by the value ratio of the predicted probability score dimension data in the data set, and the sample samples are determined to be representative for the large-scale data by the positive and negative sample rates. And through the predicted probability score dimension data, the preset probability scores corresponding to each quantile are calculated. As shown in Table 2 below, the sample set score is the predicted probability score. According to the preset probability scores corresponding to the 9 quantiles divided by the sampled samples, the large-scale data is divided into 10 equal parts to obtain 10 small-scale data sets, as shown in Table 3 below. Use the function ROW_NUMBER() to sort the 10 small-scale data sets respectively to form a locally ordered data set. Then, according to the order of the predicted probability scores, the above 10 locally ordered small-scale data sets are spliced ​​to achieve an overall ordered arrangement, which speeds up the sorting process. The sorting time of this scheme is 1.02 hours, which saves 3.3 hours compared to the previous time of sorting large-scale data directly in the ROW_NUMBER() function. It also avoids the problem that the ROW_NUMBER() function cannot produce a sorting result when the large-scale data level is too high, such as hundreds of millions of data.

[0126] Table 2

[0127]

[0128] Table 3

[0129]

[0130] Furthermore, the data sorting device comprises:

[0131] A judgment module, used to judge whether the size of each file in the specified task is greater than the processing capacity corresponding to each scoring processor Mapper;

[0132] The first cutting module is used to cut the files larger than the processing capacity corresponding to each scoring processor Mapper according to the processing capacity corresponding to the scoring processor Mapper if the size of each file in the specified task is larger than the processing capacity corresponding to each scoring processor Mapper, and allocate a scoring processor Mapper to each of the cut files.

[0133] Furthermore, the data sorting device comprises:

[0134] A merging module, configured to merge files smaller than the processing capacity of the scoring processor Mapper to form a merged file if the size of each file in the specified task is not larger than the processing capacity of each scoring processor Mapper;

[0135] The second cutting module is used to cut the merged file according to the processing volume corresponding to the scoring processor Mapper, and to allocate a scoring processor Mapper to each sub-file of the merged file after cutting.

[0136] In this embodiment, by adjusting the correspondence between the file and the scoring processor Mapper in the map stage, the hardware resources are reasonably adjusted so that the appropriate number of scoring processor Mappers can be used to ensure timeliness while saving computing resources. When the number of small input files is too large, too many scoring processor Mappers will be created, which will consume a large amount of computing resources, resulting in the inability to implement the sorting process under the existing computing resources. On the contrary, if the number of scoring processor Mappers is too small, the concurrency is too small, and the Job execution time is too long, the distributed hardware resources cannot be fully utilized, affecting the execution efficiency. The number of scoring processor Mappers is controlled by file size and file merging, so as to make full use of the existing computing resources.

[0137] Reference Figure 3 In an embodiment of the present application, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store all data required for the data sorting process. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the method of data sorting is implemented.

[0138] The processor executes the data sorting method, comprising: obtaining the data volume of the computing resources that can be called in the system when the specified task is executed; according to the data volume of the computing resources, and Calculate the number of data blocks into which the data set corresponding to the specified task is split to determine the number of quantiles, where KBlock is the number of data blocks split into the maximum data set, S is the data volume of the current computing resources, max_n is the maximum data set that can be sorted within the business-allowed time, and pmod() is the modulus; obtain the quantiles corresponding to each data block corresponding to the specified task according to the number of quantiles, where the quantiles are n in descending order, or n in ascending order, and n is a positive integer; split the data of each specified task into a specified number of small-order-of-magnitude data sets according to the quantiles, where the specified number is equal to n+1; input each of the small-order-of-magnitude data sets into the function ROW_NUMBER() for sorting; splice the sorted small-order-of-magnitude data sets according to the sorting of the quantiles corresponding to each of the small-order-of-magnitude data sets to form a full sorting corresponding to the specified task.

[0139] The above-mentioned computer equipment, under the limitation of the existing available computing resources, can smoothly execute the task by dynamically adjusting the size of the data block, without waiting for the full amount of data to meet the task requirements, thereby improving the utilization rate of the system computing resources and the efficiency of task execution. During the execution process, the small-scale data set is sampled, the scores corresponding to the sample quantiles are calculated, mapped to the quantiles of the large data set, and the ordered full data is finally formed by scoring the blocks and merging the data sets, thus realizing the sorting of large-scale data.

[0140] In one embodiment, the processor obtains the steps of obtaining the quantiles corresponding to each data block corresponding to the specified task according to the number of quantiles, including: sampling the data of the specified task to obtain a sampling sample corresponding to the data of the specified task, wherein the order of magnitude of the sampling sample is smaller than the amount of data in the data block; determining whether the sampling sample meets a preset condition; if so, calculating the value corresponding to the specified quantile of the sampling sample through the score information and dimension information of the sampling sample; and using the value of the specified quantile as the value of the quantile corresponding to the data block.

[0141] In one embodiment, before the processor uses the assigned value of the designated quantile as the assigned value of the quantile corresponding to the data block, the processor includes: calculating a first conversion rate corresponding to the sampled sample based on the statistical proportion of users who meet the business requirements in the sampled sample, and calculating a second conversion rate corresponding to the designated data block based on the statistical proportion of users who meet the business requirements in the number of designated data blocks; determining whether the difference between the first conversion rate and the second conversion rate is within a preset range; if so, determining whether the quantile distribution trend obtained from the sampled sample is consistent with the quantile distribution trend of the designated data block; and generating an instruction to use the assigned value of the designated quantile as the assigned value of the quantile corresponding to the data block.

[0142] In one embodiment, the data of the designated task is two-dimensional data consisting of a predicted probability score value and number information. The processor calculates the assignment of the designated quantile corresponding to the sampled sample through the score information and dimension information of the sampled sample, including: respectively obtaining the assignment of the field corresponding to the predicted probability score value in the sampled sample; arranging the assignment of the field corresponding to each predicted probability score value in ascending or descending order to form an assignment sorting table; and equally dividing the assignment sorting table according to n quantiles to obtain the assignment corresponding to each quantile.

[0143] In one embodiment, the sampling samples include specified positive samples and specified negative samples, the data of the specified task include positive samples and negative samples, and the step in which the processor determines whether the sampling samples meet the preset conditions includes: calculating a first ratio based on the specified number of positive samples / (specified number of positive samples + specified number of negative samples), and calculating a second ratio based on the number of positive samples / (number of positive samples + number of negative samples); determining whether the difference between the first ratio and the second ratio is within a preset range; if so, determining that the sampling samples meet the preset conditions, otherwise not.

[0144] In one embodiment, before the step of the processor obtaining the data volume of the computing resources that can be called in the system when executing the specified task, it includes: judging whether the size of each file in the specified task is larger than the processing volume corresponding to each scoring processor Mapper; if the size of each file in the specified task is larger than the processing volume corresponding to each scoring processor Mapper, cutting the files larger than the processing volume corresponding to each scoring processor Mapper according to the processing volume corresponding to the scoring processor Mapper, and assigning a scoring processor Mapper to each of the cut files.

[0145] In one embodiment, after the step of determining whether the size of each file in the specified task is greater than the processing capacity corresponding to each scoring processor Mapper, the processor further comprises: if the size of each file in the specified task is not greater than the processing capacity corresponding to each scoring processor Mapper, merging files smaller than the processing capacity of the scoring processor Mapper to form a merged file; cutting the merged file according to the processing capacity corresponding to the scoring processor Mapper, and assigning a scoring processor Mapper to each sub-file of the merged file after cutting.

[0146] Those skilled in the art will understand that Figure 3 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0147] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a method for implementing data sorting is provided, comprising: obtaining the data volume of computing resources that can be called in the system when a specified task is executed; according to the data volume of the computing resources, and Calculate the number of data blocks into which the data set corresponding to the specified task is split to determine the number of quantiles, where KBlock is the number of data blocks split into the maximum data set, S is the data volume of the current computing resources, max_n is the maximum data set that can be sorted within the business-allowed time, and pmod() is the modulus; obtain the quantiles corresponding to each data block corresponding to the specified task according to the number of quantiles, where the quantiles are n in descending order, or n in ascending order, and n is a positive integer; split the data of each specified task into a specified number of small-order-of-magnitude data sets according to the quantiles, where the specified number is equal to n+1; input each of the small-order-of-magnitude data sets into the function ROW_NUMBER() for sorting; splice the sorted small-order-of-magnitude data sets according to the sorting of the quantiles corresponding to each of the small-order-of-magnitude data sets to form a full sorting corresponding to the specified task.

[0148] The above computer-readable storage medium, under the limitation of the existing available computing resources, dynamically adjusts the size of the data block, so that the task can be smoothly executed without waiting for the full amount of data to meet the task requirements, thereby improving the utilization of system computing resources and the efficiency of task execution. During the execution process, the small-scale data set is sampled, the scores corresponding to the sample quantiles are calculated, mapped to the quantiles of the large data set, and the ordered full data is finally formed by scoring the blocks and merging the data sets, thus realizing the sorting of large-scale data.

[0149] In one embodiment, the processor obtains the steps of obtaining the quantiles corresponding to each data block corresponding to the specified task according to the number of quantiles, including: sampling the data of the specified task to obtain a sampling sample corresponding to the data of the specified task, wherein the order of magnitude of the sampling sample is smaller than the amount of data in the data block; determining whether the sampling sample meets a preset condition; if so, calculating the value corresponding to the specified quantile of the sampling sample through the score information and dimension information of the sampling sample; and using the value of the specified quantile as the value of the quantile corresponding to the data block.

[0150] In one embodiment, before the processor uses the assigned value of the designated quantile as the assigned value of the quantile corresponding to the data block, the processor includes: calculating a first conversion rate corresponding to the sampled sample based on the statistical proportion of users who meet the business requirements in the sampled sample, and calculating a second conversion rate corresponding to the designated data block based on the statistical proportion of users who meet the business requirements in the number of designated data blocks; determining whether the difference between the first conversion rate and the second conversion rate is within a preset range; if so, determining whether the quantile distribution trend obtained from the sampled sample is consistent with the quantile distribution trend of the designated data block; and generating an instruction to use the assigned value of the designated quantile as the assigned value of the quantile corresponding to the data block.

[0151] In one embodiment, the data of the designated task is two-dimensional data consisting of a predicted probability score value and number information. The processor calculates the assignment of the designated quantile corresponding to the sampled sample through the score information and dimension information of the sampled sample, including: respectively obtaining the assignment of the field corresponding to the predicted probability score value in the sampled sample; arranging the assignment of the field corresponding to each predicted probability score value in ascending or descending order to form an assignment sorting table; and equally dividing the assignment sorting table according to n quantiles to obtain the assignment corresponding to each quantile.

[0152] In one embodiment, the sampling samples include specified positive samples and specified negative samples, the data of the specified task include positive samples and negative samples, and the step in which the processor determines whether the sampling samples meet the preset conditions includes: calculating a first ratio based on the specified number of positive samples / (specified number of positive samples + specified number of negative samples), and calculating a second ratio based on the number of positive samples / (number of positive samples + number of negative samples); determining whether the difference between the first ratio and the second ratio is within a preset range; if so, determining that the sampling samples meet the preset conditions, otherwise not.

[0153] In one embodiment, before the step of the processor obtaining the data volume of the computing resources that can be called in the system when executing the specified task, it includes: judging whether the size of each file in the specified task is larger than the processing volume corresponding to each scoring processor Mapper; if the size of each file in the specified task is larger than the processing volume corresponding to each scoring processor Mapper, cutting the files larger than the processing volume corresponding to each scoring processor Mapper according to the processing volume corresponding to the scoring processor Mapper, and assigning a scoring processor Mapper to each of the cut files.

[0154] In one embodiment, after the step of determining whether the size of each file in the specified task is greater than the processing capacity corresponding to each scoring processor Mapper, the processor further comprises: if the size of each file in the specified task is not greater than the processing capacity corresponding to each scoring processor Mapper, merging files smaller than the processing capacity of the scoring processor Mapper to form a merged file; cutting the merged file according to the processing capacity corresponding to the scoring processor Mapper, and assigning a scoring processor Mapper to each sub-file of the merged file after cutting.

[0155] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0156] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, device, article or method including the element.

[0157] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for sorting data, characterized in that: include: Get the amount of computing resources available in the system when executing a specified task; According to the amount of data of the computing resources, and Calculate the number of data blocks into which the data set corresponding to the specified task is split to determine the number of quantiles, where KBlock is the number of data blocks split into the maximum data set, S is the amount of data in the current computing resources, max_n is the maximum data set that can be sorted within the business-allowed time, and pmod() is modulo; According to the number of quantiles, obtain quantiles corresponding to each data block corresponding to the specified task, wherein the quantiles are n in descending order, or the quantiles are n in ascending order, and n is a positive integer; Splitting the data of each of the specified tasks into a specified number of small-order-of-magnitude data sets according to the quantiles, wherein the specified number is equal to n+1; Input each of the small-order-of-magnitude data sets into the function ROW_NUMBER() for sorting; The sorted small-order-of-magnitude data sets are concatenated according to the sorting of the quantiles corresponding to the small-order-of-magnitude data sets, to form a full sorting corresponding to the specified task; The step of obtaining the quantiles corresponding to the data blocks corresponding to the specified task respectively according to the number of quantiles includes: Sampling the data of the designated task to obtain a sampling sample corresponding to the data of the designated task, wherein the order of magnitude of the sampling sample is smaller than the data volume of the data block; Determine whether the sample meets the preset conditions; If yes, then the value corresponding to the specified quantile of the sample is calculated using the score information and dimension information of the sample; Using the value of the specified quantile as the value of the quantile corresponding to the data block; Before the step of assigning the value of the designated quantile as the value of the quantile corresponding to the data block, the method includes: Calculate the first conversion rate corresponding to the sampled sample according to the percentage of users who meet the business requirements in the sampled sample, and calculate the second conversion rate corresponding to the specified data block according to the percentage of users who meet the business requirements in the number of specified data blocks; Determining whether a difference between the first conversion rate and the second conversion rate is within a preset range; If yes, determining that the distribution trend of the quantiles obtained according to the sample is consistent with the distribution trend of the quantiles of the specified data block; An instruction is generated to use the value of the designated quantile as the value of the quantile corresponding to the data block.

2. The method for data sorting according to claim 1, characterized in that: The data of the designated task is two-dimensional data consisting of a predicted probability score value and number information, and the step of calculating the value of the designated quantile corresponding to the sampled sample by using the score information and dimension information of the sampled sample comprises: Respectively obtain the values ​​of the fields corresponding to the predicted probability scores in the sampled samples; Arrange the values ​​of the fields corresponding to the predicted probability scores in ascending or descending order to form a value sorting table; The assignment ranking table is evenly divided according to n quantiles to obtain the assignment corresponding to each quantile.

3. The method for data sorting according to claim 1, characterized in that: The sampling samples include a specified positive sample and a specified negative sample, the data of the specified task includes a positive sample and a negative sample, and the step of judging whether the sampling samples meet the preset conditions includes: Calculate a first ratio according to the specified number of positive samples / (specified number of positive samples+specified number of negative samples), and calculate a second ratio according to the number of positive samples / (number of positive samples+number of negative samples); Determining whether a difference between the first ratio and the second ratio is within a preset range; If so, it is determined that the sample meets the preset conditions, otherwise it does not meet the conditions.

4. The method for data sorting according to claim 1, characterized in that: Before the step of obtaining the data volume of computing resources that can be called in the system when the specified task is executed, the method includes: Determine whether the size of each file in the specified task is greater than the processing capacity of each scoring processor Mapper; If the size of each file in the specified task is larger than the processing capacity corresponding to each scoring processor Mapper, the files larger than the processing capacity corresponding to each scoring processor Mapper are cut according to the processing capacity corresponding to the scoring processor Mapper, and a scoring processor Mapper is allocated to each of the cut files.

5. The method for data sorting according to claim 4, characterized in that: After the step of determining whether the size of each file in the specified task is greater than the processing amount corresponding to each scoring processor Mapper, the method includes: If the size of each file in the specified task is not greater than the processing capacity of each scoring processor Mapper, the files smaller than the processing capacity of the scoring processor Mapper are merged to form a merged file; The merged file is cut according to the processing volume corresponding to the scoring processor Mapper, and a scoring processor Mapper is respectively allocated to the sub-files of the merged file after cutting.

6. A data sorting device, characterized in that: A method for implementing data sorting as claimed in any one of claims 1 to 5, comprising: The first acquisition module is used to obtain the data volume of the computing resources that can be called in the system when the specified task is executed; A computing module, for computing the amount of data of the computing resources, and Calculate the number of data blocks into which the data set corresponding to the specified task is split to determine the number of quantiles, where KBblock is the number of data blocks into which the maximum data set is split, S is the amount of data in the current computing resources, max_n is the maximum data set that can be sorted within the time allowed by the business, and pmod() is modulo; A second acquisition module is used to acquire the quantiles corresponding to each data block corresponding to the specified task according to the number of quantiles, wherein the quantiles are n in descending order, or the quantiles are n in ascending order, and n is a positive integer; A splitting module, used for splitting the data of each of the specified tasks into a specified number of small-order-of-magnitude data sets according to the quantiles, wherein the specified number is equal to n+1; A sorting module, used for inputting each of the small-order-of-magnitude data sets into a function ROW_NUMBER() for sorting; The splicing module is used to splice the sorted small-order-of-magnitude data sets according to the sorting of the quantiles corresponding to each of the small-order-of-magnitude data sets, so as to form a full sorting corresponding to the specified task.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images