A method and apparatus for ordering data

Abstract

The disclosure provides a data sorting method and device, and relates to the technical field of big data. The specific implementation of the method comprises the following steps: receiving a data sorting request; wherein the data sorting request comprises to-be-sorted data; randomly selecting quantile reference data from the to-be-sorted data, determining the mapping quantile value of the corresponding mapping bucket of the to-be-sorted data from the quantile reference data according to a preset bucket size; distributing the to-be-sorted data to each mapping bucket according to the mapping quantile value to obtain the in-bucket data corresponding to each mapping bucket; sorting the in-bucket data of each mapping bucket to determine the global serial number of each in-bucket data; and combining the global serial number and the in-bucket data to obtain a sorting result in response to the data sorting request. The implementation can obtain a global bucketing target close to complete uniformity at low cost, greatly improve the global sorting efficiency, relieve the node resource and computing pressure, and ensure the stability of the data processing system.

Description

Technical Field

[0001] This disclosure relates to the field of big data technology, and in particular to a data sorting method and apparatus. Background Technology

[0002] The utilization of big data refers to the process of cleaning, transforming, filtering, classifying, summarizing, comparing, accumulating, deleting, counting, deduplication, mean, variance, standard deviation, normalization, sorting and other methods on massive amounts of data to obtain a variety of statistical indicators to provide decision-making references for users. The dimensions of the indicators can be selectively defined as needed.

[0003] In existing data sorting processes, equal-length bucketing is typically used to evenly divide the data value into multiple segments, with each segment corresponding to a different bucket node. During sorting, multiple read nodes judge the data they read according to the data value and send it to the bucket node corresponding to the different segments. Then, each bucket node sorts the received data.

[0004] However, due to the massive scale of big data and the different distribution patterns of data values ​​in different scenarios (such as normal, U-shaped, J-shaped distributions, etc.), equal-length bucketing will cause the data to be concentrated in some bucket nodes, resulting in excessively long sorting time, low sorting efficiency, and in severe cases, even node crashes due to insufficient memory in the bucket nodes, causing the sorting task to fail and the nodes to be unstable. Summary of the Invention

[0005] In view of this, the present disclosure provides a data sorting method and apparatus that can solve the problems of traditional equal-length bucketing, which results in excessively concentrated data, leading to excessively long sorting time, low sorting efficiency, and in severe cases, node crashes that prevent the sorting task from being executed, resulting in poor node stability.

[0006] To achieve the above objectives, according to one aspect of this disclosure, a data sorting method is provided, comprising:

[0007] Receive a data sorting request; wherein the data sorting request includes data to be sorted;

[0008] Randomly select quantile reference data from the data to be sorted, and determine the mapping quantile value of the mapping bucket corresponding to the data to be sorted from the quantile reference data according to the preset bucket size;

[0009] According to the mapping quantile value, the data to be sorted is distributed to each of the mapping buckets to obtain the data in each of the mapping buckets;

[0010] The data within each of the mapped buckets is sorted, and a global sequence number for each piece of data within the bucket is determined. In response to the data sorting request, the global sequence number and the data within the bucket are combined to obtain the sorting result.

[0011] According to another aspect of this disclosure, a data sorting apparatus is provided, comprising:

[0012] A receiving module is configured to receive a data sorting request; wherein the data sorting request includes data to be sorted;

[0013] The quantile determination module is used to randomly select quantile determination data from the data to be sorted, and determine the mapping quantile value of the mapping bucket corresponding to the data to be sorted from the quantile determination data according to the preset bucket size.

[0014] The mapping module is used to distribute the data to be sorted to each of the mapping buckets according to the mapping quantile value, so as to obtain the data in each of the mapping buckets.

[0015] The sorting module is used to sort the data within each of the mapped buckets, determine the global sequence number of each data item within the bucket, and, in response to the data sorting request, combine the global sequence number and the data within the bucket to obtain the sorting result.

[0016] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0017] Processor; and

[0018] Stored program memory,

[0019] The program includes instructions that, when executed by the processor, cause the processor to perform a sorting method for the data.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform a sorting method for the data.

[0021] One or more technical solutions provided in this application embodiment obtain the mapping quantile values ​​of the mapping buckets of the data to be sorted by random sampling, and simulate the global mapping quantiles with the definite quantiles of random sampling. This can improve the global sorting of data with a scale of over one billion. By using random sampling to provide mapping quantile value references, it can achieve approximately uniform bucketing of non-uniformly distributed data, avoid the data concentration problem of traditional bucketing, improve global sorting efficiency, greatly shorten sorting time, reduce sorting costs, improve global sorting efficiency, and balance node resources and pressure. Attached Figure Description

[0022] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A flowchart illustrating a method for sorting data according to an exemplary embodiment of the present disclosure is shown;

[0024] Figure 2 A flowchart is shown of a method for determining the mapping quantile value of mapping buckets according to an exemplary embodiment of the present disclosure;

[0025] Figure 3 A flowchart illustrating a method for distributing unsorted data according to an exemplary embodiment of the present disclosure is shown;

[0026] Figure 4 A flowchart illustrating a method for determining sorting results according to exemplary embodiments of the present disclosure is shown;

[0027] Figure 5 A schematic block diagram of a data sorting apparatus according to an exemplary embodiment of the present disclosure is shown;

[0028] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0029] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0030] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0031] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "in embodiments of this disclosure" means "at least one embodiment". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., mentioned in this disclosure are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0032] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0033] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0034] Taking the use of big data in testing scenarios as an example, in order to improve the accuracy of the model, the data user needs to purchase a large amount of data from the data owner for testing. However, in order to prevent data leakage, the data owner usually only provides limited data to the data user, such as the global sequence number of the model.

[0035] However, the scale of data that needs to be sorted in big data utilization is enormous, often far exceeding one billion, and the numerical distribution is uneven, exhibiting various forms such as a normal distribution with "high in the middle and low at both ends," a U-shaped distribution with "low in the middle and high at both ends," and a J-shaped distribution with "gradually increasing" or "gradually decreasing." When sorting using equal-length bucketing, the amount of data in each bucket in the dataset becomes too large (for example, the model divides the data values ​​evenly into "(0,10], (10,20], (20,30], ..., (90,100)"). The data segments, corresponding to the data volumes of "1 million, 4 million, 4 million, 12 million, 100 million, 500 million, 350 million, 18 million, 10 million, and 1 million", are concentrated in the bucket nodes corresponding to (50, 60], (60, 70], and (70, 80]. The sorting process is very slow, and may even lead to insufficient memory or sorting failure due to the large amount of data in the bucket nodes. As a result, the existing equal-length bucketing method is too time-consuming and inefficient when applied to big data.

[0036] The data sorting method disclosed herein determines the quantile values ​​of each bucket through random sampling, which are then mapped to the mapping buckets of the data to be sorted. After distributing the data to be sorted to each mapping bucket according to the quantile values, the global sequence number of the data in each bucket is determined based on the bucket number of the mapping bucket and the amount of data in the previous mapping bucket. This achieves the goal of obtaining a near-perfectly uniform global bucketing at low cost, greatly improving the global sorting efficiency, alleviating node resource and computational pressure, and ensuring the stability of the data processing system.

[0037] The present disclosure is described below with reference to the accompanying drawings.

[0038] Figure 1 A flowchart illustrating a data sorting method according to an exemplary embodiment of the present disclosure is shown, such as... Figure 1 As shown, the data sorting method disclosed herein includes the following steps:

[0039] In this embodiment of the disclosure, the data sorting method is executed by a sorting server.

[0040] Step S101: Receive a data sorting request; wherein the data sorting request includes data to be sorted.

[0041] In this embodiment of the disclosure, the data to be sorted in different application scenarios can be different. For example, the data sorting request is a data sorting request for a test scenario, and the data to be sorted includes the model ID and model score of the test model; or the data sorting request is a data sorting request for a financial scenario, and the data to be sorted includes the user ID and user rating score.

[0042] Furthermore, the data sorting method disclosed herein is applicable to non-uniformly distributed big data sorting scenarios, especially for non-uniformly distributed data with a data volume of more than one billion, and can improve the sorting efficiency of non-uniformly distributed data with a scale of more than one billion.

[0043] Step S102: Randomly select quantile reference data from the data to be sorted, and determine the mapping quantile value of the mapping bucket corresponding to the data to be sorted from the quantile reference data according to the preset bucket size.

[0044] In this embodiment of the disclosure, the mapping quantile value of the final mapping bucket of the data to be sorted is determined by randomly selected quantile calibrator data, so as to facilitate the subsequent distribution of the data to be sorted to the corresponding mapping bucket, thereby achieving nearly perfectly uniform data bucketing during global sorting, improving the sorting efficiency of global sequence numbers, and alleviating node resource and computational pressure.

[0045] Furthermore, such as Figure 2 As shown, the method for determining the mapping quantile value of the mapping buckets disclosed herein includes the following steps:

[0046] Step S201: Determine the standard data volume based on the product of the total data volume of the data to be sorted and the preset selection ratio.

[0047] Because traditional equal-length binning is too time-consuming and inefficient when sorting massive amounts of non-uniformly distributed data, in this embodiment of the disclosure, a preset proportion of quantile benchmark data is randomly selected from the data to be sorted to predict the distribution trend of the data to be sorted to achieve near-perfectly uniform binning.

[0048] Furthermore, the selection ratio can be selectively set according to actual needs, for example, the selection ratio can be 1%.

[0049] Step S202: Using a random selection function, select the quantile standard data that matches the standard data quantity from the unsorted data.

[0050] In this embodiment of the disclosure, the random selection function can be an SQL function, such as t_sample as(select id,score from test.test_bigdata_sort where substr(id,32,1)='1'limit 1000000), which means starting from the 32nd id in test.test_bigdata_sort, using the random selection function substr() to select 1,000,000 ids and scores as quantile benchmark data. Here, id is the model id or user id, score is the model score or user rating score, test.test_bigdata_sort is the data source file, and t_sample is the selected quantile benchmark data file.

[0051] Step S203: Count the data to be sorted according to the size of the buckets to determine the number of buckets for the fixed buckets.

[0052] In this embodiment of the disclosure, the bucket size refers to the amount of data that a bucket can store and process. When sorting data by bucket globally, the bucket size is usually determined by the total amount of data to be sorted. For a data volume of one billion, the bucket size is usually no more than 10 million. For example, the bucket size can be set between 1 million and 10 million, and correspondingly, the number of buckets is between 1,000 and 100.

[0053] Furthermore, the size of the bucket for the fixed bucket can be selectively set according to the total amount of data to be sorted. For example, the size of the bucket for the fixed bucket can be 1 million.

[0054] Furthermore, the number of buckets for determining the bucket size is determined by counting the data to be sorted in units of bucket size. The determining function can be an SQL function, such as t_group_cnt as(select count(1) / 1000000as group_cntfrom test.test_bigdata_sort), which means that the data to be sorted in test.test_bigdata_sort is counted using the counting function count() in units of 1 million, and the resulting number of buckets group_cnt = 162, where t_group_cnt is the bucket count file.

[0055] Step S204: Based on the number of bins, calculate the quantile of each of the calibrated bins using a sequence function and a transformation function.

[0056] In this embodiment, the quantile bins are represented as an array to store the quantile quantile data. The function for determining the quantile can be an SQL function, such as: t_percentile_array as(select transform(sequence(1, group_cnt-1, 1), x->x*1.00000000 / group_cnt)as percentile_array The function `fromt_group_cnt` generates a sequence of quantile numbers, where `sequence()` is the sequence generation function and `transform()` is the transformation function. `sequence(1, group_cnt-1, 1)` generates a sequence of quantile numbers starting at 1, ending at (group_cnt-1), and with a step size of 1. `transform(sequence(1, group_cnt-1, 1), x->x*1.00000000 / group_cnt)` takes the sequence obtained from `sequence(1, group_cnt-1, 1)` as `x`, calculates the quotient of each quantile number in the sequence divided by `group_cnt`, and obtains the quantile array `percentile_array` corresponding to the sequence. `1.00000000` represents the precision of the quantiles. `t_percentile_array` is the quantile array file. It should be noted that when distributing quantile calibrated data, after the first (group_cnt-1) calibrated buckets have been distributed, the remaining data is the quantile calibrated data belonging to the last calibrated bucket. Therefore, when calculating the calibrated quantiles, it is only necessary to calculate the calibrated quantiles of the first (group_cnt-1) calibrated buckets.

[0057] Furthermore, taking a bin number group_cnt equal to 162 as an example, the function creation means dividing the sequence of indices from 1 to 161 by 100 with a step size of 1, to obtain the percentile array percentile_array for the 1st to 161st fixed bins:

[0058] 0.00617284, 0.01234568, 0.01851852, 0.02469136, 0.03086420, 0.03703704, 0.04320988, 0.04938272, 0.05555556, 0.06172840, 0.06790123, 0.07407407, 0.08024691, 0.08641975, ...

[0059] Step S205: Using the quantiles of each of the quantile buckets and the quantile quantile data, calculate the mapping quantile value of the mapping bucket of the data to be sorted.

[0060] In this embodiment, the calculation process from the definite quantile to the mapped quantile can be implemented using the SQL function `approx_percentile()`. The `approx_percentile()` function is an approximate inverse distribution function, which can input a percentage value and a sorting criterion, and output the value corresponding to the percentage value that conforms to the sorting criterion. For example, `t_bucket_arrayas(select approx_percentile(score, percentile_array, 0.0001)as bucket_arrayfrom t_sample, t_percentile_array)` means sorting the `score` in `t_sample` and outputting the mapped quantile array `bucket_array` corresponding to the `percentile_array` in `t_percentile_array`.

[0061] 0.006257048124935549,0.012500299341169918,0.018758634828670573,0.025036981282956727,0.03131693188992779,0.03747748509802484,0.04362021131348093,0.04984589016128982, where t_bucket_array is the mapped quantile value array file.

[0062] In this embodiment of the disclosure, the calculation process from the quantile to the mapped quantile value includes the following steps:

[0063] Step S2051: Sort the data values ​​of the quantile standard data in ascending order.

[0064] In this embodiment of the disclosure, for example, if the data sorting request is a test scenario, the data value of the quantile-accurate data is the model score; or if the data sorting request is a financial scenario, the data value of the quantile-accurate data is the user evaluation score.

[0065] Step S2052: Determine the amount of data to be distributed to each of the quantiles and the amount of data to be distributed to each quantile according to the amount of data to be distributed.

[0066] In this embodiment of the disclosure, the sorted quantile-based data is distributed according to the quantile distribution data volume of each quantile bucket, wherein:

[0067] For the first calibrated bucket, the calibrated data distribution volume of the calibrated bucket is the product of the calibrated quantile and the calibrated data volume of the first calibrated bucket; for example, the calibrated data distribution volume of the first calibrated bucket = 0.00617284 * calibrated data volume;

[0068] For the second to penultimate calibrated buckets, the calibrated data distribution amount for each calibrated bucket is the product of the difference between the calibrated quantile of each calibrated bucket and the calibrated quantile of the previous calibrated bucket, and the calibrated data amount. That is, the calibrated data distribution amount for the second calibrated bucket is the product of the difference between the second and first calibrated quantiles and the calibrated data amount; the calibrated data distribution amount for the third calibrated bucket is the product of the difference between the third and second calibrated quantiles and the calibrated data amount, and so on. For example, the calibrated data distribution amount for the second calibrated bucket = (0.01234568 - 0.00617284) * calibrated data amount; the calibrated data distribution amount for the fourteenth calibrated bucket = (0.08641975 - 0.08024691) * calibrated data amount.

[0069] For the last fixed-point bucket, the fixed-point distribution data volume of the fixed-point bucket is the amount of fixed-point data remaining after the first to second-to-last fixed-point buckets have been distributed.

[0070] Step S2053: Determine the calibrated quantile value of the calibrated bucket using the maximum data value of the quantile calibrated data distributed to each of the calibrated buckets.

[0071] In this embodiment of the disclosure, in order to distribute the data to be sorted to each mapping bucket in a nearly completely uniform manner, the local quantile standard data selected from the data to be sorted is distributed in a nearly completely uniform manner to obtain the maximum value of the target quantile standard data distributed to each standard bucket, which is used as the data value corresponding to the mapping quantile of the mapping bucket in which the data to be sorted is distributed in a nearly completely uniform manner, thereby avoiding the data concentration problem of traditional equal-length buckets.

[0072] Furthermore, since the target quantile calibrated data in the calibrated buckets is determined using the sorted quantile calibrated data, the maximum data value of the quantile calibrated data in each calibrated bucket is the last data value.

[0073] Step S2054: Use the quantile values ​​of each quantile bucket as the mapping quantile values ​​of the corresponding mapping buckets of the data to be sorted.

[0074] In this embodiment of the disclosure, since the mapping buckets are mapped from the quasi-buckets, the number of mapping buckets for the data to be sorted is the same as the number of quasi-buckets. That is, the mapping buckets for the data to be sorted and the quasi-buckets for the quasi-data correspond one-to-one.

[0075] In this embodiment, the method for determining the quantile values ​​of the mapped buckets disclosed herein randomly selects quantile benchmark data from the data to be sorted as a reference for near-perfectly uniform distribution. This quantile benchmark data is then distributed perfectly uniformly. The number of buckets for each benchmark is calculated based on the bucket size and the total amount of data to be sorted. The benchmark quantiles of each benchmark bucket are then calculated. Based on these benchmark quantiles, the mapping quantiles of the mapped buckets at the final bucketing stage of the data to be sorted are determined. The mapping quantiles of each mapped bucket reflect the data distribution trend of the data to be sorted, achieving near-perfectly uniform distribution. This provides a reference for the near-perfectly uniform distribution of the data to be sorted when performing global sorting of the data, balancing the resources and computational pressure of each bucket node, and improving global sorting efficiency while saving storage and computational resources of the nodes.

[0076] Step S103: Distribute the data to be sorted to each of the mapping buckets according to the mapping quantile values ​​to obtain the data in each of the mapping buckets.

[0077] In this embodiment, after determining the mapping quantile values ​​of each mapping bucket corresponding to the data to be sorted, the data to be sorted can be distributed. The sorting server includes read nodes and mapping bucket nodes, with a one-to-one correspondence between the read nodes and the mapping bucket nodes. During sorting, the sorting server distributes the data to be sorted evenly to each read node according to the ratio of the total data volume to the number of buckets. Each read node then compares each piece of data to be sorted in its allocated data pile with its corresponding mapping quantile value and distributes it to the corresponding mapping bucket node. Finally, each mapping bucket node performs global sorting.

[0078] Furthermore, the distribution of data to be sorted can be done using SQL functions, such as `t_bucket as(select id, score, width_bucket(score, bucket_array) as bucket_no from test.test_bigdata_sort, t_bucket_array)`. This means that the mapped bucket number `bucket_no` to which `id` and `score` belong in `test.test_bigdata_sort` is determined by using the mapped quantile value array `bucket_array` in `t_bucket_array` and the aggregation function `width_bucket()`. Here, `t_bucket` is the mapping distribution file, which includes the correspondence between `bucket_no`, `id`, and `score` for each mapped bucket.

[0079] In this disclosed embodiment, or, as Figure 3 As shown, the method for distributing unsorted data in this disclosure includes the following steps:

[0080] Step S301: Determine the numerical range of each mapping bucket based on the mapping quantile value of each mapping bucket.

[0081] In this embodiment of the disclosure, since the mapping quantile value of each mapping bucket is the maximum data value of the mapping bucket, for data falling between the maximum data values ​​of different mapping buckets, it is necessary to first determine the numerical range to which the data belongs. Specifically:

[0082] For the first mapped bucket, the numerical range is the range that is greater than or equal to the lower threshold of the data value to be sorted and less than or equal to the mapped quantile value of the first mapped bucket. For example, if the lower threshold of the data value is 0 and the mapped quantile value of the first mapped bucket is 5, then the numerical range of the first mapped bucket is [0, 5].

[0083] For the last mapped bucket, the numerical range is the range greater than the mapped quantile value of the second-to-last mapped bucket and less than or equal to the upper threshold of the data value to be sorted. For example, if the upper threshold of the data value is 100 and the mapped quantile value of the second-to-last mapped bucket is 95, then the numerical range of the last mapped bucket is (95, 100].

[0084] For the remaining mapped buckets, the numerical range is the range that is greater than the mapped quantile value of the previous mapped bucket and less than or equal to the mapped quantile value of its own mapped bucket. For example, if the maximum data value of the fifth mapped bucket is 12 and the maximum data value of the sixth mapped bucket is 16, then the numerical range of the sixth mapped bucket is (12, 16).

[0085] Step S302: Generate the correspondence between the mapping bucket number and the numerical range of the mapping bucket.

[0086] In this embodiment of the disclosure, a correspondence is generated between the mapping bucket number of each mapping bucket and the corresponding numerical range of the mapping bucket. For example, the correspondence is {⑥, (12, 16]}.

[0087] Step S303: Based on the correspondence between the mapping bucket number and the numerical range, the unsorted data belonging to the numerical range is distributed to the corresponding mapping bucket, thereby obtaining the data within each mapping bucket.

[0088] In this embodiment of the disclosure, the data to be sorted is distributed to each mapping bucket according to the mapping quantile value of the mapping bucket, thereby achieving an approximately uniform distribution of data exceeding one billion, thus alleviating the computational and resource pressure on nodes and improving data sorting efficiency.

[0089] Step S104: Sort the data within each of the mapped buckets, determine the global sequence number of each data entry within the bucket, and in response to the data sorting request, combine the global sequence number and the data within the bucket to obtain the sorting result.

[0090] In this embodiment of the disclosure, after the data to be sorted is distributed, each mapped bucket sorts the data within the bucket according to the data value of the data in the bucket to obtain the bucket sequence number of each data in the bucket. Then, based on the amount of data in each mapped bucket, the global sequence number of each data in the bucket is determined. Alternatively, based on the amount of data in each mapped bucket, the global sequence number of the first data in the bucket of each mapped bucket is determined. Then, each mapped bucket sorts the data within the bucket according to the data value of the data in the bucket to obtain the bucket sequence number of each data in the bucket. The global sequence number of each data in the bucket is determined by the global sequence number of the first data in the bucket and the bucket sequence number. Combining the global sequence number and the data in the bucket yields the final sorting result.

[0091] Furthermore, such as Figure 4 As shown, the method for determining the sorting result in this disclosure includes the following steps:

[0092] Step S401: For each of the mapped buckets, sort the data in the bucket according to the data value in the bucket from low to high, and determine the bucket sequence number of each piece of data in the bucket.

[0093] In this embodiment, the bucket indices can be determined using SQL functions, such as `t_bucket_inner_sortas(select id, score, bucket_no, row_number()over(partition by bucket_no orderby score)as bucket_inner order from t_bucket)`. This means using the sorting function `row_number()` to sort the buckets mapped to `bucket_no` in `t_bucket` according to the score, obtaining the bucket indices `bucket_inner order`. Then, using the window function `over()`, the bucket indices are output in ascending order of score within the mapped bucket indices `bucket_no`, resulting in the bucket sorting file `t_bucket_inner_sort`, which includes the correspondence between `bucket_no`, `id`, `score`, and `bucket_inner order` for each mapped bucket.

[0094]

[0095] Furthermore, the serial numbers inside the buckets start counting from 1.

[0096] Step S402: Determine the global sequence number of each piece of data within a bucket using the bucket sequence number and data value of the data within each of the mapped buckets.

[0097] Step S4021: Using the bucket sequence number, count the amount of data in each of the mapped buckets.

[0098] In this embodiment of the disclosure, the determination of the bucket data volume can be a SQL function, such as t_bucket_record_cnt as(select bucket_no, count(1)as cnt from t_bucket_inner_sort group by bucket_no), which means using the counting function count() to count the bucket indices of each bucket_no mapped in t_bucket_inner_sort, and obtain the bucket data volume cnt of each mapped bucket. Here, t_bucket_record_cnt is a bucket data volume file, including the correspondence between bucket_no and cnt of each mapped bucket:

[0099] 0 1017405 1 1008061 2 995668 3 1019919 4 1032873 5 1004657 6 986236 7 983244 8 1003636 9 1023596 10 1036988 …… ……

[0100] As shown in the table above, the amount of data in each mapped bucket is not completely equal. That is, the global bucket of the data to be sorted is a globally distributed bucket that is approximately uniform, rather than a globally distributed bucket that is necessarily completely uniform (it may be completely uniform or approximately uniform). However, it has achieved the purpose of this disclosure: preventing node crashes, balancing the resources and computational pressure between nodes, shortening sorting time, and improving sorting efficiency. It avoids the defect of excessive data concentration in traditional bucketing methods and improves the stability of the data sorting system.

[0101] Step S4022: Calculate the global sequence number of the first data entry in each mapped bucket according to the bucket number in ascending order and the amount of data in each bucket.

[0102] In this embodiment, the global sequence number of the first data in the bucket can be determined using an SQL function, such as `t_bucket_start_order as(select bucket_no, (sum(cnt)over(order by bucket_no)-cnt)as bucket_start_order from t_bucket_record_cnt)`. This means that the global sequence number `bucket_start_order` of the first data in each mapped bucket is determined by summing the bucket data volume of each `bucket_no` using the sum() function, and the `over()` function is used to output the `bucket_start_order` of each `bucket_no` one by one, resulting in the first sorted file `t_bucket_start_order`, which includes the correspondence between the `bucket_no` of each mapped bucket and the global sequence number `bucket_start_order` of the first data in the bucket.

[0103] 0 0 1 1017405 2 2025466 3 3021134 4 4041053 5 5073926 6 6078583 7 7064819 8 8048063 9 9051699 10 10075295 …… ……

[0104] Furthermore, the global sequence number of the first data in the first mapped bucket is counted starting from 0.

[0105] Step S4023: For each of the mapped buckets, calculate the global sequence number of the remaining data in each of the mapped buckets based on the global sequence number of the first data in the first bucket and the sequence numbers of the remaining data in the remaining buckets.

[0106] In this embodiment of the disclosure, the global sequence number of the data in the remaining buckets can be determined by an SQL function, such as: (select id, score, t_bucket_inner_sort.bucket_no, bucket_start_order+bucket_inner order as global_order from t_bucket_inner_sort left join t_bucket_start_order on t_bucket_inner_sort.bucket_no=t_bucket_start_order.bucket_no), which means that for each id and score that are the same in each t_bucket_inner_sort.bucket_no and t_bucket_start_order.bucket_no, the sum of bucket_start_order and bucket_inner order is calculated as the global_order of id and score.

[0107] Step S403: According to the bucket sequence number of the mapped bucket in ascending order, the combination of the bucket sequence number, the global sequence number and the data in the bucket is concatenated to obtain the sorting result.

[0108] In this embodiment of the disclosure, for example, in a test scenario where the data sorting request is a test scenario, the combination of bucket number, global number, model ID, and model segment is concatenated to obtain the sorting result:

[0109]

[0110]

[0111] For example, in a financial scenario, a data sorting request might involve concatenating a combination of bucket number, global number, user ID, and user rating score.

[0112] In this embodiment, the sorting result determination method first sorts the data to be sorted distributed to each mapped bucket to obtain the bucket index. Then, the amount of data in each mapped bucket is counted based on the bucket index. Next, the global index of the first data item in each mapped bucket is calculated according to the bucket index. Finally, the global index of each bucket's data is obtained by summing the global index of the first data item and the bucket index. This global index is then combined with the bucket index to obtain the final global data. All mapped buckets are then concatenated to obtain the sorting result. Since the data volume in each mapped bucket is approximately or completely uniformly distributed, the sorting time is similar, greatly improving the global sorting efficiency and avoiding node crashes caused by single-node backlog. It should be noted that due to the non-uniform distribution of the data to be sorted, multiple buckets in an approximately uniformly distributed global bucket may have the same data value.

[0113] In the embodiments of this disclosure, the data sorting method avoids the problems of traditional data bucket node backlog and crash, realizes near-uniform bucketing of non-uniformly distributed data, shortens sorting time, improves global sorting efficiency, can be applied to global sorting of more than one billion data, reduces sorting cost, improves data sorting stability, makes reasonable use of sorting computing and storage resources, and further improves the processing efficiency of computing tasks.

[0114] Figure 5 This is a schematic diagram of the main modules of a data sorting apparatus according to an embodiment of the present disclosure, such as... Figure 5 As shown, the data sorting apparatus 500 of this disclosure includes:

[0115] The receiving module 501 is used to receive a data sorting request; wherein the data sorting request includes data to be sorted.

[0116] The quantile quantile module 502 is used to randomly select quantile quantile quantile data from the data to be sorted, and determine the mapping quantile value of the mapping bucket corresponding to the data to be sorted from the quantile quantile quantile data according to the preset bucket size.

[0117] The mapping module 503 is used to distribute the data to be sorted to each of the mapping buckets according to the mapping quantile value, so as to obtain the data in each of the mapping buckets.

[0118] The sorting module 504 is used to sort the data within each of the mapped buckets, determine the global sequence number of each data item within the bucket, and, in response to the data sorting request, combine the global sequence number and the data within the bucket to obtain the sorting result.

[0119] Exemplary embodiments of this disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this disclosure.

[0120] Exemplary embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to embodiments of this disclosure.

[0121] Exemplary embodiments of this disclosure also provide a computer program product, including a computer program, wherein, when executed by a processor of a computer, the computer program is used to cause the computer to perform a method according to an embodiment of this disclosure.

[0122] refer to Figure 6 The present invention describes a structural block diagram of an electronic device 600 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0123] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0124] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, output unit 607, storage unit 608, and communication unit 609. Input unit 606 can be any type of device capable of inputting information to electronic device 600. Input unit 606 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 607 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 604 may include, but is not limited to, disk and optical disk. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMa6 devices, cellular communication devices, and / or the like.

[0125] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, Figures 1 to 5 The method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 600 via ROM 602 and / or communication unit 609. In some embodiments, computing unit 601 can be configured to execute by any other suitable means (e.g., by means of firmware). Figures 1 to 5 The method.

[0126] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0130] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0131] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. A data sorting method, characterized in that, The sorting method is applied in a sorting server, which includes mapped bucket nodes; the sorting method includes: Receive a data sorting request; wherein the data sorting request includes data to be sorted; Randomly selecting quantile benchmark data from the unsorted data includes: determining the benchmark data volume based on the product of the total data volume of the unsorted data and a preset selection ratio; and using a random selection function to select the quantile benchmark data from the unsorted data that matches the benchmark data volume. Based on a preset bucket size, the mapping quantile value of the corresponding mapping bucket for the data to be sorted is determined from the quantile standard data, including: counting the data to be sorted according to the bucket size to determine the number of standard buckets; calculating the standard quantile of each standard bucket using a sequence function and a transformation function based on the number of buckets; and calculating the mapping quantile value of the mapping bucket for the data to be sorted using the standard quantile of each standard bucket and the quantile standard data. According to the mapping quantile value, the data to be sorted is distributed to each of the mapping buckets to obtain the data in each of the mapping buckets; The data within each of the mapped buckets is sorted, and a global sequence number for each piece of data within the bucket is determined. In response to the data sorting request, the global sequence number and the data within the bucket are combined to obtain a sorting result. Each mapped bucket node in the sorting server is used to carry the distributed data within the bucket for global sorting.

2. The sorting method as described in claim 1, characterized in that, The step of calculating the mapped quantile value of the mapped bucket of the data to be sorted using the quantile values ​​of each of the quantile buckets and the quantile quantile data includes: The data values ​​of the quantile standard data are sorted in ascending order; Based on the quantiles and data volume of each quantile bucket, the data volume of each quantile bucket is determined, and the sorted quantile data is distributed to each quantile bucket according to the data volume. The quantile value of each quantile bucket is determined by using the maximum data value of the quantile data distributed to each quantile bucket, and the quantile value of each quantile bucket is used as the mapping quantile value of the corresponding mapping bucket of the data to be sorted.

3. The sorting method as described in claim 1, characterized in that, The step of distributing the data to be sorted to each of the mapped buckets according to the mapped quantile value, to obtain the data within each mapped bucket, includes: The numerical range of each mapping bucket is determined based on the mapping quantile value of each mapping bucket. Generate the correspondence between the mapping bucket number and the numerical range of the mapping bucket; Based on the correspondence between the mapping bucket number and the numerical range, the unsorted data belonging to the numerical range is distributed to the corresponding mapping bucket, thereby obtaining the data within each mapping bucket.

4. The sorting method as described in claim 1, characterized in that, The step of sorting the data within each of the mapped buckets and determining the global sequence number of each data entry within a bucket includes: For each of the mapping buckets, the data in the buckets are sorted in ascending order of the data values ​​to determine the bucket sequence number of each data item. The global sequence number of each piece of data within a bucket is determined by using the bucket sequence number and data value of the data within each of the mapped buckets.

5. The sorting method as described in claim 4, characterized in that, The step of determining the global sequence number of each piece of data within a bucket using the bucket sequence number and data value of the data within each of the mapped buckets includes: Using the bucket index, the amount of data in each mapped bucket is counted. According to the bucket number of the mapped buckets in ascending order, and based on the amount of data in each bucket, calculate the global sequence number of the first data in each mapped bucket. For each of the mapped buckets, the global sequence number of the remaining data in each mapped bucket is calculated based on the global sequence number of the first data in the first bucket and the sequence numbers of the remaining data in the other buckets.

6. A data sorting device, characterized in that, The sorting device operates on a sorting server, which includes mapped bucket nodes; the sorting device includes: A receiving module is configured to receive a data sorting request; wherein the data sorting request includes data to be sorted; A quantile-based module is used to randomly select quantile-based quantile data from the data to be sorted, including: determining the quantile data volume based on the product of the total data volume of the data to be sorted and a preset selection ratio; selecting the quantile-based quantile data matching the quantile data volume from the data to be sorted using a random selection function; and determining the mapped quantile value of the mapped bucket corresponding to the data to be sorted from the quantile-based quantile data according to a preset bucket size, including: counting the data to be sorted according to the bucket size to determine the number of quantile buckets; calculating the quantile value of each quantile bucket using a sequence function and a transformation function based on the number of buckets; and calculating the mapped quantile value of the mapped bucket of the data to be sorted using the quantile values ​​of each quantile bucket and the quantile-based quantile data. The mapping module is used to distribute the data to be sorted to each of the mapping buckets according to the mapping quantile value, so as to obtain the data in each of the mapping buckets. The sorting module is used to sort the data within each of the mapped buckets, determine the global sequence number of each piece of data within the bucket, and, in response to the data sorting request, combine the global sequence number and the data within the bucket to obtain the sorting result; wherein, each mapped bucket node in the sorting server is used to carry the distributed data within the bucket for global sorting.

7. An electronic device, comprising: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the data sorting method according to any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the data sorting method according to any one of claims 1-5.

Images