Data sorting methods, devices and storage media

Abstract

This invention discloses a data sorting method, device, and storage medium, relating to the field of data processing technology. It obtains ordered candidate sequences of principal elements corresponding to each data sorting process. These ordered candidate sequences are obtained by the data sorting process through data sample exchange and sorting based on the data sample sequence. When a principal element is determined to meet a preset load balancing condition based on the ordered candidate sequence, it checks whether the principal elements adjacent to that principal element also meet the preset load balancing condition. If adjacent principal elements are found not to meet the preset load balancing condition, it assists the adjacent principal elements in calculation. When all principal elements meet the preset load balancing condition, the principal element and / or adjacent principal elements are broadcast to obtain a first principal element sequence. The data sequence is divided according to the first principal element sequence to obtain a first displacement array. Based on the first displacement array, data exchange and local sorting between data sequences are performed to obtain an ordered result sequence, providing high-efficiency computing power for large-scale data sorting.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data sorting method, device and storage medium. Background Technology

[0002] In the era of big data, the amount of data in various applications is growing rapidly, and massive data processing relies heavily on large-scale parallel computing. Sorting methods are widely used in various scientific research and applications. Making full use of computer resources to achieve large-scale parallel sorting helps to increase the scale of data to be sorted, improve data sorting performance, and prevent data sorting from becoming a bottleneck for application performance.

[0003] However, existing merge-based data sorting methods suffer from high memory overhead and are prone to memory overflow, making them unsuitable for large-scale data sorting. Parallel sampling-based data sorting methods also suffer from uneven data load between processes, which negatively impacts the sorting efficiency of large-scale data sorting. Therefore, existing data sorting methods cannot provide high-efficiency computational capabilities for large-scale data sorting. Summary of the Invention

[0004] The main objective of this invention is to provide a data sorting method, device, and storage medium, aiming to solve the technical problem that existing data sorting methods cannot provide high-efficiency computing power for large-scale data sorting.

[0005] To achieve the above objectives, the present invention provides a data sorting method, characterized in that the data sorting method includes the following steps:

[0006] Obtain the ordered candidate sequence of principal elements corresponding to each data sorting process, wherein each data sorting process includes a data sample sequence, and the ordered candidate sequence is obtained by the data sorting process sorting the data sample sequence after data sample exchange based on the data sample sequence;

[0007] When the principal element corresponding to the ordered candidate sequence satisfies the preset load balancing condition, the neighboring principal elements of the principal element are detected to satisfy the preset load balancing condition. When the neighboring principal element is detected not to satisfy the preset load balancing condition, the neighboring principal element is assisted in the calculation.

[0008] When it is determined that all the master elements meet the preset load balancing conditions, the master element and / or the adjacent master element are broadcast to obtain a first master element sequence, wherein the master element and / or the adjacent master element is an initialization master element or an updated master element;

[0009] The data sequence is divided into a first displacement array based on the first principal component sequence. Data exchange is performed between the data sequences based on the first displacement array. The data sequences after data exchange are then sorted locally to obtain an ordered result sequence.

[0010] Optionally, the data sorting method further includes:

[0011] Sort the local data to obtain a data sequence;

[0012] The data sequence is subjected to regular sampling to obtain the data sample sequence.

[0013] Optionally, the data sorting method further includes:

[0014] Non-target data samples in the data sample sequence are sent to the corresponding target data sorting process, and target data samples are received for data sample exchange. The first sample sequence number of the non-target data sample is not the same as the first process sequence number of the data sorting process to which the data sample sequence belongs, and the first sample sequence number is the same as the second process sequence number of the target data sorting process. The second sample sequence number of the target data sample is the same as the first process sequence number, and the second sample sequence number is not the same as the second process sequence number.

[0015] Optionally, the data sorting method further includes:

[0016] The median in the ordered candidate sequence is determined as the initial value, and the corresponding pivot element is initialized based on the initial value to obtain the initialized pivot element;

[0017] After broadcasting the initial principal element and obtaining the second principal element sequence, the data sequence is divided according to the second principal element sequence to obtain the second displacement array;

[0018] After calculating the longitudinal sum of the second displacement array, determine the relationship between the longitudinal sum and the preset lower limit and / or preset upper limit.

[0019] When the vertical axis is between a preset lower limit and a preset upper limit, it is determined that the initialization master element satisfies the preset load balancing condition.

[0020] Optionally, after the step of determining the size relationship between the vertical sum and the preset lower limit and / or preset upper limit, the data sorting method further includes;

[0021] When the vertical sum is less than the preset lower limit, a binary search is performed according to the increasing direction of each value in the ordered candidate sequence to obtain an updated value. The initial principal element is then updated based on the updated value to obtain the updated principal element.

[0022] Broadcast the updated principal element, execute the step of obtaining the second principal element sequence, until the updated principal element satisfies the preset load balancing condition when the vertical sum determined based on the updated principal element is between the preset lower limit value and the preset upper limit value.

[0023] Optionally, after the step of determining the size relationship between the vertical sum and a preset lower limit and / or a preset upper limit, the data sorting method further includes:

[0024] When the vertical sum is greater than the preset upper limit, a binary search is performed according to the decreasing direction of each value in the ordered candidate sequence to obtain an updated value. The initial principal element is then updated based on the updated value to obtain the updated principal element.

[0025] Broadcast the updated principal element, execute the step of obtaining the second principal element sequence, until the updated principal element satisfies the preset load balancing condition when the vertical sum determined based on the updated principal element is between the preset lower limit value and the preset upper limit value.

[0026] Optionally, the step of detecting whether the neighboring primary elements of the primary element satisfy the preset load balancing conditions includes:

[0027] Based on the adjacent principal elements, candidate values ​​are determined, and based on the candidate values, it is determined whether the adjacent principal elements meet the preset load balancing conditions.

[0028] After the step of determining whether the adjacent principal elements meet the preset load balancing conditions based on the candidate values, the data sorting method further includes:

[0029] When it is determined that the adjacent primary elements meet the preset load balancing conditions, it is determined that each primary element and the adjacent primary elements meet the preset load balancing conditions.

[0030] Optionally, the step of exchanging data between the data sequences based on the first displacement array includes:

[0031] Data partitioning criteria are generated based on the first displacement array;

[0032] Data exchange between the data sequences is performed based on the data partitioning criteria.

[0033] To achieve the above objectives, the present invention also provides an electronic device, which includes: a memory, a processor, and a computer processing program stored in the memory and executable on the processor. When the computer processing program is executed by the processor, it implements the steps of the data sorting method described above.

[0034] Furthermore, to achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a computer processing program, which, when executed by a processor, implements the steps of the data sorting method described above.

[0035] In this invention, an ordered candidate sequence of principal elements corresponding to each data sorting process is obtained. Each data sorting process includes a data sample sequence. The ordered candidate sequence is obtained by sorting the data sample sequences after data sample exchange between the data sorting processes. By exchanging data sample sequences between data sorting processes, the computational and memory overhead of each data sorting process is balanced, which is beneficial for large-scale data sorting. When determining that the corresponding principal element meets the preset load balancing condition based on the ordered candidate sequence, it is checked whether the adjacent principal elements of the principal element meet the preset load balancing condition. If it is detected that an adjacent principal element does not meet the preset load balancing condition... When conditions are met, the algorithm assists neighboring principals in calculations. When all principals meet the preset load balancing conditions, the principal and / or neighboring principals are broadcast to obtain the first principal sequence. The data sorting process, which completes the load balancing calculation of the corresponding principals, assists in the load balancing calculation of the undetermined principals, making full use of computing resources, accelerating the load balancing calculation of principals, and further improving the efficiency of data sorting. The data sequence is divided into a first displacement array based on the first principal sequence. Data exchange is performed between data sequences based on the first displacement array. The data sequences after data exchange are sorted locally to obtain an ordered result sequence, providing high-efficiency computing capabilities for large-scale data sorting. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiments of the present invention;

[0037] Figure 2 This is a flowchart illustrating an embodiment of the data sorting method of the present invention;

[0038] Figure 3 This is a schematic diagram of the process steps for adjacent principal component assistance calculation during the data sorting process of the present invention.

[0039] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0040] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0041] like Figure 1 As shown, Figure 1 This is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiments of the present invention.

[0042] The data sorting method of this invention is applied to an electronic device, such as... Figure 1 As shown, the electronic device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display area and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0043] Optionally, the electronic device may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, and so on. Sensors may include light sensors, motion sensors, and other sensors. Specifically, light sensors may include ambient light sensors and proximity sensors. The ambient light sensor can adjust the brightness of the display screen according to the ambient light level, while the proximity sensor can turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, a gravity accelerometer can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity, and can be used for applications that identify the mobile terminal's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition functions (such as pedometers, taps), etc. Of course, the mobile terminal may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, which will not be elaborated here.

[0044] Those skilled in the art will understand that Figure 1 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0045] like Figure 1 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a computer processing program.

[0046] exist Figure 1In the terminal shown, network interface 1004 is mainly used to connect to the backend server and communicate data with it; user interface 1003 is mainly used to connect to the client (user terminal) and communicate data with it; while processor 1001 can be used to call the computer processing program stored in memory 1005 and perform the following operations:

[0047] Obtain the ordered candidate sequence of principal elements corresponding to each data sorting process, wherein each data sorting process includes a data sample sequence, and the ordered candidate sequence is obtained by the data sorting process sorting the data sample sequence after data sample exchange based on the data sample sequence;

[0048] When the principal element corresponding to the ordered candidate sequence satisfies the preset load balancing condition, the neighboring principal elements of the principal element are detected to satisfy the preset load balancing condition. When the neighboring principal element is detected not to satisfy the preset load balancing condition, the neighboring principal element is assisted in the calculation.

[0049] When it is determined that all the master elements meet the preset load balancing conditions, the master element and / or the adjacent master element are broadcast to obtain a first master element sequence, wherein the master element and / or the adjacent master element is an initialization master element or an updated master element;

[0050] The data sequence is divided into a first displacement array based on the first principal component sequence. Data exchange is performed between the data sequences based on the first displacement array. The data sequences after data exchange are then sorted locally to obtain an ordered result sequence.

[0051] Furthermore, the processor 1001 can call a computer program stored in the memory 1005 and also perform the following operations:

[0052] Sort the local data to obtain a data sequence;

[0053] The data sequence is subjected to regular sampling to obtain the data sample sequence.

[0054] Furthermore, the processor 1001 can call a computer program stored in the memory 1005 and also perform the following operations:

[0055] Non-target data samples in the data sample sequence are sent to the corresponding target data sorting process, and target data samples are received for data sample exchange. The first sample sequence number of the non-target data sample is not the same as the first process sequence number of the data sorting process to which the data sample sequence belongs, and the first sample sequence number is the same as the second process sequence number of the target data sorting process. The second sample sequence number of the target data sample is the same as the first process sequence number, and the second sample sequence number is not the same as the second process sequence number.

[0056] Furthermore, the processor 1001 can call a computer program stored in the memory 1005 and also perform the following operations:

[0057] The median in the ordered candidate sequence is determined as the initial value, and the corresponding pivot element is initialized based on the initial value to obtain the initialized pivot element;

[0058] After broadcasting the initial principal element and obtaining the second principal element sequence, the data sequence is divided according to the second principal element sequence to obtain the second displacement array;

[0059] After calculating the longitudinal sum of the second displacement array, determine the relationship between the longitudinal sum and the preset lower limit and / or preset upper limit.

[0060] When the vertical axis is between a preset lower limit and a preset upper limit, it is determined that the initialization master element satisfies the preset load balancing condition.

[0061] Furthermore, the processor 1001 can call a computer program stored in the memory 1005 and also perform the following operations:

[0062] When the vertical sum is less than the preset lower limit, a binary search is performed according to the increasing direction of each value in the ordered candidate sequence to obtain an updated value. The initial principal element is then updated based on the updated value to obtain the updated principal element.

[0063] Broadcast the updated principal element, execute the step of obtaining the second principal element sequence, until the updated principal element satisfies the preset load balancing condition when the vertical sum determined based on the updated principal element is between the preset lower limit value and the preset upper limit value.

[0064] Furthermore, the processor 1001 can call a computer program stored in the memory 1005 and also perform the following operations:

[0065] When the vertical sum is greater than the preset upper limit, a binary search is performed according to the decreasing direction of each value in the ordered candidate sequence to obtain an updated value. The initial principal element is then updated based on the updated value to obtain the updated principal element.

[0066] Broadcast the updated principal element, execute the step of obtaining the second principal element sequence, until the updated principal element satisfies the preset load balancing condition when the vertical sum determined based on the updated principal element is between the preset lower limit value and the preset upper limit value.

[0067] Furthermore, the processor 1001 can call a computer program stored in the memory 1005 and also perform the following operations:

[0068] Based on the adjacent principal elements, candidate values ​​are determined, and based on the candidate values, it is determined whether the adjacent principal elements meet the preset load balancing conditions.

[0069] After the step of determining whether the adjacent principal elements meet the preset load balancing conditions based on the candidate values, the data sorting method further includes:

[0070] When it is determined that the adjacent primary elements meet the preset load balancing conditions, it is determined that each primary element and the adjacent primary elements meet the preset load balancing conditions.

[0071] Furthermore, the processor 1001 can call a computer program stored in the memory 1005 and also perform the following operations:

[0072] Data partitioning criteria are generated based on the first displacement array;

[0073] Data exchange between the data sequences is performed based on the data partitioning criteria.

[0074] like Figure 2 As shown, Figure 2 This is a flowchart illustrating an embodiment of the data sorting method of the present invention. In this embodiment, the data sorting method includes the following steps:

[0075] Step S10: Obtain the ordered candidate sequence of principal elements corresponding to each data sorting process, wherein each data sorting process includes a data sample sequence, and the ordered candidate sequence is obtained by the data sorting process sorting the data sample sequence after data sample exchange based on the data sample sequence.

[0076] In existing merge-based data sorting methods, the number of merges increases with the number of data sorting processes, and the overhead of data exchange and synchronization between data sorting processes also increases. In distributed memory systems, dense and large-scale data exchange can become a performance bottleneck. In partition-based data sorting methods, a specific data sorting process needs to collect data samples from all data sorting processes and sort them side by side. This approach has high space complexity, and the space overhead increases with the scale of data sorting, which can easily lead to memory overflow problems and is not conducive to large-scale data sorting.

[0077] Based on the above, this embodiment proposes to exchange data sample sequences included in each data sorting process, thereby balancing the computational and memory overhead of each data sorting process, which is beneficial for sorting large-scale data. This differs from the general approach of sorting p data processes at O(p...)... 2 All data samples are sent to a specific data sorting process. The time complexity of sorting the data sample sequence is O(p). 2 logp 2 This embodiment proposes a full-interchange regular sampling method that facilitates the exchange of data samples between p data sorting processes using the set communication operation MPI_Alltoall.

[0078] Specifically, the i-th data sorting process in p data sorting processes sends all data samples except the i-th data sample in the data sample sequence included in the data sorting process to other corresponding data sorting processes, for example, sending the j-th data sample to the j-th data sorting process. At the same time, it receives the i-th data sample sent by other data sorting processes, for example, receiving the i-th data sample sent by the j-th data sorting process. Based on this data sample exchange, each data sorting process has only O(p) data samples, so that the time complexity of sorting the data sample sequence after the data sample exchange is only O(plogp), which is significantly reduced compared to existing data sorting methods.

[0079] By reordering the data sample sequences after data sample exchange in each data sorting process, an ordered candidate sequence of principal components corresponding to each data sorting process can be obtained based on the reordered data sample sequences. Based on this ordered candidate sequence, the iterative update range of the corresponding principal components can be narrowed, and the computational efficiency of calculating whether the principal components are load-balanced can be improved, thereby avoiding the situation of uneven load among data sorting processes.

[0080] Optionally, the data sorting method further includes:

[0081] Step S101: Sort the local data to obtain a data sequence.

[0082] Step S102: Perform regular sampling on the data sequence to obtain the data sample sequence.

[0083] In this embodiment, the local data is the data to be sorted. Each data sorting process inputs the same amount of local data. After receiving the local data, the data sorting process first sorts the local data to obtain a data sequence. Then, by performing regular sampling on the data sequence, a data sample sequence is obtained. Regular sampling divides the data sequence into data sample groups using equally spaced sampling points. The equal interval is set according to the number of data sorting processes. Based on p data sorting processes, regular sampling divides the data sequence into a data sample sequence consisting of p data samples, facilitating the orderly exchange of data samples between the data sorting processes.

[0084] Optionally, the data sorting method further includes:

[0085] Step S103: Send the non-target data samples in the data sample sequence to the corresponding target data sorting process, and receive the target data samples for data sample exchange. The first sample sequence number of the non-target data sample is not the same as the first process sequence number of the data sorting process to which the data sample sequence belongs, and the first sample sequence number is the same as the second process sequence number of the target data sorting process. The second sample sequence number of the target data sample is the same as the first process sequence number, and the second sample sequence number is not the same as the second process sequence number.

[0086] Suppose there are 5 data sorting processes: the 1st, 2nd, 3rd, 4th, and 5th data sorting processes. Each process includes a data sample sequence of the same size as the others [data 1, data 2, data 3, data 4, data 5]. During the data sample exchange between these sequences, the 1st data sorting process sends non-target data samples ([data 2, data 3, data 4, data 5]) from its sequence to the corresponding target process. Specifically, it sends data 2 to the 2nd process, data 3 to the 3rd process, data 4 to the 4th process, and data 5 to the 5th process. Simultaneously, it receives data from the 2nd, 3rd, 4th, and 5th data sorting processes. The first data sorting process sends the target data sample (data 1) included in its respective data sample sequence. The second data sorting process sends the non-target data samples ([data 1, data 3, data 4, data 5]) in its included data sample sequence to the corresponding target data sorting process. That is, data 1 is sent to the first data sorting process, data 3 is sent to the third data sorting process, data 4 is sent to the fourth data sorting process, and data 5 is sent to the fifth data sorting process. At the same time, it receives the target data sample (data 2) included in its respective data sample sequence sent by the first, third, fourth, and fifth data sorting processes. The remaining data sorting processes perform the same operation as the first and second data sorting processes. This operation is to balance the computational and memory overhead among the data sorting processes, so that the data sorting method can be applied to large-scale data sorting.

[0087] Step S20: When the principal element corresponding to the ordered candidate sequence satisfies the preset load balancing condition, detect whether the adjacent principal elements of the principal element satisfy the preset load balancing condition, and when it is detected that the adjacent principal element does not satisfy the preset load balancing condition, assist the adjacent principal element in calculation.

[0088] In this embodiment, when any one or more data sorting processes determine the principal load balancing of their pairs based on their respective ordered candidate sequences, i.e., when the preset load balancing conditions are met, the data sorting processes whose principals meet the preset load balancing conditions will detect whether the principals corresponding to adjacent data sorting processes meet the preset load balancing conditions. When a principal that does not meet the preset load balancing conditions is detected, the data sorting processes whose principals meet the preset load balancing conditions will assist the data sorting processes whose principals do not meet the preset load balancing conditions in calculation, helping them find the principal that guarantees load balancing. By fully utilizing the computing resources of the data sorting processes whose principals meet the preset load balancing conditions, the calculation process of all principals is accelerated, thereby further speeding up the efficiency of data sorting.

[0089] Optionally, the data sorting method further includes:

[0090] Step S201: Determine the median in the ordered candidate sequence as the initial value, and initialize the corresponding principal component based on the initial value to obtain the initialized principal component;

[0091] Step S202: Broadcast the initial principal element, obtain the second principal element sequence, and then divide the data sequence according to the second principal element sequence to obtain the second displacement array;

[0092] Step S203: After calculating the longitudinal sum of the second displacement array, determine the relationship between the longitudinal sum and the preset lower limit value and / or the preset upper limit value.

[0093] Step S204: When the vertical axis is between a preset lower limit and a preset upper limit, it is determined that the initialization master element satisfies the preset load balancing condition.

[0094] Suppose that p data sorting processes are used to sort a total of n data, where X is the local data in a certain data sorting process. i The size is n i , Using p data sorting processes, find p-1 principal elements that satisfy load balancing partitioning of data, with an acceptable deviation rate not greater than b.

[0095] Each data sorting process sorts its corresponding local data, resulting in a data sequence X'. i Each data sorting process then sorts the data sequence X'. i After performing regular sampling, the resulting data sample sequence of size p is as follows:

[0096] y i ={x i,0 ,x i,w ,......,x i,(p-1)w}

[0097] Where w = n i / p.

[0098] After exchanging data samples based on the data sample sequence, an ordered candidate sequence of size p is obtained as follows:

[0099] y' i ={y' i,0 ,......,y' i,(p-1)}

[0100] Each data sorting process selects the median of the corresponding ordered candidate sequence as the initial value, i.e., with the ordered candidate sequence as y'. i For example, in this case, choose y' i,p / 2 The corresponding pivot elements are initialized using these as initial values. After obtaining the initialized pivot elements, each data sorting process broadcasts its corresponding initialized pivot elements to other data sorting processes running in parallel, so that each data sorting process can obtain a second pivot element sequence z = {z1, ..., z2} of size p. p-1 At this point, each data sorting process sorts the data sequence X' based on the second principal component sequence. i Divide into p segments to obtain the second displacement array:

[0101] s i ={s i,0 ,......,s i,p-1}

[0102] Among them, s i,p-1 Indicates in the data sequence X' i Find z that is not less than z p-1 The first data position.

[0103] Each data sorting process calculates the vertical sum of the i-th displacement array, that is, the data sorting process that sorts to 1 calculates the s-th displacement array. i,0 The vertical sum of the displacement arrays, sorted by p, is used to calculate the s-th data sorting process. i,p-1 The longitudinal sum of the displacement arrays is:

[0104] Among them, element v i Let v represent the total amount of data in the first i processes. Ideally, after data redistribution, the amount of data in each process is n / p. Then, the ideal vertical sum v... i It should be a multiple of n / p, but finding a data partitioning scheme that can achieve absolute equivalence is very costly. Therefore, by defining an acceptable deviation b, as long as the value of the longitudinal sum falls between the preset lower limit and the preset upper limit, the i-th principal element is considered to satisfy load balancing, i.e., it satisfies the preset load balancing condition.

[0105] The preset lower limit and preset upper limit are the range of conditions within which the data sorting process can meet the load balancing requirements.

[0106] Optionally, after determining the size relationship between the vertical sum and the preset lower limit and / or preset upper limit in step S203, the data sorting method further includes;

[0107] Step S205: When the vertical sum is less than the preset lower limit, perform a binary search according to the increasing direction of each value in the ordered candidate sequence to obtain an updated value, and update the initial principal element based on the updated value to obtain the updated principal element;

[0108] Step S206: Broadcast the updated principal element, execute the step of obtaining the second principal element sequence, until the vertical sum determined based on the updated principal element is between the preset lower limit value and the preset upper limit value, and determine that the updated principal element satisfies the preset load balancing condition.

[0109] If the vertical sum is less than the preset lower limit, that is:

[0110]

[0111] Then, a binary search is performed based on the median value of the ordered candidate sequence in the direction of increasing median value to iteratively update the principal element until the vertical sum is determined based on the updated principal element and can be between the preset lower limit value and the preset upper limit value.

[0112] in, The lower limit is set as b, which is the acceptable deviation of the principal element. Binary search is a search algorithm for finding a specific element in an ordered array. In this embodiment, it is to find a value in the ordered candidate sequence that can update the principal element and calculate whether the updated principal element meets the preset load balancing condition. Binary search can effectively narrow the update range of the principal element and run the process of the principal element satisfying load balancing at a lower cost, avoiding uneven overhead.

[0113] Optionally, after determining the magnitude relationship between the longitudinal direction and the preset lower limit and / or preset upper limit in step S203, the method further includes:

[0114] Step S207: When the vertical sum is greater than the preset upper limit value, perform a binary search according to the decreasing direction of each value in the ordered candidate sequence to obtain an updated value, and update the initial principal element based on the updated value to obtain the updated principal element;

[0115] Step S208: Broadcast the updated principal element, execute the step of obtaining the second principal element sequence, until the vertical sum determined based on the updated principal element is between the preset lower limit value and the preset upper limit value, and determine that the updated principal element satisfies the preset load balancing condition.

[0116] If the vertical sum is greater than the preset upper limit, that is:

[0117]

[0118] Then, a binary search is performed based on the median value of the ordered candidate sequence in the direction of decreasing median value to iteratively update the principal elements until the vertical sum is determined based on the updated principal elements and can be between the preset lower limit and the preset upper limit.

[0119] in, To set a preset upper limit, in this embodiment, by searching in the ordered candidate sequence for a value that can update the principal element and calculate whether the updated principal element meets the preset load balancing condition, the range of principal element updates is effectively narrowed through binary search, and the process of the principal element satisfying load balancing is run at a lower cost, avoiding uneven overhead.

[0120] In this embodiment, the value of the vertical sum v i If the value falls between the preset lower limit and the preset upper limit, the corresponding principal element is considered to satisfy the load balancing requirement. Furthermore, the update direction of the corresponding principal element in subsequent iterations can be determined by the relationship between the vertical sum and the preset lower and upper limits. If the vertical sum is less than the preset lower limit, the principal element performs a binary search on the numerical sampling along the increasing direction. If the vertical sum is greater than the preset upper limit, the principal element performs a binary search on the numerical sampling along the decreasing direction. The updated value of the principal element can reflect the distribution of data in each process to a certain extent.

[0121] Optionally, the step of detecting whether the neighboring primary elements of the primary element meet the preset load balancing conditions in step S20 includes:

[0122] Step S209: Determine candidate values ​​based on the adjacent principal elements, and determine whether the adjacent principal elements meet the preset load balancing conditions based on the candidate values.

[0123] Optionally, after the step S209 in which the adjacent principal elements are determined based on the candidate values ​​to satisfy the preset load balancing conditions, the data sorting method further includes:

[0124] Step S2010: When it is determined that the adjacent primary elements meet the preset load balancing conditions, it is determined that each primary element and the adjacent primary elements meet the preset load balancing conditions.

[0125] To fully utilize idle computing resources and accelerate the finding of all master elements that satisfy load balancing, in this embodiment, the data sorting process that satisfies load balancing for master elements assists the data sorting processes that do not satisfy load balancing for other master elements. When the master element corresponding to any one or more data sorting processes has already satisfied load balancing, the data sorting process will determine whether the master elements corresponding to other adjacent data sorting processes satisfy load balancing. If it is detected that the master element corresponding to any adjacent data sorting process (i.e., the adjacent master element) does not satisfy load balancing, the data sorting process that satisfies load balancing will provide candidate values ​​other than the values ​​in the ordered candidate sequence of the data sorting process corresponding to the adjacent master element based on the values ​​already determined by the adjacent master element (i.e., the values ​​updated for the adjacent master element). Among them, the values ​​in the ordered candidate sequence between the data sorting process that satisfies load balancing for master elements and the data sorting process corresponding to the adjacent master element can all be candidate values, and there can be multiple candidate values. After determining the candidate values, a vertical sum is calculated based on the candidate values. The specific calculation steps refer to step S204. The results of the multiple vertical sums are used to detect whether the adjacent master elements satisfy load balancing.

[0126] If the results of the multiple vertical sums indicate that the adjacent principal elements still do not meet the load balancing requirement, the determination of candidate values ​​is repeated (where the re-determined candidate values ​​can be previously determined candidate values ​​or candidate values ​​that have not been used before). The results of the multiple vertical sums are used to re-determine whether the adjacent principal elements meet the load balancing requirement until the result that the adjacent principal elements meet the load balancing requirement is obtained.

[0127] After obtaining the result that adjacent principal elements satisfy load balancing, the data sorting process that satisfies load balancing continues to check whether the principal elements corresponding to any adjacent data sorting process do not satisfy load balancing. If no such case is detected, it means that the principal elements corresponding to all data sorting processes satisfy load balancing, making full use of computing resources, avoiding idle waiting of the data sorting process, which helps to accelerate the iterative update process of principal elements and thus improves computing efficiency.

[0128] Specifically, refer to Figure 3 As shown, assume there are four data sorting processes: data sorting process 1, data sorting process 2, data sorting process 3, and data sorting process 4. Figure 3It can be seen that in the first step of determining whether the principal element meets the preset load balancing condition, only the principal element corresponding to data sorting process 1 meets the preset load balancing condition. Therefore, data sorting process 1 will assist the adjacent data sorting process 2 in determining whether the principal element meets the preset load balancing condition. However, data sorting processes 3 and 4 do not have any other data sorting processes assisting them and can only determine whether the principal element meets the preset load balancing condition based on their own processes. In the second stage, with the assistance of data sorting process 1, data sorting process 2 obtains the result that its principal element meets the preset load balancing condition, while data sorting processes 3 and 4... Since the principal element of data sorting process 1 still fails to meet the preset load balancing condition, data sorting process 2 assists data sorting process 4 based on adjacent detection. This results in the principal element of data sorting process 3 meeting the preset load balancing condition, while the principal element of data sorting process 4 does not. Therefore, data sorting process 2 will once again check whether the principal element of data sorting process 4 meets the preset load balancing condition, until the principal element of data sorting process 4 meets the preset load balancing condition. This process continues until the principal element of data sorting process 4 meets the preset load balancing condition, resulting in the result that the principal elements of all four data sorting processes meet the preset load balancing condition.

[0129] In step S30, when it is determined that each of the master elements meets the preset load balancing conditions, the master element and / or the adjacent master element are broadcast to obtain a first master element sequence, wherein the master element and / or the adjacent master element is an initialization master element or an updated master element.

[0130] When no data sorting process whose principal element meets the preset load balancing condition detects a data sorting process whose principal element does not meet the preset load balancing condition, it means that the principal elements corresponding to all data sorting processes meet the preset load balancing condition. At this time, each data sorting process broadcasts its corresponding principal element to other data sorting processes running in parallel, so that all data sorting processes can receive all principal elements except for the data sorting process itself, and generate a first principal element sequence based on the received principal elements. This first principal element sequence can divide the data sequence so that each data sorting process can obtain a displacement array composed of data of the same size.

[0131] The preset load balancing condition refers to the fact that after the data sequence is divided and the data sequence is exchanged based on the initial principal element or the updated principal element, the amount of computational task for each data sorting process to perform local sorting of the data sequence after the data exchange is the same.

[0132] Step S40: Divide the data sequence according to the first principal component sequence to obtain a first displacement array, perform data exchange between the data sequences based on the first displacement array, and perform local sorting on the data exchanged data sequences to obtain an ordered result sequence.

[0133] Each data sorting process, after partitioning its respective data sequence based on the corresponding first principal component sequence, obtains the first shift array, i.e., in the data sequence X' i Find z that is not less than z j The first data position s i,j This yields the first displacement array, namely:

[0134] s i ={s i,0 ,......,s i,p_1}

[0135] Among them, s i,0 =0,s i,p_1 =n i s i,j ≤s i,j+1 , that is s i,0 Indicates in the data sequence X' i The data position ranked first in the middle, s i,p-1 Indicates in the data sequence X' i The first data position in the (p-1)th data group is s. i,j Indicates in the data sequence X' i The first data position in the j-th partition, s i,j+1 Indicates in the data sequence X' i The first data position in the (j+1)th part is divided, and all the principal elements included in the first principal element sequence will generate a partitioning scheme for their respective data sequences.

[0136] After obtaining the first displacement array, based on the data groups divided in the first displacement array, each data sorting process exchanges its own data sequence with the data sequences of other data sorting processes. Each data sorting process then performs local sorting on the exchanged data sequences, thus obtaining an ordered sequence of data sorting results. This satisfies the requirement that data is ordered both within and between processes, improving the efficiency of large-scale data sorting computation.

[0137] Optionally, step S30, which involves exchanging data between the data sequences based on the first displacement array, includes:

[0138] Step S301: Generate a data partitioning standard based on the first displacement array;

[0139] Step S302: Data exchange between the data sequences is performed based on the data partitioning standard.

[0140] Based on the grouping of data included in the data sequence already divided by the first displacement array, a data partitioning standard is generated to divide the data sequence. Each data sorting process exchanges its own data sequence with the data sequences of other data sorting processes according to its own data partitioning standard. That is, data sorting process i will exchange data s in its data sequence. i,j to s i,j+1 Data between processes is sent to data sorting process j, while data sorting process i receives data from other data sorting processes s. j,i to s j,i+1 The data between them completes the data exchange between data sequences.

[0141] In this embodiment, an ordered candidate sequence of principal elements corresponding to each data sorting process is obtained. Each data sorting process includes a data sample sequence. The ordered candidate sequence is obtained by sorting the data sample sequences after data sample exchange between the data sorting processes. By exchanging data sample sequences between data sorting processes, the computational and memory overhead of each data sorting process is balanced, which is beneficial for large-scale data sorting. When a principal element is determined to meet a preset load balancing condition based on the ordered candidate sequence, it is checked whether its adjacent principal elements meet the preset load balancing condition. If it is detected that an adjacent principal element does not meet the preset load balancing condition... When conditions are met, the algorithm assists neighboring principals in calculations. When all principals meet the preset load balancing conditions, the principal and / or neighboring principals are broadcast to obtain the first principal sequence. The data sorting process, which completes the load balancing calculation of the corresponding principals, assists in the load balancing calculation of the undetermined principals, making full use of computing resources, accelerating the load balancing calculation of principals, and further improving the efficiency of data sorting. The data sequence is divided into a first displacement array based on the first principal sequence. Data exchange is performed between data sequences based on the first displacement array. The data sequences after data exchange are sorted locally to obtain an ordered result sequence, providing high-efficiency computing capabilities for large-scale data sorting.

[0142] Furthermore, this invention also proposes an electronic device, which includes a memory, a processor, and a computer processing program stored in the memory and executable on the processor. When the processor executes the computer processing program, it implements the steps of the above-mentioned data sorting method.

[0143] Furthermore, the present invention also proposes a computer-readable storage medium storing a computer processing program, wherein the modeling and simulation program, when executed by a processor, implements the steps of the above-described data sorting method.

[0144] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0145] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0147] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural changes made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A data sorting method, characterized in that, The data sorting method includes the following steps: Obtain the ordered candidate sequence of principal elements corresponding to each data sorting process, wherein each data sorting process includes a data sample sequence, and the ordered candidate sequence is obtained by the data sorting process sorting the data sample sequence after data sample exchange based on the data sample sequence; When the principal element corresponding to the ordered candidate sequence satisfies the preset load balancing condition, the neighboring principal elements of the principal element are detected to satisfy the preset load balancing condition. When the neighboring principal element is detected not to satisfy the preset load balancing condition, the neighboring principal element is assisted in the calculation. When it is determined that all the master elements meet the preset load balancing conditions, the master element and / or the adjacent master element are broadcast to obtain a first master element sequence, wherein the master element and / or the adjacent master element is an initialization master element or an updated master element; The data sequence is divided into a first displacement array based on the first principal component sequence. Data exchange is performed between the data sequences based on the first displacement array. The data sequences after data exchange are sorted locally to obtain an ordered result sequence. The data sorting method further includes: The median in the ordered candidate sequence is determined as the initial value, and the corresponding pivot element is initialized based on the initial value to obtain the initialized pivot element; After broadcasting the initial principal element and obtaining the second principal element sequence, the data sequence is divided according to the second principal element sequence to obtain the second displacement array; After calculating the vertical sum of the second displacement array, the relationship between the vertical sum and the preset lower limit and / or preset upper limit is determined. When the ordered candidate sequence is the ordered candidate sequence of the principal element corresponding to the i-th data sorting process, the vertical sum of the second displacement array is the total amount of data of the first i processes after data exchange between the data sequences based on the second displacement array. When the vertical axis is between a preset lower limit and a preset upper limit, it is determined that the initialization master element satisfies the preset load balancing condition.

2. The data sorting method as described in claim 1, characterized in that, The data sorting method further includes: Sort the local data to obtain a data sequence; The data sequence is subjected to regular sampling to obtain the data sample sequence.

3. The data sorting method as described in claim 1, characterized in that, The data sample sequence includes several data samples arranged sequentially, and the data sorting method further includes: Non-target data samples in the data sample sequence are sent to the corresponding target data sorting process, and target data samples are received for data sample exchange. The first sample sequence number of the non-target data sample is not the same as the first process sequence number of the data sorting process to which the data sample sequence belongs, and the first sample sequence number is the same as the second process sequence number of the target data sorting process. The second sample sequence number of the target data sample is the same as the first process sequence number, and the second sample sequence number is not the same as the second process sequence number.

4. The data sorting method as described in claim 1, characterized in that, After the step of determining the size relationship between the vertical sum and the preset lower limit and / or preset upper limit, the data sorting method further includes; When the vertical sum is less than the preset lower limit, a binary search is performed according to the increasing direction of each value in the ordered candidate sequence to obtain an updated value. The initial principal element is then updated based on the updated value to obtain the updated principal element. Broadcast the updated principal element, execute the step of obtaining the second principal element sequence, until the updated principal element satisfies the preset load balancing condition when the vertical sum determined based on the updated principal element is between the preset lower limit value and the preset upper limit value.

5. The data sorting method as described in claim 1, characterized in that, After the step of determining the relationship between the vertical sum and the preset lower limit and / or preset upper limit, the data sorting method further includes: When the vertical sum is greater than the preset upper limit, a binary search is performed according to the decreasing direction of each value in the ordered candidate sequence to obtain an updated value. The initial principal element is then updated based on the updated value to obtain the updated principal element. Broadcast the updated principal element, execute the step of obtaining the second principal element sequence, until the updated principal element satisfies the preset load balancing condition when the vertical sum determined based on the updated principal element is between the preset lower limit value and the preset upper limit value.

6. The data sorting method as described in claim 1, characterized in that, The step of detecting whether the adjacent primary elements of the primary element meet the preset load balancing conditions includes: Based on the adjacent principal elements, candidate values ​​are determined, and based on the candidate values, it is determined whether the adjacent principal elements meet the preset load balancing conditions. After the step of determining whether the adjacent principal elements meet the preset load balancing conditions based on the candidate values, the data sorting method further includes: When it is determined that the adjacent primary elements meet the preset load balancing conditions, it is determined that each primary element and the adjacent primary elements meet the preset load balancing conditions.

7. The data sorting method as described in claim 1, characterized in that, The step of exchanging data between the data sequences based on the first displacement array includes: Data partitioning criteria are generated based on the first displacement array; Data exchange between the data sequences is performed based on the data partitioning criteria.

8. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer processing program stored in the memory and executable on the processor, the computer processing program being configured to implement the steps of the data sorting method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer processing program, which, when executed by a processor, implements the steps of the data sorting method as described in any one of claims 1 to 7.

Images