A parallel sorting-based log collection method and system

Abstract

The application provides a log collection method and system based on parallel sorting, which specifically comprises the following steps: (1) data sampling: collecting log data from different log sources, transmitting the log data to a data processing node, and using a big data-oriented scalable regular sampling method to obtain representative samples from the log data; (2) parallel sorting: distributing the representative samples to multiple nodes in the system according to certain rules, and sorting the log data on each node using a parallel sorting algorithm; (3) parallel writing: collecting the sorted log data representative samples into a file system in parallel; (4) monitoring and optimization: monitoring the space utilization of the file system in real time through a verification file and a checking tool, and regularly checking the integrity of the log data. The application uses a big data-oriented scalable regular sampling method to preprocess and sample the log, and uses parallel sorting technology to efficiently collect the log data. Then, the sampled log sample data is distributed to multiple processing nodes to realize parallel sorting operation. The application disperses sample storage and processing tasks to each process for sorting and iteratively updates the pivot element, thereby improving parallel efficiency, reducing processing bottlenecks caused by uneven data distribution, and improving the overall performance and reliability of the log management system.

Description

Technical Field

[0001] This invention relates to a log collection method and system based on parallel sorting optimization, belonging to the field of parallel computing. Background Technology

[0002] In today's digital age, distributed systems have permeated various fields and industries, from cloud computing to the Internet of Things, from big data to artificial intelligence. With the rapid development of distributed systems and the horizontal scaling of complex applications, log management faces new challenges. Logs are no longer limited to generation by a single device or application, but are distributed and stored across different devices and systems. As various applications run, the number of logs is exploding. The traditional method of troubleshooting by accessing machine logs is becoming increasingly impractical. On the one hand, because logs are distributed across different machines, troubleshooting consumes a lot of manpower; on the other hand, all engineers have a need to query logs, and granting everyone access to machines cannot guarantee security. In this context, establishing a comprehensive server log collection tool to collect server log data, analyze system status, and quickly identify system faults is of great significance for improving operational efficiency and ensuring normal business operations. Currently, common log collection solutions suffer from poor performance, low reliability, and poor scalability, specifically manifested in the following ways:

[0003] (1) Syslog is a standard logging protocol that can send log information generated by systems and applications to a centralized log server or centralized storage, and is easy to implement and deploy. However, Syslog has low security, and plaintext log transmission is vulnerable to network sniffing and hijacking threats.

[0004] (2) Fluentd is an open-source log collector that supports multiple input and output sources and provides a flexible plugin mechanism, making it suitable for large-scale log collection and processing. However, in some specific application scenarios, Fluentd's performance is not efficient enough and requires proper configuration and optimization.

[0005] In addition, with business expansion, the amount of log data is growing rapidly, and traditional log collection solutions often cannot be effectively scaled, making it difficult for the system to cope with the high load and large-scale log data processing needs. Attached Figure Description

[0006] Figure 1 This is a system flowchart.

[0007] Figure 2 This is a flowchart of the data sampling process. Summary of the Invention

[0008] To address the problems existing in the above-mentioned technologies, this application proposes a log collection optimization method based on parallel sorting, the specific steps of which include:

[0009] (1) Data sampling: Collect log data from different log sources, transmit the log data to the data processing node, and use a scalable regular sampling method for big data to obtain representative samples from the log data;

[0010] (2) Parallel sorting: The above representative samples are distributed to multiple nodes in the system according to certain rules, and the log data is sorted on each node using a parallel sorting algorithm;

[0011] (3) Parallel writing: Representative samples of sorted log data are collected in parallel into the file system;

[0012] (4) Monitoring and optimization: Monitor the file system space utilization in real time through verification files and inspection tools, and check the integrity of log data regularly.

[0013] Furthermore, in step (1), given locally ordered data Uniformly extract (p-1) samples, where p is the number of processes, and construct the sample set. n i For a locally ordered data sequence x i 'size'; i is the process ID; k is the number of samples to be extracted during the iteration. U represents the data uniformity.

[0014] Furthermore, in step (1), sampling is performed using distributed processing, with each process i processing its sample set Y. i Sample y in i,j Send to process j, and simultaneously receive sample y from process j. i,j Processes i and j exchange samples, where i∈{0,...,p-1}, j∈{0,...,p-1}, and each process receives a sample sequence Y of size O(p). i ′={y′ i,j =y i,j Each process receives the sample sequence Y. i Sort the samples to obtain an ordered sample sequence Y. i ";Ordered sequence Y i "Used to determine the i-th pivot element z" i The candidate values, where the first Sample As the i-th pivot z i The initial candidate values; each process determines a candidate value for a pivot element, forming a pivot sequence Z = {z1, ..., z2}. p-1} is used to partition the data sequence X′ i It forms p disjoint subsequences.

[0015] Furthermore, in step (1), when obtaining representative samples, all pivot elements Z = {z1, ..., z} are collected. p-1}, for z i Sort and calculate longitudinal displacement. Where S i,j Let i be the displacement array from process i to j. This indicates the pivot element z j The condition of the j-th pivot of the load-balanced data partition is met, where, Let b be the ideal longitudinal displacement, b be the deviation rate, and n be the total amount of data.

[0016] Based on the above methods, this application proposes a log collection system optimized by parallel sorting, specifically including:

[0017] (1) Data sampling module: This module collects log data from different log sources, transmits the log data to the data processing node, and uses a scalable regular sampling method for big data to obtain representative samples from the log data;

[0018] (2) Parallel sorting module: This module distributes the above representative samples to multiple nodes in the system according to certain rules, and uses a parallel sorting algorithm to sort the log data on each node;

[0019] (3) Parallel writing module: This module collects representative samples of sorted log data in parallel into the file system;

[0020] (4) Monitoring and optimization module: This module monitors the space utilization of the file system in real time through verification files and inspection tools, and periodically checks the integrity of log data.

[0021] Furthermore, in the data sampling module, given locally ordered data... Uniformly extract (p-1) samples, where p is the number of processes, and construct the sample set. n i For a locally ordered data sequence x′ i Size; i is the process ID; k is the number of samples to be extracted during the iteration. U represents the data uniformity.

[0022] Furthermore, the data sampling module employs distributed processing, with each process i processing its sample set Y. i Sample y in i,j Send to process j, and simultaneously receive sample y from process j. i,jProcesses i and j exchange samples, where i∈{0,...,p-1}, j∈{0,...,p-1}, and each process receives a sample sequence Y of size O(p). i ′={y′ i,j =y i,j Each process receives the sample sequence Y. i Sort the samples to obtain an ordered sample sequence Y. i ";Ordered sequence Y i "Used to determine the i-th pivot element z" i The candidate values, where the first Sample As the i-th pivot z i The initial candidate values; each process determines a candidate value for a pivot element, forming a pivot sequence Z = {z1, ..., z2}. p-1} is used to partition the data sequence X′ i It forms p disjoint subsequences.

[0023] Furthermore, in the data sampling module, when acquiring representative samples, all pivot elements Z = {z1,…,z} are collected. p-1}, for z i Sort and calculate longitudinal displacement. Where S i,j Let i be the displacement array from process i to j. This indicates the pivot element z j The condition of the j-th pivot of the load-balanced data partition is met, where, Let b be the ideal longitudinal displacement, b be the deviation rate, and n be the total amount of data.

[0024] The following technical effects are achieved by adopting this invention:

[0025] (1) Efficient Log Data Sampling and Sorting: A scalable regularized sampling algorithm for big data is adopted, which can effectively handle large-scale log data. Furthermore, based on the characteristics of data distribution, different sampling densities are used in different segments, adaptively adjusting the sampling density. Considering the distribution characteristics of the data, this invention introduces an adaptive α value to adaptively adjust the sampling density based on factors such as data uniformity, thereby improving the representativeness of the sampling. Utilizing multi-node parallel processing and a distributed sorting algorithm, log data can be sorted quickly, improving sorting efficiency and performance.

[0026] (2) Improved system stability and reliability: Through the collective communication operation in the data redistribution stage, the integrity and consistency of the data are ensured, and by distributing sample storage and processing tasks to each process, the storage pressure of the main process will be further reduced, and the reliability will be enhanced.

[0027] (3) Good scalability: It can adapt to the rapid growth of log data volume and business expansion. Through distributed processing and parallel sorting, it can effectively cope with the processing needs of high load and large-scale log data, ensuring that the system can still operate efficiently as it continues to evolve.

[0028] This invention employs a scalable regularized sampling method for big data preprocessing and sampling of logs, and utilizes parallel sorting technology for efficient log data collection. Subsequently, the sampled log data is distributed across multiple processing nodes to achieve parallel sorting operations. This invention improves parallel efficiency, reduces processing bottlenecks caused by uneven data distribution, and enhances the overall performance and reliability of the log management system by distributing sample storage and processing tasks across various processes and iteratively updating pivot elements. Detailed Implementation

[0029] The process of this invention is as follows Figure 1 As shown. The specific details of each step in the method are as follows:

[0030] (1) Data Sampling: Log data is collected from different log sources and transmitted to the data processing node. Then, a scalable regularized sampling algorithm for big data is used to obtain representative samples from the large-scale log data. The data sampling flowchart is as follows: Figure 2 As shown.

[0031] The algorithm adaptively determines the sampling ratio based on the data volume and distribution, and filters the sample set according to certain selection rules to ensure the quality and integrity of the sampled data. Secondly, it performs distributed processing, allocating data to multiple nodes for parallel processing to fully utilize computing resources for handling large-scale data and accelerate the entire sampling process. During data distribution, it is necessary to consider issues of uniform data distribution and load balancing to ensure that the computational tasks on each node are balanced. Through the data sampling stage, a representative set of data samples is obtained, laying the foundation for subsequent sorting and writing optimizations.

[0032] The sampling algorithm is as follows:

[0033] In a distributed system, given locally ordered data X′ i Each process (with a total of p processes) will extract p-1 samples evenly from the sample according to the sampling rules. These samples maintain an ordered structure, i.e., Y i Also constituting an ordered sequence, and considering the varying data distributions, this application will use different sampling densities in different segments based on the characteristics of the data distribution, adaptively adjusting the sampling density. Taking into account the data distribution characteristics, this invention introduces an adaptive α value, adaptively adjusting the sampling density based on factors such as data uniformity to improve the representativeness of the sampling. The sampling formula is as follows: Where, n i For a locally ordered data sequence x′ i Size, where i is the process ID and k is the number of samples to be extracted during iteration. U represents the uniformity of the data.

[0034] By distributing sample storage and processing tasks across various processes, the storage pressure on the main process will be further reduced. The operation flow is as follows: each process i stores its sample set Y. i Sample y in i,j Send to process j, and simultaneously receive sample y from process j. i,j Processes i and j exchange samples, where i∈{0,...,p-1}, j∈{0,...,p-1}. Therefore, each process receives a sample set Y of size O(p). i ′={y′ i,j =y i,j}

[0035] Next, each process processes the received sample set Y i Sort the samples to obtain an ordered sample sequence Y. i ". Ordered sequence Y i "Used to determine the i-th pivot element z" i The candidate values, where the first Sample Selected as the i-th pivot z i The initial candidate values. Each process is responsible for determining the candidate values ​​for one pivot element, forming the pivot sequence Z = {z1, ..., z2}. p-1} is used to partition the data sequence X′ i Given p disjoint subsequences, the goal is to redistribute the data so that each process has a relatively balanced amount of data after redistribution.

[0036] To achieve dynamic adjustment and load balancing of pivot elements, the pivot elements will be iteratively updated. This will be done after collecting all pivot elements Z = {z1, ..., z...}. p-1 After that, z needs to be... i Sort the data to ensure its order. To improve data processing efficiency, the calculation of the vertical displacement sum is added to make the pivot elements more reasonably divide the data. Each process will independently use the pivot array Z to sort the locally sorted data X. i Divide the data into segments and determine the offset array S representing the data partitions. i Calculate the longitudinal displacement and (S i,j (where the shift array is from process i to process j), if This indicates the pivot element z jThe j-th pivot satisfies the load balancing requirements of the data partition. By dynamically adjusting and iteratively updating pivot elements, it is possible to better adapt to changes in data distribution and reduce load imbalance issues during data processing. in Let b be the ideal longitudinal displacement, b be the deviation rate, and n be the total amount of data.

[0037] This sampling method improves parallel efficiency by distributing sample storage and processing tasks to various processes for sorting and iteratively updating pivot elements, reducing processing bottlenecks caused by uneven data distribution, thereby enhancing the overall performance and reliability of the log management system.

[0038] (2) Parallel sorting: The sampled log data is distributed to multiple nodes in the system according to certain rules, and the distributed database log files are sorted on each node using a parallel sorting algorithm.

[0039] The parallel sorting algorithm process is as follows:

[0040] a. Local sorting: Each process sorts the local data sequence X. i Perform a local sort to obtain a new ordered sequence X. i ′, making the All satisfy x′ i,j ≤x′ i,j+1 .

[0041] b. Determine data partitions: Perform histogram analysis on the local data sequence to determine the data distribution characteristics. Based on the histogram analysis results, each process will determine the data partitioning points and the shift array S. i ={s i,j |j=0,...,p-1} is used to represent the data partitioning situation, and the locally ordered data sequence X′ is used to represent the partitioning situation. i Divide into p disjoint subsequences X′ i,j ={x′ i,k |s i,j ≤k i,j+1 The shift array S is dynamically adjusted based on the actual size of each subsequence. i The position of the dividing point satisfies s i,o =0,s i,p =n i ,s i,j ≤s i,j+1 Note that if s i,j =s i,j+1 Then X′ i,j Empty.

[0042] c. Data redistribution: Each process redistributes data according to the offset array S representing the data partitioning situation.​i Sending different numbers of data elements to p-1 other processes and receiving different numbers of data elements from the other p-1 processes through set communication operations, resulting in p locally ordered data sequences X″. i .

[0043] d. Re-sorting locally: Re-sorting the received data sequence X″ after data redistribution. i Perform a local sort to obtain a new ordered sequence. After this stage of operation is completed, the data in each process and between processes is ordered.

[0044] (3) Parallel writing: The sorted log data samples are collected into the file system in parallel. The goal of this stage is to leverage the advantages of parallel processing to improve the efficiency of log file collection operations.

[0045] Using multi-threading or multi-processing, log text collection operations can be performed simultaneously on multiple nodes. Threads or processes will store each log entry into the corresponding text system in the order they are recorded. Since log files may contain a large number of records, it is necessary to ensure system performance and stability. Therefore, consider using batch insertion or batch update methods to collect multiple records into the file system at once.

[0046] (4) Monitoring and Optimization: First, it is necessary to monitor the file system's space utilization in real time to ensure sufficient storage space for log data and avoid data loss or write failures due to insufficient storage space. Second, the integrity of log files should be checked regularly. By verifying the file's checksum or using file integrity checking tools, it should be ensured that the log files are not corrupted or tampered with, and abnormal log files should be handled promptly to ensure data reliability. In addition, monitoring the file system's read and write performance is essential. Read and write latency and throughput should be tracked, and performance optimization should be performed to ensure that data can be written and read efficiently.

[0047] A more specific example:

[0048] Suppose there is a log file containing user activity records user queries, inserts, updates, and other operations. Each log entry contains information such as timestamp, operation type, and user ID.

[0049] (1) These log files are sampled for use in subsequent data processing. Each process samples locally ordered data X′. i p-1 samples are sampled using regularization to form the subsequence Y. i Assume there are 4 processes (p=4) processing data in parallel, and the log file has a total of 4000 records (n=1000 for each node). Each node obtains samples from its local data according to a certain sampling ratio to form a sample set Y.i .

[0050] Each node sorts its acquired sample sequence, selects a portion as candidate values ​​for the pivot element, forming the pivot sequence Z. Each node divides its local data according to the pivot sequence and calculates the longitudinal displacement.

[0051] Assume the deviation rate is b = 0.1. Then... Therefore, for the j-th pivot, the upper and lower bounds of the longitudinal displacement are 8.206 and 8.352, respectively, and the pivot elements are iteratively updated based on these results.

[0052] After applying a regularized sampling algorithm, the data is distributed across multiple nodes for parallel processing. This fully utilizes computing resources to process large-scale data and accelerates the entire sampling process. Through the data sampling stage, a representative sample of data is obtained.

[0053] (2) Next, these log files will be sorted in parallel, following the process below: Assume there is a distributed system with 4 nodes, each with the following database log files:

[0054] Node 1: [A,D,G,J]

[0055] Node 2: [B,E,H,K]

[0056] Node 3: [C,F,I,L]

[0057] Node 4: [M, N, O, P]

[0058] a. Local sorting: Each node sorts its local database log files. Assume the sorting rule is alphabetical order.

[0059] Node 1: [A,D,G,J]->[A,D,G,J]

[0060] Node 2: [B,E,H,K]->[B,E,H,K]

[0061] Node 3: [C,F,I,L]->[C,F,I,L]

[0062] Node 4: [M,N,O,P]->[M,N,O,P]

[0063] b. Determine data partitioning: Each node determines how to divide the data. Let's assume the data is divided into two parts.

[0064] Node 1: S1 = [0, 2, 4], indicates that the data is divided into two subsequences: [A, D], [G, J].

[0065] Node 2: S2 = [0, 2, 4] indicates that the data is divided into two subsequences: [B, E], [H, K]

[0066] Node 3: S3 = [0, 2, 4], indicates that the data is divided into two subsequences: [C, F], [I, L]

[0067] Node 4: S4 = [0, 2, 4] indicates that the data is divided into two subsequences: [M, N], [O, P]

[0068] c. Data redistribution: Each node sends and receives data to other nodes according to the data partitioning situation.

[0069] Node 1: Sends [G,J] to Node 2 and receives [B,E] and [C,F].

[0070] Node 2: Sends [H,K] to Node 3 and receives [A,D] and [I,L].

[0071] Node 3: Sends [I,L] to Node 4 and receives [B,E] and [G,J].

[0072] Node 4: Sends [O,P] to Node 1 and receives [C,F] and [H,K].

[0073] d. Local sorting again: Each node sorts the received data locally.

[0074] Node 1: [A,D,G,J,B,E,C,F]->[A,B,C,D,E,F,G,J]

[0075] Node 2:[B,E,H,K,A,D,I,L]->[A,B,D,E,H,I,K,L]

[0076] Node 3: [C,F,I,L,B,E,G,J]->[B,C,E,F,G,I,J,L]

[0077] Node 4: [M,N,O,P,C,F,H,K]->[C,F,H,K,M,N,O,P]

[0078] After these stages are completed, the data on each node is ordered, and the data across all nodes is also ordered. Parallel sorting is complete.

[0079] (3) The parallel write phase uses four write nodes to process data in parallel. Assuming each write node processes one record, records can be assigned to different write nodes. For example, the first write node processes the first record, the second node processes the second record, and so on.

[0080] (4) Monitoring and optimization phase: Based on the performance and stability of parallel acquisition operations, monitor the space utilization of the file system in real time, check the integrity of log files regularly, and handle abnormal log files in a timely manner to ensure data reliability.

[0081] This embodiment utilizes four processes to process data in parallel, fully leveraging the computing power of multi-core processors to significantly improve the parallel sorting speed. Experimental results show that compared to traditional serial processing, the parallel processing system can reduce processing time by more than 50% when handling large-scale datasets. Furthermore, the regularized sampling algorithm ensures the representativeness of samples selected from log files. When comparing multiple datasets, the system using regularized sampling achieves higher sample coverage and better captures the data distribution characteristics. Moreover, the local sorting and data redistribution strategies ensure that data across all nodes is ordered in the final output.

[0082] The units, devices, or modules described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above devices are described by dividing them into various modules according to their functions. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection between the devices or units shown or discussed can be through some interfaces, and the indirect coupling or communication connection between the devices or units can be electrical, mechanical, or other forms.

[0083] Those skilled in the art will also know that, besides implementing the controller using purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices within it used to implement various functions can also be considered structures within that hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the method and structures within a hardware component.

[0084] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0085] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0086] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. This application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0087] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A log collection method based on parallel sorting optimization, characterized in that: The specific steps include: (1) Data sampling: During data sampling, log data is collected from different log sources, the log data is transmitted to the data processing node, and a scalable regular sampling method for big data is used to obtain representative samples from the log data; (2) Parallel sorting: The above representative samples are distributed to multiple nodes in the system according to certain rules, and the log data is sorted on each node using a parallel sorting algorithm; (3) Parallel writing: Representative samples of sorted log data are collected in parallel into the file system; (4) Monitoring and optimization: Monitor the file system space utilization in real time through verification files and inspection tools, and check the integrity of log data regularly; In step (1), a local ordered data sequence is given. Uniform extraction Given a sample set, where p is the number of processes, construct the sample set. , Locally ordered data sequence size; i Assign a process number; k To iterate through the number of samples to be extracted, an adaptive value is needed. , U For data uniformity; Sampling is performed using distributed processing, with each process i processing its own sample set. Samples in Send to process j, and simultaneously receive samples from process j. ,process i Sample exchange is performed between process A and process B, where Each process receives a size of sample sequence Each process receives the sample sequence. Sort the samples to obtain an ordered sequence. ; ordered sample sequence Used to determine the i-th pivot element The candidate values, where the first Sample As the i-th pivot The initial candidate values; each process determines a candidate value for a pivot element, forming a pivot sequence. Used to divide data sequences Form p distinct subsequences; When obtaining representative samples, collect all pivot elements. ,right Sort and calculate longitudinal displacement. ,in Let i be the displacement array from process i to j. This indicates the pivot element The first conforming to load balancing data partitioning j The conditions of a pivot, among which, , Let b be the ideal longitudinal displacement, b be the deviation rate, and n be the total amount of data.

2. A log acquisition system based on parallel sorting optimization, characterized in that: The system includes: (1) Data sampling module: The data sampling module collects log data from different log sources, transmits the log data to the data processing node, and uses a scalable regular sampling method for big data to obtain representative samples from the log data; (2) Parallel sorting module: This module distributes the above representative samples to multiple nodes in the system according to certain rules, and sorts the log data on each node using a parallel sorting algorithm; (3) Parallel writing module: This module collects representative samples of sorted log data in parallel into the file system; (4) Monitoring and optimization module: This module monitors the file system space utilization in real time through file verification and inspection tools, and periodically checks the integrity of log data; In the data sampling module, given a local ordered data sequence Uniform extraction One sample, p To construct a sample set for the number of processes. , Locally ordered data sequence size; i Assign a process number; k To iterate through the number of samples to be extracted, an adaptive value is needed. , U For data uniformity; In the data sampling module, sampling is carried out in a distributed manner, with each process i processing its sample set. Samples in Send to process j, and simultaneously from process j j Received samples Samples are exchanged between process i and process 𝑗, where Each process receives a size of sample sequence Each process receives the sample sequence. Sort the samples to obtain an ordered sequence. ; ordered sample sequence Used to determine the first i Pivot elements The candidate values, where the first Sample As the first i Pivot The initial candidate values; each process determines a candidate value for a pivot element, forming a pivot sequence. Used to divide data sequences become p A number of disjoint subsequences; In the data sampling module, when obtaining representative samples, all pivot elements are collected. ,right Sort and calculate longitudinal displacement. ,in For process i arrive j The displacement array, if This indicates the pivot element The first conforming to load balancing data partitioning j The conditions of a pivot, among which, , For the ideal longitudinal displacement, b The deviation rate, n This represents the total amount of data.

Images