Method and system for optimizing log collection based on parallel sorting

Abstract

The invention provides a log collection method and system based on parallel sorting optimization, and the method specifically comprises the steps: (1) data sampling: collecting log data from different log sources, transmitting the log data to a data processing node, and obtaining a representative sample from the log data through a big data-oriented extensible regular sampling method; (2) parallel sorting: distributing the representative samples to a plurality of nodes in a system according to a certain rule, and sorting the log data on each node by using a parallel sorting algorithm; (3) parallel writing: collecting the sorted representative samples of the log data into a file system in parallel; and (4) monitoring and optimizing: monitoring the space utilization rate of a file system in real time through a verification file and an inspection tool, and regularly inspecting the integrity of log data. According to the method, a big data-oriented extensible regular sampling method is adopted to pre-process and sample logs, and a parallel sorting technology is utilized to efficiently collect log data. And then, the sampled log sample data is distributed to a plurality of processing nodes, so that parallel sorting operation is realized. According to the method, sample storage and processing tasks are scattered to each process for sorting, and pivot elements are iteratively updated, so that the parallel efficiency is improved, the processing bottleneck caused by non-uniform data distribution is reduced, and the overall performance and reliability of a log management system are improved.

Description

Technical Field

[0001] The invention relates to a log collection method and system based on parallel sorting optimization, and belongs to the field of parallel computing. Background Art

[0002] In today's digital age, from cloud computing to the Internet of Things, from big data to artificial intelligence, distributed systems have been used in all fields and industries. With the rapid development of distributed systems and the horizontal expansion of complex applications, log management faces new challenges. Logs are no longer limited to a single device or application, but are stored in different devices and systems. With the operation of various applications, the number of logs is showing an explosive growth trend. The traditional way of accessing machines to query logs to troubleshoot problems has become increasingly unfeasible. On the one hand, because logs are scattered on different machines, a lot of manpower is consumed when troubleshooting problems; on the other hand, all engineers have the need to query logs, and giving everyone access to machines cannot guarantee security. In this case, establishing a complete server log collection tool, collecting server log data, analyzing system status, and quickly discovering system failures are of great significance to improving operation and maintenance efficiency and ensuring the normal operation of business. At present, common log collection solutions have problems such as poor performance, low reliability, and poor scalability, which are specifically manifested in:

[0003] (1) Syslog is a standard log protocol that can send log information generated by systems and applications to a centralized log server or centralized storage. It is easy to implement and deploy. However, the Syslog security of this protocol is low, and plaintext log transmission is vulnerable to network sniffing and hijacking.

[0004] (2) Fluentd is an open source log collector that supports multiple input and output sources and provides a flexible plug-in mechanism, which is suitable for large-scale log collection and processing. However, in some specific application scenarios, Fluentd's performance is not efficient enough and requires reasonable configuration and optimization.

[0005] In addition, as the business expands, the amount of log data grows rapidly. Traditional log collection solutions often cannot be effectively expanded, making it difficult for the system to cope with high-load and large-scale log data processing requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 It is the system flow chart.

[0007] Figure 2 This is the data sampling flow chart. Summary of the invention

[0008] In view of the problems existing in the above prior art, this application proposes a log collection method based on parallel sorting optimization, and the specific steps include:

[0009] (1) Data sampling: Collect log data from different log sources, transmit the log data to data processing nodes, and use a scalable regular sampling method for big data to obtain representative samples from the log data;

[0010] (2) Parallel sorting: Distribute the representative samples to multiple nodes in the system according to certain rules, and use a parallel sorting algorithm to sort the log data on each node;

[0011] (3) Parallel writing: Collect representative samples of sorted log data into the file system in parallel;

[0012] (4) Monitoring and optimization: Use validation files and inspection tools to monitor the space utilization of the file system in real time and regularly check the integrity of log data.

[0013] Furthermore, in step (1), given the local ordered data Uniformly extract (p-1) samples, where p is the number of processes, to construct a sample set n i is the local ordered data sequence x i ′ size; i is the process number; k is the number of samples to be extracted during the traversal, U is the data uniformity.

[0014] Furthermore, in step (1), distributed processing is adopted for sampling, and each process i distributes its sample set Y i Sample y in i,j Send to process j and receive sample y from process j i,j , process i and process j exchange samples, where i∈{0,...,p-1},j∈{0,...,p-1}, each process receives a sample sequence Y of size O(p) i ′={y′ i,j =y i,j}; Each process receives the sample sequence Y i ′ is sorted to obtain the ordered sample sequence Y i ″; ordered sequence Y i ″ is used to determine the i-th pivot element z i The candidate values ​​of Samples As the i-th pivot z i Each process determines a candidate value for a pivot element, forming a pivot sequence Z = {z1,…,z p-1}, used to divide the data sequence X′ i into p mutually disjoint subsequences.

[0015] Furthermore, in step (1), when obtaining representative samples, all pivot elements Z = {z1, ..., z p-1}, for z i Sorting, calculating the longitudinal and Where S i,j is the displacement array from process i to j, if Indicates the pivot element z j The jth pivot of the load-balanced data partition is qualified as follows: is the longitudinal sum of ideal displacement, b is the deviation rate, and n is the total amount of data.

[0016] Based on the above method, this application proposes a log collection system based on parallel sorting optimization, which specifically includes:

[0017] (1) Data sampling module: This module collects log data from different log sources, transmits the log data to the data processing node, and uses a scalable regular sampling method for big data to obtain representative samples from the log data;

[0018] (2) Parallel sorting module: This module distributes the above representative samples to multiple nodes in the system according to certain rules, and uses a parallel sorting algorithm to sort the log data on each node;

[0019] (3) Parallel writing module: This module collects representative samples of sorted log data into the file system in parallel;

[0020] (4) Monitoring and optimization module: This module monitors the space utilization of the file system in real time through validation files and inspection tools, and regularly checks the integrity of log data.

[0021] Furthermore, in the data sampling module, given the local ordered data Uniformly extract (p-1) samples, where p is the number of processes, to construct a sample set n i is the local ordered data sequence x′ i size; i is the process number; k is the number of samples to be extracted during the traversal. U is the data uniformity.

[0022] Furthermore, the sampling in the data sampling module adopts distributed processing, and each process i takes its sample set Y i Sample y in i,j Send to process j and receive sample y from process j i,j, process i and process j exchange samples, where i∈{0,...,p-1},j∈{0,...,p-1}, each process receives a sample sequence Y of size O(p) i ′={y′ i,j =y i,j}; Each process receives the sample sequence Y i ′ is sorted to obtain the ordered sample sequence Y i ″; ordered sequence Y i ″ is used to determine the i-th pivot element z i The candidate values ​​of Samples As the i-th pivot z i Each process determines a candidate value for a pivot element, forming a pivot sequence Z = {z1,…,z p-1}, used to divide the data sequence X′ i into p mutually disjoint subsequences.

[0023] Furthermore, in the data sampling module, when obtaining representative samples, all pivot elements Z = {z1,…,z p-1}, for z i Sorting, calculating the longitudinal and Where S i,j is the displacement array from process i to j, if Indicates the pivot element z j The jth pivot of the load-balanced data partition is qualified as follows: is the longitudinal sum of ideal displacement, b is the deviation rate, and n is the total amount of data.

[0024] After adopting the present invention, the following technical effects are achieved:

[0025] (1) Efficient log data sampling and sorting: The scalable regular sampling algorithm for big data can effectively process large-scale log data, and according to the characteristics of data distribution, different sampling densities are used in different segments to adaptively adjust the sampling density. Considering the distribution characteristics of the data, the present invention introduces an adaptive α value to adaptively adjust the collection density according to the uniformity of the data, thereby improving the representativeness of the sampling. By using multi-node parallel processing and distributed sorting algorithms, log data can be quickly sorted, which improves sorting efficiency and performance.

[0026] (2) Improved system stability and reliability: The integrity and consistency of the data are ensured through collective communication operations in the data redistribution phase. In addition, by distributing sample storage and processing tasks to various processes, the storage pressure of the main process is further reduced and reliability is enhanced.

[0027] (3) Good scalability: It can adapt to the rapid growth of log data volume and business expansion. Through distributed processing and parallel sorting, it can effectively cope with the processing needs of high-load and large-scale log data, ensuring that the system can still operate efficiently under continuous development.

[0028] The present invention uses a scalable regular sampling method for big data to preprocess and sample logs, and uses parallel sorting technology to efficiently collect log data. Subsequently, the sampled log sample data will be distributed to multiple processing nodes to achieve parallel sorting operations. The present invention disperses sample storage and processing tasks to each process for sorting, and iteratively updates the pivot elements, thereby improving parallel efficiency, reducing processing bottlenecks caused by uneven data distribution, and improving the overall performance and reliability of the log management system. DETAILED DESCRIPTION

[0029] The process of the present invention is as follows Figure 1 The specific contents of each step in the method are as follows:

[0030] (1) Data sampling: Collect log data from different log sources and transmit the data to the data processing node. Then, use a scalable regular sampling algorithm for big data to obtain representative samples from large-scale log data. The data sampling flow chart is as follows: Figure 2 shown.

[0031] The algorithm adaptively determines the sampling ratio according to the amount and distribution of data, and screens the sample set according to certain screening rules to ensure the quality and integrity of the sampled data. Secondly, the data is distributed and distributed to multiple nodes for parallel processing, making full use of computing resources to process large-scale data and accelerate the entire sampling process. In the process of data distribution, it is necessary to consider the problems of uniform data distribution and load balancing to ensure that the computing tasks of each node are balanced. Through the processing of the data sampling stage, a part of representative data samples is obtained, which lays the foundation for subsequent sorting and writing optimization.

[0032] The specific sampling algorithm is:

[0033] In a distributed system, given locally ordered data X′ i , each process (the total number of processes is p) will uniformly extract p-1 samples from it according to the sampling rule. These samples maintain order, that is, Y i It is also an ordered sequence. Considering that the data distribution is different, this application will use different sampling densities in different segments according to the characteristics of data distribution and adaptively adjust the sampling density. Considering the distribution characteristics of data, the present invention introduces an adaptive α value to adaptively adjust the acquisition density according to the uniformity of data, etc., to improve the representativeness of sampling. The sampling formula is Among them, n i is the local ordered data sequence x′ i size, i is the process number, k is the number of samples to be extracted during the traversal, U is the uniformity of the data.

[0034] By distributing sample storage and processing tasks to each process, the storage pressure of the main process will be further reduced. The operation process is as follows: each process i will store its sample set Y i Sample y in i,j Send to process j and receive sample y from process j i,j , processes i and j exchange samples, where i∈{0,...,p-1},j∈{0,...,p-1}. Therefore, each process receives a sample set Y of size O(p) i ′={y′ i,j =y i,j}.

[0035] Next, each process receives the sample set Y i ′ is sorted to obtain the ordered sample sequence Y i ″. Ordered sequence Y i ″ is used to determine the i-th pivot element z i The candidate values ​​of Samples is chosen as the i-th pivot z i Each process is responsible for determining a candidate value for a pivot element, forming a pivot sequence Z = {z1,…,z p-1}, used to divide the data sequence X′ i It is expected that the amount of data owned by each process after redistribution is relatively balanced.

[0036] In order to achieve dynamic adjustment and load balancing of pivot elements, the pivot elements are iteratively updated. After all the pivot elements Z = {z1,…,z p-1}, you need to i To improve the efficiency of data processing, add the calculation of the longitudinal sum of the displacement, and make the pivot element divide the data more reasonably. Each process will independently use the pivot array Z to sort the locally sorted data X. i ' to divide and determine the displacement array S representing the data partition i , calculate the longitudinal and (S i,j is the displacement array from process i to process j), if Indicates the pivot element z jThe jth pivot that satisfies the load balancing data partition. By dynamically adjusting and iteratively updating the pivot elements, it can better adapt to changes in data distribution and reduce the load imbalance problem in the data processing process. in is the longitudinal sum of ideal displacement, b is the deviation rate, and n is the total amount of data.

[0037] This sampling method can improve parallel efficiency by distributing sample storage and processing tasks to various processes for sorting and iterative updating of pivot elements, reducing processing bottlenecks caused by uneven data distribution, thereby improving the overall performance and reliability of the log management system.

[0038] (2) Parallel sorting: The sampled log data is distributed to multiple nodes in the system according to certain rules, and the parallel sorting algorithm is used on each node to sort the allocated database log files.

[0039] The parallel sorting algorithm process is as follows:

[0040] a. Local sorting: Each process sorts the local data sequence X i Perform a local sort to obtain a new ordered sequence X i ′, so that All satisfy x′ i,j ≤x′ i,j+1 .

[0041] b. Determine data partitions: Perform histogram analysis on the local data sequence to determine the distribution characteristics of the data. Based on the histogram analysis results, each process determines the partition point for splitting the data and the displacement array S i ={s i,j |j=0,...,p-1} to represent the data partition situation, and the locally ordered data sequence X′ i Split into p mutually disjoint subsequences X′ i,j ={x′ i,k |s i,j ≤k i,j+1}. Dynamically adjust the displacement array S according to the actual size of each subsequence i The partition point position satisfies s i,o =0,s i,p =n i ,s i,j ≤s i,j+1 . Note that if s i,j =s i,j+1 , then X′ i,j Is empty.

[0042] c. Data redistribution: Each process is based on the displacement array S representing the data partition situation​i , send different numbers of data elements to other p-1 processes, and receive different numbers of data elements from other p-1 processes through collective communication operations, and obtain p segments of locally ordered data sequence X″ i .

[0043] d. Local sorting again: the data sequence X″ received after data redistribution i Perform a local sort to get a new ordered sequence After completing this stage of operation, the data on each process and between processes are in order.

[0044] (3) Parallel writing: collect the sorted log data samples in parallel to the file system. The goal of this stage is to take advantage of parallel processing to improve the efficiency of log file collection operations.

[0045] Use multi-threading or multi-process to collect log text on multiple nodes at the same time. The threads or processes will store the log records one by one in the corresponding text system in the order of the log records. Since the log file may contain a large number of records, it is necessary to ensure the performance and stability of the system. Consider using batch insert or batch update to collect multiple records into the file system at one time.

[0046] (4) Monitoring and optimization: First, it is necessary to monitor the space utilization of the file system in real time to ensure that there is enough storage space to store log data and avoid data loss or write failure due to insufficient storage space. Secondly, regularly check the integrity of the log file by verifying the checksum of the file or using a file integrity check tool to ensure that the log file is not damaged or tampered with, and handle abnormal log files in a timely manner to ensure data reliability. In addition, it is necessary to monitor the read and write performance of the file system, track read and write latency and throughput, and perform performance optimization to ensure that data can be written and read efficiently.

[0047] A more specific embodiment:

[0048] Assume that the log file about user activities records the user's query, insert, update and other operations. Each log contains information such as timestamp, operation type, user ID, etc.

[0049] (1) Sample these log files for use in subsequent data processing. Each process starts from the locally ordered data X′ i In the regular sampling, p-1 samples are sampled to form a subsequence Y i , assuming that there are 4 processes (p=4) processing data in parallel, assuming that the log file has a total of 4000 records (n=1000 for each node), and each node obtains samples from local data according to a certain sampling ratio to form a sample set Yi .

[0050] Each node sorts the sample sequence it obtains, selects a part of it as the candidate value of the pivot element, and forms the pivot sequence Z. Each node divides the local data according to the pivot sequence and calculates the longitudinal sum of the displacement.

[0051] Assume the deviation rate is b = 0.1. Then Therefore, for the jth pivot, the upper and lower bounds of the longitudinal displacement are 8.206 and 8.352, respectively, and the pivot element is iteratively updated based on the results.

[0052] After the regular sampling algorithm, the data is distributed to multiple nodes for parallel processing. The computing resources are fully utilized to process large-scale data and accelerate the entire sampling process. Through the processing of the data sampling stage, a portion of representative data samples will be obtained.

[0053] (2) Next, these log files are sorted in parallel according to the following process: Assume that there is a distributed system that contains 4 nodes, and each node has the following database log files:

[0054] Node 1: [A,D,G,J]

[0055] Node 2: [B,E,H,K]

[0056] Node 3: [C, F, I, L]

[0057] Node 4: [M,N,O,P]

[0058] a. Local sorting: Each node sorts the local database log files. Assume that the sorting rule is alphabetical order.

[0059] Node 1: [A,D,G,J]->[A,D,G,J]

[0060] Node 2: [B,E,H,K]->[B,E,H,K]

[0061] Node 3: [C, F, I, L] -> [C, F, I, L]

[0062] Node 4: [M, N, O, P] -> [M, N, O, P]

[0063] b. Determine data partition: Each node determines how to split the data. Suppose the data is divided into 2 parts.

[0064] Node 1: S1 = [0, 2, 4], which means the data is divided into two subsequences: [A, D], [G, J]

[0065] Node 2: S2 = [0, 2, 4], which means the data is divided into two subsequences: [B, E], [H, K]

[0066] Node 3: S3 = [0, 2, 4], which means the data is divided into two subsequences: [C, F], [I, L]

[0067] Node 4: S4 = [0, 2, 4], which means the data is divided into two subsequences: [M, N], [O, P]

[0068] c. Data redistribution: Each node sends and receives data to other nodes based on the data partition situation.

[0069] Node 1: sends [G,J] to node 2, receives [B,E] and [C,F].

[0070] Node 2: sends [H, K] to node 3, receives [A, D] and [I, L].

[0071] Node 3: sends [I,L] to node 4, receives [B,E] and [G,J].

[0072] Node 4: sends [O,P] to node 1, receives [C,F] and [H,K].

[0073] d. Local sorting again: Each node sorts the received data locally.

[0074] Node 1: [A,D,G,J,B,E,C,F] -> [A,B,C,D,E,F,G,J]

[0075] Node 2:[B,E,H,K,A,D,I,L]->[A,B,D,E,H,I,K,L]

[0076] Node 3: [C, F, I, L, B, E, G, J] -> [B, C, E, F, G, I, J, L]

[0077] Node 4: [M,N,O,P,C,F,H,K]->[C,F,H,K,M,N,O,P]

[0078] After completing these stages, the data on each node is in order, and the data between all nodes is also in order. The parallel sort is complete.

[0079] (3) The parallel writing phase uses four write nodes to process data in parallel. Assuming that each write node processes one record, records can be assigned to different write nodes. For example, the first write node processes the first record, the second node processes the second record, and so on.

[0080] (4) Monitoring and optimization phase: Based on the performance and stability of parallel acquisition operations, the space utilization of the file system is monitored in real time, the integrity of log files is checked regularly, and abnormal log files are processed in a timely manner to ensure data reliability.

[0081] This embodiment uses 4 processes to process data in parallel, making full use of the computing power of multi-core processors, and significantly improving the parallel sorting speed of data. Experimental results show that compared with traditional serial processing, the system using parallel processing can shorten the time by more than 50% when processing large-scale data sets. On the other hand, the regular sampling algorithm ensures that the samples selected from the log file are representative. When comparing multiple data sets, the system using regular sampling has a higher sample coverage rate and can better capture the distribution characteristics of the data. Moreover, the strategy of local sorting and data redistribution ensures that the data between all nodes is ordered when it is finally output.

[0082] The units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this application, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0083] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0084] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0085] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0086] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0087] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A log collection method based on parallel sorting optimization, the specific steps include: (1) Data sampling; (2) Parallel sorting: distribute the above representative samples to multiple nodes in the system according to certain rules, and use a parallel sorting algorithm to sort the log data on each node; (3) Parallel writing: representative samples of sorted log data are collected in parallel into the file system; (4) Monitoring and optimization: Use validation files and inspection tools to monitor the space utilization of the file system in real time and regularly check the integrity of log data. This is characterized by: during data sampling, log data is collected from different log sources, the log data is transmitted to the data processing node, and a scalable regular sampling method for big data is used to obtain representative samples from the log data.

2. The method for optimizing log collection based on parallel sorting as claimed in claim 1, characterized in that: In step (1), given the local ordered data Uniformly extract (p-1) samples, where p is the number of processes, to construct a sample set n i is the local ordered data sequence x i ′ size; i is the process number; k is the number of samples to be extracted during the traversal, U is the data uniformity.

3. The method for optimizing log collection based on parallel sorting as claimed in claim 1, characterized in that: In step (1), distributed processing is adopted for sampling, and each process i takes its sample set Y i Sample y in i,j Send to process j and receive sample y from process j i,j , process i and process j exchange samples, where i∈{0,...,p-1},j∈{0,...,p-1}, each process receives a sample sequence Y of size O(p) i ′={y i ' ,j =y i,j }; Each process receives the sample sequence Y i ′ is sorted to obtain the ordered sample sequence Y i ″; ordered sequence Y i ″ is used to determine the i-th pivot element z i The candidate values ​​of Samples As the i-th pivot z i Each process determines a candidate value for a pivot element, forming a pivot sequence Z = {z1,…,z p-1 }, used to divide the data sequence X i ′ into p mutually disjoint subsequences.

4. The method for optimizing log collection based on parallel sorting according to claim 1, characterized in that: In step (1), when obtaining representative samples, collect all pivot elements Z = {z1, ..., z p-1 }, for z i Sorting, calculating the longitudinal and Where S i,j is the displacement array from process i to j, if Indicates the pivot element z j The jth pivot of the load-balanced data partition is qualified as follows: is the longitudinal sum of ideal displacement, b is the deviation rate, and n is the total amount of data.

5. A log collection system based on parallel sorting optimization, comprising: (1) Data sampling module; (2) Parallel sorting module: This module distributes the above representative samples to multiple nodes in the system according to certain rules, and uses a parallel sorting algorithm to sort the log data on each node; (3) Parallel writing module: This module collects representative samples of sorted log data into the file system in parallel; (4) Monitoring and optimization module: This module monitors the space utilization of the file system in real time through validation files and inspection tools, and regularly checks the integrity of log data; its characteristics are: the data sampling module collects log data from different log sources, transmits the log data to the data processing node, and uses a scalable regular sampling method for big data to obtain representative samples from the log data.

6. A log collection system based on parallel sorting optimization as claimed in claim 5, characterized in that: In the data sampling module, given local ordered data Uniformly extract (p-1) samples, where p is the number of processes, to construct a sample set n i is the local ordered data sequence x i ′ size; i is the process number; k is the number of samples to be extracted during the traversal, U is the data uniformity.

7. The log collection system based on parallel sorting optimization as claimed in claim 5, characterized in that: In the data sampling module, sampling is distributed, and each process i takes its sample set Y i Sample y in i,j Send to process j and receive sample y from process j i,j , process i and process j exchange samples, where i∈{0,...,p-1},j∈{0,...,p-1}, each process receives a sample sequence Y of size O(p) i ′={y i ' ,j =y i,j }; Each process receives the sample sequence Y i ′ is sorted to obtain the ordered sample sequence Y i ″; ordered sequence Y i ″ is used to determine the i-th pivot element z i The candidate values ​​of Samples As the i-th pivot z i Each process determines a candidate value for a pivot element, forming a pivot sequence Z = {z1,…,z p-1 }, used to divide the data sequence X i ′ into p mutually disjoint subsequences.

8. The log collection system based on parallel sorting optimization as claimed in claim 5, characterized in that: In the data sampling module, when obtaining representative samples, collect all pivot elements Z = {z1,…,z p-1 }, for z i Sorting, calculating the longitudinal and Where S i,j is the displacement array from process i to j, if Indicates the pivot element z j The jth pivot of the load-balanced data partition is qualified as follows: is the longitudinal sum of ideal displacement, b is the deviation rate, and n is the total amount of data.

Images