Parallel computing method and system suitable for large-scale data processing

Abstract

The present application relates to the technical field of large-scale data processing, and particularly relates to a parallel computing method and system suitable for large-scale data processing. The system comprises a task management unit, a distributed load balancing module, an elastic expansion architecture, an intelligent communication optimization module, and a resource monitoring unit, wherein the task management unit divides large-scale data into a plurality of sub-tasks by means of a task decomposer, and distributes the sub-tasks to computing nodes by means of a task scheduler and a priority distributor; the distributed load balancing module achieves global load balancing by means of a load sensing unit, a dynamic adjustment unit and a balance optimization unit; the elastic expansion architecture dynamically adjusts system resources by means of a node manager, a resource pool controller and an expansion decision-making device; the intelligent communication optimization module optimizes inter-node communication by means of a communication path planning unit, a bandwidth distribution unit and a delay compensation unit; and the resource monitoring unit monitors the system performance in real time by means of a performance collector, a state analyzer and an anomaly detector.

Description

Parallel computing method and system suitable for large-scale data processing TECHNICAL FIELD

[0001] The present application belongs to the technical field of parallel computing, and specifically relates to a parallel computing method and system suitable for large-scale data processing. BACKGROUND

[0002] With the advent of the big data era, the demand for large-scale data processing is increasing, and the traditional serial computing method has been unable to meet the requirements of efficient processing of massive data. Therefore, parallel computing technology has become an important means to solve the problem of large-scale data processing. However, the existing parallel computing method still has certain deficiencies in task allocation, load balancing, computing efficiency and system scalability, which affects the overall performance and resource utilization of large-scale data processing.

[0003] After searching, a matrix decomposition parallelization optimization method and system for high-performance computing are disclosed in CN118378008B, with a publication date of September 20, 2024. This patent constructs a matrix feature extractor and a hierarchical Bayesian network model, realizes intelligent parallel strategy selection based on feature quantization and probability model, and adopts block hierarchical parallel scheduling strategy and dynamic load balancing mechanism to optimize the execution efficiency of matrix decomposition tasks. However, this technical solution mainly targets the decomposition task of meteorological data matrix, and its application scenario is relatively limited, making it difficult to be directly applied to other types of large-scale data processing requirements. In addition, this solution has strong dependence on the heterogeneity of computing nodes, which may lead to performance degradation in the case of limited hardware resources.

[0004] After searching, a parallel task processing method and system for large-scale data are disclosed in CN113821329B, with a publication date of February 11, 2022. This patent realizes distributed parallel processing of large-scale data through the cooperative work of the master node and the computing node, and receives the processing results through the feedback node to improve the reliability and efficiency of the system. However, this technical solution relies too much on the scheduling ability of the master node in the task allocation process, which may cause the master node to become a performance bottleneck of the system, especially in the case of extremely large data or complex computing tasks, the scalability and stability of the system may be limited. In addition, this solution does not fully consider the optimization of node communication overhead, which may increase network load in high concurrency scenarios, affecting the overall processing efficiency. TECHNICAL PROBLEM

[0005] The above problems show that the existing large-scale data parallel processing method still has certain deficiencies in task allocation flexibility, load balancing adaptability, system expansibility, and communication overhead optimization. Therefore, the present application provides a parallel computing method and system suitable for large-scale data processing, aiming to optimize the task allocation strategy, improve the load balancing capability, enhance the system expansibility, and reduce the communication overhead, so as to realize efficient and stable large-scale data processing and meet the needs of modern big data applications. Technical solutions

[0006] The present application provides a parallel computing method and system suitable for large-scale data processing, which solves the problems of insufficient task allocation flexibility, poor load balancing adaptability, limited system expansibility, and insufficient optimization of inter-node communication overhead in existing parallel computing methods. The present application aims to build an efficient and stable large-scale data processing system by introducing an adaptive task allocation mechanism, a distributed load balancing module, an elastic expansion architecture, and an intelligent communication optimization strategy.

[0007] The technical solution adopted by the present application to solve the above technical problems is a parallel computing system suitable for large-scale data processing, which includes a task management unit, a distributed load balancing module, an elastic expansion architecture, an intelligent communication optimization module, and a resource monitoring unit. The task management unit is electrically connected to the distributed load balancing module, the elastic expansion architecture, and the intelligent communication optimization module. The task management unit includes a task decomposer, a task scheduler, and a priority allocator. The distributed load balancing module includes a load sensing unit, a dynamic adjustment unit, and a balancing optimization unit. The elastic expansion architecture includes a node manager, a resource pool controller, and an expansion decision maker. The intelligent communication optimization module includes a communication path planning unit, a bandwidth allocation unit, and a delay compensation unit. The resource monitoring unit includes a performance collector, a state analyzer, and an anomaly detector.

[0008] The task management unit is installed in the master node. The task decomposer divides the input large-scale data into multiple sub-tasks. The task scheduler distributes the sub-tasks to the computing nodes according to the priority queue generated by the priority allocator. The priority allocator prioritizes the sub-tasks based on task complexity and resource requirements. The task decomposer divides the data into uniform blocks using a hash function, with each block fixed at 128MB in size. The task scheduler uses a priority-based polling algorithm to ensure that high-priority tasks are allocated to nodes with more idle resources.

[0009] The distributed load balancing module is installed in the computing node, the load perception unit collects the CPU usage, the memory occupation rate and the disk I / O rate of each node in real time, the dynamic adjustment unit adjusts the node load by migrating part of the task or reallocating the new task according to the data of the load perception unit; the balance optimization unit globally optimizes the task distribution based on the genetic algorithm, so as to ensure that the load standard deviation of all nodes is less than 5%; the sampling period of the load perception unit is 100 ms, and the task migration delay of the dynamic adjustment unit is 300 ms.

[0010] In the elastic expansion architecture, the node manager is responsible for the registration of the new node and the logout of the exit node, the resource pool controller maintains a dynamic resource pool, and the expansion decision maker decides whether to start the expansion operation according to the current task queue length and the node load; the node manager detects the node state through the heartbeat mechanism, and the heartbeat interval is 5 seconds; the resource pool controller stores resource information based on the red-black tree data structure, and the expansion threshold of the expansion decision maker is set to be greater than 80% of the average load of the node, and the reduction threshold is set to be less than 30% of the average load of the node.

[0011] In the intelligent communication optimization module, the communication path planning unit selects the optimal communication path according to the network topology structure, the bandwidth allocation unit dynamically adjusts the bandwidth allocation proportion based on the task priority, and the delay compensation unit compensates the communication delay through a prediction model; the communication path planning unit calculates the shortest path by using the Dijkstra algorithm, the priority weight range of the bandwidth allocation unit is 1 to 10, and the prediction model of the delay compensation unit is based on the LSTM neural network, and the prediction error is less than 2%.

[0012] The resource monitoring unit is installed in the master node and the computing node, the performance collector collects real-time performance data of CPU, memory, disk and network, the state analyzer analyzes the trend of the collected data, and the abnormality detector detects abnormal states based on the sliding window algorithm; the sampling frequency of the performance collector is 1 kHz, the time window length of the state analyzer is 1 minute, and the sliding window size of the abnormality detector is 10 sampling points.

[0013] Preferably, in the task management unit, a task dependency relationship resolver is further included, the task dependency relationship resolver includes a dependency graph generator and a dependency conflict processor. The dependency graph generator represents the dependency relationship between tasks by using a directed acyclic graph, and the dependency conflict processor solves the dependency conflict based on a depth-first search algorithm; the upper limit of the number of nodes of the dependency graph generator is 10000, the upper limit of the number of edges is 50000, and the conflict solving time of the dependency conflict processor is 200 ms.

[0014] Preferably, in the distributed load balancing module, a cross-node task migration optimizer is further included, the cross-node task migration optimizer comprising a migration cost evaluator and a migration path planner, the migration cost evaluator calculates migration cost based on task size and network bandwidth, and the migration path planner selects an optimal migration path by using an A* algorithm, the cost calculation formula of the migration cost evaluator is C=T / B+P, wherein T is task size, B is network bandwidth, and P is priority weight, and the path search time complexity of the migration path planner is O(log N).

[0015] Preferably, in the intelligent communication optimization module, a multi-path redundant communication unit is further included, the multi-path redundant communication unit comprising a path selector and a data shuffler, the path selector selects multiple redundant paths according to network state, and the data shuffler divides a data packet into multiple small pieces for transmission through different paths, the upper limit of the number of paths of the path selector is 4, the shuffling size of the data shuffler is 64 KB, and the shuffling recombination time is 10 ms.

[0016] Preferably, in the resource monitoring unit, a resource utilization rate predictor is further included, the resource utilization rate predictor comprising a historical data analyzer and a future load predictor, the historical data analyzer extracts resource utilization patterns based on time series analysis, and the future load predictor predicts future load based on an ARIMA model, the time window length of the historical data analyzer is 1 hour, and the prediction accuracy of the future load predictor is 95%.

[0017] The structure, implementation and operation principle of the application are as follows: the task management unit divides large-scale data into multiple sub-tasks through a task decomposer, and distributes the sub-tasks to computing nodes through a task scheduler and a priority allocator; the distributed load balancing module collects node load information in real time through a load sensing unit, adjusts task distribution according to the load information through a dynamic adjustment unit, and globally optimizes the task distribution through a balance optimization unit; the elastic expansion architecture registers and deregisters nodes through a node manager, maintains a dynamic resource pool through a resource pool controller, and decides to expand or reduce nodes according to load conditions through an expansion decision maker; the intelligent communication optimization module selects an optimal communication path through a communication path planning unit, dynamically adjusts bandwidth allocation through a bandwidth allocation unit, and compensates for communication delay through a delay compensation unit; the resource monitoring unit collects performance data through a performance collector, performs trend analysis through a state analyzer, and detects abnormal states through an abnormality detector; the task dependency relationship resolver solves task dependency problems through a dependency graph generator and a dependency conflict processor; the cross-node task migration optimizer optimizes the task migration process through a migration cost evaluator and a migration path planner; the multi-path redundant communication unit implements multi-path redundant communication through a path selector and a data shuffler; and the resource utilization rate predictor predicts resource utilization rate through a historical data analyzer and a future load predictor. Advantages

[0018] The task management unit realizes efficient division and distribution of tasks through the task decomposer and the task scheduler, significantly improving the flexibility of task allocation; the distributed load balancing module realizes global load balancing through the load perception unit and the balancing optimization unit, ensuring the stability of the system; the intelligent communication optimization module reduces communication overhead through the communication path planning unit and the delay compensation unit, improving the overall efficiency of the system. BRIEF DESCRIPTION OF DRAWINGS

[0019] Fig. 1 is a system architecture diagram of the present application;

[0020] Fig. 2 is a structural schematic diagram of the task management unit;

[0021] Fig. 3 is a structural schematic diagram of the distributed load balancing module;

[0022] Fig. 4 is a structural schematic diagram of the elastic expansion architecture;

[0023] Fig. 5 is a structural schematic diagram of the intelligent communication optimization module;

[0024] Fig. 6 is a structural schematic diagram of the resource monitoring unit;

[0025] Fig. 7 is a structural schematic diagram of the task dependency resolver;

[0026] Fig. 8 is a structural schematic diagram of the cross-node task migration optimizer; DRAWINGS

[0027] 1, task management unit; 11, task decomposer; 12, task scheduler; 13, priority allocator; 14, task dependency resolver; 141, dependency graph generator; 142, dependency conflict processor; 2, distributed load balancing module; 21, load perception unit; 22, dynamic adjustment unit; 23, balancing optimization unit; 24, cross-node task migration optimizer; 241, migration cost evaluator; 242, migration path planner; 3, elastic expansion architecture; 31, node manager; 32, resource pool controller; 33, expansion decision maker; 4, intelligent communication optimization module; 41, communication path planning unit; 42, bandwidth allocation unit; 43, delay compensation unit; 44, multi-path redundant communication unit; 441, path selector; 442, data shuffler; 5, resource monitoring unit; 51, performance collector; 52, state analyzer; 53, exception detector; 54, resource utilization predictor; 541, historical data analyzer; 542, future load predictor. Best mode of the present application

[0028] In order to make the technical means, creative features, purposes and effects realized by the present application easy to understand, the present application is further described below in conjunction with specific embodiments.

[0029] The embodiment of the present application provides a parallel computing method and system suitable for large-scale data processing, and solves the problems of insufficient task allocation flexibility, poor load balancing adaptability, limited system expansibility and insufficient optimization of node communication overhead in the prior art. The present application aims to build an efficient and stable large-scale data processing system by introducing an adaptive task allocation mechanism, a distributed load balancing module, an elastic expansion architecture and an intelligent communication optimization strategy.

[0030] Referring to FIGS. 1-8, a parallel computing system suitable for large-scale data processing includes a task management unit 1, a distributed load balancing module 2, an elastic expansion architecture 3, an intelligent communication optimization module 4 and a resource monitoring unit 5. The task management unit 1 is electrically connected with the distributed load balancing module 2, the elastic expansion architecture 3 and the intelligent communication optimization module 4. The task management unit 1 includes a task decomposer 11, a task scheduler 12 and a priority allocator 13. The distributed load balancing module 2 includes a load awareness unit 21, a dynamic adjustment unit 22 and a balancing optimization unit 23. The elastic expansion architecture 3 includes a node manager 31, a resource pool controller 32 and an expansion decision maker 33. The intelligent communication optimization module 4 includes a communication path planning unit 41, a bandwidth allocation unit 42 and a delay compensation unit 43. The resource monitoring unit 5 includes a performance collector 51, a state analyzer 52 and an exception detector 53.

[0031] Referring to FIG. 2, the task management unit 1 is installed in a master node. The task decomposer 11 divides input large-scale data into multiple sub-tasks. The task scheduler 12 distributes the sub-tasks to computing nodes according to a priority queue generated by the priority allocator 13. The priority allocator 13 prioritizes the sub-tasks based on task complexity and resource requirements. The task decomposer 11 divides the data into uniform blocks by using a hash function, and the size of each block is fixed at 128 MB. The task scheduler 12 uses a priority-based polling algorithm to ensure that high-priority tasks are allocated to nodes with more idle resources. The task management unit 1 further includes a task dependency relationship parser 14, which includes a dependency graph generator 141 and a dependency conflict processor 142. The dependency graph generator 141 represents the dependency relationship between tasks by using a directed acyclic graph. The dependency conflict processor 142 solves dependency conflicts based on a depth-first search algorithm. The upper limit of the number of nodes of the dependency graph generator 141 is 10,000, and the upper limit of the number of edges is 50,000. The conflict resolution time of the dependency conflict processor 142 is 200 ms.

[0032] Referring to FIG. 3, the distributed load balancing module 2 is installed in the computing node, the load perception unit 21 collects the CPU usage, memory occupancy and disk I / O rate of each node in real time, the dynamic adjustment unit 22 adjusts the node load by migrating part of the task or reallocating the new task based on the data of the load perception unit 21; the balance optimization unit 23 globally optimizes the task distribution based on the genetic algorithm, ensuring that the load standard deviation of all nodes is less than 5%; the sampling period of the load perception unit 21 is 100 ms, and the task migration delay of the dynamic adjustment unit 22 is 300 ms. The distributed load balancing module 2 further comprises a cross-node task migration optimizer 24, which comprises a migration cost evaluator 241 and a migration path planner 242. The migration cost evaluator 241 calculates the migration cost based on the task size and network bandwidth, and the migration path planner 242 selects the optimal migration path using the A* algorithm; the cost calculation formula of the migration cost evaluator 241 is C=T / B+P, where T is the task size, B is the network bandwidth, and P is the priority weight, and the path search time complexity of the migration path planner 242 is O(logN).

[0033] Referring to FIG. 4, in the elastic expansion architecture 3, the node manager 31 is responsible for the registration of new nodes and the logout of exiting nodes, the resource pool controller 32 maintains a dynamic resource pool, and the expansion decision maker 33 decides whether to start the expansion operation according to the current task queue length and node load; the node manager 31 detects the node state through the heartbeat mechanism, and the heartbeat interval is 5 seconds; the resource pool controller 32 stores resource information using a data structure based on the red-black tree, and the expansion threshold of the expansion decision maker 33 is set to be greater than 80% of the average node load, and the reduction threshold is set to be less than 30% of the average node load.

[0034] Referring to FIG. 5, in the intelligent communication optimization module 4, the communication path planning unit 41 selects the optimal communication path according to the network topology, the bandwidth allocation unit 42 dynamically adjusts the bandwidth allocation proportion based on the task priority, and the delay compensation unit 43 compensates the communication delay through a prediction model; the communication path planning unit 41 calculates the shortest path using the Dijkstra algorithm, the priority weight range of the bandwidth allocation unit 42 is 1 to 10, and the prediction model of the delay compensation unit 43 is based on the LSTM neural network, and the prediction error is less than 2%. The intelligent communication optimization module 4 further comprises a multi-path redundant communication unit 44, which comprises a path selector 441 and a data shuffler 442. The path selector 441 selects multiple redundant paths according to the network state, and the data shuffler 442 divides the data packet into multiple small fragments for transmission through different paths; the upper limit of the number of paths of the path selector 441 is 4, the shuffling size of the data shuffler 442 is 64 KB, and the shuffling recombination time is 10 ms.

[0035] Referring to FIG. 6, the resource monitoring unit 5 is installed in the master node and the computing node, the performance collector 51 collects real-time performance data of CPU, memory, disk and network, the state analyzer 52 performs trend analysis on the collected data, and the abnormality detector 53 detects abnormal states based on a sliding window algorithm; the sampling frequency of the performance collector 51 is 1 kHz, the time window length of the state analyzer 52 is 1 minute, and the sliding window size of the abnormality detector 53 is 10 sampling points. The resource monitoring unit 5 further includes a resource utilization rate predictor 54, which includes a historical data analyzer 541 and a future load predictor 542. The historical data analyzer 541 extracts resource utilization patterns based on time series analysis, and the future load predictor 542 predicts future load based on an ARIMA model; the time window length of the historical data analyzer 541 is 1 hour, and the prediction accuracy of the future load predictor 542 is 95%. Embodiments of the application

[0036] In particular, when large-scale data needs to be processed, the input large-scale data is first divided into multiple sub-tasks by the task decomposer 11 in the task management unit 1. The size of each sub-task is fixed at 128 MB, and the data block size is ensured to be uniform by a hash function. Then, the priority allocator 13 prioritizes the sub-tasks based on task complexity and resource requirements, and generates a priority queue. The task scheduler 12 distributes the sub-tasks to the computing nodes according to the priority queue. In this process, if there is a task dependency relationship, the dependency graph generator 141 in the task dependency relationship resolver 14 generates a directed acyclic graph to represent the dependency relationship between tasks, and the dependency conflict processor 142 solves the dependency conflict.

[0037] In the computing node, the load perception unit 21 in the distributed load balancing module 2 collects the CPU usage, memory occupancy and disk I / O rate of each node in real time, and sends these data to the dynamic adjustment unit 22. The dynamic adjustment unit 22 adjusts the node load by migrating part of the tasks or reallocating new tasks according to the data of the load perception unit 21. The balance optimization unit 23 globally optimizes the task distribution based on a genetic algorithm to ensure that the load standard deviation of all nodes is less than 5%. If the task needs to be migrated across nodes, the migration cost evaluator 241 in the cross-node task migration optimizer 24 calculates the migration cost, and the migration path planner 242 selects the optimal migration path.

[0038] In the system running process, if the current task queue length increases or the node load exceeds 80%, the expansion decision maker 33 in the elastic expansion architecture 3 starts the expansion operation. The node manager 31 is responsible for the registration of the new node, and detects the node state through the heartbeat mechanism. The resource pool controller 32 maintains a dynamic resource pool, and stores the resource information by using the red-black tree-based data structure. When the average load of the node is less than 30%, the expansion decision maker 33 starts the reduction operation.

[0039] In order to reduce the communication overhead between nodes, the communication path planning unit 41 in the intelligent communication optimization module 4 selects the optimal communication path according to the network topology, and calculates the shortest path by using the Dijkstra algorithm. The bandwidth allocation unit 42 dynamically adjusts the bandwidth allocation proportion based on the task priority, and compensates the communication delay by using the LSTM neural network prediction model. If it is necessary to realize the multi-path redundant communication, the path selector 441 in the multi-path redundant communication unit 44 selects multiple redundant paths, and the data fragmenter 442 splits the data packet into multiple small fragments for transmission through different paths.

[0040] In order to ensure the stability and reliability of the system, the performance collector 51 in the resource monitoring unit 5 collects the real-time performance data of CPU, memory, disk and network, and performs trend analysis by using the state analyzer 52. The abnormality detector 53 detects the abnormal state based on the sliding window algorithm, and alarms in time. In addition, the historical data analyzer 541 in the resource utilization rate predictor 54 extracts the resource utilization mode based on the time series analysis, and the future load predictor 542 predicts the future load based on the ARIMA model.

[0041] The basic principles, main features and advantages of the present application are shown and described above. It should be understood by those skilled in the art that the present application is not limited by the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A parallel computing system suitable for large scale data processing, characterized in that, The application relates to a task management system, which comprises a task management unit (1), a distributed load balancing module (2), an elastic expansion architecture (3), an intelligent communication optimization module (4) and a resource monitoring unit (5). The task management unit (1) further comprises a task dependency relationship parser (14), which comprises a dependency graph generator (141) and a dependency conflict processor (142).

2. The parallel computing system of claim 1, wherein, The task decomposer (11) divides data into uniform blocks through a hash function, and the size of each block is fixed at 128 MB.

3. The parallel computing system of claim 1, wherein, The distributed load balancing module (2) further comprises a cross-node task migration optimizer (24), which comprises a migration cost evaluator (241) and a migration path planner (242).

4. The parallel computing system of claim 1, wherein, The sampling period of the load perception unit (21) is 100 ms, and the task migration delay of the dynamic adjustment unit (22) is 300 ms.

5. The parallel computing system of claim 1, wherein, The resource pool controller (32) stores resource information by adopting a red-black tree-based data structure, and the expansion threshold of the expansion decision maker (33) is set to be greater than 80% of the average load of nodes, and the reduction threshold is set to be less than 30% of the average load of nodes.

6. The parallel computing system of claim 1, wherein, The communication path planning unit (41) calculates the shortest path by adopting a Dijkstra algorithm, the priority weight range of the bandwidth allocation unit (42) is 1-10, the prediction model of the delay compensation unit (43) is based on an LSTM neural network, and the prediction error is less than 2%.

7. The parallel computing system of claim 1, wherein, The intelligent communication optimization module (4) further comprises a multi-path redundant communication unit (44), which comprises a path selector (441) and a data shuffler (442).

8. The parallel computing system of claim 1, wherein, The sampling frequency of the performance collector (51) is 1 kHz, the time window length of the state analyzer (52) is 1 minute, and the size of the sliding window of the anomaly detector (53) is 10 sampling points.

9. The parallel computing system of claim 1, wherein, The resource monitoring unit (5) further comprises a resource utilization rate predictor (54), which comprises a historical data analyzer (541) and a future load predictor (542).

10. The parallel computing system of claim 1, wherein, ​

Images