A Computer Resource Load Balancing Method for Big Data
By collecting CXL link performance data at the operating system kernel layer and constructing a dynamic latency congestion topology map, combined with a latency sensitivity prediction model and a hot page migration mechanism, the problem of link layer congestion perception in big data clusters was solved, achieving fine-grained load balancing, improving resource utilization and operational efficiency, and reducing energy consumption.
Patent Information
- Application Number
- CN202611212082.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-11
- Publication Date
- 2026-11-06
AI Technical Summary
In big data clusters based on the CXL memory decoupling architecture, existing load balancing methods are unable to detect the real-time congestion and latency jitter of the link layer microarchitecture, making it difficult to effectively suppress the long tail effect of task execution in the early stages. Moreover, existing solutions can only provide post-event remedies and are difficult to actively intervene, resulting in an increase in the overall energy consumption and hardware costs of the cluster.
By deploying probe modules at the operating system kernel layer to collect CXL link performance monitoring counter data, a dynamic latency congestion topology map is constructed. Combined with a latency sensitivity prediction model, a bipartite graph matching algorithm and a hot page migration mechanism are used to achieve fine-grained load balancing scheduling and proactively trigger hot page migration to avoid congestion.
It achieves millisecond-level awareness of the link microarchitecture status, reduces the probability of memory-intensive tasks being assigned to congested links, improves cluster resource utilization and job completion efficiency, and reduces overall energy consumption.
Smart Images
Figure CN123048192A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data cluster resource scheduling technology, and in particular to a computer resource load balancing method for big data. Specifically, it relates to a dynamic load balancing method for big data tasks based on link layer microarchitecture performance status awareness, applied to a computing and memory decoupled architecture. It can be used in resource scheduling scenarios of big data clusters in fields such as cloud computing architecture, computing-network convergence and new hardware scheduling, and high-performance computing. Background Technology
[0002] As data centers evolve towards converged computing and network architectures with decoupled computing and memory, memory pooling based on CXL technology has become a crucial technical path for improving the utilization of memory resources in big data clusters. In this architecture, the local memory of compute nodes is no longer the sole source of memory supply. Compute nodes can access memory resources located in remote memory pools across nodes via the CXL switching network, thereby achieving on-demand allocation of memory resources at the cluster level. In big data distributed computing frameworks, a job is typically decomposed into a directed acyclic graph composed of multiple task nodes. Each task node has different dependencies on CPU and memory bandwidth resources. When the number of compute nodes increases and a large number of tasks concurrently access the same memory pool, significant queuing and bandwidth contention phenomena will occur on the link, as seen in Kubernetes and Apache. The load balancing scheduling method used by big data cluster resource managers, such as YARN, typically collects static resource metrics such as CPU utilization and memory capacity utilization of each computing node periodically. It then allocates tasks to be scheduled in a bin-like manner based on a preset resource threshold strategy. During task execution, it re-evaluates node load and migrates tasks through periodic polling. The above scheduling decisions are based on coarse-grained, quasi-static resource metrics such as CPU and memory capacity. They do not involve the collection and utilization of the performance status of the link layer microarchitecture, making it difficult to perceive the real-time congestion and latency jitter caused by cross-node memory access. This makes it difficult to effectively suppress the long-tail effect of task execution in the early stages. Existing solutions can only perform post-event remediation after detecting task execution timeouts, and it is difficult to actively intervene before congestion occurs. The industry can usually only passively alleviate the problem through static resource isolation or hardware over-provisioning, resulting in an increase in the overall energy consumption and hardware costs of the cluster. Summary of the Invention
[0003] The technical problem to be solved by this invention is: in a big data cluster based on the CXL memory decoupling architecture, how to perceive the real-time congestion and latency jitter status at the underlying link microarchitecture level, and accordingly perform fine-grained, congestion-aware dynamic load balancing scheduling for big data tasks, thereby eliminating the long-tail effect of task execution caused by cross-node memory access congestion and improving the overall resource utilization and job completion efficiency of the cluster.
[0004] To address the aforementioned technical problems, this invention provides a computer resource load balancing method for big data, comprising:
[0005] A probe module is deployed at the operating system kernel layer of the computing node. The probe module periodically collects performance monitoring counter data of the CXL link. The performance monitoring counter data includes link bandwidth utilization, request queue depth and round-trip latency.
[0006] The collected performance monitoring counter data is filtered to obtain a smoothed link load index.
[0007] Based on the smoothed link load metric, the comprehensive edge weight of each link pair between the compute node and the memory pool node is calculated. The comprehensive edge weight is obtained by weighted fusion of the latency metric and the congestion metric of the link pair.
[0008] Using the comprehensive edge weights as edge weights, a dynamic latency congestion topology graph is constructed between computing nodes and memory pool nodes, and the topology graph is updated according to a preset update cycle.
[0009] Extract the feature vector of the task to be scheduled, and input the feature vector into the pre-trained latency sensitivity prediction model to obtain the latency sensitivity score of the task.
[0010] The latency sensitivity score is compared with a preset sensitivity threshold to distinguish between memory-intensive tasks and compute-intensive tasks;
[0011] For memory-intensive tasks, candidate computing nodes with a comprehensive edge weight lower than a preset congestion threshold are retrieved in the dynamic delay congestion topology graph. The set of these candidate computing nodes is modeled as a bipartite graph. With the overall weighted delay cost as the optimization objective, a bipartite graph matching algorithm is used to solve the allocation scheme between the task and the candidate computing nodes.
[0012] For computationally intensive tasks, a load balancing strategy based on CPU utilization is used for allocation.
[0013] During task execution, the comprehensive edge weights and their changing trends of the links corresponding to each computing node are continuously monitored. When the preset sudden congestion judgment conditions are met, the hot page migration mechanism based on RDMA is triggered to migrate the hot pages on the memory pool nodes corresponding to the congested links to the standby memory pool nodes with lower comprehensive edge weights.
[0014] The technical effects that this invention can achieve include:
[0015] 1. By non-intrusively collecting link layer microarchitecture performance counter data through a lightweight probe deployed at the operating system kernel level, the limitation of the operating system's inability to perceive cross-node memory access congestion is overcome, and millisecond-level perception of the link microarchitecture status is achieved.
[0016] 2. By performing multi-dimensional quantification of latency sensitivity for big data tasks, scheduling decisions are upgraded from coarse-grained capacity matching to fine-grained latency-sensitive matching, reducing the probability that memory-intensive tasks are assigned to congested links.
[0017] 3. By combining the dynamic delay congestion topology graph with the task delay sensitivity score and using the bipartite graph matching algorithm to solve the allocation scheme, congestion-aware load balancing that takes into account both task characteristics and link status is achieved.
[0018] 4. By using dual judgment conditions based on link congestion values and their changing trends, the smooth migration of hot pages can be proactively triggered before the long-tail effect occurs, which can improve cluster resource utilization and reduce overall energy consumption without relying on hardware over-configuration. Attached Figure Description
[0019] Figure 1 This is a schematic diagram of the overall system architecture of the present invention;
[0020] Figure 2 This is a flowchart of the hardware-level non-intrusive dynamic delay congestion topology construction method of the present invention;
[0021] Figure 3 This is a flowchart of the congestion-aware adaptive load balancing and hot page smooth migration method of the present invention. Detailed Implementation
[0022] The following is in conjunction with the appendix Figure 1-3 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0023] I. System Overall Architecture
[0024] The load balancing method proposed in this invention operates on a CXL memory-decoupled big data cluster consisting of a compute node cluster, a CXL switching network, and a shared memory pool. In this cluster, the hardware layer includes multiple compute nodes and multiple memory pool nodes, interconnected via the CXL switching network. The CXL switching network integrates a performance monitoring counter unit to record raw performance data such as bandwidth utilization, queue depth, and round-trip latency of requests forwarded through it. At the operating system and probe layers, a lightweight probe module is deployed on each compute node. This probe establishes a data acquisition connection with the corresponding performance monitoring counter through an interface provided by the CXL driver. At the scheduling decision layer, based on… The system is configured with a dynamic delay congestion topology construction module, a task feature extraction and delay sensitivity quantification module, a congestion-aware scheduling module, and a hot page migration control module. Data collected by the probe is aggregated into the dynamic delay congestion topology construction module, whose output is connected to the congestion-aware scheduling module. The output of the task queue is connected to the task feature extraction and delay sensitivity quantification module, whose output is also connected to the congestion-aware scheduling module. The scheduling decision output of the congestion-aware scheduling module points to the computing node layer for task allocation. There is bidirectional data interaction between the congestion-aware scheduling module and the hot page migration control module. The hot page migration control module is connected to the memory pool layer through an RDMA path to enable cross-node migration of hot page data.
[0025] II. Hardware-level Non-intrusive Dynamic Delay Congestion Topology Construction Method
[0026] This invention deploys a lightweight probe module at the operating system kernel level. This probe module periodically accesses the performance monitoring counters of the corresponding CXL link through the standard interface provided by the CXL driver, and collects raw performance counter data including link bandwidth utilization, request queue depth, and round-trip latency.
[0027] The raw performance counter data is subjected to a moving average filter within a preset time window to remove noise caused by instantaneous jitter, resulting in a smoothed link load index sequence.
[0028] Based on the smoothed link load metric, for each computing node and each memory pool node it can access, the comprehensive edge weight of the link pair is calculated. This calculation can be expressed as follows:
[0029] ;
[0030] in, Indicates at time compute nodes With memory pool nodes The combined edge weights of the CXL links between them. Indicates at time The round-trip delay of this link after smoothing This represents the pre-set upper limit of delay normalization. Indicates at time The smoothed request queue depth of this link This represents the pre-defined normalized upper limit of the queue depth. and These are the delay weighting coefficient and the congestion weighting coefficient, respectively, which satisfy... In one specific implementation, The value is 0.6. The value is set to 0.4, which means that the impact of round-trip delay on the overall congestion level of the link is considered to be higher than the impact of request queue depth. Those skilled in the art can adjust this combination of values according to the hardware characteristics and business features of the specific cluster.
[0031] Using computing nodes as one type of vertex and memory pool nodes as another type of vertex, and the aforementioned comprehensive edge weights as edge weights, a dynamic latency congestion topology graph is constructed between computing nodes and memory pool nodes. The above-mentioned collection, filtering, and weight calculation processes are executed cyclically according to a preset update cycle to form a dynamic topology sequence that evolves over time, which is then called in real time by the scheduling decision module. The preset update cycle is set to the order of milliseconds to ensure that the topology graph can reflect the real-time changes in the state of the CXL link layer microarchitecture in a timely manner.
[0032] III. Multidimensional Quantification Method for Latency Sensitivity of Big Data Tasks
[0033] This invention further analyzes the operational characteristics of big data tasks and quantifies their sensitivity to CXL link latency jitter in order to distinguish between memory-intensive tasks and compute-intensive tasks.
[0034] In the offline training phase, multi-dimensional feature vectors are extracted from the historical execution logs and runtime monitoring data of big data tasks. These feature vectors include the average CPU utilization of the task, memory access frequency per unit time, cache hit rate, total amount of data read and written by the task, and task parallelism. For each historical task sample, the proportion of its actual execution time relative to the baseline execution time under different CXL link latency conditions is used as the latency sensitivity label for that sample. This label, along with the feature vectors, constitutes the training dataset. The gradient boosting decision tree algorithm is used to train this training dataset offline, learning the non-linear mapping relationship between the task feature vectors and the latency sensitivity label, thus obtaining a latency sensitivity prediction model. This model consists of multiple regression trees in a weighted summation manner, and its prediction output can be expressed as:
[0035] ;
[0036] in, Indicates task Delay sensitivity score, This represents the total number of regression trees in the gradient boosting tree ensemble model. Indicates the first Tree regression for the task eigenvectors The given predicted contribution value.
[0037] In the online inference stage, before the actual task scheduling, the feature vector of the task to be scheduled is extracted in real time and input into the latency sensitivity prediction model to obtain the latency sensitivity score of the task. The score is compared with a preset sensitivity threshold. Tasks with scores higher than the threshold are classified as memory-intensive tasks, while tasks with scores lower than the threshold are classified as computation-intensive tasks. The classification result is output to the congestion-aware scheduling module for its use.
[0038] IV. Congestion-Aware Adaptive Load Balancing and Smooth Hot Page Migration Methods
[0039] The congestion-aware scheduling module periodically obtains the latest dynamic delay congestion topology map output by the dynamic delay congestion topology construction module, and at the same time obtains the delay sensitivity score output by the task delay sensitivity quantification module for the current queue of tasks to be scheduled.
[0040] For memory-intensive tasks with latency sensitivity scores exceeding a preset threshold, compute nodes with a combined edge weight lower than a preset congestion threshold are retrieved from the current dynamic latency congestion topology graph to form a candidate compute node set. The set of memory-intensive tasks to be scheduled and this candidate compute node set are modeled as a bipartite graph, where the tasks are bipartite. Figure 1 The side vertex is the candidate computation node, and the edge weight is the weighted delay cost corresponding to the task being assigned to the corresponding node. Using the overall weighted delay cost as the optimization objective, a bipartite graph matching algorithm is employed to solve the task and candidate computation node allocation scheme. This optimization process can be expressed as:
[0041] ;
[0042] The constraints are:
[0043] ;
[0044] in, This indicates the total number of memory-intensive tasks currently awaiting scheduling. This represents the total number of nodes in the candidate computation node set. For a binary decision variable taking the value 0 or 1, Indicates task Assigned to candidate node , Indicates unallocated. Indicates task Delay sensitivity score, Indicates at time Candidate Nodes The corresponding CXL link's comprehensive edge weights, with constraints indicating that each task is assigned to exactly one candidate node.
[0045] For computationally intensive tasks with latency sensitivity scores not exceeding a preset threshold, a conventional CPU utilization-based load balancing strategy is used for allocation. This avoids occupying CXL link resources with lower current comprehensive edge weights, thus prioritizing these resources for memory-intensive tasks.
[0046] During task execution, the scheduler continuously monitors the real-time integrated edge weights and their changing trends of the CXL links corresponding to each computing node. When a link meets the following criteria, it is determined that the link is about to trigger a long-tail effect:
[0047] ;
[0048] in, This indicates the pre-set threshold for sudden congestion. Represents a node The rate of change of the overall edge weight of the corresponding link over time; a value greater than zero indicates that the congestion level is increasing.
[0049] When the above conditions are met, the RDMA-based hot page smooth migration mechanism is triggered: First, memory pages on the memory pool nodes corresponding to the current congested links with access frequencies higher than a preset frequency threshold are counted and marked as hot pages to be migrated. Then, the batch of hot page data is migrated to the standby memory pool nodes with a comprehensive edge weight lower than the preset congestion threshold through the RDMA path. During the migration process, a copy-on-write strategy is used to maintain data consistency between the source memory pages and the target memory pages. After the migration is completed, the source memory pages are released. After the hot page migration is completed, the comprehensive edge weights of the relevant edges in the dynamic delay congestion topology graph and the mapping relationship between the computing nodes and memory pool nodes of the affected tasks are updated. The triggering conditions, migration objects, and migration effects of this migration are recorded in the scheduling log for subsequent adjustments to the scheduling threshold and matching strategy.
[0050] V. Alternative Implementation Methods
[0051] In an alternative implementation, the dynamic latency congestion topology construction process can be carried out without relying on the native performance monitoring counters of the CXL switching network. Instead, the cross-node memory access latency can be indirectly estimated on the compute node side through the performance monitoring unit of the operating system kernel. Specifically, the number of completion cycles of a specific memory access instruction is counted and combined with the known local memory access baseline to deduce the additional latency increment caused by accessing the remote memory pool, thereby constructing an approximate dynamic latency topology. This alternative method is suitable for CXL hardware platforms that do not yet fully support the performance monitoring counter exposure interface.
[0052] In another alternative implementation, the task and computing node matching decision-making process can be made without using a bipartite graph matching algorithm. Instead, a reinforcement learning-based scheduling strategy can be adopted, which uses the dynamic delay congestion topology state and the task delay sensitivity score as the environmental state input, and the negative value of the average task completion time as the reward function. The parameters of the scheduling strategy network are continuously optimized through online learning, so that the scheduler approaches the globally optimal task allocation scheme in the long run. This alternative approach is suitable for application scenarios where the cluster size increases and the state space increases significantly.
[0053] VI. System Implementation Examples
[0054] This invention also provides a computer resource load balancing system for big data, used to implement the aforementioned method. The system includes a probe module, a dynamic latency congestion topology construction module, a task latency sensitivity quantification module, a congestion-aware scheduling module, and a hot page migration module deployed at the operating system kernel layer of each computing node. The probe module periodically collects performance monitoring counter data from the CXL link. The dynamic latency congestion topology construction module constructs and updates a dynamic latency congestion topology map based on this data. The task latency sensitivity quantification module extracts features and scores latency sensitivity of tasks to be scheduled, distinguishing task types. The congestion-aware scheduling module combines the topology map and scores to complete task allocation. The hot page migration module triggers hot page migration before congestion worsens. Each of these modules can be implemented by a processor executing computer programs stored in memory, or by hardware circuits such as a field-programmable gate array (FPGA).
[0055] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the various steps of the aforementioned computer resource load balancing method for big data.
[0056] VII. Technical Effects Description
[0057] By employing the above-described specific implementation methods, this invention can achieve millisecond-level non-intrusive awareness of congestion status in the CXL link layer microarchitecture, and perform fine-grained congestion-aware scheduling by combining task latency-sensitive quantification results. Simultaneously, it actively triggers smooth migration of hot pages based on dual judgment conditions of link congestion degree and its changing trend, thereby resolving the long-tail effect before it occurs. This improves the resource utilization and job completion efficiency of big data clusters without relying on hardware over-configuration.
[0058] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.
Claims
1. A computer resource load balancing method for big data, characterized in that, include: A probe module is deployed at the operating system kernel layer of the computing node. The probe module periodically collects performance monitoring counter data of the CXL link. The performance monitoring counter data includes link bandwidth utilization, request queue depth and round-trip latency. The collected performance monitoring counter data is filtered to obtain a smoothed link load index. Based on the smoothed link load metric, the comprehensive edge weight of each link pair between the computing node and the memory pool node is calculated. The comprehensive edge weight is obtained by weighted fusion of the latency metric and the congestion metric of the link pair. Using the comprehensive edge weights as edge weights, a dynamic delay congestion topology graph is constructed between computing nodes and memory pool nodes, and the dynamic delay congestion topology graph is updated according to a preset update cycle. Extract the feature vector of the task to be scheduled, and input the feature vector into a pre-trained latency sensitivity prediction model to obtain the latency sensitivity score of the task. The latency sensitivity score is compared with a preset sensitivity threshold. Tasks with a score higher than the preset sensitivity threshold are identified as memory-intensive tasks, and tasks with a score lower than the preset sensitivity threshold are identified as compute-intensive tasks. For tasks identified as memory-intensive tasks, computing nodes with a comprehensive edge weight lower than a preset congestion threshold are retrieved in the dynamic delay congestion topology graph to form a candidate computing node set. The memory-intensive tasks to be scheduled and the candidate computing node set are modeled as a bipartite graph. With the overall weighted delay cost as the optimization objective, a bipartite graph matching algorithm is used to solve the allocation scheme between tasks and candidate computing nodes. For tasks identified as computationally intensive, a load balancing strategy based on CPU utilization is used for allocation. During task execution, the comprehensive edge weight of the link corresponding to the computing node and its changing trend are continuously monitored. When the comprehensive edge weight of a link exceeds the preset sudden congestion threshold and shows an upward trend, the hot page migration mechanism based on RDMA is triggered to migrate the memory pages on the memory pool node corresponding to the link that have an access frequency higher than the preset frequency threshold to the standby memory pool node whose comprehensive edge weight is lower than the preset congestion threshold.
2. The method according to claim 1, characterized in that, The step of filtering the collected performance monitoring counter data to obtain smoothed link load indicators includes: performing moving average filtering on the performance monitoring counter data within a preset time window to remove instantaneous jitter noise; and obtaining the comprehensive edge weight by weighted summation based on the ratio of the smoothed round-trip delay to a preset normalized upper limit of delay and the ratio of the smoothed request queue depth to a preset normalized upper limit of queue depth, wherein the sum of the delay weight coefficient and the congestion weight coefficient is 1.
3. The method according to claim 1, characterized in that, The pre-trained latency sensitivity prediction model is constructed as follows: a multi-dimensional feature vector is extracted from the historical execution logs and runtime monitoring data of big data tasks. The multi-dimensional feature vector includes the average CPU utilization of the task, the frequency of memory access per unit time, the cache hit rate, the total amount of data read and written by the task, and the parallelism of the task. The proportion of the actual execution time of the historical task under different CXL link latency conditions relative to the baseline execution time is used as the latency sensitivity label of the task sample. Together with the multi-dimensional feature vector, the model forms a training dataset. The gradient boosting decision tree algorithm is used to train the training dataset to obtain the latency sensitivity prediction model composed of the weighted sum of multiple regression trees.
4. The method according to claim 1, characterized in that, The step of modeling the memory-intensive tasks to be scheduled and the set of candidate computing nodes as a bipartite graph, with the overall weighted latency cost as the optimization objective, and using a bipartite graph matching algorithm to solve the task and candidate computing node allocation scheme includes: taking the memory-intensive tasks as vertices on one side of the bipartite graph, taking the candidate computing nodes as vertices on the other side of the bipartite graph, and using the weighted latency cost corresponding to the task after it is allocated to the corresponding candidate computing node as the edge weight, and using a bipartite graph matching algorithm to solve the task and candidate computing node allocation scheme that satisfies the constraint that each memory-intensive task is allocated to a candidate computing node.
5. The method according to claim 1, characterized in that, The triggering of the RDMA-based hot page migration mechanism includes: counting memory pages on the memory pool node corresponding to the link whose access frequency is higher than a preset frequency threshold, marking them as hot pages to be migrated, migrating the hot pages to be migrated to a standby memory pool node with a comprehensive edge weight lower than a preset congestion threshold through the RDMA path, using a copy-on-write strategy to maintain data consistency between the source memory page and the target memory page during the migration process, and releasing the source memory page after the migration is completed.
6. The method according to claim 5, characterized in that, After the migration of the hot page to be migrated is completed, the process also includes: updating the comprehensive edge weight of the relevant edges in the dynamic delay congestion topology graph, updating the mapping relationship between the computing nodes and memory pool nodes corresponding to the affected tasks, and recording the triggering conditions, migration objects and migration effects of this migration, which are used to adjust the scheduling threshold and matching strategy.
7. A computer resource load balancing system for big data, characterized in that, include: The probe module, deployed in the operating system kernel layer of the computing node, is used to periodically collect performance monitoring counter data of the CXL link. The performance monitoring counter data includes link bandwidth utilization, request queue depth, and round-trip latency. The dynamic delay congestion topology construction module is used to filter the performance monitoring counter data, calculate the comprehensive edge weight of each link pair between the computing node and the memory pool node based on the filtered link load index, and construct and update the dynamic delay congestion topology between the computing node and the memory pool node according to the comprehensive edge weight as the edge weight. The task latency sensitivity quantification module is used to extract the feature vector of the task to be scheduled, and output the latency sensitivity score of the task according to the pre-trained latency sensitivity prediction model. Based on the latency sensitivity score, the task is divided into memory-intensive tasks and computation-intensive tasks. The congestion-aware scheduling module is used to search for candidate computing nodes with a comprehensive edge weight lower than a preset congestion threshold in the dynamic delay congestion topology graph for memory-intensive tasks, form a candidate computing node set, solve the allocation scheme between tasks and candidate computing nodes using a bipartite graph matching algorithm, and allocate tasks based on CPU utilization using a load balancing strategy for computing-intensive tasks. The hot page migration module is used to continuously monitor the comprehensive edge weights and their changing trends of the links corresponding to computing nodes. When the preset sudden congestion judgment conditions are met, the RDMA-based hot page migration mechanism is triggered.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the computer resource load balancing method for big data as described in any one of claims 1 to 6.