Heterogeneous Spark cluster parallel scheduling optimization method and system based on data tilt correction

Abstract

The invention discloses a heterogeneous Spark cluster parallel scheduling optimization method and system based on data skew correction. The method comprises the following steps: (1) establishing a job scheduling model, and constructing a calculation cost model; (2) generating an initial parallelism degree through a parallelism degree estimation algorithm based on the total input data amount, the memory demand scaling factor xp and the total node memory capacity; (3) correcting data inclination; (4) heterogeneous node task allocation; (5) dynamically optimizing the degree of parallelism; (6) reinforcement learning-driven dynamic scheduling decision making; (7) data skew recognition and correction of graph structure perception; (8) a multi-stage self-adaptive caching mechanism is adopted; (9) energy efficiency perception and carbon footprint optimization scheduling; and (10) a privacy protection prediction model under a federated learning framework. The system comprises a cost modeling module, a dynamic splitting engine, a capability sequencer, a closure cache and a prediction execution module, the dynamic splitting engine constructs a splitting decision tree with a data type self-sensing capability, and the splitting decision tree dynamically generates a splitting strategy by analyzing data element characteristics.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a heterogeneous Spark cluster parallel scheduling optimization method and system based on data skew correction. BACKGROUND

[0002] Data skew is a common performance killer in distributed computing, especially in the Shuffle phase of Spark. If the data volume corresponding to some key values is much larger than that of other keys, the task of processing that key will become a long-tail task, slowing down the entire job progress. Traditional Spark can only detect partition-level skew and is difficult to locate specific skew keys, and lacks a dynamic correction mechanism.

[0003] Heterogeneous Spark cluster nodes have different computing capabilities, memories, and hardware types, such as GPU / NPU and ordinary CPU nodes. Traditional scheduling often uses an average allocation strategy, which leads to idle resources on high-performance nodes and excessive load on low-performance nodes, resulting in low overall efficiency. Parallelism, the number of tasks, is a key parameter that affects Spark performance: too low parallelism can lead to resource waste, and too high parallelism can cause memory overflow or a surge in Shuffle overhead. Traditional solutions rely on manual configuration and are difficult to adapt to dynamic changes in data volume and cluster state. In actual production, there are various types of jobs such as batch processing and stream processing, and dynamic changes in cluster state such as node failure and resource preemption, and traditional scheduling lacks adaptive ability.

[0004] Based on the above problems, the present application designs a heterogeneous Spark cluster parallel scheduling optimization method and system based on data skew correction, with the core goal of solving the performance bottleneck and resource utilization problems of traditional Spark scheduling mechanism in a heterogeneous environment. SUMMARY

[0005] The technical solution adopted by the present application to solve its technical problems is: a heterogeneous Spark cluster parallel scheduling optimization method and system based on data skew correction, which solves the problems of low efficiency, poor adaptability, and prominent performance bottleneck of traditional Spark in a heterogeneous environment through three core mechanisms: accurate correction of data skew, intelligent scheduling of heterogeneous nodes, and dynamic optimization of parallelism. Ultimately, the job running time is shortened, the resource utilization is improved, the system robustness is enhanced, and the system is more suitable for large-scale, diversified distributed computing scenarios such as big data analysis and machine learning training.

[0006] A heterogeneous Spark cluster parallel scheduling optimization method based on data skew correction, comprising the following steps:

[0007] (1) Establish a job scheduling model, collect the partition data volume, node computing capacity and network bandwidth of each Stage through distributed monitoring agent, and build a computing cost model; (2) Based on the total input data volume, memory demand ratio factor x_p and total node memory capacity, generate initial parallelism through parallelism estimation algorithm; (3) Data skew correction; (4) Heterogeneous node task allocation; (5) Dynamic optimization of parallelism; (6) Dynamic scheduling decision driven by reinforcement learning; (7) Graph structure perception-based data skew identification and correction; (8) Multi-level adaptive caching mechanism; (9) Energy efficiency perception and carbon footprint optimization scheduling; (10) Privacy protection prediction model under the federated learning framework.

[0008] Further, the step (1) of building a computing cost model comprises:

[0009] 1.1 Partition pull time , wherein is the Shuffle data volume, is the effective bandwidth between nodes;

[0010] 1.2 Computing time , wherein is the task computing complexity, is the node normalized computing power;

[0011] 1.3 Resource vacancy rate , the formula shows that when the computing time is significantly greater than the pull time, the resource vacancy rate rises. The model dynamically calibrates the parameters through regression analysis to ensure consistency with the physical cluster performance.

[0012] Further, the step (2) comprises:

[0013] 2.1 Constraint condition: if node memory, then node memory , wherein is the memory demand ratio factor (default 0.6);

[0014] 2.2 LSTM prediction: historical job data is trained through feature extraction (including input data volume, operator type, Shuffle ratio, etc.) LSTM network to predict the S' value of the next Stage;

[0015] 2.3 Dynamic correction: when the prediction confidence is >80%, use the predicted value S' as the initial parallelism, otherwise start the rule engine to adjust S in combination with CPU / network constraints.

[0016] Further, the step (3) of data skew correction comprises:

[0017] 3.1 Skew detection: Stage-level detection: Calculate the standard deviation of the partition data volume σ, if σ > 2 × mean, determine the skew stage, key-level detection: Build key value distribution histogram, filter the keys whose data volume is more than 3 times the average as skew keys Kskew;

[0018] 3.2 Dynamic splitting: Sub-key generation: Add a hash suffix to Kskew (such as Kskew_1, Kskew_2), and the splitting times tskn are searched by bisection method to minimize the splitting cost function Cost = α × network overhead + β × computing overhead; Load awareness: If the average CPU utilization of the cluster is > 70%, adjust the splitting granularity to ceil(tskn × 1.5) to avoid excessive splitting; Redistribute strategy: The split sub-keys are merged through hash mapping in the Shuffle stage to ensure logical consistency.

[0019] Further, the step (4) heterogeneous node task allocation includes:

[0020] 4.1 Capability normalization: Calculate the node normalized computing power Pnode= , where Ttask is the historical task completion time, n is the number of tasks; GPU / NPU nodes are additionally multiplied by the acceleration factor (such as GPU nodes × 5).

[0021] 4.2 Greedy allocation: Sort the nodes in descending order of Pnode, and allocate the largest data bucket to the strongest node first, and execute the formula: allocation amount = min(bucket size, Pnode × remaining time);

[0022] 4.3 Dynamic reallocation: When node failure is detected, pause the task queue, and use healthy nodes Nhealthy to reallocate unfinished tasks, and execute the reallocation amount .

[0023] Further, the step (5) parallelism dynamic optimization includes:

[0024] 5.1 Closure reuse: If the current operation closure (such as Map operator) is consistent with the historical closure signature, directly reuse its optimal parallelism configuration;

[0025] 5.2 Memory-aware strategy: If the node idle memory ratio is > 30%, enable full parallel strategy (Snew = Scurrent × 2), if the memory is insufficient, downgrade according to .

[0026] Further, the step (6) reinforcement learning driven dynamic scheduling decision includes:

[0027] 6.1 Construct a scheduling strategy model based on the PPO (Proximal Policy Optimization) algorithm, with the cluster state vector (CPU, memory, network, I / O) and job characteristics (data size, operation type, dependency relationship) as inputs.

[0028] 6.2 Design a multi-objective reward function R = w1 × throughput + w2 × resource utilization - w3 × long-tail delay - w4 × energy consumption;

[0029] 6.3 Online learning mechanism: The model is updated iteratively every 100 assignments, and the performance of the new strategy is verified through A / B testing;

[0030] 6.4 Offline-Online Collaborative Training: The offline phase involves pre-training on historical data, while the online phase involves fine-tuning based on actual execution results.

[0031] Furthermore, step (7) of data tilt identification and correction for graph structure perception includes:

[0032] 7.1 Construct a data lineage graph G=(V,E), where V represents a data partition and E represents a data dependency relationship;

[0033] 7.2 Applying Graph Neural Networks (GNNs) to calculate node importance scores: ;

[0034] 7.3 Skew propagation prediction based on graph structure: skewfuture(v) = α × skewcurrent(v) + β × ∑u∈N(v)skewcurrent(u);

[0035] 7.4 Structure-aware splitting: Prioritize splitting skewing keys that are associated with multiple downstream tasks to reduce global impact.

[0036] Furthermore, step (8) of the multi-level adaptive caching mechanism includes:

[0037] 8.1 Construct a three-level caching system: L1 (CPU cache), L2 (node ​​memory), and L3 (distributed shared cache);

[0038] 8.2 Dynamic decision-making regarding cached content: ;

[0039] 8.3 Cross-job cache sharing: Construct a job similarity matrix Sim(i,j)=cos(featurei,featurej), and jobs with similarity > 0.7 share the cache.

[0040] 8.4 Learning-based cache eviction: Using LSTM to predict future access probabilities and prioritizing the eviction of low-probability data.

[0041] Furthermore, step (9) of energy efficiency sensing and carbon footprint optimization scheduling includes:

[0042] 9.1 Constructing the energy efficiency model: Ptotal = Pbase + k1 × CPUutil + k2 × Memoryutil + k3 × Networkutil;

[0043] 9.2 Carbon Intensity Sensing and Scheduling: Integrating real-time carbon intensity data from the power grid, carbon_intensity(t) = f(time,location,energy_mix);

[0044] 9.3 Task scheduling optimization objective: min∑(w1×runtime+w2×energy_consumption+w3×carbon_footprint);

[0045] 9.4 Dynamic frequency adjustment: Adjust the CPU frequency fnew=fbase×(1-λ×carbon_intensity) in real time according to the mission critical path and carbon intensity changes.

[0046] Furthermore, the privacy-preserving prediction model under the federated learning framework in step (10) includes:

[0047] 10.1 Construct a cross-organization federated learning architecture where data from different organizations does not leave their domains, and only model gradients are exchanged;

[0048] 10.2 Differential Privacy Protection: Adding Laplace Noise To gradient update;

[0049] 10.3 Model Aggregation Optimization: ;

[0050] 10.4 Personalized Model Branch: Retain the global model backbone and deploy personalized headers in each organization.

[0051] A parallel scheduling optimization system for heterogeneous Spark clusters based on data skew correction includes: a cost modeling module, integrating a cost calculation model and a resource vacancy rate formula library; a key-value analyzer, which uses a sliding window to statistically analyze the distribution of key-value data and generate histograms for skew detection; a dynamic splitting engine, which performs subkey splitting, hash redistribution, and incremental parallelism calculation; a capability sorter, which calculates normalized capability based on the historical task completion time of nodes; a closure cache, which stores the closures of each stage operation and records of the optimal parallelism; and a prediction execution module, including multi-dimensional metric analysis, feature extraction, model service, and feedback learning.

[0052] Furthermore, the dynamic splitting engine constructs a splitting decision tree with data type self-awareness capabilities. The splitting decision tree dynamically generates splitting strategies by analyzing data element characteristics, including: structured data splitting priority rules, unstructured data subkey generation mechanism, and short key and long key difference processing strategy: establishing a key length threshold θ (default 32 bytes), splitting short keys according to the number of nodes n, and splitting long keys according to the amount of data.

[0053] The beneficial effects of this invention are as follows:

[0054] 1. By combining technologies such as computational cost model, dynamic parallelism adjustment, intelligent splitting, and heterogeneous node allocation, we can accurately identify and efficiently process data skew. Through computational cost model and resource vacancy rate formula, we can accurately quantify the utilization rate of cluster resources, provide a theoretical basis for dynamic scheduling, and improve the utilization rate of cluster resources and job execution efficiency.

[0055] 2. By adopting LSTM+XGBoost ensemble prediction, the accuracy of parallelism prediction is improved and the resource waste caused by inaccurate prediction is reduced. An innovative subkey splitting type-aware strategy is proposed, which adopts different splitting strategies for different data types.

[0056] 3. Heterogeneous environment adaptation node capability normalization, accelerator-specific evaluation, and resource preemption awareness are combined to allocate tasks based on the characteristics of heterogeneous nodes, maximizing the use of cluster computing power. Through operation closure reuse and dynamic reallocation protocols, computational overhead is reduced and system stability is improved.

[0057] 4. Based on a data type-aware dynamic splitting execution engine, this invention effectively solves the performance bottleneck problem caused by data skew in distributed computing by constructing data type awareness capabilities, a long-short key differentiation processing mechanism, and an adaptive splitting decision tree. Compared with existing technologies, this invention achieves precise matching between data characteristics and splitting strategies, significantly improving resource utilization and job execution efficiency, and has significant technical value and broad application prospects. Attached Figure Description

[0058] Figure 1 : Job scheduling model architecture diagram.

[0059] Figure 2 Flowchart of parallelism prediction algorithm.

[0060] Figure 3 Flowchart for data skew correction.

[0061] Figure 4 Flowchart of heterogeneous node task allocation.

[0062] Figure 5 System module interaction flowchart.

[0063] Figure 6 : Reinforcement learning scheduling decision-making flowchart.

[0064] Figure 7 : Schematic diagram of tilt recognition of graph structure perception data.

[0065] Figure 8 Multi-level adaptive caching architecture diagram.

[0066] Figure 9 Flowchart for energy efficiency and carbon footprint optimization scheduling.

[0067] Figure 10 : Framework diagram for privacy protection in federated learning.

[0068] Figure 11 Flowchart of a dynamic splitting execution engine based on data type self-awareness.

[0069] Figure 12 Experimental framework diagram.

[0070] Figure 13 Heterogeneous node scheduling verification Web-Google task allocation comparison chart.

[0071] Figure 14 Parallelism dynamic optimization verification model accuracy comparison chart.

[0072] Figure 15 Comparison chart of closure reuse effects.

[0073] Figure 16 : Comparison of overall system performance gains of the method presented in this paper. Detailed Implementation

[0074] The following describes in detail the implementation process of the technical solution of this invention, in conjunction with the design background, accompanying drawings, and embodiments. The inventors of this solution, Bian Chen et al., proposed a job scheduling model in their paper 1, "Parallelism Inference Algorithm for Spark Framework," which includes concepts such as wide-dependency computation cost, resource vacancy rate, and spill probability. They designed a parallelism inference algorithm (PDA) to optimize task parallelism by considering multiple factors, maximizing resource utilization and minimizing overhead. Paper 2, "Data Skew Correction Scheduling Strategy for Heterogeneous Spark Clusters," addresses the data skew problem in heterogeneous Spark clusters by proposing the DSCH strategy, which includes a parallelism prediction algorithm, a data skew correction algorithm, and a heterogeneous node task allocation algorithm. This intelligent task allocation algorithm improves the adaptability between data computation volume and node computational capabilities, reduces the probability of external memory spill on worker nodes, and improves overall performance. This solution encompasses the core ideas of the above two papers, not only solving the problems of data skew and parallelism settings but also further expanding their application scope and technology. Specifically, a series of reinforcement learning, graph neural networks, multi-level caching mechanisms, energy efficiency optimization, and federated learning were introduced to form a complete method and system for parallel scheduling optimization of heterogeneous Spark clusters.

[0075] The following implementation is run in a heterogeneous Spark cluster equipped with GPU and CPU nodes, deployed on a Kubernetes platform, supporting Pod resource isolation and HPA elastic scaling. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0076] Example 1

[0077] A parallel scheduling optimization method for heterogeneous Spark clusters based on data skew correction includes the following steps: (1) Establishing a job scheduling model, collecting partition data volume, node computing power and network bandwidth of each Stage through a distributed monitoring agent, and constructing a computational cost model; (2) Generating the initial parallelism based on the total amount of input data, the memory requirement ratio factor x_p and the total memory capacity of the nodes through a parallelism prediction method; (3) Data skew correction; (4) Task allocation for heterogeneous nodes; (5) Dynamic optimization of parallelism; (6) Dynamic scheduling decision driven by reinforcement learning; (7) Data skew identification and correction based on graph structure awareness; (8) Multi-level adaptive caching mechanism; (9) Energy efficiency awareness and carbon footprint optimization scheduling; (10) Privacy protection prediction model under the federated learning framework.

[0078] Among them, (1) establish a job scheduling model, collect the partition data volume, node computing power and network bandwidth of each Stage through a distributed monitoring agent, and construct a computing cost model including:

[0079] 1.1 Partition fetch time ,in This represents the amount of Shuffle data. Effective bandwidth between nodes;

[0080] 1.2 Calculation Time ,in Calculate the complexity of the task. Normalize the computing power of nodes;

[0081] 1.3 Resource vacancy rate The formula shows that resource vacancy rate increases when computation time is significantly longer than retrieval time. The model dynamically calibrates parameters through regression analysis to ensure consistency with physical cluster performance.

[0082] Figure 1 The architecture of the job scheduling model is demonstrated, comprising three main components: a distributed monitoring agent, a computational cost model construction, and dynamic parameter calibration. The monitoring agent collects partition data volume, node computing power, and network bandwidth to construct a computational cost model that includes partition fetch time, computation time, and resource idle rate. The parameter calibration module dynamically adjusts the model parameters through regression analysis to ensure consistency with the physical cluster performance.

[0083] Among them, (2) the initial parallelism is generated based on the total amount of input data, the memory requirement ratio factor x_p, and the total memory capacity of the nodes, using the parallelism pre-estimation method, including:

[0084] 2.1 Constraints: If Node memory is then forcibly limited. Node memory ,in This is the memory requirement scaling factor (default 0.6).

[0085] 2.2 LSTM Prediction: Historical job data is used to train an LSTM network after feature extraction (including input data volume, operator type, shuffle ratio, etc.) to predict the S′ value of the next stage;

[0086] 2.3 Dynamic correction: When the prediction confidence is >80%, the predicted value S′ is used as the initial parallelism; otherwise, the rule engine is started, and S is adjusted in combination with CPU / network constraints.

[0087] Figure 2 The complete process of the parallelism prediction algorithm is demonstrated, starting from data acquisition, through memory constraint checking, LSTM prediction, confidence verification, and finally outputting the initial parallelism.

[0088] Among them, (3) data skew correction includes:

[0089] 3.1 Skew detection: Stage-level detection: Calculate the standard deviation σ of the data volume of the partition. If σ > 2 × mean, it is judged as a skewed stage. Key-level detection: Construct a key-value distribution histogram and filter keys with a data volume exceeding 3 times the average as skewed keys (Kskew).

[0090] 3.2 Dynamic Splitting: Subkey Generation: Append hash suffixes (e.g., Kskew_1, Kskew_2) to Kskew. The number of splits, tskn, is minimized using a binary search to minimize the splitting cost function Cost = α × network overhead + β × computational overhead. Load Awareness: If the cluster's average CPU utilization is >70%, the splitting granularity is adjusted to ceil(tskn × 1.5) to avoid over-split. Redistribution Strategy: The split subkeys are merged through hash mapping during the Shuffle phase to ensure logical consistency.

[0091] Figure 3 The document details the data skew correction process, including a dual detection mechanism at both the stage and key-value levels, dynamic subkey splitting, load-aware adjustment, and redistribution strategies. This process accurately identifies skewed data and intelligently splits it while maintaining logical data consistency.

[0092] Among them, (4) heterogeneous node task allocation includes: 4.1 Capacity Normalization: Normalized computing power of computing nodes Pnode = Where Ttask is the historical task completion time, and n is the number of tasks; GPU / NPU nodes are multiplied by an additional acceleration factor (e.g., GPU node × 5).

[0093] 4.2 Greedy Allocation: Sort nodes in descending order by Pnode, and allocate the largest data bucket to the strongest node first. Execute the formula: Allocation amount = min(bucket size, Pnode × remaining time);

[0094] 4.3 Dynamic Reallocation: When a node failure is detected, the task queue is paused, and unfinished tasks are reallocated using a healthy node, Nhealthy, for execution. .

[0095] Figure 4 This demonstrates the complete process of task allocation across heterogeneous nodes, from capacity normalization calculation and greedy allocation to exception handling and elastic scaling. This process fully considers node heterogeneity, maximizes resource utilization, and handles various anomalies to ensure system stability.

[0096] Among them, (5) dynamic optimization of parallelism includes:

[0097] 5.1 Closure Reuse: If the signature of the current operation closure (such as the Map operator) is consistent with that of a historical closure, its optimal parallelism configuration is directly reused;

[0098] 5.2 Memory-Aware Strategy: If the node's free memory percentage is >30%, enable the full parallel strategy (Snew = Scurrent × 2); if memory is insufficient, proceed according to... Downgraded.

[0099] Figure 5 The interaction flow of the system's various modules is demonstrated, with the central scheduling controller coordinating the work of the key-value analyzer, dynamic splitting engine, capability sorter, closure cache, and predictive execution module. The modules exchange data through clearly defined interfaces, forming a closed-loop optimization system.

[0100] Among them, (6) reinforcement learning-driven dynamic scheduling decisions include:

[0101] 6.1 Construct a scheduling strategy model based on the PPO (Proximal Policy Optimization) algorithm, with the cluster state vector (CPU, memory, network, I / O) and job characteristics (data size, operation type, dependency relationship) as inputs.

[0102] 6.2 Design a multi-objective reward function R = w1 × throughput + w2 × resource utilization - w3 × long-tail delay - w4 × energy consumption;

[0103] 6.3 Online learning mechanism: The model is updated iteratively every 100 assignments, and the performance of the new strategy is verified through A / B testing;

[0104] 6.4 Offline-Online Collaborative Training: The offline phase involves pre-training on historical data, while the online phase involves fine-tuning based on actual execution results.

[0105] Figure 6 This demonstrates the complete process of reinforcement learning-driven scheduling decision-making, from environmental state acquisition to policy update and validation. The PPO algorithm learns the optimal scheduling policy by maximizing a multi-objective reward function (including throughput, resource utilization, long-tail latency, and energy consumption), and the effect of the new policy is validated through A / B testing for every 100 jobs.

[0106] Among them, (7) data tilt identification and correction for graph structure perception includes:

[0107] 7.1 Construct a data lineage graph G=(V,E), where V represents a data partition and E represents a data dependency relationship;

[0108] 7.2 Applying Graph Neural Networks (GNNs) to calculate node importance scores: ;

[0109] 7.3 Skew propagation prediction based on graph structure: skewfuture(v) = α × skewcurrent(v) + β × ∑u∈N(v)skewcurrent(u);

[0110] 7.4 Structure-aware splitting: Prioritize splitting skewing keys that are associated with multiple downstream tasks to reduce global impact.

[0111] Figure 7 This paper demonstrates the process of graph structure-aware data skew identification. First, a data lineage graph is constructed, where nodes represent data partitions (e.g., users, orders, products) and edges represent data dependencies. Then, a graph neural network is applied to predict the risk of skew propagation. Finally, the splitting priority is calculated based on node importance and skewness to achieve intelligent splitting with structure awareness.

[0112] Among them, (8) the multi-level adaptive caching mechanism includes:

[0113] 8.1 Construct a three-level caching system: L1 (CPU cache), L2 (node ​​memory), and L3 (distributed shared cache);

[0114] 8.2 Dynamic decision-making regarding cached content: ;

[0115] 8.3 Cross-job cache sharing: Construct a job similarity matrix Sim(i,j)=cos(featurei,featurej), and jobs with similarity > 0.7 share the cache;

[0116] 8.4 Learning-based cache eviction: Using LSTM to predict future access probabilities and prioritizing the eviction of low-probability data.

[0117] Figure 8 The system showcases a multi-level adaptive caching architecture, comprising three levels: L1 (CPU cache), L2 (node ​​memory), and L3 (distributed shared cache). The caching strategy decision engine dynamically determines the cache location and lifecycle of data based on job characteristics, data access patterns, and historical reuse information. The system also supports intelligent features such as job similarity analysis, LSTM access prediction, dynamic lifecycle management, and cross-level cache consistency.

[0118] Among them, (9) energy efficiency sensing and carbon footprint optimization scheduling includes:

[0119] 9.1 Constructing the energy efficiency model: Ptotal = Pbase + k1 × CPUutil + k2 × Memoryutil + k3 × Networkutil;

[0120] 9.2 Carbon Intensity Sensing and Scheduling: Integrating real-time carbon intensity data from the power grid, carbon_intensity(t) = f(time,location,energy_mix);

[0121] 9.3 Task scheduling optimization objective: min∑(w1×runtime+w2×energy_consumption+w3×carbon_footprint);

[0122] 9.4 Dynamic frequency adjustment: Adjust the CPU frequency fnew=fbase×(1-λ×carbon_intensity) in real time according to the mission critical path and carbon intensity changes.

[0123] Figure 9 This demonstrates the complete process of energy efficiency and carbon footprint optimization scheduling, starting with the acquisition of grid carbon intensity data, proceeding through task characteristic analysis, energy efficiency model calculation, multi-objective optimization, and finally generating scheduling decisions and making dynamic frequency adjustments. While ensuring SLA (Service Level Agreement), the system significantly reduces energy consumption and carbon footprint through intelligent scheduling, while keeping the performance impact within acceptable limits.

[0124] Among them, (10) privacy-preserving prediction models under the federated learning framework include:

[0125] 10.1 Construct a cross-organization federated learning architecture where data from different organizations does not leave their domains, and only model gradients are exchanged;

[0126] 10.2 Differential Privacy Protection: Adding Laplace Noise To gradient update;

[0127] 10.3 Model Aggregation Optimization: ;

[0128] 10.4 Personalized Model Branch: Retain the global model backbone and deploy personalized headers in each organization.

[0129] Figure 10 This paper demonstrates a privacy-preserving framework for federated learning, comprising a central coordinating server and Spark clusters from multiple participating organizations. Each organization trains its model locally, with the raw data remaining within its domain; only encrypted model updates are exchanged. The coordinating server adds differential privacy noise when aggregating these updates and then distributes the global model. Each organization retains portions of the model layers as personalized branches to adapt to local data characteristics. This framework achieves cross-organizational knowledge sharing and model performance improvements while protecting data privacy.

[0130] This solution presents a parallel scheduling optimization method for heterogeneous Spark clusters. It features the following key innovations:

[0131] 1. Dynamic Parallelism Adjustment and Prediction: Not limited to static parameter settings, but uses LSTM+XGBoost integrated prediction to dynamically adjust the parallelism, which improves the accuracy of parallelism prediction.

[0132] 2. Data skew identification and correction: Using graph structure perception data skew identification technology, we can more accurately discover data skew and its propagation path, and take corresponding splitting strategies to correct it.

[0133] 3. Heterogeneous node task allocation: Based on the actual computing power and current load of the nodes, tasks are intelligently allocated to maximize the use of cluster resources.

[0134] 3. Reinforcement Learning-Driven Scheduling Decisions: A scheduling strategy model is constructed using the PPO algorithm, and resource utilization, throughput, long-tail latency, and energy consumption are optimized through a multi-objective reward function.

[0135] 4. Multi-level adaptive caching mechanism: Through a three-level caching system (L1 / L2 / L3) and a learning-based cache eviction strategy, I / O overhead is reduced.

[0136] 5. Energy efficiency and carbon footprint optimization scheduling: Combine real-time carbon intensity data to optimize task scheduling, reduce energy consumption and carbon footprint.

[0137] 6. Privacy Protection in Federated Learning: Protecting data privacy in cross-organizational collaboration while improving model accuracy through differential privacy and personalized branching techniques.

[0138] Example 2

[0139] A parallel scheduling optimization system for heterogeneous Spark clusters based on data skew correction includes: a cost modeling module, integrating a cost calculation model and a resource vacancy rate formula library; a key-value analyzer, which uses a sliding window to statistically analyze the distribution of key-value data and generate histograms for skew detection; a dynamic splitting engine, which performs subkey splitting, hash redistribution, and incremental parallelism calculation; a capability sorter, which calculates normalized capability based on the historical task completion time of nodes; a closure cache, which stores the closures of each stage operation and records of the optimal parallelism; and a prediction execution module, including multi-dimensional metric analysis, feature extraction, model service, and feedback learning.

[0140] The dynamic splitting engine constructs a splitting decision tree with data type self-awareness. This decision tree dynamically generates splitting strategies by analyzing data element characteristics, including: priority rules for structured data splitting, a subkey generation mechanism for unstructured data, and a strategy for handling differences between short and long keys: establishing a key length threshold θ (default 32 bytes), splitting short keys based on the number of nodes n, and splitting long keys based on the amount of data. (Reference) Figure 11 The specific process is as follows:

[0141] 1. Data type self-aware splitting decision tree

[0142] (1) Priority rules for structured data splitting:

[0143] 1if(field.isPrimaryKey()&&joinCount>1000){ 2splitPriority-=0.3; / / Reduce splitting priority by 30% to avoid excessive splitting of high-frequency associative keys. 3}

[0144] (2) Unstructured data subkey generation mechanism:

[0145] 1StringcontentFingerprint=MD5Utils.hash(content).substring(0,4); 2StringsubKey=originalKey+"_"+contentFingerprint;

[0146] (3) Strategy for handling differences between short and long keys:

[0147] Set a key length threshold θ (default 32 bytes), and use a differentiated splitting formula for key values ​​of different lengths: 1. Short key (length ≤ θ): tskn_short = min(α × N × (1 + S), L_max), 2 long keys (length > θ): .

[0148] in: α is the short bond splitting factor (default 0.8), and β is the long bond splitting factor (default 0.3). N is the number of effective nodes in the cluster, and D is the total amount of data (MB). S is the data skew factor (S=σ / μ, where σ is the standard deviation and μ is the mean). M is the memory stress factor (M = (memory utilization rate - 0.5)). + (Only effective when memory pressure is high) L_max is the system's maximum split limit.

[0149] 2. Cost Evaluation Model for Splitting

[0150] Construct a multi-dimensional cost evaluation function to quantify the resource overhead of different splitting strategies: Cost=w1×NetworkOverhead+w2×ComputeOverhead+w3×MemoryOverhead,

[0151] in:

[0152] NetworkOverhead=max(0.3,1.0 / splitCount)×baseNetworkCost, ComputeOverhead=(1.0+0.1×splitCount)×baseComputeCost, MemoryOverhead=splitCount×keyLength×10×memoryFactor, memoryFactor = 1.0 + (keyLength / 1000.0) (when key length > θ).

[0153] 3. Adaptive Split Execution Algorithm

[0154] The algorithm dynamically determines the optimal number of splits, including the following steps:

[0155] Initial split calculation based on skewness: baseSplits=max(2,ceil(5×skewFactor)), Data type feature adjustment: Limit the maximum number of splits for special data types such as categorical and timestamp data. Dynamic cluster load adjustment: The granularity of data splitting is dynamically reduced based on CPU and memory utilization. Boundary condition constraints: Ensure that the number of splits is within the range of [2, L_max].

[0156] This solution is a parallel scheduling optimization system for heterogeneous Spark clusters. It features the following key innovations:

[0157] 1. Adaptive processing of data types: Differentiated splitting strategies are adopted for different data types. Structured data retains business semantics, while unstructured data is split based on content features.

[0158] 2. Intelligent differentiation between short and long keys: Short keys maximize the use of cluster parallel capabilities, while long keys prioritize memory efficiency and reduce metadata overhead.

[0159] 3. Dynamic adaptive capability: Adjust the splitting granularity based on real-time cluster status, and automatically optimize the splitting strategy when resources are limited.

[0160] 4. High-frequency key protection mechanism: Reduce the splitting priority of primary keys and high-frequency associated keys to avoid business logic breakage caused by excessive splitting.

[0161] 5. Multi-objective optimization balance: By quantifying network, computation, and memory overhead, resource consumption is minimized and end-to-end job execution time is reduced. Experimental verification

[0162] To verify the innovative value of the above-mentioned parallel scheduling optimization method and system for heterogeneous Spark clusters based on data skew correction, a PageRank experiment was designed using two undirected graph datasets: Web-Google and com-YouTube. The experimental framework is as follows: Figure 12 As shown in Table 1, the control group design systematically verifies the innovativeness of this scheme.

[0163] Table 1 Group Configuration Verification target Baseline group Native Spark 3.3 + static resource allocation Traditional scheme performance baseline Experimental group Optimized Spark + dynamic scheduling system Innovation mechanism effectiveness

[0164] The environment configuration is as follows:

[0165] Heterogeneous cluster: 8-node hybrid configuration (3 high-frequency CPUs / 64G, 3 mid-frequency CPUs / 32G, 2 low-frequency CPUs / 16G).

[0166] Fault injection: The ChaosMesh tool randomly terminates worker processes.

[0167] Monitoring component: Prometheus collects 15-second granular metrics (CPU / RAM / network I / O).

[0168] The data skew correction efficiency is verified as shown in Table 2 below.

[0169] Table 2 Verification item Web-Google result YouTube result Split decision time 0.9 s (vs. baseline 3.5 s) 1.2 s (vs. baseline 4.8 s) False negative rate (double detection) 0% 0% (baseline 33%) Dynamic granularity adjustment trigger times 3 times (CPU > 70%) 7 times (including pre-split) Hash redistribution data integrity 100% 100%

[0170] Heterogeneous node scheduling verification Web-Google task allocation pair, for example Figure 13 As shown, the parallelism dynamic optimization verifies the accuracy of the model, for example... Figure 14 As shown, the closure reuse effect is as follows: Figure 15 As shown. The overall system performance gain of the method presented in this paper is as follows. Figure 16 As shown, experiments verify that the maximum task time reduction on the Web-Google dataset is ≥40%, the number of Shuffle writes to disk on the YouTube dataset is ≤1 / 5 of the baseline group, and the fault recovery time from detection to completion of reallocation is ≤60 seconds. With a lightweight deployment and the LSTM prediction module disabled, performance only decreases by 9.7%. Furthermore, even in scenarios with extreme data skew (YouTube Gini coefficient 0.92) and heterogeneous interference (CPU downclocking by 30% + GPU preemption by 40%), this solution still achieves a PageRank job acceleration of >40% and a 60% reduction in resource idle rate. Experiments demonstrate its significant improvement over traditional Spark in data skew correction, resource utilization optimization, and heterogeneous environment adaptation.

[0171] Finally, it should be noted that the above embodiments and implementation methods are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A parallel scheduling optimization method for heterogeneous Spark clusters based on data skew correction, characterized by the following steps: (1) Establish a job scheduling model, collect the partition data volume, node computing power and network bandwidth of each stage through a distributed monitoring agent, and construct a computing cost model; (2) Based on the total amount of input data, the memory requirement ratio factor x_p and the total memory capacity of the nodes, generate the initial parallelism through the parallelism prediction method; (3) Data skew correction; (4) Task allocation of heterogeneous nodes; (5) Dynamic optimization of parallelism; (6) Dynamic scheduling decision driven by reinforcement learning; (7) Data skew identification and correction with graph structure awareness; (8) Multi-level adaptive caching mechanism; (9) Energy efficiency awareness and carbon footprint optimization scheduling; (10) Privacy protection prediction model under the federated learning framework.

2. The parallel scheduling optimization method for heterogeneous Spark clusters based on data skew correction according to claim 1, characterized in that: The data skew correction mentioned in (3) includes: 3.1 Skew detection: Stage-level detection: Calculate the standard deviation σ of the data volume of the partition. If σ > 2 × mean, it is judged as a skewed stage. Key-level detection: Construct a key-value distribution histogram and filter keys with a data volume exceeding 3 times the average as skewed keys (Kskew). 3.2 Dynamic Splitting: Subkey Generation: Append a hash suffix to Kskew, and minimize the splitting cost function Cost = α × network overhead + β × computational overhead by searching the binary search method for the number of splits tskn; Load Awareness: If the average CPU utilization of the cluster is >70%, the splitting granularity is adjusted to ceil(tskn × 1.5) to avoid over-split; Redistribution Strategy: The split subkeys are merged through hash mapping during the Shuffle phase to ensure logical consistency.

3. The parallel scheduling optimization method for heterogeneous Spark clusters based on data skew correction according to claim 1, characterized in that: The heterogeneous node task allocation mentioned in (4) includes: 4.1 Capacity Normalization: Normalized computing power of computing nodes Pnode = Where Ttask is the historical task completion time, and n is the number of tasks; GPU / NPU nodes are additionally multiplied by an acceleration factor. 4.2 Greedy Allocation: Sort nodes in descending order by Pnode, and allocate the largest data bucket to the strongest node first. Execute the formula: Allocation amount = min(bucket size, Pnode × remaining time); 4.3 Dynamic Reallocation: When a node failure is detected, the task queue is paused, and unfinished tasks are reallocated using a healthy node Nhealthy. The reallocation amount is equal to the number of tasks that need to be reallocated. .

4. The parallel scheduling optimization method for heterogeneous Spark clusters based on data skew correction according to claim 1, characterized in that: The aforementioned (5) parallelism dynamic optimization includes: 5.1 Closure reuse: If the signature of the current operation closure is the same as that of a historical closure, its optimal parallelism configuration is directly reused; 5.2 Memory-Aware Strategy: If the node's free memory percentage is >30%, enable the full parallel strategy (Snew = Scurrent × 2); if memory is insufficient, proceed according to... Downgraded.

5. The parallel scheduling optimization method for heterogeneous Spark clusters based on data skew correction according to claim 1, characterized in that: The reinforcement learning-driven dynamic scheduling decision mentioned in (6) includes: 6.1 Construct a scheduling strategy model based on the PPO (Proximal Policy Optimization) algorithm, with the cluster state vector and job characteristics as inputs; 6.2 Design a multi-objective reward function R = w1 × throughput + w2 × resource utilization - w3 × long-tail delay - w4 × energy consumption; 6.3 Online learning mechanism: The model is updated iteratively every 100 assignments, and the performance of the new strategy is verified through A / B testing; 6.4 Offline-Online Collaborative Training: The offline phase involves pre-training on historical data, while the online phase involves fine-tuning based on actual execution results.

6. The parallel scheduling optimization method for heterogeneous Spark clusters based on data skew correction according to claim 1, characterized in that: The data tilt identification and correction for graph structure perception mentioned in (7) includes: 7.1 Construct a data lineage graph G=(V,E), where V represents a data partition and E represents a data dependency relationship; 7.2 Applying Graph Neural Networks (GNNs) to calculate node importance scores: ; 7.3 Skew propagation prediction based on graph structure: skewfuture(v) = α × skewcurrent(v) + β × ∑u∈N(v)skewcurrent(u); 7.4 Structure-aware splitting: Prioritize splitting skewing keys that are associated with multiple downstream tasks to reduce global impact.

7. The parallel scheduling optimization method for heterogeneous Spark clusters based on data skew correction according to claim 1, characterized in that: The aforementioned (8) multi-level adaptive caching mechanism includes: 8.1 Construct a three-level caching system: L1 (CPU cache), L2 (node ​​memory), and L3 (distributed shared cache); 8.2 Dynamic decision-making regarding cached content: ; 8.3 Cross-job cache sharing: Construct a job similarity matrix Sim(i,j)=cos(featurei,featurej), and jobs with similarity > 0.7 share the cache; 8.4 Learning-based cache eviction: Using LSTM to predict future access probabilities and prioritizing the eviction of low-probability data.

8. The parallel scheduling optimization method for heterogeneous Spark clusters based on data skew correction according to claim 1, characterized in that: The (9) energy efficiency sensing and carbon footprint optimization scheduling includes: 9.1 Constructing the energy efficiency model: Ptotal = Pbase + k1 × CPUutil + k2 × Memoryutil + k3 × Networkutil; 9.2 Carbon Intensity Sensing and Scheduling: Integrating real-time carbon intensity data from the power grid, carbon_intensity(t) = f(time,location,energy_mix); 9.3 Task scheduling optimization objective: min∑(w1×runtime+w2×energy_consumption+w3×carbon_footprint); 9.4 Dynamic frequency adjustment: Adjust the CPU frequency fnew=fbase×(1-λ×carbon_intensity) in real time according to the mission critical path and carbon intensity changes.

9. The parallel scheduling optimization method for heterogeneous Spark clusters based on data skew correction according to claim 1, characterized in that: The privacy-preserving prediction model under the federated learning framework (10) includes: 10.1 Construct a cross-organization federated learning architecture where data from different organizations does not leave their domains, and only model gradients are exchanged; 10.2 Differential Privacy Protection: Adding Laplace Noise To gradient update; 10.3 Model Aggregation Optimization: ; 10.4 Personalized Model Branch: Retain the global model backbone and deploy personalized headers in each organization.

10. A parallel scheduling optimization system for heterogeneous Spark clusters based on data skew correction, characterized in that: include: The cost modeling module integrates a cost calculation model and a resource vacancy rate formula library; the key-value analyzer uses a sliding window to statistically analyze the distribution of key-value data and generates histograms for skew detection; the dynamic splitting engine performs subkey splitting, hash redistribution, and incremental parallelism calculation; the capability sorter calculates normalized capabilities based on the historical task completion time of nodes; the closure cache stores the closures of each stage operation and the optimal parallelism record; and the prediction execution module includes multi-dimensional metric analysis, feature extraction, model service, and feedback learning. The dynamic splitting engine constructs a splitting decision tree with data type self-awareness. The splitting decision tree dynamically generates splitting strategies by analyzing data element characteristics, including: structured data splitting priority rules, unstructured data subkey generation mechanism, and short key and long key difference processing strategy.

Images