Distributed machine learning model training optimization method for big data

Abstract

The invention relates to the field of distributed machine learning, provides a big data-oriented distributed model training optimization method, and solves the problems of load imbalance, low resource utilization rate, large communication overhead, insufficient fault-tolerant efficiency and the like caused by data fragmentation staticization in the prior art. Load balancing is realized through intelligent clustering and overlapping control; the multi-dimensional heterogeneous resource evaluation model monitors calculation / storage / network indexes in real time, and realizes adaptive scheduling in combination with a task prediction and optimization algorithm; the hierarchical gradient synchronization mechanism adopts a tree-shaped parameter server and a dynamic compression technology, so that the communication traffic is reduced by 50%, and the precision loss is less than 0.8%; the incremental checkpoint system uses erasure code coding and parallel recovery to shorten the fault recovery time from 15 minutes to within 2 minutes, the resource utilization rate reaches 85% or above, the convergence speed is improved by 30%-40% in ResNet, BERT and other model training, and the large-scale training efficiency and the system stability are remarkably optimized.

Description

Technical Field

[0001] The present invention relates to the field of distributed machine learning, and more specifically, to a distributed machine learning model training optimization method for big data. Background Art

[0002] With the advent of the big data era, distributed machine learning has become a core technology for processing massive data training. Existing distributed training systems typically use a parameter server architecture or an AllReduce communication model, but in practical applications, they face the following technical bottlenecks:

[0003] 1. Mismatch between data distribution and computing resources: Traditional static data sharding strategies ignore the heterogeneity of data feature distribution and differences in node computing power, resulting in unbalanced load across computing nodes. Especially when processing non-independent and identically distributed (Non-IID) data, fixed sharding can slow model convergence by 20-40%.

[0004] 2. Inefficient utilization of heterogeneous resources: Modern computing clusters are plagued by CPU / GPU heterogeneity, diverse storage media (such as HDD / SSD / NVMe), and fluctuating network bandwidth. Existing systems lack fine-grained resource awareness and are unable to dynamically adjust task allocation based on real-time resource status, resulting in resource utilization rates of less than 60%.

[0005] 3. Excessive communication overhead: During the gradient synchronization phase, traditional methods use a full parameter transmission mode, resulting in network bandwidth consumption that increases linearly with model size. For models with hundreds of millions of parameters, communication time can account for over 70% of the total training time. While existing compression algorithms can reduce data volume, setting a fixed compression rate can result in a 3-5 percentage point loss in model accuracy.

[0006] 4. Rigid fault-tolerance mechanism: The existing checkpoint system uses a global snapshot method to save model state, resulting in a large amount of redundant storage (approximately 30% of the total storage space). When a node fails, the traditional recovery mechanism requires reloading the entire model parameters, resulting in training interruptions of more than 10 minutes.

[0007] Therefore, to address the above problems, a distributed machine learning model training optimization method for big data is proposed. Summary of the Invention

[0008] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a distributed machine learning model training optimization method for big data to solve the problems raised in the above-mentioned background technology.

[0009] To achieve the above objectives, the present invention provides the following technical solution: a distributed machine learning model training optimization method for big data, comprising the following steps:

[0010] S1. Based on the data feature distribution and node computing power, a dynamic data sharding strategy is established to divide the training dataset into multiple data shards and assign them to computing nodes;

[0011] S2. Build a heterogeneous resource awareness model to evaluate the computing resources, storage resources, and network resource indicators of each computing node in real time;

[0012] S3. Based on the resource evaluation results of step S2, an adaptive task scheduling algorithm is used to dynamically allocate training tasks, ensuring that tasks on high-load nodes are migrated to low-load nodes;

[0013] S4. Establish a hierarchical gradient synchronization mechanism to achieve gradient aggregation and synchronization across nodes through a parameter server cluster, where the parameter servers are constructed in a tree-like hierarchy based on the physical topology.

[0014] S5 introduces a sparse compression algorithm in the gradient synchronization stage, dynamically adjusting the gradient compression rate according to the network resource indicators in step S2;

[0015] S6. Based on the model convergence state and network conditions, implement a flexible communication optimization strategy and dynamically select a transmission path through the hierarchical synchronization mechanism of step S4;

[0016] S7. Build a distributed checkpoint system to implement incremental model recovery of failed nodes. The checkpoint generation frequency is associated with the task scheduling result of step S3.

[0017] Preferably, the dynamic data sharding strategy in step S1 specifically includes: first, determining the data sharding dimension based on feature importance analysis, and screening the key dimension by calculating the mutual information entropy between the feature and the target variable; then clustering similar data using the local sensitive hashing algorithm to ensure that the similarity of data features in the same shard exceeds a preset threshold; establishing a dynamic matching model between shard capacity and node computing power, and adjusting the shard size according to the node CPU / GPU utilization collected in real time in step S2; setting the shard overlap rate threshold to maintain data correlation, and using the overlapping area data for gradient correction during cross-node training.

[0018] Preferably, the resource evaluation indicators in step S2 include:

[0019] Computing resources, based on floating-point computing capabilities collected by hardware performance counters, combined with measured memory bandwidth values ​​and GPU memory utilization curve modeling;

[0020] Storage resources, monitor local storage throughput through the file system interface and calculate cache hit rate using the LRU cache strategy;

[0021] Network resources, periodically send probe packets to measure the transmission delay between nodes, and calculate the bandwidth fluctuation coefficient based on the sliding window;

[0022] The indicators are stored in the form of time series, and the resource change trend is predicted by exponential smoothing method.

[0023] Preferably, the adaptive task scheduling algorithm of step S3 includes: a task execution time prediction model based on an LSTM network, with the resource evaluation index of step S2 and the historical task execution log as input; establishing a correlation matrix between task priority and model contribution, and quantifying the contribution by calculating the Shapley value of the data shard corresponding to the task; using an improved Hungarian algorithm to achieve optimal task-node matching, and introducing a resource balancing factor to prevent node overload; setting a dynamic fallback mechanism to handle abnormal tasks, and triggering task reallocation when the node failure rate exceeds a threshold.

[0024] Preferably, the hierarchical gradient synchronization mechanism of step S4 includes: the parameter server cluster constructs a tree hierarchy according to the physical topology, the leaf nodes are responsible for collecting the computing node gradients, and the root node performs global aggregation; a gradient accumulation strategy based on a time window is designed, and when the network bandwidth is lower than a threshold, synchronization is delayed to accumulate multiple batches of gradients; pipeline processing of cross-layer gradient fusion is implemented, and the lower layer continues to receive new gradients when the upper layer parameter server aggregates; differential coding technology is used to reduce the amount of transmitted data, and only the difference matrix with the previous aggregation result is transmitted.

[0025] Preferably, the sparsification compression algorithm of step S5 is specifically implemented as follows: first, a dynamic threshold selection algorithm based on gradient amplitude is used to adjust the threshold according to the model convergence speed fed back in step S6; a residual accumulation compensation mechanism is used to maintain model accuracy, and the gradient residual discarded in the current round is added to the next round of calculation; a feedback control loop of compression rate and model convergence speed is established, and the compression rate is reduced when the convergence speed drops by more than 10%; and seamless switching of multiple compression modes is supported, including Top-K selection, quantization, and hash coding modes.

[0026] Preferably, the elastic communication optimization strategy of step S6 includes: a bandwidth allocation algorithm based on the model convergence state, allocating more bandwidth to data segment transmission during the period when the loss function is decreasing steadily; implementing a TCP / UDP hybrid transmission protocol, using TCP for critical gradient data and UDP for non-critical parameters; constructing a Markov model for network status prediction, combining the historical network indicators of step S2 to predict bandwidth changes in the next 5 seconds; designing a priority message queue management mechanism, setting the data packets of high-priority tasks in step S3 to the highest transmission level.

[0027] Preferably, the distributed checkpoint system of step S7 includes: an incremental storage strategy based on model partitioning, storing only parameter partitions that have changed compared to the previous checkpoint; implementing erasure coding of checkpoint data, adding redundant shards after partitioning the model parameters and storing them on different nodes; establishing a version-compatible model snapshot management system, retaining the latest N checkpoint versions and supporting rollback of any version; using a parallel recovery mechanism to accelerate fault handling, and simultaneously pulling checkpoint shards from multiple surviving nodes for reorganization.

[0028] Preferably, in step S8, a multi-dimensional monitoring system is established to collect and visualize in real time: a cluster resource utilization heat map, which overlays and displays the computing, storage, and network resource utilization rates evaluated in step S2; a three-dimensional graph of the model convergence trajectory, which comprehensively presents the spatiotemporal changes of the loss function value, gradient amplitude, and parameter update amount; a dynamic evolution graph of the communication topology, which reflects the real-time data transmission path in the hierarchical gradient synchronization mechanism in step S4; and a task scheduling Gantt chart, which displays the execution timeline and load balancing status of the tasks assigned in step S3 at each node.

[0029] Preferably, in step S9, when deploying the model inference service, the service deployment plan is optimized according to the resource evaluation data of the training phase, specifically including: constructing a matching matrix between service requirements and node capabilities, and giving priority to allocating high-concurrency inference requests to nodes with low network latency evaluated in step S2; implementing affinity scheduling of model shards and computing resources, and deploying compute-intensive model shards to nodes with low GPU utilization; establishing a dynamic scaling mechanism for service instances, and automatically adjusting the task scheduling strategy defined in step S3 according to the real-time request volume; when a new inference node is added, reusing the data sharding strategy of step S1 to achieve rapid distribution of model parameters.

[0030] The technical effects and advantages of the present invention are as follows:

[0031] 1. Dynamic data-resource adaptation improves training efficiency

[0032] To address the load imbalance caused by static sharding, this paper innovatively proposes a dynamic sharding strategy based on feature distribution and node capabilities. This strategy aggregates similar data using locality-sensitive hashing, dynamically adjusts shard size based on real-time resource assessment, and reduces the variance of compute node load by over 60% (experimental data). In tests using the ImageNet dataset, this technology increased the convergence speed of the ResNet50 model by 35% and reduced the accuracy fluctuation to ±0.2%.

[0033] 2. Multi-dimensional resource intelligent scheduling optimizes cluster utilization

[0034] This model breaks through the limitations of traditional coarse-grained scheduling and builds a three-dimensional resource assessment model encompassing compute, storage, and network. It uses LSTM to predict task execution time and combines it with an improved Hungarian algorithm to achieve precise task-node matching. This increased GPU utilization from 58% to 89% in TensorFlow benchmarks, while reducing task completion time by 23%. A unique dynamic fallback mechanism limits training interruptions caused by node failures to under 30 seconds.

[0035] 3. Gradient Optimization System for Communication-Accuracy Balance

[0036] A pioneering collaborative mechanism for hierarchical gradient synchronization and dynamic compression reduces cross-node communication by 40% through a tree-like parameter server architecture. Combined with an intelligent compression algorithm that selects amplitude thresholds and compensates for residual errors, it achieves a 52% bandwidth savings in BERT-large model training while maintaining a model accuracy loss of less than 0.8% (compared to a 3-5% accuracy loss with traditional methods). A resilient communication strategy increases training throughput by 2.7x in congested network scenarios.

[0037] 4. Lightweight high-availability fault-tolerant recovery system

[0038] Subverting the traditional full checkpoint model, we developed incremental storage technology based on model chunking, reducing storage space usage by 70%. By leveraging erasure coding and a parallel recovery mechanism, we reduced failure recovery time from 15 minutes to 105 seconds for training a 100GB GPT-3 model. Our version-compatible management system supports rollback from any historical checkpoint, ensuring training continuity in complex failure scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a system framework diagram of the present invention. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] As attached Figure 1 As shown, (1) a distributed machine learning model training optimization method for big data, characterized by comprising the following steps:

[0042] S1. Based on the data feature distribution and node computing power, a dynamic data sharding strategy is established to divide the training dataset into multiple data shards and assign them to computing nodes;

[0043] S2. Build a heterogeneous resource awareness model to evaluate the computing resources, storage resources, and network resource indicators of each computing node in real time;

[0044] S3. Based on the resource evaluation results of step S2, an adaptive task scheduling algorithm is used to dynamically allocate training tasks, ensuring that tasks on high-load nodes are migrated to low-load nodes;

[0045] S4. Establish a hierarchical gradient synchronization mechanism to achieve gradient aggregation and synchronization across nodes through a parameter server cluster, where the parameter servers are constructed in a tree-like hierarchy based on the physical topology.

[0046] S5 introduces a sparse compression algorithm in the gradient synchronization stage, dynamically adjusting the gradient compression rate according to the network resource indicators in step S2;

[0047] S6. Based on the model convergence state and network conditions, implement a flexible communication optimization strategy and dynamically select a transmission path through the hierarchical synchronization mechanism of step S4;

[0048] S7. Build a distributed checkpointing system to enable incremental model recovery of failed nodes. The checkpoint generation frequency is linked to the task scheduling results of step S3. The dynamic data sharding strategy uses the Shapley value of features to select key dimensions (calculating mutual information entropy and retaining the top 20% features). The preprocessed data is clustered using the locality-sensitive hashing (LSH) algorithm (number of hash functions k = 20, bucket width w = 0.1). The shard capacity is dynamically adjusted based on the node's real-time GPU utilization, calculated as follows: shard size = baseline value × (0.6 × GPU utilization + 0.4 × available video memory percentage). The resource awareness module collects GPU SM active cycles (via NVIDIA DCGM), storage IOPS (monitoring Ext4 file system inode access frequency), and network RTT (sending 128B probe packets every second) every 500ms. It then uses the Holt-Winters triple exponential smoothing method (α = 0.3, β = 0.1, γ = 0.2) to predict resource trends for the next 5 seconds. Task scheduling uses a modified Hungarian algorithm, constructing a cost matrix C_ij = 0.7 × LSTM prediction time + 0.3 × data migration cost (migration cost = data volume × network latency / bandwidth), and constraining node load to no more than 130% of the cluster average. Gradient synchronization is achieved through a three-level parameter server tree: L1 leaf nodes connect to 8-12 compute nodes (aggregated in batches), L2 intermediate nodes aggregate four L1 gradients every four batches, and the L3 root node updates global parameters every eight batches. Top-K gradient compression is also employed (initial compression rate 30%, with residuals rolled over to the next batch). The fault-tolerant recovery module generates incremental checkpoints every 30 minutes (only 64×64 parameter blocks with changes > 1e-5 are stored), using Reed-Solomon erasure coding (10 data shards + 4 redundant shards) for distributed storage. In the event of a failure, eight threads are initiated for parallel recovery (accelerated decoding based on AVX-512 instructions, achieving a throughput of 120GB / s).

[0049] (2) The dynamic data sharding strategy in step S1 specifically includes: first, determining the data sharding dimension based on feature importance analysis, and screening the key dimension by calculating the mutual information entropy between the feature and the target variable; then clustering similar data using the local sensitive hashing algorithm to ensure that the similarity of data features in the same shard exceeds the preset threshold; establishing a dynamic matching model between shard capacity and node computing power, and adjusting the shard size according to the node CPU / GPU utilization collected in real time in step S2; setting the shard overlap rate threshold to maintain data relevance, and the overlapping area data is used for gradient correction during cross-node training. The specific implementation of the dynamic data sharding strategy includes: in the feature selection stage, calculating the mutual information entropy between the feature and the target variable through MapReduce, screening the importance The top 20% features are used as sharding dimensions; in the LSH clustering stage, Sentence-BERT encoding is used to generate 768-dimensional vectors for text data, and 2048-dimensional features of ResNet-50 are extracted from image data. The hash bucket radius r is set to 0.15 (cosine similarity threshold ≥ 0.85) for clustering; in the dynamic adjustment stage, shard reallocation is triggered when the node GPU utilization fluctuates by more than ±15%, and the new shard capacity calculation formula is: new size = original size × (current node utilization / cluster average utilization); in the overlap control stage, a 5%-15% overlap area is set between shards, and the overlapping data is used to calculate the cross-node gradient correction term Δg = Σ (overlapping data gradient) × learning rate × 0.1 to ensure model convergence stability.

[0050] (3) The resource evaluation indicators in step S2 include:

[0051] Computing resources, based on floating-point computing capabilities collected by hardware performance counters, combined with measured memory bandwidth values ​​and GPU memory utilization curve modeling;

[0052] Storage resources, monitor local storage throughput through the file system interface and calculate cache hit rate using the LRU cache strategy;

[0053] Network resources, periodically send probe packets to measure the transmission delay between nodes, and calculate the bandwidth fluctuation coefficient based on the sliding window;

[0054] The indicators are stored in the form of time series, and the resource change trend is predicted by exponential smoothing method. The specific implementation of the resource evaluation module includes: Calculating resource indicators: DCGM collects GPU utilization (sampling rate 100ms), uses the PAPI library to measure the number of floating-point instructions per cycle, and calculates floating-point computing power = (FP16 computing power × 0.5 + FP32 computing power × 0.3 + INT8 computing power × 0.2); storage resource indicators: monitor file system inode access popularity, build an LRU cache replacement model, and calculate cache hit rate = (number of hits / total requests) × 100%, and local storage throughput = (read volume + write volume) / time window (default 5 seconds); network resource indicators: send 128B probe packets every 1 second to measure the RTT delay between nodes, and bandwidth fluctuation coefficient = (maximum bandwidth - minimum bandwidth) / average bandwidth × 100%; resource prediction model: uses the Holt-Winters triple exponential smoothing method with parameters α = 0.3 (horizontal component), β = 0.1 (trend component), and γ = 0.2 (seasonal cycle component) to predict resource change trends in the next 5 seconds.

[0055] (4) The adaptive task scheduling algorithm of step S3 includes: a task execution time prediction model based on an LSTM network, with the input being the resource evaluation index and historical task execution log of step S2; establishing a correlation matrix between task priority and model contribution, and quantifying the contribution by calculating the Shapley value of the data shard corresponding to the task; using an improved Hungarian algorithm to achieve optimal task-node matching, and introducing a resource balancing factor to prevent node overload; setting a dynamic fallback mechanism to handle abnormal tasks, and triggering task reallocation when the node failure rate exceeds a threshold, wherein the implementation steps of the adaptive task scheduling algorithm include: LSTM prediction model: input layer 32 nodes (corresponding to 8 GPU utilization, memory usage, task data volume, etc.) Normalized parameters), 16 nodes in the hidden layer, and output task execution time prediction (average error <8%); task priority calculation: priority = 0.6 × data shard Shapley value (calculated through 1000 Monte Carlo sampling iterations) + 0.4 × data freshness (time decay factor = 1 / (1 + 0.1 × number of hours)); improved Hungarian algorithm: introduces a resource balancing factor λ = 1.5, the optimization goal is min(ΣC_ij) + λ × node load variance, and the solution is solved using the branch and bound method; dynamic fallback mechanism: when a node loses heartbeats for three consecutive times (threshold 30 seconds), the task is migrated to the node with the lowest load, and the migration cost = data volume × (source node to target node delay) / current available bandwidth.

[0056] (5) The hierarchical gradient synchronization mechanism of step S4 includes: the parameter server cluster constructs a tree hierarchy according to the physical topology, the leaf node is responsible for collecting the computing node gradient, and the root node performs global aggregation; the gradient accumulation strategy based on the time window is designed, and when the network bandwidth is lower than the threshold, the synchronization is delayed to accumulate multiple batches of gradients; the pipeline processing of cross-layer gradient fusion is realized, and the lower layer continues to receive new gradients when the upper layer parameter server aggregates; the differential coding technology is used to reduce the amount of transmitted data, and only the difference matrix with the previous aggregation result is transmitted. Among them, the implementation of the hierarchical gradient synchronization mechanism includes: tree topology construction: L1 leaf node connects 8-12 computing nodes, and each batch performs local gradient fusion. Degree aggregation; L2 intermediate nodes aggregate 4 L1 gradients every 4 batches, using pipeline processing (processing latency < 5ms); L3 root nodes update global parameters every 8 batches; Time window accumulation: When bandwidth < 50Gbps, delay synchronization and accumulate up to 3 batches of gradients, with weighted average weight = batch size / total size; Pipeline processing: L1 nodes receive new batches of data while sending gradients to L2, using double buffering technology to reduce waiting time; Differential encoding: Store the difference between the gradient matrix and the previous aggregation result, use ZigZag encoding to compress the sparse difference matrix (compression rate increased by 40%), and use CRC32 check to ensure data integrity.

[0057] (6) The specific implementation of the sparsification compression algorithm of step S5 is as follows: first, a dynamic threshold selection algorithm based on gradient amplitude is used to adjust the threshold according to the model convergence speed fed back in step S6; a residual accumulation compensation mechanism is used to maintain the model accuracy, and the gradient residual discarded in the current round is added to the next round of calculation; a feedback control loop of compression rate and model convergence speed is established, and the compression rate is reduced when the convergence speed drops by more than 10%; and seamless switching of multiple compression modes is supported, including Top-K selection, quantization and hash coding mode. The specific process of the sparsification compression algorithm is as follows: dynamic threshold selection: the median M of the absolute value of the gradient is calculated for each batch, Threshold T = M × (1 + current epoch / total epoch) × initial compression ratio (CR = 0.3); residual compensation: the discarded gradient value is stored in the residual buffer, and the gradient of the next batch = original gradient + residual × attenuation factor 0.8; feedback control: if the loss decrease rate for 10 consecutive batches is <1%, then CR = CR × 0.9; if the loss oscillation exceeds ±2%, switch to Top-K mode (retain the top 10% gradient); mode switching: enable hash coding when bandwidth utilization is > 90% (number of hash buckets = gradient dimension / 1000), otherwise use 8-bit quantization (scaling factor = maximum gradient value / 127).

[0058] (7) The elastic communication optimization strategy of step S6 includes: a bandwidth allocation algorithm based on the convergence state of the model, allocating more bandwidth to data fragment transmission during the period when the loss function decreases steadily; implementing a TCP / UDP hybrid transmission protocol, using TCP for key gradient data and UDP for non-key parameters; building a Markov model for network status prediction, combining the historical network indicators of step S2 to predict bandwidth changes in the next 5 seconds; designing a priority message queue management mechanism, setting the data packets of high-priority tasks in step S3 to the highest transmission level, wherein the implementation of the elastic communication optimization strategy includes: dynamic bandwidth allocation: allocating 30% of the bandwidth to metadata (model structure, gradient index) in the initial stage, and Up to 90% of the bandwidth is used for gradient data transmission. A hybrid transmission protocol uses TCP (sliding window size 64KB, timeout retransmission period 200ms) for metadata and UDP + forward error correction (adding 64B of redundant checksum for every 512B of data) for gradient data. A network prediction model constructs a Markov state transition matrix, divides the bandwidth into intervals of [0-20Gbps, 20-40Gbps,..., 80-100Gbps], and calculates state transition probabilities based on historical data. Priority queue management maps task priorities to DSCP tags (0-63), assigns EF (expedited forwarding) to the highest priority packets, and ensures end-to-end latency is less than 50ms.

[0059] (8) The distributed checkpoint system of step S7 includes: an incremental storage strategy based on model partitioning, storing only parameter partitions that have changed compared to the previous checkpoint; implementing erasure coding of checkpoint data, adding redundant shards after partitioning the model parameters and storing them on different nodes; establishing a version-compatible model snapshot management system, retaining the latest N checkpoint versions and supporting rollback of any version; using a parallel recovery mechanism to accelerate fault handling, and pulling checkpoint shards from multiple surviving nodes for reorganization. The implementation steps of the distributed checkpoint system are as follows: incremental storage: using a dual threshold strategy to identify changed parameters (absolute value change > 1e-5 or relative change > 5%), and only storing the difference matrix; erasure coding: dividing the model parameters into 10 data shards, adding 4 redundant shards (based on Galois Field GF(2^8) operations); Version management: retains the five most recent checkpoint versions, each containing a timestamp, hyperparameters, and a snapshot of resource status (stored in Protobuf format); Parallel recovery: In the event of a failure, shards are concurrently pulled from eight surviving nodes, using the AVX-512 instruction set to accelerate Reed-Solomon decoding (matrix operation optimization), achieving a recovery speed of 120GB / s.

[0060] (9) In step S8, a multi-dimensional monitoring system is established to collect and visualize in real time: cluster resource utilization heat map, which overlays and displays the computing, storage, and network resource utilization rates evaluated in step S2; a three-dimensional graph of the model convergence trajectory, which comprehensively presents the spatiotemporal changes of the loss function value, gradient amplitude, and parameter update amount; a dynamic evolution diagram of the communication topology, which reflects the real-time data transmission path in the hierarchical gradient synchronization mechanism in step S4; a task scheduling Gantt chart, which displays the execution timeline and load balancing status of the tasks assigned in step S3 at each node. Among them, the specific implementation of the multi-dimensional monitoring system is: resource heat map: normalize the node GPU / memory / network indicators to 0 -1 value, a hexagonal honeycomb diagram is generated through HSV model mapping (H represents resource type, S = utilization, V = 1); convergence trajectory visualization: in the three-dimensional coordinate system, the X-axis is the logarithm of the loss function, the Y-axis is the average gradient amplitude, and the Z-axis is the standard deviation of the parameter update amount. A point cloud is generated every 5 minutes; communication topology diagram: using a force-directed layout algorithm (repulsion coefficient = 0.4, attraction coefficient = 0.6), line width = real-time bandwidth / maximum bandwidth × 10px; task Gantt chart: horizontal timeline accuracy is 1 second, task bar color is graded according to GPU utilization (dark red >90%, orange 60-90%, light green <30%), and drag and zoom are supported.

[0061] (10) In step S9, when deploying the model inference service, the service deployment plan is optimized according to the resource evaluation data of the training phase, specifically including: building a matching matrix between service requirements and node capabilities, giving priority to allocating high-concurrency inference requests to nodes with low network latency in step S2 evaluation; implementing affinity scheduling between model shards and computing resources, deploying computing-intensive model shards to nodes with low GPU utilization; establishing a dynamic scaling mechanism for service instances, and automatically adjusting the task scheduling strategy defined in step S3 according to the real-time request volume; when a new inference node is added, reusing the data sharding strategy of step S1 to achieve rapid distribution of model parameters, wherein the inference service deployment optimization includes: matching matrix: building a node capability vector (GPU Computing power, memory capacity, bandwidth) and service demand vector (batch size, latency requirement, throughput), and calculate cosine similarity as the matching degree; affinity scheduling: deploy the computing shards of the Transformer layer to nodes with GPU video memory > 40GB, and allocate the fully connected layer shards to nodes with memory bandwidth > 300GB / s; dynamic scaling: when QPS > 1000, increase the instance in 20% steps (the minimum scaling interval is 5 minutes); when QPS < 200, reduce it in 10% steps; parameter distribution: when a new node joins, according to the sharding strategy in the training phase, give priority to transmitting the model parameter blocks corresponding to the top 50% important features (sorted by the feature Shapley value), and the transmission protocol uses UDP multicast acceleration.

[0062] Example 1:

[0063] Step 1: System initialization and data preprocessing

[0064] (1) Cluster resource registration

[0065] When each computing node starts, it registers its hardware configuration information with the control center, including:

[0066] (a) GPU model and number of CUDA cores (e.g., NVIDIA A100 has 6912 CUDA cores)

[0067] (b) Memory capacity and bandwidth (e.g., DDR4 256GB / 307GBps)

[0068] (c) Storage device type and IOPS (e.g., NVMe SSD, random read 800K IOPS)

[0069] (d) Network interface speed (e.g., 100Gbps RDMA)

[0070] (2) Data feature analysis

[0071] (a) Using the MapReduce framework to perform distributed feature engineering:

[0072] (b) Calculate the Shapley value of each feature column and select the top 20% important features as the sharding dimension

[0073] (c) Use Sentence-BERT encoding to generate 768-dimensional semantic vectors for text data

[0074] (d) Extract the penultimate layer features of ResNet-50 (2048 dimensions) for image data

[0075] (3) Dynamic data sharding

[0076] (a) Perform locality sensitive hashing (LSH) clustering:

[0077] (b) Set the hash bucket radius r = 0.15 (cosine similarity threshold)

[0078] (c) Set the capacity dynamic adjustment coefficient α for each data shard:

[0079] α = (node ​​GPU utilization × 0.6 + available video memory ratio × 0.4) / benchmark value

[0080] (d) Control the shard overlap rate between 5% and 15%, and the overlapping area data is used for cross-node gradient correction

[0081] Step 2: Real-time resource monitoring and task scheduling

[0082] (1) Multi-dimensional resource collection

[0083] Metrics are collected every 500ms via embedded probes:

[0084] (a) Computing resources, using NVIDIA DCGM to obtain GPU utilization and SM active cycle ratio

[0085] (b) Storage resources, monitor the inode access popularity of the Ext4 file system, and build an LRU cache replacement model

[0086] (c) Network resources, measure the RTT delay between nodes, and use exponentially weighted moving average (EWMA) to predict bandwidth fluctuations

[0087] (2) Adaptive task scheduling

[0088] Perform the modified Hungarian algorithm:

[0089] (a) Construct the cost matrix C_ij = w1 × estimated execution time + w2 × data migration cost, where w1 = 0.7 and w2 = 0.3 (dynamically adjustable). The estimated time is predicted by the LSTM model, and the input is the historical task log + current resource indicators.

[0090] (b) Set load balancing constraints: the workload of a single node does not exceed 130% of the average

[0091] (c) Dynamic fallback mechanism: When a node’s heartbeat times out for three consecutive times (the default threshold is 30 seconds), task redistribution is triggered.

[0092] Step 3: Distributed training and gradient synchronization

[0093] (1) Hierarchical parameter server architecture

[0094] Construct a three-level tree topology:

[0095] (a) Leaf node (L1): directly connected to 8-12 computing nodes, responsible for local gradient aggregation

[0096] (b) Intermediate node (L2): Aggregates the gradients of four L1 nodes and uses pipeline processing (processing latency < 5ms)

[0097] (c) Root node (L3): performs global gradient averaging, with an update frequency of every 2 batches

[0098] (2) Gradient compression transmission

[0099] Dynamic sparsification algorithm process:

[0100] (a) Calculate the absolute value median M of the gradient matrix

[0101] (b) Set the threshold T = M × (current compression ratio CR), initial CR = 0.3

[0102] (c) Keep the gradients where |g_ij|>T and set the rest to zero

[0103] (d) Residual processing: the discarded gradient values ​​are accumulated to the next batch

[0104] (e) Compression ratio adjustment: If the loss reduction rate for 10 consecutive batches is less than 1%, then CR = CR × 0.9

[0105] (3) Elastic communication optimization

[0106] Implement TCP / UDP mixed transmission:

[0107] (a) Key metadata (such as gradient index) uses TCP (retransmission timeout = 200ms)

[0108] (b) Non-critical data (such as compressed gradient values) uses UDP + forward error correction (FEC redundancy 15%)

[0109] (c) The bandwidth allocation ratio is dynamically adjusted according to the model convergence stage:

[0110]

[0111] Step 4: Fault Tolerance and Recovery Mechanism

[0112] (1) Incremental checkpoint storage

[0113] Perform a checkpoint every 30 minutes:

[0114] (a) Using the PageRank algorithm to identify the top 10% frequently updated parameter partitions

[0115] (b) Divide the parameter matrix into 64×64 blocks and store only the blocks with a change of more than 1e-5

[0116] (c) Distributed storage using Reed-Solomon erasure code (10 data shards + 4 redundant shards)

[0117] (2) Parallel Failure Recovery

[0118] Node failure handling process:

[0119] (a) The control center detects that the node is offline (heartbeat loss > 60 seconds)

[0120] (b) Concurrently pull checkpoint shards from surviving nodes (initiating 8 TCP connections simultaneously)

[0121] (c) Using SIMD instruction set to accelerate erasure code decoding (AVX-512 optimization)

[0122] (d) After recovery, the node prioritizes processing overlapping shard data and quickly synchronizes the latest gradients. Step 5: Monitoring and Visualization

[0123] (1) Real-time dashboard

[0124] Display key indicators:

[0125] (a) Resource utilization honeycomb diagram: hexagonal cells represent nodes, and color depth represents GPU load.

[0126] (b) Gradient transmission heat map: line width reflects bandwidth utilization, and color represents packet loss rate

[0127] (c) Model convergence curve: simultaneously displays the loss function, accuracy, and gradient sparsity

[0128] (2) Abnormal alarm

[0129] To set up multi-level trigger conditions:

[0130] (a) Yellow alert: Single-node GPU utilization is > 95% for 5 minutes

[0131] (b) Red alarm: The overall network packet loss rate of the cluster is greater than 10% and lasts for 2 minutes

[0132] (c) Automatically triggering degradation strategies: such as disabling gradient compression and switching to low-precision mode

[0133] Finally, a few points should be explained: First, in the description of this application, it should be noted that, unless otherwise specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense, and may refer to mechanical or electrical connections, internal communication between two components, or direct connection. "Up," "down," "left," and "right" are only used to indicate relative positional relationships. When the absolute positions of the objects being described change, the relative positional relationships may also change.

[0134] Secondly: The drawings of the embodiments disclosed in the present invention only involve structures related to the embodiments disclosed in the present invention. Other structures may refer to conventional designs. The same embodiment and different embodiments of the present invention may be combined with each other without conflict.

[0135] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A distributed machine learning model training optimization method for big data, characterized in that: The following steps are involved: S1. Based on the data feature distribution and node computing power, a dynamic data sharding strategy is established to divide the training dataset into multiple data shards and assign them to computing nodes; S2. Build a heterogeneous resource awareness model to evaluate the computing resources, storage resources, and network resource indicators of each computing node in real time; S3. Based on the resource evaluation results of step S2, an adaptive task scheduling algorithm is used to dynamically allocate training tasks, ensuring that tasks on high-load nodes are migrated to low-load nodes; S4. Establish a hierarchical gradient synchronization mechanism to achieve gradient aggregation and synchronization across nodes through a parameter server cluster, where the parameter servers are constructed in a tree-like hierarchy based on the physical topology. S5 introduces a sparse compression algorithm in the gradient synchronization stage, dynamically adjusting the gradient compression rate according to the network resource indicators in step S2; S6. Based on the model convergence state and network conditions, implement a flexible communication optimization strategy and dynamically select a transmission path through the hierarchical synchronization mechanism of step S4; S7. Build a distributed checkpoint system to implement incremental model recovery of failed nodes. The checkpoint generation frequency is associated with the task scheduling result of step S3.

2. A distributed machine learning model training optimization method for big data according to claim 1, characterized in that: The dynamic data sharding strategy in step S1 specifically includes: first, determining the data sharding dimension based on feature importance analysis, and screening the key dimension by calculating the mutual information entropy between the feature and the target variable; then clustering similar data using the local sensitive hashing algorithm to ensure that the similarity of data features in the same shard exceeds a preset threshold; establishing a dynamic matching model between shard capacity and node computing power, and adjusting the shard size according to the node CPU / GPU utilization collected in real time in step S2; setting a shard overlap rate threshold to maintain data correlation, and using the overlapping area data for gradient correction during cross-node training.

3. The distributed machine learning model training optimization method for big data according to claim 1, characterized in that: The resource evaluation indicators in step S2 include: Computing resources, based on floating-point computing capabilities collected by hardware performance counters, combined with measured memory bandwidth values ​​and GPU memory utilization curve modeling; Storage resources, monitor local storage throughput through the file system interface and calculate cache hit rate using the LRU cache strategy; Network resources, periodically send probe packets to measure the transmission delay between nodes, and calculate the bandwidth fluctuation coefficient based on the sliding window; The indicators are stored in the form of time series, and the resource change trend is predicted by exponential smoothing method.

4. The distributed machine learning model training optimization method for big data according to claim 1, characterized in that: The adaptive task scheduling algorithm of step S3 includes: a task execution time prediction model based on an LSTM network, with the resource evaluation indicators and historical task execution logs of step S2 as input; establishing a correlation matrix between task priority and model contribution, and quantifying the contribution by calculating the Shapley value of the data shard corresponding to the task; using an improved Hungarian algorithm to achieve optimal task-node matching, and introducing a resource balancing factor to prevent node overload; setting a dynamic fallback mechanism to handle abnormal tasks, and triggering task reallocation when the node failure rate exceeds a threshold.

5. The distributed machine learning model training optimization method for big data according to claim 1, characterized in that: The hierarchical gradient synchronization mechanism of step S4 includes: the parameter server cluster is constructed into a tree-like hierarchy according to the physical topology, with leaf nodes responsible for collecting computing node gradients and root nodes performing global aggregation; a time window-based gradient accumulation strategy is designed, and synchronization is delayed when the network bandwidth is lower than a threshold to accumulate multiple batches of gradients; pipeline processing is implemented for cross-layer gradient fusion, so that the lower layers continue to receive new gradients while the upper-layer parameter servers are converging; differential coding technology is used to reduce the amount of transmitted data, and only the difference matrix with the previous aggregation result is transmitted.

6. A distributed machine learning model training optimization method for big data according to claim 1, characterized in that: The sparsification compression algorithm of step S5 is specifically implemented as follows: first, a dynamic threshold selection algorithm based on gradient amplitude is used to adjust the threshold according to the model convergence speed fed back in step S6; a residual accumulation compensation mechanism is used to maintain model accuracy, and the gradient residuals discarded in the current round are added to the next round of calculation; a feedback control loop is established between the compression rate and the model convergence speed, and the compression rate is reduced when the convergence speed drops by more than 10%; Supports seamless switching of multiple compression modes, including Top-K selection, quantization, and hash coding modes.

7. The distributed machine learning model training optimization method for big data according to claim 1, characterized in that: The elastic communication optimization strategy of step S6 includes: a bandwidth allocation algorithm based on the model convergence state, allocating more bandwidth to data segment transmission during the period when the loss function is decreasing steadily; implementing a TCP / UDP hybrid transmission protocol, using TCP for critical gradient data and UDP for non-critical parameters; constructing a Markov model for network status prediction, combining the historical network indicators of step S2 to predict bandwidth changes in the next 5 seconds; and designing a priority message queue management mechanism to set the data packets of high-priority tasks in step S3 to the highest transmission level.

8. The distributed machine learning model training optimization method for big data according to claim 1, characterized in that: The distributed checkpoint system of step S7 includes: an incremental storage strategy based on model partitioning, storing only parameter partitions that have changed compared to the previous checkpoint; implementing erasure coding of checkpoint data, partitioning model parameters and adding redundant shards to store them on different nodes; establishing a version-compatible model snapshot management system, retaining the most recent N checkpoint versions and supporting rollback of any version; and adopting a parallel recovery mechanism to accelerate fault handling, while pulling checkpoint shards from multiple surviving nodes for reorganization.

9. The distributed machine learning model training optimization method for big data according to claim 1, characterized in that: Also includes: In step S8, a multi-dimensional monitoring system is established to collect and visualize the following data in real time: Cluster resource utilization heat map, which overlays and displays the computing, storage, and network resource utilization rates evaluated in step S2; A 3D graph of the model convergence trajectory, comprehensively presenting the spatiotemporal changes in the loss function value, gradient amplitude, and parameter update amount; The dynamic evolution diagram of the communication topology reflects the real-time data transmission path in the hierarchical gradient synchronization mechanism in step S4; The task scheduling Gantt chart shows the execution timeline and load balancing status of the tasks assigned in step S3 on each node.

10. A distributed machine learning model training optimization method for big data according to claim 1, characterized in that: Also includes: In step S9, when deploying the model inference service, the service deployment plan is optimized based on the resource evaluation data in the training phase. Specifically, this includes: building a matching matrix between service requirements and node capabilities, and prioritizing high-concurrency inference requests to nodes with low network latency as evaluated in step S2; Implement affinity scheduling between model shards and computing resources, deploying compute-intensive model shards to nodes with low GPU utilization; establish a dynamic scaling mechanism for service instances, and automatically adjust the task scheduling strategy defined in step S3 based on the real-time request volume; when new inference nodes are added, reuse the data sharding strategy in step S1 to achieve rapid distribution of model parameters.

Images