A classification model training method based on big data distributed computing

Abstract

The present invention discloses a classification model training method based on big data distributed computing, which relates to the field of big data processing and machine learning technology. The method includes S1, obtaining communication delay data between distributed computing nodes, and analyzing delay characteristics to generate control parameters, S2, dynamically determining the batch size of each training batch based on the control parameters to ensure optimal synchronization efficiency, S3, distributing the adjusted batch size to each node, executing the model forward and backward propagation calculation process, S4, collecting the calculation results of each node, performing parameter aggregation and update, and completing the iterative training of the classification model; the classification model training method based on big data distributed computing controls the batch size according to the communication delay between nodes to solve the problem of low distributed synchronization efficiency.

Description

Technical Field

[0001] The present invention relates to the field of big data processing and machine learning technology, and in particular to a classification model training method based on big data distributed computing. Background Art

[0002] In today's big data landscape, machine learning model training typically relies on distributed computing architectures to improve training speed and the ability to process extremely large datasets. Distributed computing systems distribute data and computational tasks across multiple compute nodes and coordinate parameter synchronization through inter-node communication to achieve joint model training. Distributed synchronization mechanisms, such as synchronous SGD, are widely used during training to ensure consistency of model parameters across nodes. However, due to uneven network communication latency between nodes, especially when there are a large number of nodes or they are widely distributed, synchronization efficiency decreases significantly. The overall training process is susceptible to slow node effects, resulting in extended model training time and reduced resource utilization. To address the synchronization issues caused by communication latency, various optimization strategies have been proposed, such as increasing the batch size to reduce synchronization frequency or employing asynchronous update mechanisms, such as asynchronous SGD, to alleviate synchronization pressure. However, these approaches still have limitations: batch size adjustments are typically static and difficult to dynamically adapt to real-time communication conditions between nodes, resulting in waiting bottlenecks in high-latency scenarios. While asynchronous updates can increase training parallelism, they can also introduce parameter consistency issues, impacting the convergence speed and accuracy of the final model. Summary of the Invention

[0003] The purpose of the present invention is to provide a classification model training method based on big data distributed computing, which regulates the batch size according to the communication delay between nodes to solve the problem of low distributed synchronization efficiency.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a classification model training method based on big data distributed computing, the method comprising:

[0005] S1. Obtain communication delay data between distributed computing nodes and analyze delay characteristics to generate control parameters. This includes collecting network delay data from each node at fixed intervals, setting a delay threshold, accumulating sample periods where the delay is less than the threshold as effective communication time, and calculating the communication efficiency of each node.

[0006] S2. Dynamically determine the batch size of each training batch based on control parameters to ensure optimal synchronization efficiency. This includes calculating the actual training batch size for each node based on the communication efficiency of each node, discarding or padding non-integer parts, and writing the adjusted results to the distributed training management scheduler for task distribution.

[0007] S3: Distribute the adjusted batch size to each node and execute the model forward and backward propagation calculation process, including collecting the average response time of the previous training round for each node, calculating the computational pressure corresponding to the current batch task, and allocating the number of concurrent threads and the length of the I / O waiting queue based on the computational pressure corresponding to the current batch task;

[0008] S4. Collect the calculation results of each node and perform parameter aggregation and update to complete the iterative training of the classification model. This includes uploading a copy of the local model parameters to each node, calculating the node gradient volatility, taking its inverse as the weight, aggregating the final global parameters, and synchronizing the aggregated model update parameters to all nodes to start the next round of iteration.

[0009] Preferably, the calculation of the communication efficiency of each node in S1 includes collecting network communication delay data from each distributed computing node in turn at fixed time intervals, judging all collected delay samples in each time interval, setting a delay threshold, filtering out samples that are less than the set threshold, and counting the total observation time of the current time window, that is, the total time period from the start of collection to the end, dividing the effective communication time by the total observation time to obtain the communication efficiency value of the node, and using the communication efficiency value as the communication control parameter of the node.

[0010] Preferably, the calculation of the actual training batch size of each node in S2 includes receiving the calculated communication efficiency value of each node, setting a standard batch size as a reference benchmark for the initial training task amount, using the communication efficiency value of each node as a weight, dividing it by the sum of the communication efficiency value of each node as the weight and one, and then multiplying it by the standard batch size to obtain the initial training batch size of the node.

[0011] Preferably, the calculation of the computing pressure corresponding to the current batch task in S3 specifically includes collecting the average response time experienced by each distributed computing node in the previous round of training, combining the size of the training batch task currently assigned to the node, and multiplying it by the average response time to obtain the overall computing pressure of the node in this round of tasks.

[0012] Preferably, the S3 also includes automatically allocating corresponding hardware execution resources to each node according to the calculated pressure value, including the number of threads that can be used in parallel, the length of the input and output buffer queues, and task prefetching parameters.

[0013] Preferably, the S3 also includes starting the training calculation process immediately after the node completes resource configuration, including starting the training calculation process immediately after the node completes resource configuration, including executing forward propagation for predictive calculation of model output, executing error calculation for measuring the difference between the output and the actual target, executing back propagation for updating the internal parameters of the model, and outputting corresponding gradient information or model update results.

[0014] Preferably, the aggregation of the final global parameters in S4 specifically includes receiving a copy of the model parameters uploaded from each distributed computing node, counting the fluctuation degree of the parameter gradient, using the fluctuation degree as a reference basis for the weight, and adjusting it by taking its inverse, and integrating the model parameters of all nodes in a weighted fusion manner to calculate a final global parameter representing the collaborative training result of all nodes, and synchronously distributing the integrated global model parameters to all nodes for use as the initial parameters for the next round of training.

[0015] Preferably, the fluctuation degree of the statistical parameter gradient in S4 includes calculating the standard deviation of each parameter item in the model parameters uploaded by each node in the current training round, and using the standard deviation as a measure of the stability of the node model update.

[0016] Preferably, the weighted fusion method in S4 is to sum the products of the weights of each node and the model parameter values, and then divide the sum by the sum of all node weights to obtain the final global model parameters.

[0017] Preferably, the synchronous distribution of the global model parameters after integration in S4 adopts a broadcast mechanism, and uniformly sends the final aggregation result to all distributed computing nodes with the same timestamp to ensure the consistency and synchronization of parameter updates.

[0018] It can be seen from the above technical solution that the present invention has the following beneficial effects:

[0019] This classification model training method based on big data distributed computing obtains communication delay data between distributed computing nodes and analyzes delay characteristics to generate control parameters. Based on the control parameters, the batch size of each training batch is dynamically determined to ensure optimal synchronization efficiency. The adjusted batch size is distributed to each node, the model forward and backward propagation calculation processes are executed, and the calculation results of each node are collected for parameter aggregation and update to complete the iterative training of the classification model. This method achieves quantitative modeling of node network status, improves the accuracy and real-time adaptability of training scheduling, avoids resource waste and training blockage caused by traditional static batch strategies in high-latency or heterogeneous computing environments, improves parallel efficiency and task matching during training, enhances the flexibility of node-level training scheduling and the overall resource utilization efficiency of the system, assigns greater weight to nodes with high training stability during parameter fusion, and achieves enhanced robustness and convergence quality of model parameter updates. It realizes adaptive parallel training control in heterogeneous network environments, effectively alleviates the impact of slow node effects on the training cycle, and improves model training speed and global coordination capabilities. It can significantly improve model training efficiency, resource utilization, and parameter aggregation quality in complex and dynamically changing distributed environments, and has good engineering practical value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, the present invention provides a technical solution: a classification model training method based on big data distributed computing, the method comprising:

[0023] S1. Obtain communication delay data between distributed computing nodes and analyze delay characteristics to generate control parameters. This includes collecting network delay data from each node at fixed intervals, setting a delay threshold, accumulating sample periods where the delay is less than the threshold as effective communication time, and calculating the communication efficiency of each node.

[0024] S2. Dynamically determine the batch size of each training batch based on control parameters to ensure optimal synchronization efficiency. This includes calculating the actual training batch size for each node based on the communication efficiency of each node, discarding or padding non-integer parts, and writing the adjusted results to the distributed training management scheduler for task distribution.

[0025] S3: Distribute the adjusted batch size to each node and execute the model forward and backward propagation calculation process, including collecting the average response time of the previous training round for each node, calculating the computational pressure corresponding to the current batch task, and allocating the number of concurrent threads and the length of the I / O waiting queue based on the computational pressure corresponding to the current batch task;

[0026] S4. Collect the calculation results of each node and perform parameter aggregation and update to complete the iterative training of the classification model. This includes uploading a copy of the local model parameters to each node, calculating the node gradient volatility, taking its inverse as the weight, aggregating the final global parameters, and synchronizing the aggregated model update parameters to all nodes to start the next round of iteration.

[0027] The classification model training method based on big data distributed computing proposed in this paper employs an adaptive optimization mechanism tailored to dynamic network environments and heterogeneous computing resources, encompassing key aspects such as communication delay assessment, batch size control, computational load management, and parameter aggregation. Its core concept is to dynamically perceive changes in communication and computational performance between nodes and adjust training resource allocation and collaboration strategies in real time to achieve high efficiency and convergence stability during distributed classification model training. In step S1, the system samples network delays from all participating nodes at set intervals to obtain real-time delay information for inter-node communication links. After thresholding and comparing this delay data, only low-latency, effective communication periods are retained for statistical analysis. The effective ratio of each node's available time for synchronous communication is then calculated and defined as the communication efficiency parameter. This parameter not only reflects the network status but also indirectly characterizes the node's ability to participate in global synchronization, serving as an important input for subsequent control strategies. In step S2, the system leverages this communication efficiency to dynamically allocate the optimal training batch size for each node. By leveraging the positive correlation between batch size and communication efficiency, the system ensures that nodes with stronger communication performance take on a larger training workload, while nodes with weaker communication capabilities are burdened less, thereby shortening overall synchronization latency. To maintain computational consistency and data integrity, the non-integer portion of the batch size is discretized using a discarding or minimum padding algorithm. Simultaneously, instructions are uniformly issued by the scheduling module to achieve distributed scheduling of training tasks. In step S3, each node performs forward and backward propagation based on the assigned training task. To further balance computational pressure across nodes, the system monitors each node's average response time from the previous round and calculates a predicted computational pressure based on the current workload. Based on this pressure, the number of concurrent threads and I / O queue lengths are dynamically adjusted to align node processing capacity with the task load, improving local execution efficiency and avoiding resource congestion. In step S4, after all nodes complete computation, they upload their local copies of the model parameters. The master node or a designated aggregation node is responsible for collecting all parameters and calculating the gradient volatility based on the gradient fluctuations of each node. The inverse of this gradient volatility is then defined as the parameter weight. This strategy automatically enhances the contribution of nodes with minimal differences in training stability and suppresses the influence of nodes with significant fluctuations, effectively preventing training instability caused by gradient oscillation. Finally, parameter aggregation is achieved through a weighted summation approach, and new global model parameters are constructed. This iterative update is completed and synchronized to all compute nodes, preparing for the next round of training. Overall, this method opens up the feedback loop between the three levels of data communication, resource scheduling and model synchronization, realizes the collaborative optimization of the training process in a distributed environment, and ensures the high speed, robustness and adaptability of model training.

[0028] This method dynamically adapts to differences in network communication and computing resources in a distributed training environment, optimizing model training efficiency and stability. Specifically, it leverages communication efficiency to guide batch size allocation, effectively reducing inter-node latency and improving synchronization efficiency. A resource scheduling mechanism driven by computational pressure ensures load balancing and system responsiveness. Furthermore, an aggregation mechanism based on the inverse weight of gradient volatility improves model convergence stability and accuracy. Furthermore, this method exhibits excellent scalability and environmental adaptability, making it suitable for a variety of large-scale machine learning scenarios.

[0029] Calculating the communication efficiency of each node in S1 involves collecting network communication delay data from each distributed computing node in turn at fixed time intervals. Within each time interval, all collected delay samples are judged, a delay threshold is set, and samples less than the set threshold are screened out. The total observation time of the current time window, that is, the total time period from the start of collection to the end, is counted. The effective communication time is divided by the total observation time to obtain the communication efficiency value of the node, and the communication efficiency value is used as the communication control parameter of the node.

[0030] The communication efficiency evaluation mechanism proposed in this embodiment quantitatively analyzes the network status of distributed nodes using time windows as units. At fixed intervals, the system sequentially polls all computing nodes, collecting network delay data from each node. This data can be measured using the response time between the node and the central scheduler. During each collection cycle, the system centrally processes the collected delay samples, first setting a predetermined upper delay limit as a baseline for determining communication quality. The collected delay samples are then evaluated individually, filtering out those with delays below the upper limit. The duration of these samples within the current time window is accumulated. This accumulated time is the effective communication time for the node within the current cycle, representing the total time it can maintain good communication conditions. Furthermore, the total time elapsed from the start to the end of the collection cycle is defined as the total observation time. The system then compares the effective communication time with the total observation time to determine the node's communication efficiency within the current time window. Specifically, if a node experiences a total of ten minutes in the time window, and six of those minutes are in good communication, the node's communication efficiency can be inferred as the ratio of six minutes to ten minutes. This ratio is the communication efficiency value, which indicates the node's ability to maintain high-quality communication during the observed period. The resulting communication efficiency value is stored as a node control parameter and used to guide strategies such as batch task allocation, computing resource scheduling, and synchronization frequency tuning during subsequent training. This dynamically matches node resources with network status, ensuring synchronization efficiency and global stability during model training.

[0031] By refining the communication efficiency calculation process, this implementation can more accurately distinguish differences in network communication capabilities between different nodes, making distributed training task scheduling more scientific and targeted. Specific advantages include: first, avoiding the reduction in overall communication performance evaluation accuracy due to extremely delayed samples; second, adopting a sliding time window mechanism to enhance real-time and adaptability; and third, enabling automatic dynamic updates of node control parameters, improving communication efficiency and resource utilization during the model training phase. Compared with coarse-grained network detection methods, this approach is more suitable for stable training needs in complex network environments.

[0032] The actual training batch size of each node is calculated in S2, including receiving the calculated communication efficiency value of each node, setting a standard batch size as a reference benchmark for the initial training task volume, using the communication efficiency value of each node as a weight, dividing it by the sum of the communication efficiency value of each node as the weight and one, and then multiplying it by the standard batch size to obtain the initial training batch size of the node.

[0033] This implementation addresses the impact of differences in node network capabilities on synchronous training efficiency in distributed computing and proposes a batch size determination method based on communication efficiency-weighted distribution of training tasks. The core concept is to dynamically adjust task loads based on node communication efficiency values, ensuring that each node receives a workload commensurate with its communication capabilities during training, thereby avoiding training delays caused by communication bottlenecks. In specific operation, the system first receives communication efficiency parameters calculated from each distributed node in the previous phase. The system then sets a standard batch size, which serves as a benchmark for the total number of training tasks in this round. Based on this, the communication efficiency of each node is used as a weighting factor, and the communication efficiency values of all nodes are summed to obtain a total communication capacity reference value. The system then uses each node's communication efficiency value as its contribution, which is then combined with the overall weight ratio of other nodes. By allocating the standard batch size based on the relative contribution of node communication efficiency, the initial training batch size for each node is determined. This batch size is proportional to the node's network communication capability: nodes with better communication receive more training tasks, while nodes with poorer communication receive fewer tasks. Because communication efficiency values can be decimal or non-integer, the initially calculated batch size may also contain decimals. To maintain the integrity of the training data structure and system consistency, the non-integer portion can be appropriately rounded or interpolated. The adjusted value is ultimately used as the training data task for the node's current iteration and is input into the scheduler for distribution to each node for execution.

[0034] By using this method of weighted batch size calculation based on communication efficiency, the system can more precisely match training load with node capabilities, significantly improving synchronization efficiency and data throughput throughout the distributed training process. This effectively reduces the impact of slow communication nodes on overall training progress and minimizes the waste of computing resources caused by waiting. Furthermore, this mechanism is highly scalable and versatile, allowing the standard batch reference value and weight composition to be adjusted based on different training objectives and network conditions, enhancing the system's adaptive optimization capabilities.

[0035] Calculating the computational pressure corresponding to the current batch task in S3 specifically involves collecting the average response time experienced by each distributed computing node in the previous round of training, combining it with the size of the training batch task currently assigned to the node, and multiplying it by the average response time to obtain the overall computational pressure of the node in this round of tasks.

[0036] This embodiment provides a node computing pressure assessment method based on a combination of historical response characteristics and current task size, aiming to achieve dynamic awareness and load regulation of computing resource usage. During each round of model training, the system records the average response time required for each distributed computing node to complete its assigned training task. This response time is calculated by dividing the total time from the start of the training task to the node submitting the results by the number of batches, reflecting the node's execution efficiency and operational status. When the next round of training task allocation begins, the system reads the training batch size assigned to the current node and performs quantitative analysis based on the node's average response time in the previous round. By multiplying the current training batch size by the average response time, a comprehensive indicator is obtained, representing the level of computing pressure the node will bear in the current training round. This computing pressure value reflects both the number of tasks and the rate at which the node processes tasks. It is a key parameter for determining whether a node is at risk of overload and whether resource compensation or scheduling adjustments are required. Based on this pressure value, the system can adjust resource parameters such as thread concurrency configuration and I / O queue length to optimize node processing paths, balance internal resource distribution within the system, and dynamically reduce or reallocate tasks when necessary.

[0037] By integrating historical response times with the current task scale to form a computational pressure assessment mechanism, we can achieve quantitative modeling of the computational load of each node, making resource scheduling more accurate and dynamic. This approach avoids the allocation errors caused by relying solely on static hardware indicators, reflects operational status changes in real time, and effectively prevents computational overload or idle resources. This further improves the stability, execution efficiency, and fault tolerance of the training system, and enhances the reliability and flexibility of model training in large-scale distributed environments.

[0038] S3 also includes automatically allocating corresponding hardware execution resources to each node based on the calculated pressure value, including the number of threads that can be used in parallel, the length of the input and output buffer queues, and task prefetch parameters.

[0039] In this embodiment, to further improve node execution efficiency and system resource utilization during distributed model training, the system introduces an automated hardware resource allocation mechanism based on the calculated current computational load of each node. This mechanism uses computational load as input and dynamically adjusts the node's hardware operating parameters based on its magnitude, precisely matching resource allocation to the task load. After the system calculates the computational load of a training batch for a node, it uses this load value to find the optimal resource allocation strategy within pre-set resource scheduling mapping rules. For example, for nodes with higher workloads, the system allocates more parallel computing threads to accelerate processing; increases the length of their input and output buffers to reduce I / O bottlenecks; and increases the task prefetch depth, allowing tasks to be pre-loaded into the execution queue before execution, thereby reducing scheduling delays. Conversely, for nodes with lower workloads, the system tightens resource allocation to avoid resource waste and shifts excess resources to higher-load nodes. This allocation process is performed automatically by the distributed training scheduler, requiring no human intervention and achieving millisecond-level response adjustments to address sudden imbalances caused by changes in node load. Once resource allocation is complete, all adjusted parameters take effect immediately and adapt to the next round of model training. During the next training round, the system reassesses the computational pressure and updates the resource strategy, forming a closed-loop optimization control mechanism.

[0040] This implementation achieves a dynamic match between node computing power and task load by introducing a computationally intensive automatic hardware resource configuration mechanism, effectively improving the system's overall computing efficiency and parallel performance. This avoids the inefficiencies of static resource configuration, where node resources are overloaded or idle, and significantly improves the responsiveness and stability of the training process. Furthermore, this method requires no human intervention and is highly adaptive, making it particularly suitable for large-scale distributed environments with numerous nodes and fluctuating computational loads.

[0041] S3 also includes starting the training calculation process immediately after the node completes resource configuration, including executing forward propagation for predictive calculation of model output, executing error calculation for measuring the difference between output and actual target, executing back propagation for updating internal parameters of the model, and outputting corresponding gradient information or model update results.

[0042] This implementation further expands the specific training process after computational pressure assessment and resource allocation, proposing a design pattern in which training task execution should be tightly coupled with the resource allocation process. After completing node resource scheduling and hardware parameter adjustment in the aforementioned steps, each compute node immediately initiates the core computation phase of a new round of model training, ensuring continuous resource utilization and immediate scheduling response. The training process consists of three key steps. First, the node initiates the forward propagation process based on the current batch data. During this process, the model sequentially executes neural computations at each layer, mapping the input feature data into output predictions. This output represents the model's predicted behavior in the current state and is used for comparison with the true label. The system then enters the error calculation phase, comparing the predictions generated by the forward propagation with the actual target label to quantify the prediction deviation. This error value is used to evaluate the current model training performance. The error calculation method can be selected based on the model type, such as cross entropy for classification models and mean squared error for regression models. This is followed by the backpropagation process, which uses the error as input, propagates back layer by layer, and calculates gradient values, gradually updating the model parameters to optimize the model weights. During this process, each layer adjusts its weights based on the previous round of gradient information and the learning rate strategy, and outputs the corresponding gradient information or intermediate model update results for subsequent aggregation or local caching. The entire computation process is completed locally on the node, and after completion, the relevant results are transmitted back to the scheduling center or participate in the parameter aggregation process to implement the next stage of the training system.

[0043] By immediately initiating the training process after resource allocation, this implementation reduces resource idle time, improving the responsiveness and compactness of training tasks. Integrated management of forward propagation, error calculation, and backpropagation ensures a continuous and smooth model training process, optimizing computing resource utilization and data flow structure, further reducing system latency and enhancing overall system throughput. Training results are directly output as aggregatable gradients or model update information, facilitating subsequent unified processing.

[0044] Aggregating the final global parameters in S4 specifically includes receiving a copy of the model parameters uploaded from each distributed computing node, counting the fluctuation degree of the parameter gradient, using the fluctuation degree as a reference for the weight, and adjusting it by taking its inverse. Using a weighted fusion method, the model parameters of all nodes are integrated to calculate a final global parameter representing the collaborative training result of all nodes. The integrated global model parameters are synchronously distributed to all nodes and used as the initial parameters for the next round of training.

[0045] This embodiment proposes a model parameter aggregation strategy based on adaptive weighting of gradient fluctuations. The goal is to overcome aggregation instability caused by node performance differences or data heterogeneity in multi-node distributed training. After completing a training round, the system sequentially receives local model parameter replicas uploaded by all distributed nodes. Each parameter replica represents the weighted result obtained through independent training on the current node. To ensure the stability and representativeness of the aggregation results, the system first evaluates the gradient fluctuation of each parameter replica. Specifically, the system measures the magnitude of change in each weight in the model parameters across nodes over multiple training rounds to measure the stability of the node parameter updates. If a node's model parameters fluctuate dramatically, its training process may be subject to noise, data bias, or an abnormal learning rate. Nodes with more moderate fluctuations may be trained on a more stable dataset and thus have greater convergence stability. The system uses gradient fluctuation as a reference metric for assessing reliability. Next, the system constructs its aggregation weight by taking the inverse of each node's fluctuation, ensuring that nodes with less fluctuation have a larger share in the global parameter fusion, and vice versa. The system then fuses all parameter replicas in a weighted manner to generate a new global model parameter. This fusion process balances training effectiveness and node reliability, effectively preventing individual abnormal nodes from negatively impacting model convergence. Finally, the system synchronously pushes the fused global parameters to all compute nodes, serving as the starting model for the next round of training. This ensures that all nodes continue training with the same initial state, maintaining model consistency and collaborative optimization.

[0046] A global parameter aggregation method based on weighting by the inverse of gradient volatility significantly improves the robustness and convergence quality of distributed training. By minimizing the impact of highly volatile nodes and effectively suppressing disturbances caused by abnormal updates, the stability and accuracy of the aggregated model are improved. Furthermore, this approach adaptively reflects node health and training stability, effectively enhancing the robustness and fault tolerance of the training system. It is particularly suitable for complex distributed environments with unbalanced data, heterogeneous nodes, or high training instability.

[0047] The fluctuation degree of the statistical parameter gradient in S4 includes calculating the standard deviation of each parameter item in the model parameters uploaded by each node in the current training round, and using the standard deviation as the basis for measuring the stability of the node model update.

[0048] This implementation further quantifies the "gradient volatility" in global parameter aggregation, using standard deviation as a metric to construct a more operational model update stability assessment system. After each round of training, each distributed node uploads a copy of its local model parameters obtained from the current training to the aggregation node. This parameter copy is typically a set of weight values corresponding to each neuron connection or computational unit in the model structure. After receiving the model parameter copies of all nodes, the aggregation node counts the parameter values of all nodes at each model parameter item (i.e., each specific weight value position) at that position. The system treats this set of values for the same parameter item from different nodes as a distribution and calculates its standard deviation. This standard deviation reflects the degree of deviation in the training results of each node for that parameter, and thus reflects the degree of consistency of the model along this parameter dimension. For each node, the system summarizes or weighted averages the standard deviation values corresponding to all its parameter items to form an overall standard deviation metric, which serves as a quantitative basis for the stability of the node's model update in this round. A small standard deviation indicates a relatively stable node training process, with the update results tending towards the global average. A large standard deviation may indicate that the node is affected by factors such as data perturbations, learning rate fluctuations, or model non-convergence. This standard deviation metric is ultimately used to construct weights in weighted aggregation. By taking the inverse of the standard deviation or constructing a scaling factor, the system assigns higher weights to nodes with smaller fluctuations, thereby highlighting their dominant role in global parameter integration and suppressing the contributions of nodes that may introduce instability.

[0049] By introducing standard deviation as a metric for model parameter fluctuation, this implementation can more accurately and objectively measure the consistency and reliability of node training results. This method is easy to implement, computationally inexpensive, and scalable to large parameter spaces, demonstrating its applicability. Its weighting strategy improves the robustness of the aggregation process and model convergence accuracy, reducing the risk of error amplification caused by individual anomalous nodes and contributing to higher-quality global model training.

[0050] The weighted fusion method in S4 is to sum the products of each node weight and the model parameter value, and then divide it by the sum of all node weights to obtain the final global model parameters.

[0051] This embodiment specifically defines the weighted fusion strategy of global model parameters, adopting a weighted average mechanism based on the sum of the product of node weight and corresponding parameter value and then normalization. This mechanism aims to achieve precise control in the parameter aggregation process, so that the aggregation results can more fully reflect the differences in training stability and contribution of each node model. After each round of model training, all nodes in the distributed system upload their locally updated copies of model parameters to the central aggregation module. The model parameters of each node include multiple scalar values, each scalar corresponds to a weight item in the model structure. At the same time, the system has obtained the weight coefficient corresponding to each node based on the aforementioned standard deviation calculation method or other stability assessment mechanism to indicate its influence in the global aggregation. In each model parameter item dimension, the system multiplies the parameter value of all nodes with their corresponding node weight, and then adds all the product results to form the weighted cumulative value of the parameter item. Subsequently, the cumulative value is divided by the sum of all node weights to obtain the fusion value of the current parameter item in the global model. This operation is repeated for each model parameter, ultimately generating a global model parameter set encompassing all parameters. This unified model represents the training results for the entire system after the current training round. This fusion mechanism ensures that nodes with low volatility and stable contributions have a higher proportion in the final model, while effectively mitigating the risk of bias introduced by abnormal nodes.

[0052] By employing an aggregation strategy that takes a weighted average of the product of node weights and parameter values, this implementation achieves a logical unification between training stability and aggregation weights while maintaining the mathematical rigor of parameter fusion. This method, with its clear structure and simple computation, is suitable for parallel computation in large parameter spaces and is particularly well-suited for distributed training frameworks for deep learning models. The resulting results exhibit excellent numerical stability and global representation capabilities, effectively improving model convergence quality and overall performance.

[0053] The integrated global model parameters in S4 are synchronously distributed using a broadcast mechanism, which uniformly sends the final aggregation results to all distributed computing nodes with the same timestamp to ensure the consistency and synchronization of parameter updates.

[0054] This implementation focuses on the distribution mechanism for global parameter updates, the final step in model training. It proposes using broadcasting to uniformly control distribution at a unified time, ensuring that all distributed nodes receive the latest model parameters synchronously at the same logical point in time. The core goal of this design is to prevent inconsistent model states due to asynchronous parameter reception by nodes, which in turn affects training stability and convergence efficiency. After completing the weighted fusion of global model parameters, the system generates a new global parameter set. This set represents a unified expression of the training results of each node in the current training round, and it must be ensured that all subsequent training operations are based on this result. Therefore, the system uses a broadcast mechanism to distribute this parameter set to all nodes. The broadcast mechanism is initiated by the scheduling center, and all receiving nodes are in a listening state. Once a broadcast signal is received, the local model state can be immediately updated. To ensure logical synchronization, the system includes a unified timestamp in the broadcast message, which marks the training number or time series index of the current round. After receiving the broadcast parameters, all nodes must verify the consistency of the timestamp and load the parameters into the local model only if the timestamp matches the current round identifier. If there are delays or out-of-sync nodes, the system can set up a retry mechanism or reissue strategy to ensure that all nodes eventually successfully synchronize the parameters of that version. This synchronization process is triggered immediately after parameter aggregation is completed, and the next round of training begins after all nodes confirm receipt. This ensures that all nodes are always in the same training state during the model iteration process, avoiding accuracy drift or training errors caused by inconsistent model versions.

[0055] By employing a timestamped broadcast mechanism for parameter synchronization, this implementation significantly improves consistency control within distributed model training systems. This mechanism ensures that all nodes maintain consistency during global state updates, preventing model divergence caused by asynchronous updates or network latency. It also enhances the overall synchronization and convergence stability of the training system, making it particularly suitable for tasks requiring high consistency, such as federated learning and multi-task concurrent training.

[0056] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A classification model training method based on big data distributed computing, characterized in that: The method comprises: S1. Obtain communication delay data between distributed computing nodes and analyze delay characteristics to generate control parameters. This includes collecting network delay data from each node at fixed intervals, setting a delay threshold, accumulating sample periods where the delay is less than the threshold as effective communication time, and calculating the communication efficiency of each node. S2. Dynamically determine the batch size of each training batch based on the control parameters, including calculating the actual training batch size of each node based on the communication efficiency of each node, discarding or padding the non-integer part, and writing the adjustment result to the distributed training management scheduler for task distribution; S3: Distribute the adjusted batch size to each node and execute the model forward and backward propagation calculation process, including collecting the average response time of the previous training round for each node, calculating the computational pressure corresponding to the current batch task, and allocating the number of concurrent threads and the length of the I / O waiting queue based on the computational pressure corresponding to the current batch task; S4. Collect the calculation results of each node and perform parameter aggregation and update to complete the iterative training of the classification model. This includes uploading a copy of the local model parameters to each node, calculating the node gradient volatility, taking its inverse as the weight, aggregating the final global parameters, and synchronizing the aggregated model update parameters to all nodes to start the next round of iteration.

2. The classification model training method based on big data distributed computing according to claim 1, characterized in that: The calculation of the communication efficiency of each node in S1 includes collecting network communication delay data from each distributed computing node in turn at fixed time intervals. Within each time interval, all collected delay samples are judged, a delay threshold is set, and samples less than the set threshold are screened out. The total observation time of the current time window, that is, the total time period from the start of collection to the end, is counted. The effective communication time is divided by the total observation time to obtain the communication efficiency value of the node, and the communication efficiency value is used as the communication control parameter of the node.

3. The classification model training method based on big data distributed computing according to claim 1, characterized in that: The calculation of the actual training batch size of each node in S2 includes receiving the calculated communication efficiency value of each node, setting a standard batch size as a reference benchmark for the initial training task amount, using the communication efficiency value of each node as a weight, dividing it by the sum of the communication efficiency value of each node as the weight and one, and then multiplying it by the standard batch size to obtain the initial training batch size of the node.

4. The classification model training method based on big data distributed computing according to claim 1, characterized in that: The calculation of the computational pressure corresponding to the current batch task in S3 specifically includes collecting the average response time experienced by each distributed computing node in the previous round of training, combining the size of the training batch task currently assigned to the node with the average response time, and performing a product operation to obtain the overall computational pressure of the node in this round of tasks.

5. The classification model training method based on big data distributed computing according to claim 4 is characterized in that: The S3 also includes automatically allocating corresponding hardware execution resources to each node according to the calculated pressure value, including the number of threads that can be used in parallel, the length of the input and output buffer queues, and task prefetching parameters.

6. The classification model training method based on big data distributed computing according to claim 5, characterized in that: The S3 also includes starting the training calculation process immediately after the node completes resource configuration, including executing forward propagation for predicting the model output, executing error calculation for measuring the difference between the output and the actual target, executing back propagation for updating the internal parameters of the model, and outputting the corresponding gradient information or model update results.

7. The classification model training method based on big data distributed computing according to claim 1, characterized in that: The aggregation of the final global parameters in S4 specifically includes receiving a copy of the model parameters uploaded from each distributed computing node, counting the fluctuation degree of the parameter gradient, using the fluctuation degree as a reference for the weight, and adjusting it by taking its inverse. The model parameters of all nodes are integrated in a weighted fusion manner to calculate a final global parameter representing the collaborative training result of all nodes. The integrated global model parameters are synchronously distributed to all nodes and used as the initial parameters for the next round of training.

8. The classification model training method based on big data distributed computing according to claim 7, characterized in that: The fluctuation degree of the statistical parameter gradient in S4 includes calculating the standard deviation of each parameter item in the model parameters uploaded by each node in the current training round, and using the standard deviation as a measure of the stability of the node model update.

9. The classification model training method based on big data distributed computing according to claim 7, characterized in that: The weighted fusion method in S4 is to sum the products of the weights of each node and the model parameter values, and then divide the sum by the sum of all node weights to obtain the final global model parameters.

10. The classification model training method based on big data distributed computing according to claim 7, characterized in that: The synchronous distribution of the global model parameters after integration in S4 adopts a broadcast mechanism, and uniformly sends the final aggregation results to all distributed computing nodes with the same timestamp to ensure the consistency and synchronization of parameter updates.

Images