An asynchronous batch data parallel training method for a heterogeneous environment

Abstract

This invention relates to the field of asynchronous communication technology, and in particular provides an asynchronous batch data parallel training method for heterogeneous environments. The method includes: fine-grained load balancing: allowing worker nodes to perform multiple local iterations, first adjusting the number of iterations to balance the load of worker nodes, then adjusting the batch size, aligning the timing of worker nodes sending local model gradient information back to the server node by balancing the workload of each worker node, and balancing the gradient contribution distribution of each node; asynchronous batch aggregation based on weight pools: introducing a weight pool on the server side to collect model differences from different worker nodes, and updating the global model through smooth aggregation based on the weight pool; and layer-based model freeze training: freezing the non-convolutional layers of the global model during training, allowing them to participate in computation but restricting their parameter updates. This method improves the efficiency of model training in heterogeneous environments, accelerating model convergence while ensuring model generalization ability.

Description

Technical Field

[0001] This invention relates to the field of asynchronous communication technology, and in particular to an asynchronous batch data parallel training method for heterogeneous environments. Background Technology

[0002] As the scale of deep neural networks continues to expand, distributed training has become the mainstream paradigm for overcoming the limitations of single-machine computing power and memory. However, most existing distributed methods are designed based on homogeneous environments, resulting in high hardware costs and limited deployment, posing significant challenges in real-world heterogeneous environments. Therefore, distributed training systems must be able to effectively tolerate and adapt to cluster heterogeneity; otherwise, it will be difficult to stably, economically, and efficiently support model training in complex and diverse training environments.

[0003] The two main communication paradigms for data parallelism are batch synchronous parallelism and asynchronous parallelism. Batch synchronous parallelism requires waiting for all nodes to advance synchronously at each step. However, in heterogeneous environments, the computational and communication performance of worker nodes differs. Faster worker nodes incur significant waiting time while waiting for lagging nodes to arrive and advance synchronously, which significantly lengthens the overall training time and reduces hardware utilization. Asynchronous communication allows nodes to advance independently by eliminating global synchronization barriers, avoiding waiting time. However, heterogeneous clusters amplify the gradient staleness problem inherent in asynchronous communication, slowing down training time. Even in homogeneous environments, system factors such as resource co-location, resource preemption, network congestion, and link queuing are still prevalent, especially in cloud server scenarios. These factors collectively induce dynamic heterogeneity in the training environment. As a result, waiting time in batch synchronous training often cannot be completely eliminated. However, in real-world scenarios, heterogeneous clusters are more common, with nodes typically differing in computational power, memory capacity, and communication bandwidth. For such heterogeneous environments, existing data parallel training methods are still mainly designed based on the homogeneous cluster assumption, thus presenting the following problems in practical applications:

[0004] (1) Data allocation and training load scheduling lack the ability to adapt to heterogeneous resources.

[0005] Existing data parallelism methods typically employ uniform data partitioning, standardized local batch sizes, and fixed training paces, assuming that each worker node has approximately consistent computational performance and memory capacity. However, in heterogeneous clusters, different nodes exhibit significant differences in computational speed, memory limits, and communication capabilities. If a uniform data allocation method is still used, low-performance nodes are prone to becoming system bottlenecks due to longer computation times or memory limitations in local batches, causing high-performance nodes to experience significant idle time during synchronization or update coordination, thereby reducing overall hardware utilization. On the other hand, increasing the data load or expanding the local batch size only for high-performance nodes may lead to inconsistent training step sizes between nodes, increased communication pressure, or excessive memory usage, making it difficult to achieve a coordinated balance of training load across computation, memory, and communication dimensions.

[0006] (2) Asynchronous update mechanisms are prone to more serious gradient staleness problems in heterogeneous environments.

[0007] To reduce synchronization waiting overhead, some existing technologies employ asynchronous data parallelism, allowing each worker node to independently complete forward computation, backward computation, and gradient uploading without waiting for other nodes. However, in heterogeneous environments, due to differences in training speed among nodes, the gradients calculated by slower nodes often correspond to older versions of model parameters when uploaded to the parameter server or used for parameter updates, resulting in a significant gradient staleness phenomenon. This phenomenon causes the model update direction to deviate from the current optimal descent direction, leading to increased parameter oscillations during training, decreased convergence speed, and even affecting the model's final accuracy and generalization performance. Especially in dynamic operating environments with resource contention, network congestion, or task co-location, the differences in training speed between nodes can change over time, further exacerbating the instability of gradient staleness, making it difficult for existing asynchronous data parallel methods to balance hardware utilization and statistical efficiency.

[0008] (3) The communication and parameter aggregation strategy lacks a collaborative optimization mechanism for heterogeneous links and node states.

[0009] In data-parallel training, gradient transmission and parameter aggregation constitute one of the main system overheads. Existing technologies typically employ a uniform gradient synchronization method, a fixed update frequency, or a single communication optimization strategy, lacking the ability to adapt to heterogeneous communication links and node operating states. For example, in a hybrid interconnected environment containing both high-speed and low-speed links, if a uniform full-volume, high-frequency gradient transmission method is used, low-bandwidth links can easily become communication bottlenecks, slowing down the overall training progress. If gradient compression and delayed updates are used uniformly, although communication overhead can be reduced on low-speed links, additional compression, decompression, and error compensation overhead may be introduced in high-speed link scenarios, resulting in unnecessary consumption of computational resources. Furthermore, in parameter server architectures, existing technologies typically fail to jointly schedule the parameter uploading, aggregation, and distribution processes based on the heterogeneous state and link conditions of each node, easily leading to load concentration or hotspot congestion on the parameter server side, thereby offsetting the performance gains brought by data-parallel training. Summary of the Invention

[0010] In view of this, the present invention provides an asynchronous batch data parallel training method for heterogeneous environments, which improves the efficiency of model training in heterogeneous environments and accelerates model convergence while ensuring the model's generalization ability.

[0011] In a first aspect, the present invention provides an asynchronous batch data parallel training method for heterogeneous environments, the method comprising: Step 1, Fine-grained load balancing: Allow worker nodes to perform multiple local iterations. First, adjust the number of iterations to balance the load of worker nodes, and then adjust the batch size. By balancing the workload of each worker node, align the timing of worker nodes sending local model gradient information back to the server node, and balance the gradient contribution distribution of each node. Step 2: Based on Step 1, asynchronous batch aggregation based on weight pool: introduce a weight pool on the server side to collect model differences from different worker nodes, and update the global model through smooth aggregation based on the weight pool. Step 3: Based on Step 2, freeze the training of the model based on layers: For Convolutional Neural Networks (CNNs), the convolutional layer is the core of CNNs. The freezing criterion adopted is: freeze the non-convolutional layers of the global model during training, allowing them to participate in the calculation but restricting their parameter updates.

[0012] Optionally, step 1 includes: First, a short training period is conducted to collect the computation and communication times of each worker node. The time taken for one round trip is determined by using the slowest worker node. Next, the number of iterations of the working node is increased based on the time taken for one round trip interaction, and the gap time is calculated. The batch size of each iteration of the working node is then re-determined, and different computation time is allocated to the working node. Each worker node runs a different number of iterations locally and uploads the model differences from the local node. Distribute data subsets of different sizes according to the workload of each node. Following the recommendations in Ominilearn, the model differences uploaded by worker nodes are scaled when updating the global model. The scaling factor depends on the size of the data subset held by the worker node. The global model update rule is as follows: ; ; Where i represents worker node i. This represents the global model before the update. This represents the scaling index of worker node i. This represents the model difference for worker node i. This represents the updated global model.

[0013] Optionally, step 2 includes: Based on the model difference of the working nodes Update the current global model The backup model is obtained, added to the weight pool, and aggregated based on the weight pool to obtain the aggregated global model. Use the exponential moving average function during aggregation. : ; ; Where X is a vector composed of x, representing the function input. Indicates the function output. Indicates the smoothing exponent; It has additivity, that is This is equivalent to the model difference of all working nodes during aggregation. Simultaneously, the global model itself is smoothed within a time window; from a time scale perspective, the weight pool aggregates all global models up to deadline t. Difference from all models ,Right now + The size of the weight pool determines its ability to represent the diversity of the cluster. The size of the weight pool is greater than or equal to the number of nodes. In conjunction with the load balancing operation, the time when the model differential returned by the worker nodes arrives at the server is aligned. The weight pool can gather the updated information from all worker nodes at any given time. When the server has sufficient video memory, the weight pool should be placed on the GPU first to obtain computational resources. To achieve the desired acceleration effect, otherwise place the weight pool on the CPU; if the training environment is limited by memory, reduce the size of the weight pool; add semi-asynchronous communication restrictions to prevent dynamic heterogeneity from affecting the weight pool's ability to collect global information and balance the gradient contribution distribution.

[0014] Optionally, step 3 includes: Before training begins, the server sends the global model to each worker node, along with a subset of non-convolutional layers. Keep it local; worker nodes are trained using the local model during training, and only the convolutional layer model differences of the local model are calculated when performing backhaul operations. And then send it back to the server; the server performs a global model based on the weight pool. Difference with convolutional layer model Perform a batch aggregation operation to update the global model, and then combine the aggregated global model's convolutional layer subsets. Send to the worker node; upon receiving it, the worker node will extract a subset of the global model's convolutional layers. With local nonconvolutional layer subset The components are combined into a complete local model, aligned with the current global model, and a new round of training begins. During this process, the worker nodes only send back the convolutional layer model differences. The non-convolutional layers of the global model on the server side remain unchanged; and only a subset of the convolutional layers of the global model needs to be pulled from the server, combined with the subset of non-convolutional layers stored locally, and aligned with the global model through a combination operation; in order to accelerate communication while ensuring model quality, the model difference is symmetrically quantized in the early stages of training before being sent back to the server, thereby reducing communication time through quantization.

[0015] In a second aspect, embodiments of the present invention provide a computer-readable storage medium comprising a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to execute the asynchronous batch data parallel training method for heterogeneous environments in the first aspect or any possible implementation thereof.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the asynchronous batch data parallel training method for heterogeneous environments in the first aspect or any possible implementation of the first aspect.

[0017] The technical solution provided by this invention includes the following methods: fine-grained load balancing: allowing worker nodes to perform multiple local iterations, first adjusting the number of iterations to balance the load of worker nodes, then adjusting the batch size, aligning the timing of worker nodes sending local model gradient information back to the server node by balancing the workload of each worker node, and balancing the gradient contribution distribution of each node; asynchronous batch aggregation based on weight pool: introducing a weight pool on the server side to collect model differences from different worker nodes, and updating the global model through smooth aggregation based on the weight pool; and layer-based model freeze training: for convolutional neural networks (CNNs), where convolutional layers are the core, the freezing criterion is to freeze the non-convolutional layers of the global model during training, allowing them to participate in computation but restricting their parameter updates. This method improves the efficiency of model training in heterogeneous environments and accelerates model convergence while ensuring the model's generalization ability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of an asynchronous batch data parallel training method for heterogeneous environments provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of fine-grained load balancing provided in an embodiment of the present invention; Figure 3 A schematic diagram of asynchronous batch aggregation based on a weight pool provided in an embodiment of the present invention; Figure 4 A schematic diagram illustrating layer-based model freeze training provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0023] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0024] Figure 1 The flowchart of the asynchronous batch data parallel training method for heterogeneous environments provided in the embodiments of the present invention is as follows: Figure 1 As shown, the method includes: As the heterogeneity of the training environment becomes more complex, the gradient contribution distribution of the worker nodes becomes increasingly unbalanced, indicating that the effect of gradient staleness is becoming more severe. As a result, the model converges more slowly. The reason for the lower training efficiency of the ASP method in heterogeneous environments is attributed to the imbalance of the gradient contribution distribution of the worker nodes, that is, the aggravation of gradient staleness.

[0025] Step 1, Fine-grained load balancing: Allow worker nodes to perform multiple local iterations. First, adjust the number of iterations to balance the load of worker nodes, and then adjust the batch size. By balancing the workload of each worker node, align the timing of worker nodes sending local model gradient information back to the server node, and balance the gradient contribution distribution of each node, as shown in Table 1, Algorithm 1.

[0026] In embodiments of the present invention, such as Figure 2 As shown, step 1 includes: Taking worker node i as an example, firstly, a short training period is conducted to collect the computation time and communication time of each worker node. The time taken for one round trip is determined by the slowest worker node. : Next, based on the time taken for one round trip interaction Increase the number of iterations for worker nodes. (Line 7), and calculate the gap time. Redetermine the batch size for each iteration of the worker node, i.e. (Lines 9 to 12), by zooming in To shrink By allocating different computation times to worker nodes, memory overflow can be avoided. The principle followed in this invention is to increase the workload of high-performance nodes without reducing the workload of low-performance nodes, ensuring that the slowest worker node can complete a standard iteration and guaranteeing training efficiency. By allocating different computation times to worker nodes, the time required for all worker nodes to complete one round trip (communication time + computation time) is balanced.

[0027] Each worker node runs a different number of iterations locally and uploads the model differences from the local node. Instead of accumulating gradients, from the server's perspective, each worker node still only performs one iteration update, allocating different sized subsets of data based on the workload of each node. (Line 15) Following the recommendations in Ominilearn, the model differences uploaded by worker nodes are scaled when updating the global model. The scaling factor depends on the size of the data subset held by the worker node (Line 16). The global model update rule is as follows: ; ; Where i represents worker node i. This represents the global model before the update. This represents the scaling index of worker node i. This represents the model difference for worker node i. This represents the updated global model.

[0028] Table 1 Algorithm 1 .

[0029] The algorithm input is the number of worker nodes M and the computation time of each worker node. Communication time The output includes the initial batch size (Batchsize); the output consists of the iteration count (H) for each worker node, the data subset size (D), and the updated batch size. This represents the time taken for one round trip interaction at the initial stage for each worker node, and the time taken for the final round trip interaction is determined by the slowest node. G represents the gap time, and A represents the batch size that the working node can complete within the gap time.

[0030] Step 2: Based on Step 1, perform asynchronous batch aggregation based on the weight pool: introduce a weight pool on the server side to collect model differences from different worker nodes, and update the global model through smooth aggregation based on the weight pool.

[0031] Load balancing aligns the time when worker nodes send back model differentials to the server, balances the gradient contribution distribution of worker nodes, and reduces gradient staleness. However, the independent aggregation characteristic of asynchronous communication itself amplifies the drift of the optimization direction of worker nodes, which can easily lead to oscillations in the convergence curve and unstable model convergence.

[0032] In embodiments of the present invention, such as Figure 3 As shown, step 2 includes: Based on the model difference of the working nodes Update the current global model The backup model is obtained, added to the weight pool, and aggregated based on the weight pool to obtain the aggregated global model. Use the exponential moving average function during aggregation. : ; ; Where X is a vector composed of x, representing the function input. Indicates the function output. Indicates the smoothing exponent; It has additivity, that is This is equivalent to the model difference of all working nodes during aggregation. Simultaneously, the global model itself is smoothed within a time window; from a time scale perspective, the weight pool aggregates all global models up to deadline t. Difference from all models ,Right now + The size of the weight pool determines its ability to represent the diversity of the cluster. The size of the weight pool is greater than or equal to the number of nodes. In conjunction with the load balancing operation, the time when the model differential returned by the worker nodes arrives at the server is aligned. The weight pool can gather the updated information from all worker nodes at any given time. Considering the space overhead of the weight pool itself and the time overhead of aggregation, its size should be adjusted according to the actual situation. When the server-side GPU memory is sufficient, the weight pool should be placed on the GPU first to obtain computational resources. To achieve the desired acceleration effect, the weight pool should be placed on the CPU; otherwise, it should be placed on the CPU. If the training environment is limited by memory, the size of the weight pool should be reduced. It should be noted that the smaller the weight pool size, the weaker its ability to collect global model differences. In extreme cases, when the weight pool size is reduced to 1, it degenerates into original training and no longer achieves the effect of batch aggregation. Due to the dynamic heterogeneity that is difficult to eliminate in the training environment, a semi-asynchronous communication constraint is added to prevent dynamic heterogeneity from affecting the weight pool's ability to collect global information and to balance the gradient contribution distribution.

[0033] Step 3: Based on Step 2, freeze the training of the model based on layers: For Convolutional Neural Networks (CNNs), the convolutional layer is the core of CNNs. The freezing criterion adopted is: freeze the non-convolutional layers of the global model during training, allowing them to participate in the calculation but restricting their parameter updates.

[0034] Distributed training generates significant communication overhead, which becomes a bottleneck in improving the speed of distributed parallel training. Therefore, accelerating communication to improve overall training speed is essential. Freezing the model reduces the total amount of data that needs to be transmitted, thereby lowering communication overhead.

[0035] In embodiments of the present invention, such as Figure 4 As shown, step 3 includes: Figure 4 The left side represents the server node, and the right side represents the worker node. The model shown in the figure is a lightweight deep convolutional neural network model based on residual connections, designed specifically for feature extraction and pattern recognition tasks of image data. In the model, the input data is processed sequentially from left to right along the pipeline: First, the initial low-level features are extracted through the first convolution, followed by the first pooling for spatial downsampling to reduce computation and enhance feature translation invariance; then, the model enters the first residual block, where the features are split into two paths. The main path learns complex nonlinear feature transformations through three consecutive convolutions, while the secondary path completes dimension matching through one independent convolution. The outputs of the two paths are element-wise summed at the residual connection to obtain the output of the residual block; then, the model enters the second residual block with the same structure, repeating the above splitting-convolution-residual summation operation to further extract higher-level abstract features; finally, the model undergoes a second pooling to complete the final spatial downsampling, outputting a high-level feature map that can be used for subsequent tasks such as classification, detection, or segmentation.

[0036] Before training begins, the server node sends the global model (initial model) to each worker node, and the worker nodes then send a subset of non-convolutional layers. Keep it local. During training, worker nodes train and update the local model, such as... Figure 4 The process from step ① to step ② involves differentiating the convolutional layer model. Send to the server node (quantization is required in the early stages of training); the server node receives it. Then, aggregation is performed based on the weight pool to obtain the aggregated global model, such as... Figure 4 In steps ③ to ④, the aggregated subset of convolutional layers of the global model is sent to the worker nodes. The worker node will receive a subset of convolutional layers. Non-convolutional layer subsets preserved locally The model is assembled into a complete model based on the model dictionary, aligned with the global model at the current time step, and a new round of training begins. During this process, the worker nodes only return the model differences from the convolutional layers. The non-convolutional layers of the global model on the server side remain unchanged; and only a subset of the convolutional layers of the global model needs to be pulled from the server, combined with the subset of non-convolutional layers stored locally, and aligned with the global model through a combination operation, which effectively reduces the total amount of data that needs to be transmitted; in order to accelerate communication while ensuring model quality, the model difference is symmetrically quantized in the early stages of training before being sent back to the server, thereby reducing communication time through quantization.

[0037] In this embodiment of the invention, the overall workflow is as follows: As shown in Table 2, Algorithm 2 represents the overall training process, with subscripts indicating the overall training flow. Representing the There are 10 worker nodes. After determining the load of each worker node using Algorithm 1, the worker nodes initially perform initialization operations to obtain the initial model. Storage model And initialize the local computation Local iteration count and global loss The initialization operation is completed by obtaining the required parameters (lines 11-13). Representing the global loss, used to determine the quantization timing. `L` stores the model before local computation on the worker node, used to calculate model differences and send model update information back to the server. The worker node trains the local model (lines 14-27) and, based on... When interacting with the server, When the value is greater than a given value, worker node i will calculate the local model difference. Perform quantization operations (lines 20-22). The server maintains a global model, when... When the received model difference exceeds a given value, inverse quantization is required (lines 3-5). The standby model obtained by independently updating the global model is added to the weight pool (line 7). The aggregated global model is obtained by aggregating the weight pool (line 8). The Gloss is sent to the worker node (line 9). After receiving it, the worker node will... With local storage Combine the data to obtain the complete model (line 25), and begin a new round of training.

[0038] Table 2 Algorithm 2 .

[0039] ParameterSerer represents the server node, where This represents the model difference of the convolutional layer at worker node i. This represents the global model loss value, which requires dequantization (DQ) within a threshold range; W represents the current global model. This represents the scaling index of worker node i. This represents the aggregated global model. This represents a subset of convolutional layers in the global model.

[0040] Workers represent worker nodes, where Let L represent the local model of worker node i. A subset of convolutional layers is used to assist in calculating the differences of the convolutional layer model; These are the data subset size of worker node i, the number of iterations, the batch size, and the global model loss value, respectively. Indicates the learning rate. Represents the model gradient; This represents a subset of the convolutional layers of worker node i; represents the model difference of the convolutional layer at worker node i, and Q represents the quantization operation; This represents the aggregated subset of convolutional layers sent by the server node and the non-convolutional layers stored locally by worker node i.

[0041] Based on a parameter server framework and asynchronous communication, this invention proposes an asynchronous batch parallel distributed deep learning framework suitable for heterogeneous environments. It improves the statistical efficiency of asynchronous communication while ensuring high hardware utilization during training, and significantly accelerates the overall training time while maintaining good generalization ability.

[0042] The technical solution provided by this invention includes the following methods: fine-grained load balancing: allowing worker nodes to perform multiple local iterations, first adjusting the number of iterations to balance the load of worker nodes, then adjusting the batch size, aligning the timing of worker nodes sending local model gradient information back to the server node by balancing the workload of each worker node, and balancing the gradient contribution distribution of each node; asynchronous batch aggregation based on weight pool: introducing a weight pool on the server side to collect model differences from different worker nodes, and obtaining the aggregated global model through smooth aggregation based on the weight pool; and layer-based model freeze training: for convolutional neural networks (CNNs), where convolutional layers are the core, the freezing criterion is to freeze the non-convolutional layers of the global model during training, allowing them to participate in computation but restricting their parameter updates. This method improves the efficiency of model training in heterogeneous environments and accelerates model convergence while ensuring the model's generalization ability.

[0043] The various steps in the embodiments of the present invention can be performed by an electronic device. This electronic device includes, but is not limited to, tablet computers, portable PCs, and desktop computers.

[0044] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is running, it controls the electronic device containing the computer-readable storage medium to execute the above-described embodiment of the asynchronous batch data parallel training method for heterogeneous environments.

[0045] Figure 5 A schematic diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 5 As shown, the electronic device 21 includes a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, it implements the asynchronous batch data parallel training method for heterogeneous environments in the embodiment. To avoid repetition, it will not be described in detail here.

[0046] Electronic device 21 includes, but is not limited to, processor 211 and memory 212. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 21 and does not constitute a limitation on electronic device 21. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0047] The processor 211 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0048] The memory 212 can be an internal storage unit of the electronic device 21, such as a hard disk or RAM of the electronic device 21. The memory 212 can also be an external storage device of the electronic device 21, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard equipped on the electronic device 21. Furthermore, the memory 212 can include both internal and external storage units of the electronic device 21. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 can also be used to temporarily store data that has been output or will be output.

[0049] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for asynchronous batch data parallel training for heterogeneous environment, characterized in that, The method includes: Step 1, Fine-grained load balancing: Allow worker nodes to perform multiple local iterations. First, adjust the number of iterations to balance the load of worker nodes, and then adjust the batch size. By balancing the workload of each worker node, align the timing of worker nodes sending local model gradient information back to the server node, and balance the gradient contribution distribution of each node. Step 2: Based on Step 1, asynchronous batch aggregation based on weight pool: introduce a weight pool on the server side to collect model differences from different worker nodes, and update the global model through smooth aggregation based on the weight pool. Step 3: Based on Step 2, freeze the training of the model based on layers: For Convolutional Neural Networks (CNNs), the convolutional layer is the core of CNNs. The freezing criterion adopted is: freeze the non-convolutional layers of the global model during training, allowing them to participate in the calculation but restricting their parameter updates. Step 2 includes: Differential according to the model of the worker node Updating the current global model , obtaining a backup model, adding the backup model to a weight pool, and performing aggregation based on the weight pool to obtain an aggregated global model Using an exponential moving average function when aggregating : ； ； where X is a vector consisting of x, representing the function input, represents the function output, represents a smoothing index; It has additivity, that is This is equivalent to the model difference of all working nodes during aggregation. Simultaneously, the global model itself is smoothed within a time window; from a time scale perspective, the weight pool aggregates all global models up to deadline t. Difference from all models ,Right now + The size of the weight pool determines its ability to represent the diversity of the cluster. The size of the weight pool is greater than or equal to the number of nodes. In conjunction with the load balancing operation, the time when the model differential returned by the worker nodes arrives at the server is aligned. The weight pool can gather the updated information from all worker nodes at any given time. When the server has sufficient video memory, the weight pool should be placed on the GPU first to obtain computational resources. To achieve the desired acceleration effect, otherwise place the weight pool on the CPU; if the training environment is limited by memory, reduce the size of the weight pool; add semi-asynchronous communication restrictions to prevent dynamic heterogeneity from affecting the weight pool's ability to collect global information and balance the gradient contribution distribution.

2. The method according to claim 1, characterized in that, Step 1 includes: First, a short training period is conducted to collect the computation and communication times of each worker node. The time taken for one round trip is determined by using the slowest worker node. Next, the number of iterations of the working node is increased based on the time taken for one round trip interaction, and the gap time is calculated. The batch size of each iteration of the working node is then re-determined, and different computation time is allocated to the working node. Each worker node runs a different number of iterations locally and uploads the model differences from the local node. Distribute data subsets of different sizes according to the workload of each node. Following the recommendations in Ominilearn, the model differences uploaded by worker nodes are scaled when updating the global model. The scaling factor depends on the size of the data subset held by the worker node. The global model update rule is as follows: ； ； Where i represents worker node i. This represents the global model before the update. This represents the scaling index of worker node i. This represents the model difference for worker node i. This represents the updated global model.

3. The method according to claim 1, characterized in that, Step 3 includes: Before training begins, the server sends the global model to each worker node, along with a subset of non-convolutional layers. Keep it local; worker nodes are trained using the local model during training, and only the convolutional layer model differences of the local model are calculated when performing a backhaul operation. And then send it back to the server; the server performs a global model based on the weight pool. Difference with convolutional layer model Perform a batch aggregation operation to update the global model, and then combine the aggregated global model's convolutional layer subsets. Send to the worker node; upon receiving it, the worker node will convert a subset of the global model's convolutional layers. With local nonconvolutional layer subsets The components are combined into a complete local model, aligned with the current global model, and a new round of training begins. During this process, the worker nodes only return the convolutional layer model differences. The non-convolutional layers of the global model on the server side remain unchanged; and only a subset of the convolutional layers of the global model needs to be pulled from the server, combined with the subset of non-convolutional layers stored locally, and aligned with the global model through a combination operation; in order to accelerate communication while ensuring model quality, the model difference is symmetrically quantized in the early stages of training before being sent back to the server, thereby reducing communication time through quantization.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the asynchronous batch data parallel training method for heterogeneous environments as described in any one of claims 1 to 3.

5. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the device, cause the device to perform the asynchronous batch data parallel training method for heterogeneous environments as described in any one of claims 1 to 3.

Images