Gradient communication control method, apparatus, electronic device, storage medium, and program

Abstract

This disclosure provides gradient communication control methods, apparatus, electronic devices, storage media, and programs, relating to the computer technology field, particularly to the fields of artificial intelligence, large-scale models, deep learning, and distributed training. [Solution] A specific solution involves identifying gradient changes in a local node's target parameter group, wherein the local node is any node in a plurality of computing nodes performing distributed training on a target model, the local node stores a plurality of parameter groups of the target model, the target parameter group is any parameter group in the plurality of parameter groups, and, if the gradient change of the target parameter group satisfies communication conditions, the gradient of the target parameter group is synchronized with the target node in the plurality of computing nodes.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and particularly to the technical fields such as artificial intelligence, large-scale models, deep learning, and distributed training.

Background Art

[0002] As the scale and complexity of deep learning models increase, a large amount of computing resources and memory consumption are required to train neural network models.

[0003] Since the computing resources and memory of stand-alone devices are difficult to meet the training requirements of large-scale models, it is necessary to complete the training of the model by a distributed training method. Here, in the distributed training method, it is necessary to synchronize gradients between different computing nodes in order to complete the update of model parameters.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides a gradient communication control method, apparatus, electronic device, storage medium, and program.

Means for Solving the Problems

[0005] In a first aspect of the present disclosure, a gradient communication control method is provided, and the method includes: identifying a gradient change of a target parameter group of a local node, where the local node is any node among a plurality of computing nodes performing distributed training on a target model, the local node stores a plurality of parameter groups of the target model, and the target parameter group is any parameter group among the plurality of parameter groups; The method includes synchronizing the gradient of the target parameter group with the target node in the plurality of computing nodes if the gradient change of the target parameter group satisfies the communication conditions.

[0006] Another aspect of this disclosure provides a gradient communication control device, the device, An identification module for identifying gradient changes of a target parameter group of a local node, wherein the local node is any node in a plurality of computing nodes performing distributed training on a target model, the local node stores a plurality of parameter groups of the target model, and the target parameter group is any parameter group in the plurality of parameter groups; The system includes a synchronization module for synchronizing the gradient of the target parameter group with the target node in the plurality of computing nodes when the gradient change of the target parameter group satisfies the communication conditions.

[0007] In another aspect of this disclosure, an electronic device is provided, which is At least one processor, The system comprises at least one processor and memory that is communicated with, The memory stores instructions that are executable by the at least one processor, and when executed by the at least one processor, the instructions cause one of the methods in the embodiments of the present disclosure to be performed.

[0008] Another aspect of the present disclosure provides a non-temporary computer-readable storage medium that stores computer instructions for causing a computer to perform any one of the methods of the embodiments of the present disclosure.

[0009] In another aspect of the present disclosure, a program is provided, when executed by a processor, for performing any of the methods in the embodiments of the present disclosure.

[0010] It should be understood that the information contained herein is not intended to describe any key points or important features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Further details of other features of this disclosure will be provided in the specification below. [Brief explanation of the drawing]

[0011] The attached drawings are for the purpose of better understanding this solution and do not constitute a limitation of this disclosure.

[0012] [Figure 1] This is a schematic diagram of a scenario for a gradient communication control method according to one embodiment of the present disclosure. [Figure 2] This is a flowchart of a gradient communication control method according to one embodiment of the present disclosure. [Figure 3] This is a flowchart of a gradient communication control method according to one embodiment of the present disclosure. [Figure 4] This is a schematic diagram of a gradient communication control method according to another embodiment of the present disclosure. [Figure 5] This is a schematic diagram of the structure of a gradient communication control device according to one embodiment of the present disclosure. [Figure 6] This is a schematic diagram of the structure of a gradient communication control device according to another embodiment of the present disclosure. [Figure 7] This is a block diagram of an electronic device for a gradient communication control method to realize an embodiment of the present disclosure. [Modes for carrying out the invention]

[0013] Illustrative embodiments of the present disclosure will be described below with reference to the accompanying drawings. These drawings include various details of the embodiments of the present disclosure to aid understanding, and should be considered as illustrative only. Accordingly, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, well-known features and structural descriptions are omitted in the following description for clarity and brevity.

[0014] In this specification, the terms “first” and “second” indicate and distinguish multiple similar technical terms, and are not intended to limit their order or to limit them to only two. Furthermore, the terms “includes” and “have” and their variations are not intended to exclude others, for example, a series of steps or units. A method, system, product, or apparatus is not limited to the steps or units explicitly listed and may include other steps or units not explicitly listed or specific to these processes, methods, products, or apparatus.

[0015] In addition, unless explicitly stated otherwise, the operations shown in the flowcharts of the embodiments of this disclosure may be executed simultaneously regardless of their order of execution, and if there is an execution priority among different operations during technical implementation, the execution order among multiple operations may be unrestricted.

[0016] Distributed training is a method of training a model in parallel across multiple computing nodes (e.g., CPUs (Central Processing Units), GPUs (Graphics Processing Units), or servers). Its aim is to accelerate the training process, increase training efficiency, and enable the processing of large datasets and complex models.

[0017] The gradient is the derivative (first-order partial derivative) of the loss function with respect to the model parameters, and it represents the rate and direction of change in the loss function when a small change occurs in the model parameters. The gradient is an important basis for parameter tuning in the model optimization process, and optimization algorithms (e.g., gradient descent) determine the direction and amplitude of parameter updates through gradient information, and step by step find the model parameters that minimize the loss function.

[0018] In the process of distributed training, in order to complete the update of model parameters, it is necessary to synchronize gradients among different computing nodes. Here, gradient synchronization refers to the process of synchronizing gradients among multiple computing nodes. Through gradient synchronization, the gradients of each computing device are aggregated, and usually, a global gradient for updating model parameters is obtained by using an allreduce operation such as summing or averaging.

[0019] In related technologies, in the process of gradient synchronization, the gradients of all parameters of the model need to participate in communication. As the scale of model parameters increases, the data volume continues to increase each time different computing nodes communicate, which significantly increases the communication load. Also, in the training cycle of the entire model, gradient synchronization is not only executed once, but as the number of gradient synchronization increases, the number of communications increases, and the communication load becomes even heavier.

[0020] Therefore, embodiments of the present disclosure provide a gradient communication control method. The method can reduce the communication load by grouping the parameters of the model to obtain a plurality of parameter groups and performing gradient synchronization at appropriate times for different parameter groups.

[0021] As shown in FIG. 1, it is a schematic diagram of a scenario of the gradient communication control method according to an embodiment of the present disclosure. Here, it includes a plurality of computing nodes 110. Users of the plurality of computing nodes 110 perform distributed training on a target model. The plurality of computing nodes 110 are connected via a wireless or wired network.

[0022] Distributed training is a primary method for solving large-scale model training, and specifically there are two methods: data parallelism and model parallelism. In data parallelism, the model is divided into multiple model blocks, and each model block is placed on a corresponding computing node and trained, as shown in Figure 1. It is also possible to have a complete copy of the model on each computing node. Distributed training essentially involves distributing the model parameters across multiple computing nodes to reduce the memory pressure on a single node.

[0023] The gradient communication control method provided in the embodiments of this disclosure is applicable to model parallelism and also to model parallel + data parallel training methods. The method provided in the embodiments of this disclosure can be executed by each computing node in Figure 1. Any of these computing nodes may be referred to as a local node when executing the method.

[0024] As shown in Figure 2, a flowchart of the gradient communication control method provided in the embodiments of this disclosure is proposed, which includes the following:

[0025] In S201, the gradient change of the target parameter group of the local node is identified. The local node is any node in a group of computing nodes that perform distributed training on the target model. The local node stores multiple parameter groups of the target model, and the target parameter group is any parameter group in a group of parameter groups.

[0026] When implementing this, the target model may be a large-scale model.

[0027] Here, "large-scale model" may refer to a Large Language Model (LLM). A Large Language Model is a specific type of large-scale model specialized for processing text data. Such models are natural language processing models based on neural networks and can be used for generating, understanding, and processing text data. Large Language Models can have tens of billions of parameters, can generate high-quality text, and can be used for various natural language processing tasks, such as question and answer, text generation, and dialogue systems.

[0028] Large-scale language models possess superior inference capabilities and the ability to learn from small amounts of data. Within this context, large-scale language models are understood to have a relatively large model size or to be trained on a large number of samples. Based on these large-scale models, accurate semantic understanding can be achieved.

[0029] Furthermore, large-scale models may also be multimodal large language models (MLLMs).

[0030] During implementation, a model-parallel approach can be used to divide the target model into multiple model blocks, and these model blocks can be distributed across multiple computing nodes for training. Alternatively, a parameter-parallel approach can be used to divide the parameters of the entire model into multiple model parameters, and training can be performed on the corresponding model parameters using different nodes. Furthermore, the entire target model can be copied into multiple model copies and trained on multiple computing nodes.

[0031] During implementation, the target model performs forward propagation on the training samples and obtains output results. Furthermore, based on the output results, a loss value is calculated, and based on this loss value, the gradient of the target parameter group is obtained. Then, based on this gradient, backpropagation is performed to update the model parameters in the target parameter group.

[0032] It can be understood that the gradient communication control method provided in the embodiments of this disclosure can be implemented by treating multiple parameter groups of the target model as target parameter groups. Each compute node can store all or some of these parameter groups, which can be determined at implementation based on the model portion being executed by the compute node.

[0033] In S202, if the gradient change of the target parameter group satisfies the communication conditions, the gradient of the target parameter group is synchronized to the target nodes in multiple computing nodes.

[0034] Here, the communication condition is used to indicate whether or not there is an update to the gradient of the target parameter group, primarily indicating whether the gradient change supports the consumption of the corresponding communication overhead, and updating the model parameters through gradient synchronization. For example, if there is no change in the gradient, the model parameters performing backpropagation have hardly changed, and there is generally no need for information synchronization for such gradients. If the gradient does not change, synchronizing the gradient will cause communication overhead by not matching the actual demand. Also, when dealing with large datasets or complex models, it may not be possible to calculate the gradient of all data at once due to limitations in computational resources or memory. In this case, gradient accumulation techniques can be used, i.e., gradients are calculated for multiple mini-batch data, and these gradients are accumulated, and after accumulating a certain amount, a parameter update is performed once. This reduces the computational and memory burden while simulating the effect of gradient descent on the entire batch to some extent. Even if gradient accumulation is performed, if the accumulated gradient does not change, there is no need to perform communication, and communication overhead can be reduced while guaranteeing a normal training model.

[0035] In embodiments of this disclosure, gradient changes are determined by dynamically monitoring the statistical features of the gradient of a parameter group, and communication is triggered when it is detected that the gradient change meets specific conditions, thereby reducing unnecessary communication overhead.

[0036] Here, the target node is a compute node that needs to synchronize the gradients of a parameter group with other compute nodes besides the local node. For example, multiple compute nodes with the same target parameter group need to synchronize the gradients for that target parameter group. In model-parallel distributed training, taking model block 1 as an example, if we assume that the target parameter group includes the parameters of model block 1, then gradient synchronization for the target parameter group is required among the multiple compute nodes where model block 1 is deployed.

[0037] In the embodiments of this disclosure, in the distributed training process of a target model, each computing node stores multiple parameter groups of the target model. Any parameter group among them is designated as a target parameter group. By identifying gradient changes in the target parameter group, important gradient dynamic information in the model training process can be accurately grasped, and the communication timing of the gradients of each parameter group can be easily controlled. This reduces the blind full gradient communication using a single policy, and by communicating gradients according to parameter groups, unnecessary communication overhead and resource waste caused by full gradient communication can be reduced. Furthermore, by identifying changes in the parameter gradient of the target parameter group, the gradient can be synchronized to the target node on other computing nodes only when the gradient change of the target parameter group satisfies communication conditions. Such a trigger mechanism based on communication conditions enables flexible adjustment of communication policies and can effectively reduce the communication load consumed by invalid communication (e.g., communication even when the gradient does not change) in related technologies.

[0038] In some embodiments, the target parameter group includes multiple parameter tensors.

[0039] Here, a parameter tensor refers to a tensor used to represent model parameters. This includes, for example, weight tensors and bias tensors.

[0040] Parameter tensors are used specifically in the following process:

[0041] Model Representation: Parameter tensors are components of a model, and a single neural network model can be viewed as a single set containing multiple parameter tensors.

[0042] Model training: During the training process, the values ​​of the parameter tensor are updated using an optimization algorithm (e.g., gradient descent) to minimize the model's loss function.

[0043] Model inference: During the inference phase, the input data is transformed through each layer of the model, and these transformations are determined by parameter tensors.

[0044] During implementation, different types of parameter tensors can be stored in different target parameter groups whenever possible and used for subsequent training.

[0045] In the embodiments of this disclosure, a target parameter group can include multiple parameter tensors, thereby enabling communication control over the gradients of the same parameter group, i.e., communication control over the gradients of multiple parameter tensors. Furthermore, when the communication conditions are met, the gradients of multiple parameter tensors can complete gradient synchronization simultaneously. Including multiple parameter tensors in the same group and improving the partitioning quality of the parameter group can further enhance the quality of effective communication and reduce the communication load.

[0046] In some embodiments, the multiple parameter groups can be determined for the target model based on the following methods, including, as shown in Figure 3:

[0047] In S301, the gradient update frequency, parameter type, and parameter tensor shape of multiple model parameters of the target model are obtained.

[0048] Here, gradient update frequency refers to the interval or period during the training process in which model parameters are updated based on gradient information, and is usually measured in terms of the number of training steps or data batches. It determines the frequency of model parameter updates and has a significant impact on the model's convergence speed and training stability.

[0049] Relatively high update frequencies (e.g., single-step updates) typically allow for faster parameter tuning, leading the model to approach the optimal solution more quickly. However, in large-scale distributed training, frequent communication can lead to excessive communication overhead, potentially reducing training efficiency. An appropriate update frequency can balance the model's convergence speed with communication efficiency.

[0050] There are differences in the update frequency of different parameters relative to the target model, and the update frequency can be determined based on information such as the network structure of the target model.

[0051] Regarding parameter types, they can be determined based on the layer type of the target model and the corresponding neural network layer's function of the model parameters. For example, in a neural network, the weight parameter type for a convolutional layer is the convolutional kernel weight, the bias parameter type is the convolutional bias, the weight parameter type for a fully connected layer is the fully connected weight, and the bias parameter type is the fully connected bias, and so on. In practice, parameter types can include weights, biases, BatchNorm (batch normalization) parameters, etc.

[0052] The shape of a parameter tensor refers to the combination of dimensional sizes of this tensor. It is used to determine the memory consumption and data transfer volume of the parameter tensor.

[0053] In S302, multiple model parameters are grouped based on their gradient update frequency, parameter type, and parameter tensor shape to obtain multiple parameter groups.

[0054] In the embodiments of this disclosure, parameters with the same update frequency, parameter type, and parameter tensor shape are assigned to one group whenever possible. Exemplaryly, if the update frequency is the same, different parameter types or different parameter tensor shapes may be assigned to the same group.

[0055] Furthermore, after creating parameter groups, parameters with different attributes are marked into different groups, and a parameter group index table is created to facilitate subsequent management. In the embodiments of this disclosure, a parameter group may include multiple parameter tensors. The index includes a parameter group ID (Identification), and the parameter group includes information such as parameter names and parameter tensors.

[0056] The correspondence between each node and parameter group in the target model allows us to find the corresponding parameter tensor for each node through this correspondence during forward processing, making it easier to perform the forward inference process based on the parameter tensor.

[0057] During implementation, parameter tensors of the same parameter group are stored contiguously in memory, reducing memory fragments, improving the search efficiency of parameter tensors in forward processing, and thus improving the processing efficiency of forward processing.

[0058] In the embodiments of this disclosure, in order to facilitate gradient synchronization, each parameter group has a corresponding gradient cache which is used to store and update the gradient of subsequent parameter groups.

[0059] In the embodiments of this disclosure, model parameters are grouped based on a multidimensional method such as update frequency, parameter type, and parameter tensor shape. This allows parameters with similar features to be assigned to the same group during gradient synchronization, and controls the timing of gradient synchronization for different groupings, thereby reducing the communication load.

[0060] In some embodiments, the grouping rule for grouping the multiple model parameters includes at least one of the following:

[0061] 1) Minimize the differences in update frequency within the same group.

[0062] During implementation, ensure as much as possible that the update frequencies of multiple parameters within the same group are consistent. If this condition cannot be met, minimize the differences in gradient update frequencies within the same group. For example, as shown in Figure 4, if the update frequency of model parameter 1 is a, the update frequency of model parameter 2 is b, the update frequency of model parameter 3 is b, and the update frequency of model parameter 4 is a, then model parameter 1 and model parameter 4 are placed in the same model parameter group, i.e., model parameter group 1 in Figure 3, and model parameter 2 and model parameter 3 are placed in different model parameter groups, i.e., model parameter group 2 in Figure 3.

[0063] In the embodiments of this disclosure, by assigning parameters with similar gradient update frequencies to the same group, the update frequencies of different parameter groups can be differentiated, making it easier to better and independently control the synchronization timing of gradients of different parameter groups and achieving the objective of precisely managing the communication timing of gradients of different parameter groups. This improves communication efficiency, makes the allocation of communication resources more rational, and thereby enables effective use of communication resources and reduces the communication load.

[0064] 2) Minimize the differences in parameter types within the same group.

[0065] During implementation, model parameters of the same parameter type can be assigned to the same group. If the parameter type of model parameter 1 is a bias tensor, the parameter type of model parameter 2 is a bias tensor, the parameter type of model parameter 3 is a weight tensor, and the parameter type of model parameter 4 is a weight tensor, then model parameters 1 and 2 must be placed in the same model parameter group, while model parameters 3 and 4 must be placed in different model parameter groups.

[0066] In the embodiments of this disclosure, different types of parameters may have different update characteristics and importance in model training. By assigning parameters of the same type to the same group, differentiated communication policies and update mechanisms for different types can be facilitated, allowing for better utilization of different parameter groups to complete the updating of different model parameters of the target model, thereby improving the quality of effective communication and reducing the communication load.

[0067] 3) Minimize the differences in the shape of parameter tensors within the same group.

[0068] During implementation, parameters of the same shape can be assigned to the same group. Ideally, large and small parameters should be assigned to different parameter groups.

[0069] Here, by storing tensors of the same shape consecutively within the same parameter group, memory fragmentation can be reduced and memory utilization can be increased. Furthermore, in communication processes, parameters with tensors of the same shape can be packaged and transferred more efficiently, reducing communication overhead. At the same time, data of the same shape can better utilize the parallel computing capabilities of the hardware when performing operations such as matrix calculations, thus contributing to increased computational efficiency.

[0070] In the embodiments of this disclosure, differences in the shape of the parameter tensors affect memory consumption, calculation method, and communication efficiency. Assigning parameter tensors with similar shapes to the same group is advantageous for synchronizing gradients independently for different groupings and can increase the utilization of the communication load.

[0071] 4) The amount of data within the same group is below the first threshold.

[0072] When implemented, the first threshold can be set based on the actual situation, and this disclosure does not limit it. If the exemplary first threshold is 128M, the number of data points in the same group must be 128M or less.

[0073] In embodiments of this disclosure, limiting the amount of data within the same group can reduce the problem of excessive data volume in a single communication, leading to excessive communication delay and resource utilization. Setting the amount of data within the same group also facilitates access in forward processing and makes it easier to maintain different parameter groups.

[0074] In one embodiment, after dividing multiple parameter groups based on the same update frequency, if the remaining parameters cannot satisfy the same update frequency, parameters of different types or with differences in shape can be assigned to the same group at different update frequencies.

[0075] In one embodiment, in order to make better use of the computing resources of each node, this can be done so that the difference in data volume between different groups is less than a second threshold.

[0076] When implementing the test, if the amount of data for parameter group 1 is m and the amount of data for parameter group 2 is n, the difference between m and n must be less than the second threshold.

[0077] In the embodiments of this disclosure, a configuration in which the amount of data between each group is relatively balanced facilitates access to and management of different parameter groups, and further increases the utilization rate of computing resources across the entire distributed system, thereby reducing computing costs.

[0078] In some embodiments, identifying the gradient change of the target parameter group of the local node includes determining the statistical information in the gradient cache corresponding to the target parameter group and obtaining the gradient change of the target parameter group, the statistical information being The L2 norm of the gradient in the aforementioned gradient cache, The number of gradient updates in the aforementioned gradient cache, This includes one of the following: the number of target value elements in the gradient cache, where the target value represents that the corresponding element has a gradient update.

[0079] Here, the L2 norm can intuitively reflect the overall strength or magnitude of the gradient by calculating the square root of the sum of the squares of each element in the gradient vector. If the L2 norm is relatively large, we determine that there is an update in the gradient change of the target parameter group, and if the L2 norm is relatively small, we determine that there is a relatively low probability of an update in the gradient change of the target parameter group.

[0080] The target value elements are compared to the default value elements. Here, the number of default value elements usually refers to the number of zero-value elements in the gradient. In some sparse models or models containing many redundant parameters, the gradient may exhibit sparsity, meaning that the gradient values ​​of most elements may be zero or close to zero. By statistically counting the number of default value elements, the degree of gradient sparsity can be intuitively understood. For example, a high proportion of default value elements in the gradient of a parameter group may mean that the contribution of that parameter group to the model at the current training stage is relatively small, or that the corresponding features are not very important in the current data distribution. Conversely, a parameter group with a relatively large number of non-default value elements (i.e., target value elements) in the gradient may be more important for model training and require more attention. If there are a relatively large number of non-default value elements, we determine that there is an update in the gradient change of the target parameter group; if there are a relatively small number of non-default value elements, we determine that there is a relatively low probability of an update in the gradient change of the target parameter group.

[0081] In the embodiments of this disclosure, when determining the gradient change of the target parameter group, the gradient cache considers factors that affect the communication timing, such as the L2 norm, the number of updates, and the number of target value elements. These statistically targeted information allows for a simple and easy determination of whether the communication conditions are met, enabling gradient synchronization to the target parameter group at a reasonable timing, thereby realizing simple and rational use of communication resources.

[0082] During implementation, each of the aforementioned statistical information items can be treated as an independent element for determining whether or not the communication conditions are met. Each statistical information item has a corresponding identification mechanism for determining whether or not the communication conditions are met, and it can be understood that different statistical information items do not influence each other. On the other hand, appropriate statistical information can be selected for different parameter groups to determine whether or not the communication conditions are met. Different parameter groups of the same target model can have their communication conditions met by selecting different statistical information.

[0083] In some embodiments, determining the statistical information in the gradient cache corresponding to the target parameter group in order to speed up the determination of whether or not the communication conditions are met includes the following:

[0084] In step A1, the gradient cache of the target parameter group is divided into multiple gradient blocks.

[0085] During implementation, the gradient cache of the target parameter group can be evenly divided into multiple gradient blocks, and exemplary the division method can be one of the following:

[0086] Row-based partitioning: If the gradient cache is a two-dimensional tensor (matrix), it can be partitioned into multiple gradient blocks by rows. For example, assuming the shape of the gradient cache is M × N, it can be partitioned into k gradient blocks, each block containing M / k rows, and thus the shape of each gradient block is (M / k) × N. This partitioning method is suitable for processing row-based data, such as the weight gradient of all connected layers.

[0087] Columnar partitioning: Similarly, a 2D tensor can also be partitioned by columns. An M×N gradient cache can be partitioned into gradient blocks, each block containing N / l columns, and the shape of each gradient block is M×(N / l). This partitioning method can be applied to model parameters organized by column features, such as some of the weights in an embedding layer.

[0088] Channel-based partitioning (depth partitioning): A 4-dimensional tensor (for example, the gradient cache of a convolutional layer, which typically has a shape of N×C×H×W, where N is the batch size, C is the number of channels, and H and W are the height and width) can be partitioned by the channel dimension C. The gradient cache is partitioned into p gradient blocks, each block containing C / p channels, and the shape of each gradient block is N×(C / p)×H×W. In a convolutional neural network, different channels may correspond to different feature mappings, and partitioning by channel is advantageous for performing independent gradient analysis and communication for different feature channels.

[0089] Partitioning by height and width: A 4-dimensional tensor can also be partitioned in terms of height H and width W dimensions. For example, a gradient cache can be partitioned into multiple small blocks, each having the same height and width dimensions, for example, H'×W', and thus the shape of each gradient block is N×C×H'×W'. This partitioning method is suitable when processing local features in space, and allows for a more detailed analysis of changes in the spatial dimension of the gradient.

[0090] Equal-size partitioning: The gradient cache is divided into multiple equal-sized gradient blocks based on a predetermined block size. For example, each gradient block is set to have S elements, and the gradient cache is divided into blocks of S elements each, starting from the beginning. In the case of gradient caches with irregular or unevenly distributed shapes, this method ensures that each block has the same number of elements, facilitating unified processing and communication.

[0091] In another embodiment, gradient blocks of different sizes can be partitioned based on the importance and change characteristics of different regions in the gradient cache. For regions with significant or important gradient changes, relatively smaller blocks are partitioned to allow for more detailed analysis and communication. On the other hand, for regions with relatively mild or minor gradient changes, relatively larger blocks are partitioned to reduce the number of blocks and management overhead.

[0092] In another embodiment, partitioning can also be performed based on the degree of sparseness of the gradient cache. By analyzing the distribution of zero values ​​in the gradient cache, for example, zero values ​​in the cache can be divided into one gradient block, and then further divided into multiple gradient blocks based on regions where non-zero values ​​or large gradient values ​​are concentrated.

[0093] Any method that can be used to divide a gradient block can be applied to this disclosure, and this disclosure does not limit it.

[0094] In step A2, the cumulative amount of statistical information for multiple gradient blocks is accumulated in stages to determine whether or not the communication conditions are met.

[0095] During implementation, a single thread can be configured to process multiple gradient blocks sequentially. If the statistical information of the i-th gradient block is combined with the accumulated statistical information of the preceding (i-1) gradient blocks, and a communication condition can be triggered, then the gradient blocks after the i-th gradient block will not be processed.

[0096] To speed up processing, multiple threads can be configured, with each thread processing one gradient block. After obtaining the statistical information for the corresponding gradient block, each thread updates the statistical information for previously obtained gradient blocks and obtains the accumulated amount. With each update, it can be determined whether the updated statistical information (i.e., the accumulated amount) satisfies the communication conditions, and if the communication conditions are met, the detection of unprocessed gradient blocks can be terminated.

[0097] In the embodiments of this disclosure, the gradient is divided into multiple gradient blocks based on the gradient cache. Such a fine-grained analysis method makes it easier to identify whether communication conditions are met, and by progressively accumulating statistical information for the gradient blocks, the trend of gradient changes can be discovered more promptly during the training process. This improves the efficiency of identifying the appropriate communication timing.

[0098] In some embodiments, the aforementioned communication conditions may include the following:

[0099] A) When the gradient change includes the L2 norm, the communication condition is that the L2 norm is greater than the first communication threshold.

[0100] During implementation, the L2 norm of the gradient reflects the overall strength of the gradient. When the L2 norm of the gradient exceeds the first communication threshold, it indicates that the gradient strength of that gradient block is relatively large, the parameters have changed significantly in the current training iteration, and it is necessary to synchronize them with other computing nodes in a timely manner.

[0101] During implementation, if the gradient change includes the L2 norm, the average value of the L2 norms of multiple gradient blocks may be used as the storage amount. For example, if the gradient cache includes gradient block 1, gradient block 2, and gradient block 3, if the average value of the L2 norms of gradient block 1 and gradient block 2 is greater than the first communication threshold, communication is triggered, and the subsequent gradient block 3 does not need to be processed.

[0102] B) When the gradient change includes the number of gradient updates, the communication condition is that the number of gradient updates is greater than the second communication threshold.

[0103] The gradient update count reflects how often the parameters are updated during the training process. When the gradient update count exceeds the second communication threshold, it indicates that the parameters of that gradient block are active during the training process, meaning that frequent updates have a relatively large impact on model training and require timely synchronization.

[0104] When the gradient change includes the number of gradient updates, the sum of the number of gradient updates for the parameter tensors corresponding to multiple gradient blocks may be used as the accumulated amount.

[0105] C) When the gradient change includes the number of target value elements, the communication condition is that the number of target value elements is greater than the third communication threshold.

[0106] Here, the target value is used to indicate that there is a gradient update for the corresponding element.

[0107] Here, the number of elements with default values ​​usually refers to the number of elements in the gradient vector that are zero or close to zero. If the number of target value elements is greater than the third communication threshold, it indicates that there are a relatively large number of non-zero elements in the gradient of that gradient block, and synchronization is required.

[0108] When implementing the system, if the gradient change includes the number of target value elements, it is necessary to gradually accumulate the cumulative value of the number of target value elements of multiple gradient blocks as the aforementioned accumulated amount.

[0109] In the embodiments of this disclosure, corresponding communication conditions are set for different gradient change situations, making the communication trigger mechanism clearer, more specific, and easier to implement, and facilitating the rational use of communication resources.

[0110] In some embodiments, if the gradient update frequencies of multiple parameter tensors in a target parameter group are the same, the first communication threshold and / or the second communication threshold is the gradient update frequency of the target parameter group.

[0111] When implementing the method described above, if the target parameter groups are divided based on the update frequency, the update frequency within the same group should be the same. Therefore, the gradient update frequency can be set to the first communication threshold and / or the second communication threshold.

[0112] Here, if the gradient update frequencies of multiple parameters included in a target parameter group differ, the first and second communication thresholds for the parameter group can be determined based on the update frequencies of these parameters. The specific determination method can be determined based on the network structure of the target model or through experimentation, and the embodiments of this disclosure are not limited thereto.

[0113] In the embodiments of this disclosure, when the gradient update frequencies of multiple parameter tensors in a target parameter group are the same, the first or second communication threshold is set as the gradient update frequency of the target parameter group, allowing the communication conditions to automatically adapt according to the actual update frequency of the parameter group. This increases the flexibility and adaptability of the communication policy and allows for better adaptation to the parameter update rules in the actual training process.

[0114] In some embodiments, if the gradient update frequencies of multiple parameter tensors in the target parameter group are different, the third communication threshold is determined based on the number of parameter tensors in the target parameter group.

[0115] During implementation, if the gradient update frequencies of multiple parameter tensors in a target parameter group differ, it is usually shown that the gradient of the parameter group has a certain sparsity. The third threshold can be set as the number of parameter tensors included in the target parameter group.

[0116] In the embodiments of this disclosure, a third communication threshold can be determined based on the number of parameter tensors, and optimization processing can be performed on the features of the sparse parameter group. By setting a reasonable third communication threshold, effective information in the sparse gradient can be identified and processed more effectively, improving the efficiency of identifying reasonable communication timings, and thereby enabling the rational use of communication resources.

[0117] In some embodiments, if it is determined that the communication conditions are not met, it can be understood that not performing a communication operation on the target parameter group in order to conserve communication resources does not require synchronizing the gradient of that group.

[0118] In some embodiments, if it is determined that the communication conditions are met, a gradient rule communication operation can be performed to synchronize the gradient of the target parameter group with the target node among the plurality of computation nodes.

[0119] Here, by performing a standardized gradient communication operation, gradient synchronization can be completed as follows:

[0120] AllReduce: Achieves data synchronization by summing and averaging the results across multiple computing nodes that meet the communication conditions and have the same model parameters.

[0121] Compressed communication: When the scale of the gradient of a target parameter group is larger than a predetermined scale, i.e., when dealing with large-scale gradient data, gradient compression techniques (e.g., quantization, pruning) can be used to reduce the amount of data transmitted.

[0122] Asynchronous communication: Utilizes parallel processing of communication and computation to further increase efficiency.

[0123] In some embodiments, once the parameter updates for the target parameter group are complete, the gradient of the target parameter group is updated to its default value.

[0124] During implementation, model parameters can be updated based on the synchronized gradient using an optimization algorithm. Examples of optimization algorithms include SGD (Stochastic Gradient Descent) and Adam (Adaptive Moment Estimation). Furthermore, the updated parameters are continuously stored in their corresponding parameter groups.

[0125] During implementation, if the parameters of the target parameter group have finished updating, the gradient of the target parameter group can be updated to its default value, and the gradient of the target parameter group can be set to zero as an example.

[0126] The following techniques can be used to update the gradient of the aforementioned target parameter group to its default value.

[0127] Memory optimization techniques (e.g., the memset function (memory fill function), multithreaded clearing) are used to quickly reset the gradient to its default value. It is also possible to reduce the number of iterations by resetting the gradient to its default value before calculating the gradient in the next iteration.

[0128] In embodiments of the present disclosure, the gradient is updated to a default value after the parameters of the target parameter group are updated, and the gradient changes of the target parameter group are periodically monitored to help the method provided in embodiments of the present disclosure to synchronize the gradient for the target parameter group at a reasonable communication timing, thereby increasing the utilization of communication resources.

[0129] Based on the above, the embodiments of this disclosure provide a dynamic communication optimization policy for gradients, and the method flexibly adjusts the communication policy by detecting changes in parameter gradients in real time.

[0130] Parameter and gradient fusion management: A parameter cache corresponding to a parameter group is used to cache the model parameters in that parameter group, and a gradient cache corresponding to the parameter group stores the gradients corresponding to that parameter group. This enables parameter fusion and gradient fusion, allowing for the continuous storage of parameters and gradients for the same group, improving memory access efficiency and data management convenience.

[0131] Gradient detection optimization: Design efficient gradient detection methods including thresholding and zero-value detection, and further optimize the detection process by introducing sparsity thresholds and block detection policies.

[0132] Communication load sharing: Group management and dynamic communication detection avoid imbalances in communication load and improve the overall efficiency of the distributed training system.

[0133] Applicable to sparse update models: For models with sparse update characteristics (e.g., large-scale embedding layers, sparse neural networks, etc.), this method significantly reduces communication overhead.

[0134] Based on a similar technical concept, an embodiment of the present disclosure also proposes a gradient communication control device 500, as shown in Figure 5, which is:

[0135] An identification module 501 for identifying gradient changes of a target parameter group of a local node, wherein the local node is any node in a plurality of computing nodes performing distributed training on a target model, the local node stores a plurality of parameter groups of the target model, and the target parameter group is any parameter group in the plurality of parameter groups, and the identification module 501

[0136] The system includes a synchronization module 502 for synchronizing the gradient of the target parameter group with the target node in the plurality of computing nodes when the gradient change of the target parameter group satisfies the communication conditions.

[0137] In some embodiments, a gradient communication control device 600 has also been proposed, as shown in Figure 6, which further... The gradient update frequency, parameter type, and parameter tensor shape of multiple model parameters of the target model are obtained. The system includes a decision module 601 used to determine the plurality of parameter groups by grouping the plurality of model parameters based on the gradient update frequency, parameter type, and shape of the parameter tensor of the plurality of model parameters to obtain the plurality of parameter groups.

[0138] In some embodiments, the grouping rule used to group the multiple model parameters is: 1) Minimize the difference in update frequency within the same group. 2) Minimize the differences in parameter types within the same group. 3) Minimizing the differences in the shape of parameter tensors within the same group. 4) The amount of data within the same group is less than or equal to the first threshold, and this includes at least one of these conditions.

[0139] In some embodiments, as shown in Figure 6, the identification module 501 is, Includes a determination unit 602 for determining statistical information in the gradient cache corresponding to the target parameter group and obtaining the gradient change of the target parameter group, The aforementioned statistical information is, The L2 norm of the gradient in the aforementioned gradient cache, The number of gradient updates in the aforementioned gradient cache, The number of target value elements in the gradient cache includes one of the following, where the target value represents that the corresponding element has a gradient update.

[0140] In some embodiments, the decision module 601 is A partitioning subunit 6021 for dividing the gradient cache of the target parameter group into multiple gradient blocks, It includes a storage subunit 6022 for progressively accumulating statistical information for multiple gradient blocks and determining whether or not the communication conditions are met.

[0141] In some embodiments, when the gradient change includes the L2 norm, the communication condition includes that the L2 norm is greater than the first communication threshold. When the gradient change includes the number of gradient updates, the communication condition includes that the number of gradient updates is greater than the second communication threshold. If the gradient change includes the number of target value elements, the communication condition includes the number of target value elements being greater than the third communication threshold.

[0142] In some embodiments, if the gradient update frequencies of the multiple parameter tensors in the target parameter group are the same, the first communication threshold and / or the second communication threshold is the gradient update frequency of the target parameter group.

[0143] In some embodiments, if the gradient update frequencies of multiple parameter tensors in the target parameter group are different, the third communication threshold is determined based on the number of parameter tensors in the target parameter group.

[0144] In some embodiments, the target parameter group includes multiple parameter tensors.

[0145] In some embodiments, as shown in Figure 6, the gradient communication control device 500 further... When the update of the parameters of the target parameter group is complete, an update module 603 is provided for updating the gradient of the target parameter group to its default value.

[0146] Specific functions and illustrative descriptions of each module and submodule of the apparatus according to the embodiments of this disclosure can be found in the relevant descriptions of the corresponding steps in the embodiments of the method described above, and will not be repeated here.

[0147] The technical solutions disclosed herein ensure that the acquisition, storage, and application of users' personal information comply with applicable laws and regulations and do not violate public order or morality.

[0148] According to embodiments of the present disclosure, the present disclosure further provides electronic devices, non-temporary computer-readable storage media, and program products.

[0149] Figure 7 is a schematic block diagram of an electronic device 700 for implementing an embodiment of the present disclosure. The electronic device refers to various types of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other compatible computers. The electronic device further refers to various types of mobile devices, such as personal digital assistants, cellular phones, intelligent phones, wearable devices, and other similar computer devices. The components, their connections, and functions described in this disclosure are illustrative and do not limit the implementation of anything described or specified in this disclosure.

[0150] As shown in Figure 7, device 700 includes a computing unit 701 that can perform various appropriate operations and processes based on computer program instructions stored in read-only memory (ROM) 702 or computer program instructions loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 can further store various programs and data necessary for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are connected to each other via bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0151] Multiple components in device 700 are connected to an I / O interface 705, which includes an input unit 706 such as a keyboard and mouse, an output unit 707 such as various displays and speakers, a storage unit 708 such as a magnetic disk or optical disk, and a communication unit 709 such as a network card, modem, or wireless communication transceiver. The communication unit 709 allows device 700 to exchange information / data with other devices via computer networks such as the Internet and / or various carrier networks.

[0152] The computing unit 701 may be a variety of general-purpose and / or dedicated processing components having processing and computing capabilities. Some exemplary components of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, a computing unit that executes various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs each of the methods and processes described above, for example, the gradient communication control method. For example, in some embodiments, the gradient communication control method can be implemented as a computer software program tangibly contained in a machine-readable medium such as a storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed into the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the gradient communication control method described above can be performed. In addition, in other embodiments, the computing unit 701 may be configured to perform a gradient communication control method by any other suitable method (e.g., firmware).

[0153] Various embodiments of the systems or technologies described in this disclosure can be implemented by digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standards (ASSPs), systems-on-a-chip (SOCs), complex-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. Each of these embodiments may be implemented by one or more computer programs that run and / or interpret on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0154] Program code for performing the methods of this disclosure can be written in any combination of one or more programming languages. These program codes are provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programming data processing device, so that when the program code is executed by the processor or controller, it can perform the functions / operations defined in the flowcharts and / or block diagrams. The program code may run entirely in a mainscan, partially in a mainscan, partially as an independent soft encapsulation and partially in a remote mainscan, or entirely in a remote mainscan or on a server.

[0155] In this disclosure, machine-readable media may be tangible media containing or storing programs used by or in conjunction with instruction execution systems, devices, or equipment. Machine-readable media may be machine-readable signal media or machine-readable storage media. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any suitable combination of the contents described above. Further exemplary examples of machine-readable storage media include electrical connections by one or more wires, portable computer disk cartridges, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any combination of the contents described above.

[0156] To provide user interaction, a computer may implement the systems and technologies described herein, which may include a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor), a keyboard and pointing device for the user to provide input to the computer (e.g., a mouse or trackball). Other types of devices may also be used to provide user interaction; for example, the feedback provided to the user may be any form of sensor feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be accepted in any form (e.g., acoustic input, voice input, haptic input).

[0157] The systems and technologies described herein can be implemented in computing systems that include background components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include front-end components (e.g., user computers having a graphics user interface or network browser, through which users can interact with embodiments of the systems and technologies described herein), or in any combination of such background components, middleware components, or front-end components. Components of the system can be connected to one another via digital data communication in any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0158] A computer system can include a client and a server. Typically, the client and server are geographically separated and interact via a communication network. The client-server relationship is created by a computer program that operates on the corresponding computer. The server may be a cloud server, a server in a distributed system, or a server incorporating blockchain technology, etc.

[0159] It should be understood that steps can be newly ranked, added, or deleted using the various forms of flows shown above. For example, each step described in this disclosure may be executed in parallel, sequentially, or in a different order. This disclosure does not limit this, as long as the technical aspects disclosed in this disclosure can achieve the desired results.

[0160] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, subcombinations, and substitutions are possible due to design considerations and other factors. Any changes, equivalent substitutions, and improvements within the gist and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for controlling gradient communication, Identifying gradient changes in the target parameter group of a local node, wherein the local node is any node in a plurality of computing nodes performing distributed training on a target model, the local node stores a plurality of parameter groups of the target model, and the target parameter group is any parameter group among the plurality of parameter groups. If the gradient change of the target parameter group satisfies the communication conditions, the gradient of the target parameter group is synchronized with the target node in the plurality of computing nodes, and this includes: A method for controlling gradient communication.

2. The aforementioned gradient communication control method further includes: The gradient update frequency, parameter type, and parameter tensor shape of multiple model parameters of the target model are obtained. The process includes determining the plurality of parameter groups based on the gradient update frequency, parameter type, and shape of the parameter tensor of the plurality of model parameters, by grouping the plurality of model parameters to obtain the plurality of parameter groups. The gradient communication control method according to claim 1.

3. The grouping rule used to group the aforementioned multiple model parameters is: 1) Minimize the difference in update frequency within the same group. 2) Minimize the differences in parameter types within the same group. 3) Minimizing the differences in the shape of parameter tensors within the same group. 4) The amount of data within the same group is less than or equal to the first threshold, including at least one of the above, The gradient communication control method according to claim 2.

4. Identifying the gradient change of the target parameter group of the local node is: This includes determining the statistical information in the gradient cache corresponding to the target parameter group and obtaining the gradient change of the target parameter group. The aforementioned statistical information is, The L2 norm of the gradient in the aforementioned gradient cache, The number of gradient updates in the aforementioned gradient cache, The number of target value elements in the gradient cache includes one of the following, where the target value represents that the corresponding element has a gradient update. The gradient communication control method according to claim 1.

5. Determining the statistically targeted information in the gradient cache corresponding to the aforementioned target parameter group is: The gradient cache of the aforementioned target parameter group is divided into multiple gradient blocks, This includes gradually accumulating the amount of statistical information for multiple gradient blocks and determining whether or not the communication conditions are met, The gradient communication control method according to claim 4.

6. When the gradient change includes the L2 norm, the communication condition includes that the L2 norm is greater than the first communication threshold. When the gradient change includes the number of gradient updates, the communication condition includes that the number of gradient updates is greater than the second communication threshold. When the gradient change includes the number of target value elements, the communication condition includes the number of target value elements being greater than the third communication threshold. The gradient communication control method according to claim 4.

7. If the gradient update frequencies of the multiple parameter tensors in the target parameter group are the same, the first communication threshold and / or the second communication threshold is the gradient update frequency of the target parameter group. The gradient communication control method according to claim 6.

8. If the gradient update frequencies of the multiple parameter tensors in the target parameter group are different, the third communication threshold is determined based on the number of parameter tensors in the target parameter group. The gradient communication control method according to claim 6.

9. The aforementioned target parameter group includes multiple parameter tensors, The gradient communication control method according to claim 1.

10. The aforementioned gradient communication control method further includes: When the update of the parameters of the target parameter group is complete, the gradient of the target parameter group is updated to the default value, The gradient communication control method according to claim 1.

11. An identification module for identifying gradient changes of a target parameter group of a local node, wherein the local node is any node in a plurality of computing nodes performing distributed training on a target model, the local node stores a plurality of parameter groups of the target model, and the target parameter group is any parameter group in the plurality of parameter groups; The system includes a synchronization module for synchronizing the gradient of the target parameter group with the target nodes in the plurality of computing nodes when the gradient change of the target parameter group satisfies the communication conditions. A communication control device for gradients.

12. At least one processor, The system comprises at least one processor and a memory that is communicated with by it, The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is caused to perform the method according to any one of claims 1 to 10. Electronic devices.

13. A non-temporary computer-readable storage medium storing computer instructions that cause a computer to perform the method described in any one of claims 1 to 10.

14. A program for implementing the method described in any one of claims 1 to 10, which is executed by a processor in a computer.

Images