Distributed sparse deep learning model training acceleration method
By performing in-network aggregation and thermal parameter cache on the switch, the communication bottlenecks and storage efficiency problems in distributed sparse deep learning model training are solved, efficient gradient aggregation and model training acceleration are achieved, and the model's high accuracy and training efficiency are maintained.
Patent Information
- Application Number
- CN202510478975.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-16
- Publication Date
- 2025-08-05
AI Technical Summary
The existing distributed sparse deep learning model training system has problems such as communication bottlenecks, reduced model accuracy and low storage efficiency of hot/cold parameters. Especially in large-scale distributed training, the communication overhead affects training efficiency. The existing methods fail to effectively distinguish between hot and cold parameters access methods, resulting in wasted storage resources and low cache efficiency.
The network aggregation and thermal parameter cache are carried out on the switch. The multi-layer hash bucket structure dynamically recognizes and caches thermal parameters, and the gradient aggregation task is offloaded from the server to the switch, reducing the communication pressure and computing burden of the server, while improving cache efficiency and model training speed.
It significantly reduces the communication pressure and computing burden of the server, improves the cache hit rate and storage resource utilization, maintains the high precision and training efficiency of the model, and reduces communication and training delays.
Smart Images
Figure CN120430352A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computers, and in particular to a distributed sparse deep learning model training acceleration method. Background Art
[0002] With the widespread application of deep learning in various fields, distributed deep neural network (DNN) training has become a key technology for accelerating large-scale model development. Existing distributed DNN training systems are primarily based on the parameter server (PS) architecture, which achieves efficient data-parallel training by storing model parameters on server nodes and having multiple workers compute gradients in parallel. This architecture is widely used in fields such as natural language processing, computer vision, and recommendation systems, especially when processing large-scale sparse data, such as user behavior logs and ad click-through rate prediction.
[0003] In distributed deep neural network training, existing training systems typically use centralized gradient aggregation and parameter update mechanisms. For example, in a traditional parameter server architecture, multiple worker nodes calculate local gradients and then send them to a server node for global gradient aggregation and model updates. Furthermore, when processing sparse models, existing systems typically treat hot parameters (frequently updated parameters) and cold parameters (infrequently updated parameters) equally, storing them in the same storage hierarchy.
[0004] Existing distributed sparse deep learning model training frameworks suffer from serious communication congestion and other problems. Because centralized gradient aggregation causes the server to become a communication bottleneck, especially in large-scale distributed training, the communication overhead is huge, seriously affecting training efficiency. To alleviate the communication pressure on the parameter server, existing methods adopt asynchronous update strategies. Although this improves hardware utilization, it causes gradient staleness, which in turn affects the stability of model convergence. Therefore, this method fails to effectively improve the overall quality of model training. Secondly, the existing system fails to distinguish between hot and cold parameter access methods, resulting in wasted storage resources, inefficient caching, and frequent memory access further increases the communication pressure on the parameter server. Moreover, as the model scale and data volume increase, the communication pressure of the existing system is further exacerbated, making it difficult to effectively expand and accelerate the training of existing large models.
[0005] Some researchers have proposed a gradient aggregation method based on hot-cold separation. In terms of gradient aggregation, hot-cold gradients are aggregated separately on switches and parameter servers, which will cause data staleness problems and reduce the accuracy of data models.
[0006] In summary, existing distributed sparse deep learning training systems have problems such as communication bottlenecks, decreased model accuracy, and inefficient hot / cold parameter storage. Summary of the Invention
[0007] The technical problem to be solved by the present invention is as follows: In response to the above-mentioned problems in the prior art, a distributed sparse deep learning model training acceleration method is provided. By offloading the gradient aggregation task from the server to the switch and caching hot parameters on the switch, the communication pressure and computing burden of the server are reduced, while the cache efficiency and model training speed are improved.
[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0009] A distributed sparse deep learning model training acceleration method is applied to switches in a distributed deep neural network training system with switches as the intermediate layer, the method comprising the following steps:
[0010] Obtain the gradients calculated by the working nodes and aggregate them within the network, caching the aggregated gradients into multi-layer hash buckets;
[0011] Identify the data overflowing from the hash bucket as hot parameters and cache them locally, identify the data not overflowing from the hash bucket as cold parameters and send them to the upper-layer switch, and forward the aggregated gradient data packets to the working nodes and parameter servers;
[0012] If a parameter request is received from a working node and the corresponding hot parameters are cached locally, the hot parameter data packet is sent from the local cache to the working node.
[0013] Furthermore, obtaining the gradients calculated by the worker nodes and performing in-network aggregation includes:
[0014] Obtaining the gradient data packet of the working node and determining whether the gradient data packet meets the requirements;
[0015] If the gradient data packet does not meet the requirements, the gradient data packet is sent to the upper layer switch;
[0016] If the gradient data packet meets the requirements, the gradient data packets with the same training period will be placed in the same aggregator until the bitmap of the aggregator is full. The aggregator with a full bitmap will perform weighted summation on the gradients of all corresponding gradient data packets to obtain the aggregated gradient.
[0017] Furthermore, before obtaining the gradients calculated by the working nodes and performing in-network aggregation, the working nodes may send the gradient data to the switches, including:
[0018] Add the gradient calculation result data to the INA header field of the gradient data packet, configure the opt flag option flag of the gradient data packet header to be in the INA state, and then send the configured gradient data packet to the switch;
[0019] When determining whether the gradient data packet meets the requirements, specifically, the packet header of the gradient data packet is parsed. If the opt flag option flag in the packet header is in the INA state, the gradient packet meets the requirements; otherwise, the gradient packet does not meet the requirements.
[0020] Furthermore, when sending the configured gradient data packet to the switch, the BSP protocol is used to send the gradient data packet to the switch to ensure that all working nodes use the same parameters for training.
[0021] Furthermore, caching the aggregated gradients into multi-layer hash buckets involves:
[0022] If the current gradient data record exists in the switch's local cache, or the current gradient is in the switch's waiting queue, the current gradient's level in the hash bucket is queried and the current gradient count is updated. If the updated count is greater than the threshold for the current gradient's level in the hash bucket, the current gradient's level in the hash bucket is moved up one level. If the current gradient is at the top level in the hash bucket and the updated count is greater than the threshold for the top level in the hash bucket, the current gradient data is used as overflow data in the hash bucket.
[0023] If there is no data record of the current gradient in the local cache of the switch and the current gradient is not in the waiting queue of the switch, the level of the current gradient in the hash bucket is initialized to the bottom level.
[0024] Furthermore, the method further includes the step of identifying data that has not overflowed in the hash bucket, specifically including:
[0025] If the current gradient is in a layer other than the top layer in the hash bucket, or if the current gradient is in the top layer in the hash bucket and the updated count is less than the top layer threshold in the hash bucket, the data of the current gradient is used as the data that has not overflowed in the hash bucket.
[0026] Furthermore, after identifying the data overflowing from the hash bucket as hot parameters and caching them locally, the method further includes:
[0027] If a parameter request is received from a working node and the corresponding hot parameters are cached locally, the hot parameter data packet is sent from the local to the working node;
[0028] If a parameter request is received from a working node and the corresponding hot parameters are not cached locally, the parameter request is sent to the upper-layer switch.
[0029] The present invention also proposes a distributed sparse deep learning model training acceleration system, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the distributed sparse deep learning model training acceleration method.
[0030] The present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the distributed sparse deep learning model training acceleration method.
[0031] The present invention also proposes a computer program product, including a computer program, which, when executed by a processor, implements the steps of the distributed sparse deep learning model training acceleration method.
[0032] Compared with the prior art, the advantages of the present invention are:
[0033] This invention introduces a switch as an intermediate layer within the traditional PS architecture, offloading hot parameter caching and gradient aggregation tasks from the server to the switch. The switch then identifies and stores hot data during training in real time, enabling dynamic management of hot data and more efficient use of the switch's memory space. Furthermore, the aggregation operation does not differentiate between hot and cold data, enabling efficient data-parallel training while maintaining high model accuracy and training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 Schematic diagram of the steps of the method according to an embodiment of the present invention.
[0035] Figure 2 Detailed flowchart of the method according to an embodiment of the present invention.
[0036] Figure 3 This is a comparison chart of the model accuracy convergence curves of the method of an embodiment of the present invention and other solutions under the WDL-Criteo workload.
[0037] Figure 4 This is a comparison chart of the speedup ratio of the training task completion time deployed in a cluster using the method of an embodiment of the present invention and other solutions.
[0038] Figure 5 The figure is a comparison diagram of throughput diagrams of the method according to the embodiment of the present invention and other solutions under four different data sets.
[0039] Figure 6 This is a comparison diagram of the acceleration ratios of the method according to the embodiment of the present invention and other solutions on four different data sets in the NS-3 simulation environment.
[0040] Figure 7This is a comparison diagram of the 99th percentile flow completion time diagram of the method of the embodiment of the present invention and other solutions in the NS-3 simulation environment. DETAILED DESCRIPTION
[0041] The present invention will be further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the scope of protection of the present invention is not limited thereby.
[0042] Example 1
[0043] This embodiment proposes a distributed sparse deep learning model training acceleration method, named Themis. This method is based on in-network aggregation (INA) and hot-parameter caching. It offloads gradient aggregation tasks from the server to the switch and identifies and caches hot parameters in real time on the switch to achieve reasonable management of hot parameters, reduce the communication pressure and computing burden of the server, and improve cache efficiency and model training speed.
[0044] like Figure 1 As shown, the method of this embodiment is applied to a switch in a distributed deep neural network training system with a switch as an intermediate layer, and includes the following steps:
[0045] S1) The worker node sends gradient data to the switch;
[0046] S2) The switch obtains the gradients calculated by the working nodes and aggregates them within the network, caching the aggregated gradients into multi-layer hash buckets;
[0047] S3) The switch identifies the data overflowing from the hash bucket as hot parameters and caches them locally, identifies the data not overflowing from the hash bucket as cold parameters and sends them to the upper-layer switch, and forwards the aggregated gradient data packet to the working node and parameter server.
[0048] The following combination Figure 2 , and explain each step in detail.
[0049] In this embodiment, the header of the gradient data packet is provided with an INA header field, which is used to store the data of the gradient calculation result and also records the corresponding data ID (MsgID) and data length (MsgLen), so that each message has an INA header (MsgID, MsgLen) before the tensor value. Figure 2 As shown, in step S1 of this embodiment, the step of the working node sending gradient data to the switch includes:
[0050] The data of the gradient calculation result is added to the INA header field of the gradient data packet, and the opt flag option flag in the packet header of the gradient data packet is configured to be in the INA state. Then, the configured gradient data packet is sent to the switch. In this embodiment, the configured gradient data packet is sent to the switch using the BSP protocol. The BSP synchronous update method is adopted to enable the working node to send the gradient of the same period to the control module of the switch, thereby ensuring the consistency of the global model and ensuring that all nodes use the same parameters for training.
[0051] In step S2 of this embodiment, gradients are aggregated within the switch, offloading the gradient aggregation task from the server to the switch. This leverages the switch's high-performance computing capabilities and significantly reduces the communication pressure on the server. Furthermore, hot parameters are cached within the switch, improving cache hit rates, reducing communication delays caused by frequent server accesses, and optimizing storage resource utilization.
[0052] In distributed training, the gradients calculated by worker nodes are aggregated through switches rather than sent directly to the server. The switches use their computing power to perform a weighted sum of the gradients from different worker nodes, and then send the aggregated gradients to the server for global model updates. This reduces the amount of communication from worker nodes to the server, lowering server bandwidth pressure while also preventing training delays caused by the server becoming a communication bottleneck. Specifically, obtaining the gradients calculated by worker nodes and performing in-network aggregation involves:
[0053] S21) obtaining a gradient data packet of a working node and determining whether the gradient data packet meets the requirements;
[0054] S22) If the gradient data packet does not meet the requirements, sending the gradient data packet to an upper-layer switch;
[0055] S23) If the gradient data packet meets the requirements, the gradient data packets with the same training period are placed in the same aggregator until the bitmap of the aggregator is full. The aggregator with a full bitmap performs a weighted summation on the gradients of all corresponding gradient data packets to obtain an aggregated gradient.
[0056] like Figure 2As shown, in this embodiment, determining whether a gradient packet meets the requirements involves parsing the packet header. If the opt flag in the packet header is in the INA state, the gradient packet meets the requirements; otherwise, the gradient packet does not meet the requirements. When the gradient packet arrives at the switch, the switch uses a programmable parser to parse the received gradient packet. If the opt flag is in the INA state, it is passed to an aggregator within the aggregation module. If the opt flag is not in the INA state, it is directly sent to the upper-layer switch. The INA header field is then identified and matched based on relevant information (such as MsgID), placing gradient packets with the same training epoch into the same aggregator. When the aggregator's bitmap is full, the aggregator performs a weighted sum of the valid gradients of all gradient packets in its packet buffer to obtain aggregated gradients. The aggregator then updates the packet using the original header information (such as the MsgID in the INA header field) and the aggregated data (i.e., the aggregated gradients). The aggregator then outputs the gradient packet with the latest gradients to the upper-layer switch. After the upper-layer switch obtains the aggregated gradient data packet, it also performs judgment and in-network aggregation according to the above steps S21 to S23, and sends the aggregated new gradient data packet again to the upper-layer switch or parameter server. The specific steps are not repeated here.
[0057] For the identification and caching of hot parameters, existing technologies typically adopt simple caching strategies (such as Least Recently Used, LRU or Least Frequently Used, LFU), which cannot effectively distinguish between hot and cold parameters. This embodiment designs a multi-layer hash bucket structure to dynamically adjust its storage level according to the access frequency of the parameter. When the access frequency of a parameter exceeds a preset threshold, it is identified as a hot parameter and cached to the switch. This achieves accurate identification of hot parameters, avoids the mistaken eviction of hot parameters due to simple replacement strategies, and improves cache efficiency.
[0058] like Figure 2 As shown, the updated gradient, i.e., the aggregated data, is cached into a multi-layer hash bucket. During the caching process, first determine whether there is a record of the data in the switch cache. If so, find the index corresponding to the gradient data, then find the data through the index, and add 1 to the value of the index position. If the data cannot be found in the existing cache, check whether the data exists in the waiting queue. If the data exists in the waiting queue, execute the same steps as if the data exists. If the data does not exist in the waiting queue, create a new hash bucket for the data in the cache and initialize the value to 1. Specifically, when caching the aggregated gradient into a multi-layer hash bucket, it specifically includes:
[0059] If the switch's local cache contains the current gradient data record, or the current gradient is in the switch's waiting queue, the current gradient's level in the hash bucket is queried and the current gradient count is updated. Specifically, the hash bucket index is first calculated: bucketlndex = hash(pkt.key), and the corresponding hash bucket counter is incremented: HashBucket[bucketlndex][counter]++; if the updated count is greater than the threshold for the current gradient's level in the hash bucket, the current gradient's level in the hash bucket is moved up one level; otherwise, the current gradient's level in the hash bucket is retained.
[0060] If the switch's local cache does not contain the current gradient data record and the current gradient is not in the switch's waiting queue, the current gradient is initialized to the bottom level in the hash bucket. Specifically, a new hash bucket for the current gradient is created in the cache and initialized to 1: HashMap[bucket[bucketlndex][counter]=1].
[0061] As can be seen, in step S2, when caching the aggregated gradient into the hash bucket, data with higher communication frequency will float up to the upper layers of the multi-level hash bucket. When the number of communications of a certain data exceeds the highest threshold of the multi-level hash bucket, the data will overflow. In this embodiment, the data will be determined as a hot parameter and stored in the cache of the switch. At the same time, the data that has not overflowed will be determined as a cold parameter and then sent to the upper-level switch. After receiving the cold parameter, the upper-level switch will also follow the above steps to cache the obtained data into the hash bucket of the switch, and will determine it as a hot parameter when the number of communications of the data exceeds the highest threshold of the multi-level hash bucket. The specific steps will not be repeated here.
[0062] Therefore, in step S3 of this embodiment, when identifying data overflowing from the hash bucket, if the current gradient is at the top level in the hash bucket and the updated count is greater than the top level threshold in the hash bucket, the data of the current gradient is regarded as overflow data in the hash bucket. Figure 2 As shown, this embodiment maintains a hot parameter registry through the top level of the hash bucket, identifies the overflow data as hot parameters and caches them locally, specifically registering the hot parameters in the hot parameter registry hot_parameters, to achieve dynamic identification and caching of hot parameters for frequently accessed parameters.
[0063] Similarly, when identifying data that has not overflowed in the hash bucket, if the current gradient is in a layer other than the top layer in the hash bucket, or the current gradient is in the top layer in the hash bucket and the updated count is less than the top layer threshold in the hash bucket, then the data of the current gradient is regarded as data that has not overflowed in the hash bucket.
[0064] In step S3 of this embodiment, the data packet containing the aggregated gradients is forwarded to the worker nodes and parameter server. The switch directly sends the aggregated results to the worker nodes, avoiding long-distance communication and worker node latency. Simultaneously, the parameter server receives the aggregated gradient results and uses them to update the global training model to maintain model consistency.
[0065] Through steps S1 to S3, hot parameters are identified and cached in the switch. When a working node requests parameters, the switch will prioritize providing hot parameters from the cache, reducing the frequency of access to the server. The corresponding steps are as follows:
[0066] If a parameter request is received from a working node and the corresponding hot parameters are cached locally, the hot parameter data packet is sent from the local to the working node;
[0067] If a parameter request is received from a working node and the corresponding hot parameters are not cached locally, the parameter request is sent to the upper-layer switch.
[0068] Through the above steps, the thermal parameters of the switch cache improve the cache hit rate, reduce the communication delay caused by frequent access to the server, and optimize the utilization of storage resources.
[0069] The effectiveness of the method of this embodiment is verified through test platform experiments and large-scale NS-3 simulations.
[0070] The testbed experimental setup is as follows: 8 worker nodes and 1 parameter server are used. All worker nodes and parameter server are equipped with 20-core Intel Xeon Gold 5218R CPUs. The worker nodes and parameter server are configured with 64GB and 256GB of memory, respectively. Each worker node is equipped with an Nvidia 3090 GPU. These components are connected via Mellanox ConnectX-5 interfaces and interconnected via 100Gbps Ethernet network bandwidth, with the Edgecore Wedge100BF-32X switch model. The default underlying transport protocol is TCP. The operating system on all devices is Ubuntu 18.04, configured with CUDA 11.3.19, cuDNN 8.2.0, and NCCL 2.9.9.
[0071] The method in this example is compared with HET and ATP. HET uses the SSP synchronization method, and all settings are set according to the recommendations in the relevant papers. End-to-end experiments are conducted using four representative datasets: Wide & Deep Learning (WDL-Criteo), Deep Factorization Machine (DFM-Avazu), Neural Collaborative Filter (NCF-Movie), and DCN-Criteosearch. These are all high-dimensional sparse models widely used in recommendation systems and computational advertising. In the simulation, the default batch size is 256 and the default embedding size is 512.
[0072] Figure 3 Figure 2 shows the model accuracy convergence curves for the method in this example, HET, and ATP under the WDL-Criteo workload. The method in this example is named Themis. It can be seen that both Themis and ATP achieve higher final model accuracy than HET, and they converge to the same accuracy level.
[0073] Figure 4 This is a diagram showing the acceleration ratio of Themis compared to HET and ATP deployed in a cluster. Figure 4 (a) and Figure 4 (b) shows the change in training speed improvement (measured after the preparation phase, starting from the second phase of training). Figure 4 In (a), each experiment was trained with 4 different bandwidths, and all were configured with 8 workers. Themis outperformed HET and ATP in all scenarios; specifically, at 100Gbps training bandwidth, Themis achieved a training speedup of 23.3% and 20.5% compared to HET and ATP, respectively. Figure 4 In (b), each experiment was trained with a different number of workers, all configured with 100Gbps training bandwidth. Increasing the number of workers can enhance parallel processing capabilities, but it may also increase communication overhead. Compared to HET and ATP, Themis achieves training speed improvements of 23.3% and 20.5%, respectively. We also observe even more significant speedups with Themis and ATP using in-network aggregation techniques. Caching hot parameters on switches significantly reduces communication traffic, thereby accelerating training.
[0074] Figure 5 This is a schematic diagram of the throughput of Themis compared with HET and ATP under four different data sets. Figure 5 (a) is the throughput of different strategies under the WDL-Criteo model. Figure 5 (b) The throughput of different strategies under the DFM-Avazu model. Figure 5 (c) is the throughput of different strategies under the NCF-Movie model. Figure 5 (d) shows the throughput of different strategies under the DCN-Criteosearch model. Judging from the results of the four experiments, as the number of worker nodes increases, the aggregate throughput decreases, which seems to be contrary to common sense. The main reason is that gradient aggregation needs to wait for the gradients of all workers to arrive, and the increase in the number of workers leads to an increase in waiting and processing time, which effectively reduces the aggregate throughput. It can be seen that the more workers there are, the more significant the improvement of Themis in overall throughput, and it outperforms all other solutions. Specifically, compared with HET and ATP, Themis's effective aggregate throughput in the WDL-Criteo model is improved by 43.7% and 14.6%, respectively. It is an advanced network aggregation technology that aims to reduce communication pressure through network aggregation, thereby improving the overall performance of the network.
[0075] The large-scale NS-3 simulation environment is set up as follows: a leaf-spine network topology consisting of 12 spine switches and 24 leaf switches. Each leaf switch connects to 32 hosts, with a link bandwidth of 100 Gbps and a default latency of 2 μs. The switch provides 5 MB of memory for INA and cache. TCP is used as the transport protocol.
[0076] Figure 6 This is a schematic diagram of the acceleration ratio of Themis compared with HET and ATP on four different data sets in the NS-3 simulation environment. Figure 6 (a) is the speedup ratio of different strategies under the WDL-Criteo model. Figure 6 (b) is the acceleration ratio of different strategies under the DFM-Avazu model. Figure 6 (c) is the speedup ratio of different strategies under the NCF-Movie model. Figure 6 (d) shows the speedup of different strategies under the DCN-Criteosearch model. It can be observed that Themis significantly improves training speed by 1.17 to 1.59 times compared to the baseline (PS-Lite). Specifically, when comparing four different models under a 0.8x workload, Themis outperforms HET and ATP by 30.1% and 41.9%, 11.7% and 25.9%, 7.9% and 11.9%, and 21.3% and 38.3% respectively. We also note that Themis's speedup becomes increasingly significant as the workload increases.
[0077] Figure 7 The diagram shows the flow completion time (FCT) of Themis compared with HET and ATP at the 99th percentile in the NS-3 simulation environment. Figure 7 (a) is the percentile flow completion time of different strategies under the WDL-Criteo model. Figure 7 (b) is the percentile flow completion time of different strategies under the DFM-Avazu model. Figure 7 (c) is the percentile flow completion time of different strategies under the NCF-Movie model. Figure 7 (d) Percentile flow completion time for different strategies under the DCN-Criteosearch model. Specifically, compared with PS Lite, HET, and ATP under a 0.8 workload, Themis reduced the 99th percentile flow completion time by 37.2%, 23.1% and 29.5%, 26.9%, 10.4% and 20.5%, 19.1%, 7.4% and 10.7%, and 33.2%, 17.5%, and 27.7%, respectively. ATP performs better than HET because it also performs network aggregation on the switch, reducing transmission communication cycles. Themis achieves better performance by specifically and effectively managing thermal parameters to reduce server response and communication pressure.
[0078] Experimental results show that compared with the existing technology, the method of this embodiment has significant advantages in reducing communication delay, improving training speed and maintaining model accuracy.
[0079] Example 2
[0080] This embodiment proposes a distributed sparse deep learning model training acceleration system, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the distributed sparse deep learning model training acceleration method described in Example 1.
[0081] This embodiment also proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the distributed sparse deep learning model training acceleration method described in Example 1 are implemented.
[0082] This embodiment also proposes a computer program product, including a computer program, which, when executed by a processor, implements the steps of the distributed sparse deep learning model training acceleration method described in Example 1.
[0083] In summary, the present invention proposes a distributed sparse deep learning model training acceleration method that performs in-network aggregation and caching of hot parameters on switches. Unlike the traditional PS architecture, the present invention introduces switches as the middle layer, deploys gradient aggregation tasks to the switches, and uses the switches to identify and cache hot parameters in the distributed sparse deep learning model training process.
[0084] The present invention dynamically identifies and caches hot parameters through a multi-layer hash bucket structure, and can adjust the storage level according to the access frequency, thereby avoiding the inefficiency problem caused by the fixed cache strategy in the existing technology. At the same time, the hot parameters are cached in the switch so that when the switch faces the parameter request of the working node, it will give priority to providing the hot parameters from the cache, reducing the access frequency to the server and significantly improving the cache efficiency.
[0085] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. A distributed sparse deep learning model training acceleration method, characterized by The method is applied to a switch in a distributed deep neural network training system with a switch as an intermediate layer, and the method includes the following steps: Obtain the gradients calculated by the working nodes and aggregate them within the network, caching the aggregated gradients into multi-layer hash buckets; The data overflowing from the hash bucket is identified as hot parameters and cached locally, the data not overflowing from the hash bucket is identified as cold parameters and sent to the upper-layer switch, and the aggregated gradient data packet is forwarded to the working node and parameter server.
2. The distributed sparse deep learning model training acceleration method according to claim 1 is characterized in that ,When obtaining the gradients calculated by the working nodes and performing in-network aggregation, it includes: Obtaining the gradient data packet of the working node and determining whether the gradient data packet meets the requirements; If the gradient data packet does not meet the requirements, the gradient data packet is sent to the upper layer switch; If the gradient data packet meets the requirements, the gradient data packets with the same training period will be placed in the same aggregator until the bitmap of the aggregator is full. The aggregator with a full bitmap will perform weighted summation on the gradients of all corresponding gradient data packets to obtain the aggregated gradient.
3. The distributed sparse deep learning model training acceleration method according to claim 2 is characterized in that ,Before obtaining the gradient calculated by the working node and ,aggregating it in the network, the working node also ,sends the gradient data to the switch, including: Add the gradient calculation result data to the INA header field of the gradient data packet, configure the opt flag option flag of the gradient data packet header to be in the INA state, and then send the configured gradient data packet to the switch; When determining whether the gradient data packet meets the requirements, specifically, the packet header of the gradient data packet is parsed. If the opt flag option flag in the packet header is in the INA state, the gradient packet meets the requirements; otherwise, the gradient packet does not meet the requirements.
4. The distributed sparse deep learning model training acceleration method according to claim 3, characterized in that: When sending the configured gradient data packet to the switch, the BSP protocol is used to send the gradient data packet to the switch to ensure that all working nodes use the same parameters for training.
5. The distributed sparse deep learning model training acceleration method according to claim 1, characterized in that: When caching the aggregated gradients into multi-layer hash buckets, it specifically includes: If the current gradient data record exists in the switch's local cache, or the current gradient is in the switch's waiting queue, the current gradient's level in the hash bucket is queried and the current gradient count is updated. If the updated count is greater than the threshold for the current gradient's level in the hash bucket, the current gradient's level in the hash bucket is moved up one level. If the current gradient is at the top level in the hash bucket and the updated count is greater than the threshold for the top level in the hash bucket, the current gradient data is used as overflow data in the hash bucket. If there is no data record of the current gradient in the local cache of the switch and the current gradient is not in the waiting queue of the switch, the level of the current gradient in the hash bucket is initialized to the bottom level.
6. The distributed sparse deep learning model training acceleration method according to claim 5, characterized in that: The step of identifying data that has not overflowed in the hash bucket is also included, specifically comprising: If the current gradient is in a layer other than the top layer in the hash bucket, or if the current gradient is in the top layer in the hash bucket and the updated count is less than the top layer threshold in the hash bucket, the data of the current gradient is used as the data that has not overflowed in the hash bucket.
7. The distributed sparse deep learning model training acceleration method according to claim 1, characterized in that: After identifying the data overflowing from the hash bucket as hot parameters and caching them locally, the method further includes: If a parameter request is received from a working node and the corresponding hot parameters are cached locally, the hot parameter data packet is sent from the local to the working node; If a parameter request is received from a working node and the corresponding hot parameters are not cached locally, the parameter request is sent to the upper-layer switch.
8. A distributed sparse deep learning model training acceleration system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the distributed sparse deep learning model training acceleration method according to any one of claims 1 to 7.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the distributed sparse deep learning model training acceleration method according to any one of claims 1 to 7 are implemented.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the distributed sparse deep learning model training acceleration method described in any one of claims 1 to 7 are implemented.