Gradient data transmission method for model training and related equipment

Abstract

According to the model training-oriented gradient data transmission method and the related equipment provided by the invention, the compression ratio is dynamically determined according to the change of the transmission rate by monitoring the transmission rate in real time, so that the gradient data can be intelligently compressed according to the compression ratio, the data transmission quantity is remarkably reduced, and the network burden is relieved; and meanwhile, the transmission rate of each working end is adjusted through congestion control according to the bandwidth of a transmission link between the switch and the parameter server so as to relieve congestion. Under the combined action of the two, the transmission time of the gradient data is remarkably shortened, and the data transmission efficiency is improved, so that the training process of the distributed artificial intelligence model is accelerated, and the overall training efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a gradient data transmission method and related equipment for model training. Background Art

[0002] As the model parameters and training data sets of large models grow exponentially, data centers use distributed training frameworks to utilize computing resources more efficiently. Data parallelism is a distributed artificial intelligence (AI) model training framework that allows each computing node to share the same model parameters while processing different data subsets at the same time.

[0003] However, in the data-parallel parameter server (PS) architecture, multiple worker nodes calculate gradients based on their respective data subsets, while one or more parameter servers store and update model parameters. After completing a gradient calculation, the worker transmits the calculated gradient to the parameter server to update the overall parameters of the model. This many-to-one communication mode is called gradient aggregation. In gradient aggregation, since all worker ends use the same link, bandwidth bottlenecks are prone to occur, causing network congestion. This network congestion will prolong the transmission time of the gradient, thereby reducing the efficiency of distributed artificial intelligence model training. Summary of the invention

[0004] In view of this, the purpose of this application is to propose a gradient data transmission method and related equipment for model training to solve the above technical problems.

[0005] Based on the above purpose, the first aspect of the present application provides a gradient data transmission method for model training, which is applied to a gradient data transmission control system. The system includes a switch, a parameter server and multiple working ends. The method is executed by each working end, including:

[0006] Determine the gradient data of the current training times, obtain the initial transmission rate corresponding to the gradient data of the current training times, and obtain the transmission rate at the current moment corresponding to the gradient data of the current training times;

[0007] Determine the transmission rate at the current moment as the target transmission rate, and determine the transmission rate change ratio based on the initial transmission rate and the target transmission rate;

[0008] Obtain the total number of bytes of gradient data for the current training times, count the number of bytes of gradient data that have been successfully transmitted for the current training times, and determine the number of remaining transmission bytes for the current training times based on the total number of bytes of gradient data for the current training times and the number of bytes of gradient data that have been successfully transmitted for the current training times;

[0009] Based on the number of bytes of the gradient data that have been successfully transmitted under the current training times, the total number of bytes of the gradient data of the current training times, the number of remaining transmission bytes of the current training times and the transmission rate change ratio, a compression ratio determination process is performed through a compression ratio algorithm to obtain a compression ratio of the current training times;

[0010] Determine target gradient data for the current training times based on the compression ratio of the current training times and the gradient data for the current training times;

[0011] Obtaining a first link bandwidth of a transmission link between the switch and the parameter server, and a total number of multiple working ends;

[0012] Determine a second link bandwidth of a transmission link between each working end and the switch based on the first link bandwidth and the total number of the plurality of working ends;

[0013] The target gradient data of the current training times is transmitted to the switch according to the second link bandwidth, so that the switch transmits the target gradient data of the current training times to the parameter server according to the first link bandwidth.

[0014] Optionally, determining the transmission rate at the current moment as the target transmission rate includes:

[0015] Get the transmission rate of the next moment from the current moment;

[0016] Performing a difference processing using the transmission rate at the current moment and the transmission rate at the next moment to obtain a first difference processing result;

[0017] Performing ratio processing on the first difference processing result and the transmission rate at the current moment to obtain a transmission rate change value;

[0018] In response to the transmission rate change value being within a range of a preset transmission rate change value threshold, the transmission rate at the current moment is determined to be a target transmission rate.

[0019] Optionally, determining the transmission rate change ratio based on the initial transmission rate and the target transmission rate includes:

[0020] Performing a difference processing using the initial transmission rate and the target transmission rate to obtain a second difference processing result;

[0021] The transmission rate change ratio is obtained by performing ratio processing on the second difference processing result and the initial transmission rate.

[0022] Optionally, the determining the number of remaining transmission bytes for the current training times based on the total number of bytes of the gradient data for the current training times and the number of bytes of the gradient data that have been successfully transmitted for the current training times includes:

[0023] The total number of bytes of the gradient data for the current training times and the number of bytes of the gradient data that have been successfully transmitted for the current training times are subtracted to obtain the number of remaining transmission bytes for the current training times.

[0024] Optionally, the compression ratio determination process is performed by a compression ratio algorithm based on the number of bytes of the gradient data that have been successfully transmitted under the current training times, the total number of bytes of the gradient data of the current training times, the number of remaining transmission bytes of the current training times, and the transmission rate change ratio to obtain the compression ratio of the current training times, including:

[0025] The remaining transmission bytes of the current training times and the transmission rate change ratio are used to perform product processing to obtain a product processing result;

[0026] The number of bytes of the gradient data that have been successfully transmitted under the current training times and the product processing result are summed to obtain a sum processing result;

[0027] The sum processing result and the total number of bytes of the gradient data of the current training times are used to perform ratio processing to obtain the compression ratio of the current training times.

[0028] Optionally, determining the target gradient data of the current training times based on the compression ratio of the current training times and the gradient data of the current training times includes:

[0029] The compression ratio of the current training times and the gradient data of the current training times are multiplied to obtain the target gradient data of the current training times.

[0030] Optionally, determining the second link bandwidth of the transmission link between each working end and the switch based on the first link bandwidth and the total number of the plurality of working ends includes:

[0031] A ratio processing is performed on the first link bandwidth and the total number of the plurality of working ends to obtain a second link bandwidth of a transmission link between each working end and the switch.

[0032] Based on the same inventive concept, the second aspect of the present application provides a gradient data transmission device for model training, which is applied to a gradient data transmission control system. The system includes a switch, a parameter server and multiple working ends. The device is arranged at each working end. The device includes:

[0033] A first acquisition module is configured to determine the gradient data of the current training times, acquire the initial transmission rate corresponding to the gradient data of the current training times, and acquire the transmission rate at the current moment corresponding to the gradient data of the current training times;

[0034] a change ratio determination module, configured to determine the transmission rate at a current moment as a target transmission rate, and determine a transmission rate change ratio based on the initial transmission rate and the target transmission rate;

[0035] The remaining transmission determination module is configured to obtain the total number of bytes of the gradient data of the current training times, count the number of bytes of the gradient data that have been successfully transmitted under the current training times, and determine the remaining number of bytes to be transmitted for the current training times based on the total number of bytes of the gradient data of the current training times and the number of bytes of the gradient data that have been successfully transmitted under the current training times;

[0036] The compression ratio determination module is configured to perform compression ratio determination processing based on the number of bytes of the gradient data that have been successfully transmitted under the current training times, the total number of bytes of the gradient data of the current training times, the number of remaining transmission bytes of the current training times, and the transmission rate change ratio through a compression ratio algorithm to obtain the compression ratio of the current training times;

[0037] a target gradient determination module, configured to determine target gradient data for the current training times based on the compression ratio of the current training times and the gradient data for the current training times;

[0038] A second acquisition module is configured to acquire a first link bandwidth of a transmission link between the switch and the parameter server, and a total number of multiple working terminals;

[0039] a link bandwidth determination module, configured to determine a second link bandwidth of a transmission link between each working end and the switch based on the first link bandwidth and a total number of the plurality of working ends;

[0040] The gradient data transmission module is configured to transmit the target gradient data of the current training times to the switch according to the second link bandwidth, so that the switch can transmit the target gradient data of the current training times to the parameter server according to the first link bandwidth.

[0041] Based on the same inventive concept, the third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the method described in the first aspect above when executing the computer program.

[0042] Based on the same inventive concept, the fourth aspect of the present application provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect above.

[0043] From the above, it can be seen that the gradient data transmission method and related equipment for model training provided by this application can monitor the transmission rate in real time, dynamically determine the compression ratio according to the change of the transmission rate, and then intelligently compress the gradient data according to the compression ratio, which significantly reduces the amount of data transmission and reduces the network burden; at the same time, according to the transmission link bandwidth between the switch and the parameter server, the transmission rate of each working end is adjusted through congestion control to alleviate congestion. Under the joint action of the two, the transmission time of gradient data is significantly shortened, and the data transmission efficiency is improved, thereby accelerating the training process of distributed artificial intelligence models and improving the overall training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic diagram of a parameter server architecture in data parallelism in an embodiment of the present application;

[0046] Figure 2 A schematic diagram showing the impact of bottleneck links and gradient compression on transmission time in an embodiment of the present application;

[0047] Figure 3 This is a flowchart of a gradient data transmission method for model training according to an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a processing flow of a gradient compression mechanism based on application transmission rate awareness according to an embodiment of the present application;

[0049] Figure 5 This is a structural block diagram of a gradient data transmission device for model training according to an embodiment of the present application;

[0050] Figure 6 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0052] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0053] It is understandable that before using the technical solutions of each embodiment of the present application, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0054] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to the electronic device, application, server, storage medium or other software or hardware that performs the operation of the technical solution of the present application according to the prompt message.

[0055] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0056] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of this application. Other methods that meet relevant laws and regulations may also be applied to the implementation method of this application.

[0057] As the model parameters and training data sets of large models grow exponentially, data centers use distributed training frameworks to efficiently utilize computing resources. Data parallelism is a distributed artificial intelligence (AI) model training framework that allows each computing node to share the same model parameters while processing different data subsets.

[0058] However, in the data-parallel parameter server (PS) architecture, multiple worker nodes calculate gradients based on their respective data subsets, while one or more parameter servers store and update model parameters. After completing a gradient calculation, the worker transmits the calculated gradient to the parameter server to update the overall parameters of the model. This many-to-one communication mode is called gradient aggregation. In gradient aggregation, since all worker ends use the same link, bandwidth bottlenecks are prone to occur, causing network congestion. This network congestion will prolong the transmission time of the gradient, thereby reducing the efficiency of distributed artificial intelligence model training.

[0059] In order to shorten the gradient transmission time of distributed AI training in data center networks, this application proposes a transmission rate-aware cross-layer gradient compression mechanism. This mechanism dynamically adjusts the compression ratio by the transmission rate, alleviates the impact of network congestion caused by bandwidth bottlenecks, and thus shortens the gradient transmission time of distributed AI training. This application comprehensively considers the characteristics of distributed training and the link bottlenecks in the transmission process, and improves the efficiency of distributed AI training without significantly reducing the accuracy of the model.

[0060] Among them, data center distributed training data parallel architecture: With the exponential growth of machine learning model parameters and training data sets, data centers adopt distributed training frameworks to utilize computing resources more efficiently. These frameworks involve multiple computing nodes interconnected through a network to collaboratively complete large-scale training tasks. In a distributed training framework, data parallelism allows each computing node to share the same model parameters while processing different data subsets. The PS architecture is one of the common data parallel models. In the PS architecture, multiple worker nodes (i.e., working ends) calculate gradients based on their respective data subsets, while one or more parameter servers store and update model parameters. After completing a gradient calculation iteration, the results of the working node must be synchronized to a single parameter server node to update the overall parameters of the model. This process is called gradient aggregation, which is a many-to-one communication mode.

[0061] In order to optimize the efficiency of data transmission between distributed training nodes in data parallelism, Remote Direct Memory Access (RDMA) technology is usually used. RDMA allows a computing node to read and write data directly in the memory of another node without going through the operating system, thereby significantly reducing communication latency and increasing data transmission rates. However, RDMA requires that the network has no packet loss-any packet loss may cause calculation errors or require retransmission, thereby reducing overall efficiency. In high-load networks, especially in the gradient aggregation process of data parallelism, multiple nodes transmitting data to a single node at the same time often leads to bandwidth bottlenecks, causing network congestion. To address the above challenges, this application proposes relevant solutions at both the network layer and the computing layer to alleviate potential network congestion. At the network layer, congestion control algorithms are studied to alleviate congestion by adjusting the transmission rate. At the computing layer, gradient compression is implemented to reduce the amount of data that needs to be transmitted over the network.

[0062] Congestion control algorithms: From the perspective of the Internet, network congestion will reduce the scalability and performance of distributed training. To alleviate network congestion, data centers have proposed a variety of congestion control algorithms. Data Center Quantized Congestion Notification (DCQCN) adjusts the transmission rate according to the Explicit Congestion Notification (ECN) mark, which is generated based on buffer usage. High Precision Congestion Control (HPCC) detects the number of transmitted bytes in the network through in-band Network Telemetry (INT) and adjusts the sending window size. Timely is another popular congestion control method that adjusts the sending rate based on the rate of change of the round-trip time (RTT). These congestion control algorithms effectively alleviate congestion through different detection algorithms and corresponding rate control strategies. However, although these algorithms alleviate congestion by adjusting the transmission rate, they significantly increase the gradient transmission time and fail to effectively address the impact of network congestion on the scalability of distributed training.

[0063] Gradient compression technology: From a computing perspective, large-scale tasks require more and more worker nodes. The increase in the number of worker nodes leads to an increase in the amount of communication during the gradient aggregation process, which in turn prolongs the gradient transmission time. However, not all gradients calculated by each worker node need to be transmitted to the parameter server. Existing studies have shown that distributed training is robust to inaccurate gradients, and this practice of reducing transmitted gradients is called gradient compression. Gradient compression technology reduces the size of the gradient vector that needs to be transmitted and the amount of data that needs to be transmitted for each iteration, thereby reducing the network load and improving the gradient transmission time. Common methods for achieving gradient compression include quantization and sparsification. Quantization reduces the data size by reducing the precision of the gradient value, such as reducing double-precision floating-point numbers to single precision or even half precision, thereby significantly reducing the amount of data during synchronization. Sparsification only transmits important gradient elements, usually those whose absolute values ​​exceed a certain threshold.

[0064] In the related art, gradient compression based on iteration state is applied in distributed training. Gradient compression based on iteration state comprehensively considers iteration state and gradient compression to better solve the bandwidth bottleneck problem. For example, it is proposed to add a shim layer between the training model and the transmission protocol to detect the progress of iterative transmission and adjust the compression ratio accordingly. This method enables nodes with slower transmission progress to transmit fewer gradients, thereby shortening the gradient transmission time. The delay-aware compression library infers the link status by measuring the iteration delay, and adjusts the compression ratio accordingly to reduce the extension of gradient aggregation time caused by slow nodes. However, both methods only indirectly consider changes in network conditions and fail to accurately adapt to changes in rate.

[0065] The embodiment of the present application provides a gradient data transmission method for model training, which monitors the transmission rate in real time, dynamically determines the compression ratio according to the change of the transmission rate, and then can intelligently compress the gradient data according to the compression ratio, significantly reducing the amount of data transmission and alleviating the network burden; at the same time, according to the transmission link bandwidth between the switch and the parameter server, the transmission rate of each working end is adjusted through congestion control to alleviate congestion. Under the joint action of the two, the transmission time of gradient data is significantly shortened, the data transmission efficiency is improved, thereby accelerating the training process of the distributed artificial intelligence model and improving the overall training efficiency.

[0066] The gradient data transmission method for model training proposed in this application is applied to gradient data transmission control systems, such as Figure 1 As shown, the system includes a switch (Switch), a parameter server (Server) and multiple workers (Workers).

[0067] Data parallelism effectively scales computing power to process large-scale data by partitioning the data set and processing it on different nodes.

[0068] Figure 1 The structure of a parameter server (PS) in data parallelism is shown, which includes a server node S (i.e., parameter server) for managing global model parameters and M worker nodes {W 1 , W 2 , ..., W M}(i.e. working end). Each W m It is a separate computing unit that computes a data subset D m In each iteration n, each worker node W m According to the current global model θ n and its data subset D m Independent calculation of gradients As shown in the following formula:

[0069]

[0070] In one iteration, we mainly focus on the process of sending gradients from multiple worker nodes to a single parameter server node, because this process may cause congestion. The transmission time of this process is defined as T g The link between the working node and the switch is defined as L w Each L w The capacity is C w The link between the parameter server and the switch is L s , whose capacity is C s The sum of gradients transmitted by all workers in iteration n is defined as G n , as shown in the following formula:

[0071]

[0072] in, Indicates the magnitude of the gradient.

[0073] T g There are two cases, as shown in the following formula:

[0074]

[0075] The first case is that the sum of the link capacities of the working nodes is less than or equal to C s In this case, there will be no congestion, and all working nodes will send gradients to S at the maximum link capacity. The second case is when the sum of the link capacities of the working nodes is greater than C s , L sThe congestion control algorithm will reduce the transmission rate of all working nodes to the same stable value, thereby extending T g In fact, C s Usually equal to C w Therefore, T g Mainly affected by the bottleneck link bandwidth.

[0076] Although congestion control can alleviate congestion by adjusting the transmission rate of each worker node, it does not solve the problem of increased transmission time due to bottleneck links. For distributed training, congestion control seems to solve this problem if not all gradients are transmitted. Reducing the transmission rate will extend T g , while gradient compression will shorten T g .

[0077] Sparsification is an effective technique to achieve gradient compression. It only transmits part of the gradient. After the introduction of gradient compression technology, it can reduce and T g . Transmission time T after gradient compression g_cprs It is given by the following formula:

[0078]

[0079] Among them, R cprs Represents the compression ratio of the gradient.

[0080] Although gradient compression shortens T g_cprs , but it will inevitably reduce the accuracy to a certain extent. For example, sparsification may cause gradient noise, which is an introduced error. This error accumulates during the iteration process and affects the convergence and accuracy of the final model. In order to solve the problem of increased transmission time due to congestion during the aggregation process, this application defines the following optimization objectives during the training process, which are expressed as follows:

[0081]

[0082] in, is the training accuracy loss.

[0083] According to the above analysis, the present application found that the transmission time is the shortest when there is no congestion. Once congestion occurs, the transmission time will increase. In order to make the transmission time not affected by the bottleneck link, the present application proposes a transmission rate-aware gradient compression mechanism (TRGC), which is applied to the processing process of the gradient data transmission method for model training.

[0084] like Figure 2 As shown in Figure 1, a simple topology is shown with three worker nodes and one parameter server node. During iteration n, assuming =1, L w The bandwidth is 1, L s The bandwidth is greater than or equal to 3. At this stage, no congestion will occur during the gradient aggregation process, so the transmission time T g is 1, at this time T g represents the shortest transmission time, such as Figure 2 However, L s and L w The bandwidths are usually equal, in which case L s It is the bottleneck link. Based on the existing congestion control algorithm, each worker will reduce the transmission rate to 1 / 3, T g will be expanded to 3, such as Figure 2 As shown in (b), TRGC will compress the gradient according to the change of transmission rate, T g will be shortened to 1, such as Figure 2 As shown in (c), TRGC solves the problem of increased transmission time caused by bottleneck links.

[0085] like Figure 3 As shown, the method is executed by each working end, and the method of this embodiment includes:

[0086] Step 301, determine the gradient data of the current training times, obtain the initial transmission rate corresponding to the gradient data of the current training times, and obtain the transmission rate at the current moment corresponding to the gradient data of the current training times.

[0087] In this step, during the machine learning training process, the model updates these parameters by calculating the gradient of the loss function with respect to the model parameters. Gradient data refers to these calculated gradient values, which indicate the direction in which the loss function decreases fastest.

[0088] The current training number refers to the current iteration or cycle in the training process. In each iteration, the gradient data is calculated based on the current model parameters.

[0089] At the beginning of gradient aggregation, the initial transmission rate is recorded as the baseline rate. Changes in the current transmission rate are detected. When a bottleneck link appears, network congestion will occur. The congestion control algorithm will adjust the sending rate to alleviate congestion. When changes in the transmission rate are detected, the current transmission rate is recorded.

[0090] Step 302: determine the current transmission rate as the target transmission rate, and determine the transmission rate change ratio based on the initial transmission rate and the target transmission rate.

[0091] In this step, the current transmission rate is determined as the target transmission rate, indicating that the current transmission rate has reached a stable state. The transmission rate change ratio is determined using the stable target transmission rate and the initial transmission rate.

[0092] The transmission rate change ratio is the relative difference between the target transmission rate and the initial transmission rate, usually expressed as a percentage or decimal. It is an indicator of how the data transmission rate changes over time and is used to predict future transmission rate trends or reflect changes in current network conditions.

[0093] Step 303, obtain the total number of bytes of gradient data for the current training times, and count the number of bytes of gradient data that have been successfully transmitted for the current training times, and determine the remaining number of bytes to be transmitted for the current training times based on the total number of bytes of gradient data for the current training times and the number of bytes of gradient data that have been successfully transmitted for the current training times.

[0094] In this step, a certain amount of gradient data is generated in each model training iteration (or training epoch). The size of these gradient data (in bytes) is calculated and recorded as the total number of bytes of gradient data for that training epoch. This data is very useful for understanding the data burden generated by each training iteration.

[0095] In a distributed training system, gradient data usually needs to be transmitted between different computing nodes. Count the number of bytes of gradient data that have been successfully transmitted from the source node (worker) to the target node (parameter server or switch) under the current training number. This information is crucial for monitoring the progress and performance of data transmission.

[0096] Based on the total number of bytes of gradient data for the current training times and the number of bytes of gradient data that have been successfully transmitted for the current training times, the number of bytes of gradient data that still need to be transmitted for the current training times, i.e., the number of remaining transmission bytes, can be obtained. This data is very helpful for evaluating the remaining transmission time and resource requirements.

[0097] It can provide a comprehensive perspective to monitor the generation of gradient data, transmission progress, and the amount of remaining tasks. This is crucial to ensure the efficiency and reliability of data transmission and optimize the training process. By monitoring this data in real time, potential data transmission bottlenecks or errors can be discovered and resolved in a timely manner, thereby improving overall training efficiency.

[0098] Step 304, based on the number of bytes of gradient data that have been successfully transmitted under the current training times, the total number of bytes of gradient data for the current training times, the number of remaining transmission bytes for the current training times and the transmission rate change ratio, a compression ratio is determined by using a compression ratio algorithm to obtain a compression ratio for the current training times.

[0099] In this step, a suitable compression ratio is calculated or determined using the number of bytes of the successfully transmitted gradient data, the total number of bytes, the number of remaining transmission bytes, and the transmission rate change ratio.

[0100] The compression ratio refers to the ratio between the original data size and the compressed data size, which is used to measure the degree of data compression.

[0101] The gradient data compression ratio determined in the current training round will be used to compress the gradient data to reduce bandwidth usage during transmission while trying to maintain data accuracy and training efficiency.

[0102] By dynamically adjusting the compression ratio, the utilization of network resources can be optimized while ensuring the training effect.

[0103] Step 305 : determining target gradient data for the current training times based on the compression ratio for the current training times and the gradient data for the current training times.

[0104] In this step, the gradient data of the current training times are gradient compressed based on the compression ratio of the current training times to determine the adjusted gradient data set, i.e., the target gradient data, so as to ensure that the performance of the model is maintained as much as possible while compressing the model.

[0105] The target gradient data is used to guide the subsequent training process, such as considering the impact of compression ratio when updating model parameters, or readjusting gradients after compressing the model to restore or improve performance. Gradient compression can reduce bandwidth usage during transmission.

[0106] Step 306: Obtain a first link bandwidth of a transmission link between the switch and the parameter server, and a total number of multiple working ends.

[0107] In this step, the first link bandwidth refers to the maximum data transmission rate that can be supported on the transmission link between the switch and the parameter server, usually in bits per second (bps).

[0108] The total number of multiple working ends refers to the total number of devices associated with a switch or network that are active or ready to receive service.

[0109] Step 307: Determine a second link bandwidth of the transmission link between each working end and the switch based on the first link bandwidth and the total number of the plurality of working ends.

[0110] In this step, in order to alleviate congestion, the second transmission rate of the transmission link between each working end and the switch is adjusted based on the first link bandwidth. For example, if all working ends transmit data at the maximum speed at the same time, then the bandwidth available to each working end will be the first link bandwidth divided by the total number of working ends, so as to alleviate the congestion caused by the bottleneck link bandwidth.

[0111] Step 308: Transmit the target gradient data of the current training times to the switch according to the second link bandwidth, so that the switch can transmit the target gradient data of the current training times to the parameter server according to the first link bandwidth.

[0112] In this step, in each training iteration, each working end determines the corresponding gradient data and sends the data to the switch according to the second link bandwidth. The switch then transmits the gradient data to the parameter server according to the first link bandwidth. After receiving the gradient data, the parameter server will update the parameters and distribute the updated parameters back to all working ends through the switch to prepare for the next training iteration.

[0113] This ensures that model parameters can be globally updated based on data from all computing nodes, thereby improving the efficiency and effectiveness of training.

[0114] Through the above scheme, by real-time monitoring of the transmission rate, the compression ratio is dynamically determined according to the change of the transmission rate, and then the gradient data can be intelligently compressed according to the compression ratio, which significantly reduces the amount of data transmission and reduces the network burden; at the same time, the transmission rate of each working end is adjusted through congestion control according to the transmission link bandwidth between the switch and the parameter server to alleviate congestion. Under the joint action of the two, the transmission time of gradient data is significantly shortened, the data transmission efficiency is improved, thereby accelerating the training process of the distributed artificial intelligence model and improving the overall training efficiency.

[0115] In some embodiments, in step 302, determining the current transmission rate as the target transmission rate includes:

[0116] Step A1, obtaining the transmission rate at the next moment of the current moment.

[0117] Step A2, performing a difference processing using the transmission rate at the current moment and the transmission rate at the next moment to obtain a first difference processing result.

[0118] Step A3, performing ratio processing on the first difference processing result and the transmission rate at the current moment to obtain a transmission rate change value.

[0119] Step A4: in response to the transmission rate change value being within a range of a preset transmission rate change value threshold, determining the transmission rate at the current moment as a target transmission rate.

[0120] In the above scheme, the transmission rate at the current moment is first recorded. Then the transmission rate at the next time point immediately after the current moment (i.e., the transmission rate at the next moment) is obtained. This usually means that the system can monitor the change of the transmission rate in real time or near real time.

[0121] The difference between the current transmission rate and the transmission rate at the next moment is calculated (ie, the first difference processing result). This difference reflects the change in the transmission rate between two consecutive time points.

[0122] Divide this difference (the result of the first difference processing) by the current transmission rate to obtain a ratio or percentage. This ratio (i.e., the transmission rate change value) reflects the relative change in the transmission rate relative to the current value, and provides a standardized measure for evaluating the magnitude of the transmission rate change.

[0123] Check whether the transmission rate change value falls within a preset, acceptable range (ie, the transmission rate change value threshold).

[0124] If the transmission rate change value is within this range, it means that the change of the transmission rate is stable or acceptable, so the transmission rate at the current moment is determined as the target transmission rate.

[0125] Ensure that the selected transmission rate is neither too volatile nor too conservative, thereby optimizing the performance and efficiency of data transmission to adapt to changes in network conditions and ensure the stability and efficiency of data transmission.

[0126] In some embodiments, in step 302, determining the transmission rate change ratio based on the initial transmission rate and the target transmission rate includes:

[0127] Step B1, performing a difference processing using the initial transmission rate and the target transmission rate to obtain a second difference processing result.

[0128] Step B2, performing ratio processing on the second difference processing result and the initial transmission rate to obtain a transmission rate change ratio.

[0129] In the above scheme, the difference between the initial transmission rate and the target transmission rate (ie, the second difference processing result) is determined. This difference reflects the change required from the initial transmission rate to the target transmission rate.

[0130] The second difference processing result is compared with the initial transmission rate, and their ratio (i.e., the transmission rate change ratio) is calculated. This ratio represents the ratio of the change in the transmission rate relative to the initial value, that is, it measures the relative size of the change in the transmission rate. It can help understand how the transmission rate changes and the degree of change.

[0131] In some embodiments, in step 303, the determining of the number of remaining transmission bytes for the current training times based on the total number of bytes of the gradient data for the current training times and the number of bytes of the gradient data that have been successfully transmitted for the current training times includes:

[0132] The total number of bytes of the gradient data for the current training times and the number of bytes of the gradient data that have been successfully transmitted for the current training times are subtracted to obtain the number of remaining transmission bytes for the current training times.

[0133] In the above solution, the number of remaining bytes to be transmitted can be quickly determined by subtracting the number of bytes of the successfully transmitted gradient data from the total number of bytes of the gradient data of the current training times.

[0134] This provides information about how much gradient data still needs to be transmitted in the current training step.

[0135] In some embodiments, step 304 includes:

[0136] Step C1, performing product processing using the remaining number of transmission bytes of the current training times and the transmission rate change ratio to obtain a product processing result.

[0137] Step C2, summing the number of bytes of the gradient data that have been successfully transmitted under the current training times and the product processing result to obtain a summing processing result.

[0138] Step C3, performing ratio processing using the sum processing result and the total number of bytes of the gradient data of the current training times to obtain the compression ratio of the current training times.

[0139] In the above scheme, the remaining number of transmission bytes and the transmission rate change ratio are multiplied to obtain a product processing result, which reflects the expected change in the remaining number of transmission bytes after considering the transmission rate change.

[0140] The number of successfully transmitted gradient data bytes is summed with the product processing result to obtain a sum processing result, which represents the total expected number of transmitted bytes after taking into account the change in transmission rate and the amount of data transmitted under the current number of training times.

[0141] The summation result is divided by the total number of bytes of gradient data to obtain the ratio processing result, that is, the compression ratio of the current training times. This compression ratio represents the reduction ratio of the amount of gradient data actually transmitted or expected to be transmitted relative to the total amount of gradient data.

[0142] The compression efficiency of gradient data is dynamically evaluated by considering the change in transmission rate and the amount of data transmitted. The compression ratio is an important metric because it helps understand the efficiency and optimization potential of data transmission, especially in distributed training or network transmission-constrained environments. By continuously optimizing the compression algorithm and transmission strategy of gradient data, the training efficiency can be further improved, bandwidth usage can be reduced, and the model training process can be accelerated.

[0143] In some embodiments, step 305 includes:

[0144] The compression ratio of the current training times and the gradient data of the current training times are multiplied to obtain the target gradient data of the current training times.

[0145] In the above scheme, in order to adjust the size of the gradient, the compression ratio is multiplied by the gradient data of the current training times, thereby achieving gradient compression of the gradient data of the current training times, so as to reduce bandwidth usage during transmission.

[0146] The adjusted gradient data (i.e. the target gradient data for the current number of training times) is obtained through gradient compression. These data will be used to update the parameters of the model to further reduce the prediction error of the model.

[0147] In some embodiments, step 307 includes:

[0148] A ratio processing is performed on the first link bandwidth and the total number of the plurality of working ends to obtain a second link bandwidth of a transmission link between each working end and the switch.

[0149] In the above scheme, in order to alleviate congestion, the second transmission rate of the transmission link between each working end and the switch is adjusted based on the first link bandwidth. For example, if all working ends transmit data at the maximum speed at the same time, then the bandwidth available to each working end will be the first link bandwidth divided by the total number of working ends, so as to alleviate the congestion caused by the bottleneck link bandwidth.

[0150] In some embodiments, the gradient data transmission method for model training of the present application uses a transmission rate-aware gradient compression mechanism (TRGC) as follows: Figure 4 As shown:

[0151] (1) Obtain the initial transmission rate data from the network layer: At the beginning of gradient aggregation, record the initial transmission rate r b , using this as the baseline rate.

[0152] (2) Detect changes in the current transmission rate: When a bottleneck link appears, network congestion will occur. The congestion control algorithm will adjust the sending rate to alleviate congestion. When a change in the transmission rate is detected, the current transmission rate is recorded as r. c .

[0153] (3) Monitor the stability of the transmission rate: Update the next transmission rate r n , if the transmission rate is stable (i.e., the transmission rate changes|(r c -r n ) / r c |Within the set threshold TH) the number of times CNT reaches a certain value, it is considered that the rate has reached a stable state.

[0154] (4) Calculate the transmission rate change ratio: R rate =(r b -r c ) / r b .

[0155] (5) Calculate the compression ratio: When the rate has reached a stable state, use the stable rate change ratio R rate As the remaining bytes B r The remaining transmission bytes are B r =B t -B s , where B t is the total number of bytes in the stream, B s is the number of bytes successfully transferred, B r is the remaining transmission bytes. According to the stable rate change ratio R rate To calculate the compression ratio R of the total bytes cprs . Where R cprs (B s +B r ·R eate ) / B t .

[0156] (6) Check the rationality of the compression ratio: Check the compression ratio R cprs Is it within the reasonable range [MIN_R, 1]? If R cprs If it is within a reasonable range, the total number of bytes B is updated according to this compression ratio. t . Among them B t =B s +B r ·R rate .

[0157] (7) As training iterations proceed, the above process is repeated to continuously optimize the transmission rate and gradient compression strategy to ensure that network congestion time and communication overhead are minimized without significantly reducing model accuracy.

[0158] In summary, this application comprehensively considers the congestion control of the network layer and the gradient compression of the computing layer to improve the efficiency of distributed training. And adjusts the gradient compression ratio according to the transmission rate, reducing the extension of gradient transmission time caused by bottleneck links.

[0159] In addition, compared with the original solution of adjusting the compression ratio according to changes in multiple iteration states, this application pays more fine-grained attention to the network congestion caused by bottleneck links between single iterations. When bottleneck links such as gradient aggregation appear in the network, the compression ratio is adjusted according to the rate changes to ensure that the gradient transmission time is not affected by the bottleneck link.

[0160] In addition, this application comprehensively considers data center congestion control and gradient compression strategies, reduces gradient transmission time, and improves the efficiency of distributed AI training.

[0161] In some embodiments, a simulation platform (such as NS3) is used to implement the proposed transmission rate-aware gradient compression mechanism (TRGC) scheme, and the four congestion control algorithms (DCQCN, DCTCP, TIMELY and HPCC) improved by TRGC are respectively called GC-DCQCN, GC-DCTCP, GC-TIMELY and GC-HPCC, and the congestion control algorithm improved by TRGC is compared with the original algorithm. The simulation topology consists of 8 working nodes, 1 switch and 1 parameter server, and the link bandwidth is set to 25Gbps, simulating a typical distributed AI training environment.

[0162] Through simulation, the performance (normalized gradient transmission time and compression ratio change) of GC-DCQCN, GC-DCTCP, GC-TIMELY and GC-HPCC in dealing with the "incast" phenomenon (i.e., instantaneous congestion caused by many-to-one communication) during gradient synchronization is analyzed and compared with the original algorithm.

[0163] The simulation results show that when "incast" congestion occurs, the traditional congestion control algorithm (such as DCQCN) will be affected by the bottleneck link and reduce the sending rate, thereby extending the gradient transmission time. The improved congestion control algorithm can avoid the influence of the bottleneck link to a certain extent and reduce the gradient transmission time.

[0164] Simulation results show that the normalized gradient transmission time of the original congestion control algorithm increases linearly with the number of working nodes. Compared with the original algorithm, GC-DCQCN, GC-DCTCP, GC-HPCC and GC-TIMELY slow down the growth of the normalized gradient transmission time.

[0165] Specifically, the normalized gradient transmission time of GC-DCQCN increases linearly with the number of worker nodes. When the number of worker nodes is 8, GC-DCQCN reduces the gradient transmission time by 73.94%.

[0166] Similarly, the normalized gradient transmission time of GC-DCTCP also increases linearly with the number of working nodes, but when the number of nodes is 4, the growth rate of the gradient transmission time slows down.

[0167] When the number of working nodes is 8, GC-DCTCP reduces the gradient transmission time by 65.24% compared with the original algorithm. The normalized gradient transmission time of GC-HPCC remains stable when the number of nodes increases, and its performance is better than the other three algorithms.

[0168] This is believed to be due to HPCC's ability to more accurately measure the network status based on the number of bytes being transmitted on the link. When the number of working nodes is 8, GC-HPCC reduces the gradient transmission time by 87.91%. GC-HPCC effectively solves the problem of gradient transmission time affected by bottleneck link bandwidth. The growth trend of the normalized gradient transmission time of GC-TIMELY is similar to that of GC-DCQCN and GC-DCTCP.

[0169] When the number of working nodes is 8, GC-TIMELY reduces the gradient transmission time by 76.87% compared with the original algorithm.

[0170] To evaluate the effect of TRGC on the convergence of the model, the above simulations were repeated and the compression ratios in multiple iterations were recorded.

[0171] The results show that the compression ratio will be maintained above 20%. Based on previous research, a compression ratio of about 20% will not affect the convergence of the model.

[0172] In summary, TRGC shows significant advantages in the gradient synchronization stage of distributed AI training. Compared with traditional congestion control algorithms, TRGC reduces the impact of bottleneck links and reduces the time of gradient transmission. This congestion control scheme combined with gradient compression can greatly improve the efficiency of AI model training without significantly reducing the accuracy of the model.

[0173] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0174] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0175] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a gradient data transmission device for model training.

[0176] refer to Figure 5 The gradient data transmission device for model training is applied to a gradient data transmission control system. The system includes a switch, a parameter server and a plurality of working ends. The device is arranged at each working end. The device includes:

[0177] A first acquisition module 501 is configured to determine the gradient data of the current training times, acquire the initial transmission rate corresponding to the gradient data of the current training times, and acquire the transmission rate at the current moment corresponding to the gradient data of the current training times;

[0178] A change ratio determination module 502 is configured to determine the current transmission rate as the target transmission rate, and determine the transmission rate change ratio based on the initial transmission rate and the target transmission rate;

[0179] The remaining transmission determination module 503 is configured to obtain the total number of bytes of the gradient data of the current training times, count the number of bytes of the gradient data that have been successfully transmitted under the current training times, and determine the number of remaining transmission bytes of the current training times based on the total number of bytes of the gradient data of the current training times and the number of bytes of the gradient data that have been successfully transmitted under the current training times;

[0180] The compression ratio determination module 504 is configured to perform compression ratio determination processing based on the number of bytes of the gradient data that have been successfully transmitted under the current training times, the total number of bytes of the gradient data of the current training times, the number of remaining transmission bytes of the current training times, and the transmission rate change ratio through a compression ratio algorithm to obtain the compression ratio of the current training times;

[0181] A target gradient determination module 505 is configured to determine target gradient data for the current training times based on the compression ratio of the current training times and the gradient data for the current training times;

[0182] A second acquisition module 506 is configured to acquire a first link bandwidth of a transmission link between the switch and the parameter server, and a total number of multiple working terminals;

[0183] The link bandwidth determination module 507 is configured to determine a second link bandwidth of a transmission link between each working end and the switch based on the first link bandwidth and the total number of the plurality of working ends;

[0184] The gradient data transmission module 508 is configured to transmit the target gradient data of the current training times to the switch according to the second link bandwidth, so that the switch can transmit the target gradient data of the current training times to the parameter server according to the first link bandwidth.

[0185] In some embodiments, the change ratio determination module 502 is specifically configured to:

[0186] Get the transmission rate of the next moment from the current moment;

[0187] Performing a difference processing using the transmission rate at the current moment and the transmission rate at the next moment to obtain a first difference processing result;

[0188] Performing ratio processing on the first difference processing result and the transmission rate at the current moment to obtain a transmission rate change value;

[0189] In response to the transmission rate change value being within a range of a preset transmission rate change value threshold, the transmission rate at the current moment is determined to be a target transmission rate.

[0190] In some embodiments, the change ratio determination module 502 is specifically configured to:

[0191] Performing a difference processing using the initial transmission rate and the target transmission rate to obtain a second difference processing result;

[0192] The transmission rate change ratio is obtained by performing ratio processing on the second difference processing result and the initial transmission rate.

[0193] In some embodiments, the remaining transmission determination module 503 is specifically configured to:

[0194] The total number of bytes of the gradient data for the current training times and the number of bytes of the gradient data that have been successfully transmitted for the current training times are subtracted to obtain the number of remaining transmission bytes for the current training times.

[0195] In some embodiments, the compression ratio determination module 504 is specifically configured to:

[0196] The remaining transmission bytes of the current training times and the transmission rate change ratio are used to perform product processing to obtain a product processing result;

[0197] The number of bytes of the gradient data that have been successfully transmitted under the current training times and the product processing result are summed to obtain a sum processing result;

[0198] The sum processing result and the total number of bytes of the gradient data of the current training times are used to perform ratio processing to obtain the compression ratio of the current training times.

[0199] In some embodiments, the target gradient determination module 505 is specifically configured as follows:

[0200] The compression ratio of the current training times and the gradient data of the current training times are multiplied to obtain the target gradient data of the current training times.

[0201] In some embodiments, the link bandwidth determination module 507 is specifically configured to:

[0202] A ratio processing is performed on the first link bandwidth and the total number of the plurality of working ends to obtain a second link bandwidth of a transmission link between each working end and the switch.

[0203] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0204] The device of the above embodiment is used to implement the corresponding gradient data transmission method for model training in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0205] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the gradient data transmission method for model training described in any of the above embodiments is implemented.

[0206] Figure 6 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 601, a memory 602, an input / output interface 603, a communication interface 604, and a bus 605. The processor 601, the memory 602, the input / output interface 603, and the communication interface 604 are connected to each other in communication within the device through the bus 605.

[0207] The processor 601 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0208] The memory 602 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 602 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 602 and called and executed by the processor 601.

[0209] The input / output interface 603 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0210] The communication interface 604 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0211] The bus 605 includes a path for transmitting information between various components of the device (eg, the processor 601 , the memory 602 , the input / output interface 603 , and the communication interface 604 ).

[0212] It should be noted that, although the above device only shows the processor 601, the memory 602, the input / output interface 603, the communication interface 604 and the bus 605, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0213] The electronic device of the above embodiment is used to implement the corresponding gradient data transmission method for model training in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0214] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the gradient data transmission method for model training as described in any of the above embodiments.

[0215] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0216] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the gradient data transmission method for model training as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0217] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0218] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0219] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0220] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A gradient data transmission method for model training, characterized in that: Applied to a gradient data transmission control system, the system includes a switch, a parameter server and multiple working ends, and the method is executed by each working end, including: Determine the gradient data of the current training times, obtain the initial transmission rate corresponding to the gradient data of the current training times, and obtain the transmission rate at the current moment corresponding to the gradient data of the current training times; Determine the transmission rate at the current moment as the target transmission rate, and determine the transmission rate change ratio based on the initial transmission rate and the target transmission rate; Obtain the total number of bytes of gradient data for the current training times, count the number of bytes of gradient data that have been successfully transmitted for the current training times, and determine the number of remaining transmission bytes for the current training times based on the total number of bytes of gradient data for the current training times and the number of bytes of gradient data that have been successfully transmitted for the current training times; Based on the number of bytes of the gradient data that have been successfully transmitted under the current training times, the total number of bytes of the gradient data of the current training times, the number of remaining transmission bytes of the current training times and the transmission rate change ratio, a compression ratio determination process is performed through a compression ratio algorithm to obtain a compression ratio of the current training times; Determine target gradient data for the current training times based on the compression ratio of the current training times and the gradient data for the current training times; Obtaining a first link bandwidth of a transmission link between the switch and the parameter server, and a total number of multiple working ends; Determine a second link bandwidth of a transmission link between each working end and the switch based on the first link bandwidth and the total number of the plurality of working ends; The target gradient data of the current training times is transmitted to the switch according to the second link bandwidth, so that the switch transmits the target gradient data of the current training times to the parameter server according to the first link bandwidth.

2. The method according to claim 1, characterized in that The determining the transmission rate at the current moment as the target transmission rate includes: Get the transmission rate of the next moment from the current moment; Performing a difference processing using the transmission rate at the current moment and the transmission rate at the next moment to obtain a first difference processing result; Performing ratio processing on the first difference processing result and the transmission rate at the current moment to obtain a transmission rate change value; In response to the transmission rate change value being within a range of a preset transmission rate change value threshold, the transmission rate at the current moment is determined to be a target transmission rate.

3. The method according to claim 1, characterized in that The determining the transmission rate change ratio based on the initial transmission rate and the target transmission rate comprises: Performing a difference processing using the initial transmission rate and the target transmission rate to obtain a second difference processing result; The transmission rate change ratio is obtained by performing ratio processing on the second difference processing result and the initial transmission rate.

4. The method according to claim 1, characterized in that The method of determining the number of remaining transmission bytes for the current training times based on the total number of bytes of the gradient data for the current training times and the number of bytes of the gradient data that have been successfully transmitted for the current training times includes: The total number of bytes of the gradient data for the current training times and the number of bytes of the gradient data that have been successfully transmitted for the current training times are subtracted to obtain the number of remaining transmission bytes for the current training times.

5. The method according to claim 1, characterized in that The compression ratio determination process is performed based on the number of bytes of the gradient data that have been successfully transmitted under the current training times, the total number of bytes of the gradient data of the current training times, the number of remaining transmission bytes of the current training times, and the transmission rate change ratio through a compression ratio algorithm to obtain the compression ratio of the current training times, including: The remaining transmission bytes of the current training times and the transmission rate change ratio are used to perform product processing to obtain a product processing result; The number of bytes of the gradient data that have been successfully transmitted under the current training times and the product processing result are summed to obtain a sum processing result; The sum processing result and the total number of bytes of the gradient data of the current training times are used to perform ratio processing to obtain the compression ratio of the current training times.

6. The method according to claim 1, characterized in that The step of determining target gradient data for the current training times based on the compression ratio for the current training times and the gradient data for the current training times comprises: The compression ratio of the current training times and the gradient data of the current training times are multiplied to obtain the target gradient data of the current training times.

7. The method according to claim 1, characterized in that The determining, based on the first link bandwidth and the total number of the plurality of working ends, a second link bandwidth of a transmission link between each working end and the switch comprises: A ratio processing is performed on the first link bandwidth and the total number of the plurality of working ends to obtain a second link bandwidth of a transmission link between each working end and the switch.

8. A gradient data transmission device for model training, characterized in that: Applied to a gradient data transmission control system, the system includes a switch, a parameter server and multiple working ends, the device is arranged at each working end, and the device includes: A first acquisition module is configured to determine the gradient data of the current training times, acquire the initial transmission rate corresponding to the gradient data of the current training times, and acquire the transmission rate at the current moment corresponding to the gradient data of the current training times; a change ratio determination module, configured to determine the transmission rate at a current moment as a target transmission rate, and determine a transmission rate change ratio based on the initial transmission rate and the target transmission rate; The remaining transmission determination module is configured to obtain the total number of bytes of the gradient data of the current training times, count the number of bytes of the gradient data that have been successfully transmitted under the current training times, and determine the remaining number of bytes to be transmitted for the current training times based on the total number of bytes of the gradient data of the current training times and the number of bytes of the gradient data that have been successfully transmitted under the current training times; The compression ratio determination module is configured to perform compression ratio determination processing based on the number of bytes of the gradient data that have been successfully transmitted under the current training times, the total number of bytes of the gradient data of the current training times, the number of remaining transmission bytes of the current training times, and the transmission rate change ratio through a compression ratio algorithm to obtain the compression ratio of the current training times; a target gradient determination module, configured to determine target gradient data for the current training times based on the compression ratio of the current training times and the gradient data for the current training times; A second acquisition module is configured to acquire a first link bandwidth of a transmission link between the switch and the parameter server, and a total number of multiple working terminals; a link bandwidth determination module, configured to determine a second link bandwidth of a transmission link between each working end and the switch based on the first link bandwidth and a total number of the plurality of working ends; The gradient data transmission module is configured to transmit the target gradient data of the current training times to the switch according to the second link bandwidth, so that the switch can transmit the target gradient data of the current training times to the parameter server according to the first link bandwidth.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images