Deep learning method, apparatus, network device, and readable storage medium

Abstract

The deep learning method and device, the network device and the readable storage medium provided by the embodiment of the present application perform data processing on the model copies in each GPU in the server to obtain gradient data of the current batch; when the accumulated number of gradient data in the corresponding GPU reaches a preset threshold, the accumulated gradient data is used to update the model copy; wherein the preset threshold of the gradient data corresponding to at least one GPU is greater than or equal to 2. Thus, the model is updated after the gradient data is accumulated to a certain number, which expands the actual Batch Size used during training, reduces the number of parameter updates, breaks the limitation of the single GPU display memory, can realize training of a larger Batch Size, and effectively improves the throughput and efficiency of the system.

Description

Technical Field

[0001] The embodiments of the present invention relate to, but are not limited to, the field of artificial intelligence. Specifically, they relate to, but are not limited to, a deep learning method, apparatus, network device, and readable storage medium. Background Technology

[0002] In recent years, deep learning has made groundbreaking progress, achieving impressive performance in areas such as machine vision, speech recognition, natural language processing, machine translation, data mining, and autonomous driving. However, training these deep learning network models requires significant computational power. Although major advancements have been made in GPU hardware, network architecture, and training methods in recent years, training a network model on a single machine can still be very time-consuming. To accelerate the training of deep learning models, the industry has conducted extensive work and research to achieve efficient distributed training of neural networks. Currently, data parallelism is the most widely used method in large-scale distributed model training. When we research the techniques and methods of distributed model training, we find the following problems: As the computing power of GPUs (Graphics Processing Units) continues to increase, data aggregation between GPUs has become a bottleneck for large-scale distributed model training. Synchronous training increases the overall batch size (the number of samples selected in one training session), but if experimental conditions are limited and GPU resources are insufficient, it is impossible to train models with extremely large batch sizes, hindering our research and verification of some algorithms and techniques. Moreover, for GPUs with different computing power, synchronous training can waste the performance of high-performance GPUs. Summary of the Invention

[0003] The deep learning method, apparatus, network device, and readable storage medium provided in the embodiments of the present invention mainly solve the technical problem of limited model training samples and poor GPU performance utilization in related technologies.

[0004] To address the aforementioned technical problems, embodiments of the present invention provide a deep learning method, comprising:

[0005] Data processing is performed on the model copies in each GPU of the server to obtain the gradient data for the current batch;

[0006] When the cumulative number of gradient data in the corresponding GPU reaches a preset threshold, the model copy is updated with the cumulative gradient data; wherein, the preset threshold for gradient data corresponding to at least one GPU is greater than or equal to 2.

[0007] This invention also provides a deep learning device, comprising:

[0008] The processing module is used to process the model copies in each GPU in the server to obtain the gradient data for the current batch.

[0009] An update module is used to update the model copy with the accumulated gradient data when the number of accumulated gradient data in the corresponding GPU reaches a preset threshold; wherein the preset threshold for gradient data corresponding to at least one GPU is greater than or equal to 2.

[0010] This invention also provides a network device, which includes a processor, a memory, and a communication bus;

[0011] The communication bus is used to enable communication between the processor and the memory;

[0012] The processor is used to execute one or more computer programs stored in the memory to implement the steps of the deep learning method described above.

[0013] This invention also provides a computer storage medium that stores one or more programs, which can be executed by one or more processors to implement the steps of the deep learning method described above.

[0014] The beneficial effects of this invention are:

[0015] According to the deep learning method, apparatus, network device, and readable storage medium provided in embodiments of the present invention, data processing is performed on model replicas in each GPU of a server to obtain gradient data for the current batch; when the cumulative number of gradient data in the corresponding GPU reaches a preset threshold, the accumulated gradient data is used to update the model replica; wherein, the preset threshold for gradient data corresponding to at least one GPU is greater than or equal to 2. Thus, by accumulating a certain amount of gradient data before updating the model, the actual batch size used during training is increased, while the number of parameter updates is reduced, breaking free from the limitation of a single GPU's video memory, enabling training with a larger batch size, and effectively improving the system's throughput and efficiency.

[0016] Other features and corresponding beneficial effects of the present invention will be described in the latter part of the specification, and it should be understood that at least some of the beneficial effects will become obvious from the description in the specification. Attached Figure Description

[0017] Figure 1 This is a flowchart of the deep learning method provided in Embodiment 1 of the present invention;

[0018] Figure 2 This is a flowchart of the deep learning method provided in Embodiment 2 of the present invention;

[0019] Figure 3 This is a schematic diagram of the deep learning convergence relationship provided in Embodiment 2 of the present invention;

[0020] Figure 4 This is a flowchart of the deep learning method provided in Embodiment 3 of the present invention;

[0021] Figure 5 This is a schematic diagram of the deep learning device provided in Embodiment 4 of the present invention;

[0022] Figure 6 This is a schematic diagram of the network device composition provided in Embodiment 5 of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the embodiments of this invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0024] Example 1:

[0025] This embodiment provides a deep learning method; please refer to [reference needed]. Figure 1 The method includes:

[0026] S101. Process the model copies in each GPU of the server to obtain the gradient data of the current batch.

[0027] S102. When the cumulative number of gradient data in the corresponding GPU reaches a preset threshold, the model copy is updated with the cumulative gradient data; wherein, the preset threshold for gradient data corresponding to at least one GPU is greater than or equal to 2.

[0028] The main task of a neural network is to find the optimal parameters (weights and biases) during learning; these optimal parameters are also the parameters that minimize the loss function. However, in general, the loss function is quite complex, with many parameters, making it difficult to determine where the minimum value is found. Therefore, the gradient method is used to find the minimum (or the smallest possible value) through the gradient.

[0029] It's important to note that the gradient represents the direction in which the function value decreases the most at each point, so the gradient direction doesn't necessarily point to the minimum. However, following its direction will minimize the function's value. Therefore, when searching for the location of the function's minimum (or the smallest possible value), we should use the gradient information as a clue to determine the direction to proceed.

[0030] This is where the gradient method comes in handy. In the gradient method, the function's value is moved a certain distance from the current position along the gradient direction, then the gradient is recalculated in the new direction, and so on.

[0031] This process of gradually decreasing the function value by continuously moving along the gradient direction is called the gradient method. Generally speaking, in neural networks (deep learning), the gradient method mainly refers to the gradient descent method.

[0032] The gradient descent algorithm works by moving along the parameters of the objective function L(w). The opposite direction of the gradient ( This involves continuously updating the model parameters (by taking the partial derivative with respect to the vector w) to reach the minimum point of the objective function. There are three main types: Batch Gradient Descent (BGD), Stochastic Gradient Descent (SGD), and Mini-Batch Gradient Descent (MBGD). Their difference lies in the number of samples used in each learning iteration (updating model parameters). Deep learning generally uses Mini-Batch Gradient Descent. The main formula for updating model parameters is generally as follows:

[0033]

[0034] Where λ refers to the update step size, also known as the learning rate, and B refers to the number of samples used in each iteration.

[0035] In this embodiment, during the update iteration process, instead of directly iterating based on each calculated gradient, the calculated gradients are temporarily stored during the training of the model replica. The model replica is not updated directly; instead, after sufficient gradient data has been calculated, iteration is performed on the model replica based on this gradient data. This implementation effectively increases the batch size used in GPU computation, while simultaneously reducing the number of parameter updates and the amount of data in the inter-GPU data aggregation stage, thus improving the computation-to-communication ratio.

[0036] In some embodiments, processing the model copies in each graphics processing unit (GPU) of the server to obtain gradient data for the current batch may include:

[0037] Before the gradient data in the corresponding GPU reaches a preset threshold, the gradient data for the current batch is calculated based on the previously updated model copy and the samples corresponding to the model copy. Before the formal iteration, that is, before updating the model copy, the GPU computation is based on the previously updated model copy, which can also be obtained by accumulating multiple gradient data and then iterating over them.

[0038] In some embodiments, before processing the model copies in each graphics processing unit (GPU) of the server to obtain the gradient data for the current batch, the following may also be included:

[0039] The initial weights of the model copy in the original GPU are sent to all other copies. Here, weights refer to the model's parameters. Before training begins, the model's parameters need to be randomly initialized, which are the initial weights. Each iteration updates the model using the calculated gradients to find the optimal parameters, i.e., the optimal weights. The initial weights of other copies are based on the initial weights of the model copy in the original GPU; that is, the initial parameters used on all GPUs are the same.

[0040] In some embodiments, before updating the model copy with the accumulated gradient data when the cumulative number of gradient data in the corresponding GPU reaches a preset threshold, the method may further include:

[0041] Based on the computing power of each GPU in the server, a preset threshold for gradient data is determined. This preset threshold, for a single deep learning process, may involve multiple servers, and each server may contain multiple GPUs. At least one GPU must have a preset threshold for gradient data greater than or equal to 2. The specific value is determined based on the computing power of the specific GPU, as follows:

[0042] When two GPUs have the same processing power, the preset threshold for gradient data for each GPU is the same. For example, suppose there are two servers, each with 8 GPUs, for a total of 16 GPUs. These 16 GPUs have the same processing power. When training a model, such as the ResNet-50 model, the maximum batch size that can be used on a single GPU is assumed to be 256. When using data parallel synchronous SGD for distributed training, the actual effective batch size is proportional to the number of GPUs used. For example, if 4 GPUs are used, the actual batch size is 1K. When verifying scenarios of 256, 1K, 2K, and 4K in sequence, it was found that the initial learning rate is directly proportional to the batch size. When the global batch size is 4K, all 16 GPUs are already in use. If we want to continue to verify the batch size of 8K, the gradients of two mini-batch data are accumulated on each GPU before the parameter update. At this time, the preset threshold for each GPU is 2. Similarly, for the 16K and 32K scenarios, the gradients of 4 and 8 mini-batch data need to be accumulated on each GPU, respectively. At this time, the preset thresholds are 4 and 8, respectively.

[0043] When two GPUs have different processing capabilities, the preset threshold for gradient data is higher for the GPU with stronger processing power. For example, suppose there is a server with two GPUs, a V100 GPU and a P100 GPU, where the V100 GPU has significantly more computing power than the P100 GPU. Taking the ResNet-50 model as an example, when the batch size during training is 256, the V100 processes more than twice the number of images per second as the P100 does. In this scenario, the V100 can process two batches of data in the time it takes the P100 to process one batch. This means the preset threshold for gradient data for the V100 GPU is twice that for the P100 GPU; for example, the preset threshold for gradient data for the V100 GPU is 2, while the preset threshold for gradient data for the P100 GPU is 1.

[0044] In some embodiments, updating the model copy with accumulated gradient data may include:

[0045] After summing and averaging the gradient data in this GPU, gradient aggregation is performed across GPUs.

[0046] The model replicas on each GPU are updated using the gradient data after gradient aggregation. This method involves each GPU calculating its local average gradient data, provided that the amount of gradient data meets the requirements. Then, each GPU performs gradient aggregation based on its own calculated gradient data to obtain aggregated gradient data, and finally updates the model replicas on each GPU based on this gradient data.

[0047] Specifically, summing and averaging the gradient data in this GPU can include summing and averaging the newly generated gradient data in this GPU after the last updated model copy.

[0048] The equivalent calculation formula can be as follows:

[0049]

[0050] In some embodiments, updating the model copy with accumulated gradient data further includes:

[0051] After summing and averaging the gradient data in this GPU, the model copy in this GPU is updated using the summed and averaged gradient data. In other words, the model copy in this GPU is updated by directly summing and averaging the gradient data calculated by this GPU.

[0052] The deep learning method provided in this invention processes the model replicas on each GPU in a server to obtain gradient data for the current batch. When the cumulative number of gradient data in the corresponding GPU reaches a preset threshold, the model replica is updated with the accumulated gradient data. The preset threshold for gradient data corresponding to at least one GPU is greater than or equal to 2. By accumulating a certain amount of gradient data before updating the model, the actual batch size used during training is increased, while the number of parameter updates is reduced. This breaks free from the limitations of a single GPU's video memory, enabling training with a larger batch size and effectively improving system throughput and efficiency.

[0053] Example 2

[0054] This embodiment provides a deep learning method. When training a ResNet-50 model in a multi-GPU environment, the initial learning rate should be set differently when using different batch sizes to ensure the final convergence accuracy. The general rule of thumb is that the initial learning rate should increase with the increase of the batch size, that is, there is a certain linear relationship between the initial learning rate and the batch size.

[0055] Suppose we have two GPU servers, each using 8 GPUs, for a total of 16 GPUs. When training a ResNet-50 model, the maximum batch size usable on a single GPU is 256. When using distributed training with data-parallel synchronous SGD, the actual effective batch size is proportional to the number of GPUs used. For example, using 4 GPUs results in an actual batch size of 1K. After verifying scenarios of 256, 1K, 2K, and 4K, we found that the initial learning rate is indeed proportional to the batch size. However, when the global batch size is 4K, all 16 GPUs are already utilized. To further verify scenarios with batch sizes of 8K and above, we can accumulate gradients from several batches of data before updating parameters on each GPU. Similarly, in the 8K scenario, we can accumulate 2 gradients; in the 16K and 32K scenarios, we can accumulate 4 and 8 mini-batches of data per GPU, respectively. Taking the 16K scenario as an example, the flowchart is as follows... Figure 2 As shown, the detailed process is as follows:

[0056] S201. A copy of the model is generated on each GPU, and the initial weights of the original GPU copy numbered 0 are broadcast to all other copies. In addition, a local variable and a counter are defined on each GPU to accumulate gradients.

[0057] S202. Within each replica, perform data reading, data preprocessing and data augmentation, load the data onto the corresponding GPU, and perform model calculations and gradient calculations.

[0058] S203. Determine if the current cumulative gradient count has reached 4. If not, continue executing S202. Otherwise, sum and average the 4 accumulated gradients, and then perform gradient aggregation across GPUs.

[0059] S204. Within each replica, update the local model parameters using the aggregated gradient data.

[0060] This allows for model training with batch sizes of 8K, 16K, and 32K using only 16 GPUs. Verification has shown a linear relationship between the initial learning rate and batch size within a certain range. This range refers to batch sizes below 8K; beyond 8K, the convergence accuracy of ResNet-50 deteriorates significantly. Figure 3 As shown.

[0061] Example 3

[0062] This embodiment provides a deep learning method. Assume two different GPUs, such as NVIDIA P100 and V100, were purchased at different times. The newly purchased V100 GPU has significantly higher computing power than the P100 GPU. Taking ResNet-50 as an example, when the batch size during training is 256, the V100 processes more than twice the number of images per second as the P100. If these two GPUs with different computing power are used together for synchronous training, from a computing power perspective, even if the V100 processes two batches of data consecutively, the P100 might not even finish processing one. Therefore, during the time interval between two synchronous operations, the V100 can process two batches of data, while the P100 processes one batch. Please refer to [reference needed]. Figure 4 The process is as follows:

[0063] S401. Generate a copy of the model on P100 and V100 respectively, and broadcast the initial weights of the P100 copy to the V100 copy. Define a local variable and a counter on V100 to accumulate gradients.

[0064] S402. Within each replica, perform data reading, data preprocessing and data augmentation, load the data onto the corresponding GPU, and perform model calculations and gradient calculations.

[0065] S403. On V100, check if the current cumulative gradient count has reached 2. If not, continue with S402. Otherwise, sum the two accumulated gradients, and then perform AllReduce between the two GPUs. [AllReduce is an operation that reduces the target array (representing All) in all processes to a single array (representing Reduce) and returns the result array to all processes. (For example, merging the gradient values ​​on all GPUs, assuming they are represented by an array, performing a reduce operation to form a single array, and returning it to all GPUs)], and divide by the total batch size of the two GPUs, 3, to obtain the final gradient data.

[0066] S404. Within each replica, update the local model parameters using the aggregated gradient data.

[0067] The effect of this is that the extra computing power of V100 compared to P100 is fully utilized and not wasted. The end result is that the data throughput is increased by nearly 50%, and the training speed is also greatly improved.

[0068] Example 4

[0069] This embodiment provides a deep learning device; please refer to [reference needed]. Figure 5 The device includes:

[0070] Processing module 51 is used to process the model copies in each graphics processing unit (GPU) in the server to obtain the gradient data of the current batch.

[0071] The update module 52 is used to update the model copy with the accumulated gradient data when the number of gradient data in the corresponding GPU reaches a preset threshold; wherein, the preset threshold for gradient data corresponding to at least one GPU is greater than or equal to 2.

[0072] Deep learning often employs the gradient descent algorithm. The gradient descent algorithm works by moving along the parameters of the objective function L(w). The opposite direction of the gradient ( This involves continuously updating the model parameters (by taking partial derivatives with respect to the vector w) to reach the minimum point of the objective function. There are three main types: Batch Gradient Descent (BGD), Stochastic Gradient Descent (SGD), and Mini-Batch Gradient Descent (MBGD). Their difference lies in the number of samples used in each learning iteration (updating model parameters). Deep learning generally uses Mini-Batch Gradient Descent.

[0073] In this embodiment, during the update iteration process, instead of directly iterating based on each calculated gradient, the calculated gradients are temporarily stored during the training of the model replica. The model replica is not updated directly; instead, after sufficient gradient data has been calculated, iteration is performed on the model replica based on this gradient data. This implementation effectively increases the batch size used in GPU computation, while simultaneously reducing the number of parameter updates and the amount of data in the inter-GPU data aggregation stage, thus improving the computation-to-communication ratio.

[0074] In some embodiments, processing the model copies in each graphics processing unit (GPU) of the server to obtain gradient data for the current batch may include:

[0075] Before the gradient data in the corresponding GPU reaches a preset threshold, the gradient data for the current batch is calculated based on the previously updated model copy and the samples corresponding to the model copy. Before the formal iteration, that is, before updating the model copy, the GPU computation is based on the previously updated model copy, which can also be obtained by accumulating multiple gradient data and then iterating over them.

[0076] In some embodiments, before processing the model copies in each graphics processing unit (GPU) of the server to obtain the gradient data for the current batch, the following may also be included:

[0077] The initial weights of the model copy in the original GPU are sent to all other copies.

[0078] In some embodiments, before updating the model copy with the accumulated gradient data when the cumulative number of gradient data in the corresponding GPU reaches a preset threshold, the method may further include:

[0079] Based on the computing power of each GPU in the server, a preset threshold for gradient data is determined. This preset threshold, for a single deep learning process, may involve multiple servers, and each server may contain multiple GPUs. The preset threshold for gradient data in at least one GPU is equal to or equal to 2. The specific value is determined based on the computing power of the specific GPU, as follows:

[0080] When the processing power of two GPUs is the same, the preset threshold of the gradient data corresponding to the GPUs is the same.

[0081] When two GPUs have different computing capabilities, the preset threshold for the gradient data corresponding to the GPU with stronger computing capabilities will be larger.

[0082] In some embodiments, updating the model copy with accumulated gradient data may include:

[0083] After summing and averaging the gradient data in this GPU, gradient aggregation is performed across GPUs.

[0084] The model replicas on each GPU are updated using the gradient data after gradient aggregation. This method involves each GPU calculating its local average gradient data, provided that the amount of gradient data meets the requirements. Then, each GPU performs gradient aggregation based on its own calculated gradient data to obtain aggregated gradient data, and finally updates the model replicas on each GPU based on this gradient data.

[0085] Specifically, summing and averaging the gradient data in this GPU can include summing and averaging the newly generated gradient data in this GPU after the last updated model copy.

[0086] In some embodiments, updating the model copy with accumulated gradient data further includes:

[0087] After summing and averaging the gradient data in this GPU, the model copy in this GPU is updated using the summed and averaged gradient data. In other words, the model parameters of this GPU are updated by directly summing and averaging the gradient data calculated by this GPU.

[0088] The deep learning device provided in this invention processes model replicas on each GPU in a server to obtain gradient data for the current batch. When the cumulative number of gradient data in the corresponding GPU reaches a preset threshold, the model replica is updated with the accumulated gradient data. The preset threshold for gradient data corresponding to at least one GPU is greater than or equal to 2. By accumulating a certain amount of gradient data before updating the model, the actual batch size used during training is increased, while the number of parameter updates is reduced. This overcomes the limitations of a single GPU's video memory, enabling training with a larger batch size and effectively improving system throughput and efficiency.

[0089] Example 5

[0090] This embodiment provides a network device; please refer to [reference needed]. Figure 6 The network device includes a processor 61, a memory 62, and a communication bus 63;

[0091] Communication bus 63 is used to realize the connection communication between processor 61 and memory 62;

[0092] The processor 61 is used to execute one or more computer programs stored in the memory 62 to implement the steps of the deep learning methods described in the above embodiments, which will not be repeated here.

[0093] This embodiment also provides a computer-readable storage medium, which includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage network devices, or any other medium that can be used to store desired information and is accessible by a computer.

[0094] The computer-readable storage medium in this embodiment can be used to store one or more computer programs, which can be executed by a processor to implement at least one step of the deep learning method in the above embodiments.

[0095] This embodiment also provides a computer program (or computer software) that can be distributed on a computer-readable medium and executed by a computing network device to implement at least one step of the deep learning method in the above embodiments.

[0096] This embodiment also provides a computer program product, including a computer-readable network device, on which the computer program as shown above is stored. In this embodiment, the computer-readable network device may include the computer-readable storage medium as shown above.

[0097] Therefore, those skilled in the art should understand that all or some of the steps in the methods disclosed above, the network device, and the functional modules / units in the network device can be implemented as software (which can be implemented using computer program code executable by the network device), firmware, hardware, and suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as integrated circuits, such as application-specific integrated circuits (ASICs).

[0098] Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, computer program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. Therefore, this invention is not limited to any particular combination of hardware and software.

[0099] The above description, in conjunction with specific implementation methods, provides a further detailed explanation of the embodiments of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A deep learning method, comprising: performing data processing on a model copy in each graphics processing unit (GPU) in a server to obtain gradient data of a current batch; updating the model copy with the accumulated gradient data when the number of the gradient data accumulated in the corresponding GPU reaches a preset threshold; wherein the preset threshold of the gradient data corresponding to at least one GPU is greater than or equal to 2; before the updating of the model copy with the accumulated gradient data when the number of the gradient data accumulated in the corresponding GPU reaches the preset threshold, further comprising: determining the preset threshold of the gradient data according to the operation processing capacity of each GPU in the server; wherein when the operation processing capacities of two GPUs are the same, the preset threshold of the gradient data corresponding to the GPUs is the same; when the operation processing capacities of two GPUs are different, the preset threshold corresponding to the GPU with stronger operation processing capacity is greater in the preset thresholds of the gradient data corresponding to the GPUs.

2. The deep learning method of claim 1, wherein, the performing of the data processing on the model copy in each GPU in the server to obtain the gradient data of the current batch comprises: before the gradient data in the corresponding GPU reaches the preset threshold, calculating the gradient data of the current batch according to the model copy updated last time and the sample corresponding to the model copy.

3. The deep learning method of claim 1, wherein, before the performing of the data processing on the model copy in each GPU in the server to obtain the gradient data of the current batch, further comprising: sending the initial weight of the model copy in the original GPU to all other copies.

4. The deep learning method of any one of claims 1-3, wherein, the updating of the model copy with the accumulated gradient data comprises: summing and averaging each of the gradient data in the GPU, and then performing gradient aggregation between the GPUs; updating the model copy in each GPU with the gradient data after the gradient aggregation.

5. The deep learning method of claim 4, wherein, the summing and averaging of each of the gradient data in the GPU comprises: summing and averaging each of the gradient data newly generated in the GPU after the model copy updated last time.

6. The deep learning method of any one of claims 1-3, wherein, the updating of the model copy with the accumulated gradient data comprises: updating the model copy in the GPU with the gradient data after the summing and averaging of each of the gradient data in the GPU. 7.A deep learning device, comprising: a processing module configured to perform data processing on a model copy in each graphics processing unit (GPU) in a server to obtain gradient data of a current batch; an updating module configured to update the model copy with the accumulated gradient data when the number of the gradient data accumulated in the corresponding GPU reaches a preset threshold; wherein the preset threshold of the gradient data corresponding to at least one GPU is greater than or equal to 2. The device is further configured to determine the preset threshold of the gradient data according to the operation processing capability of each GPU in the server before updating the model copy with the accumulated gradient data when the accumulated number of the gradient data in the corresponding GPU reaches the preset threshold; when the operation processing capability of two GPUs is the same, the preset threshold of the gradient data corresponding to the GPUs is the same; when the operation processing capability of two GPUs is different, the preset threshold corresponding to the GPU with stronger operation capability is larger in the preset threshold of the gradient data corresponding to the GPUs. 8.A network device, comprising a processor, a memory and a communication bus; The communication bus is configured to connect and communicate between the processor and the memory; The processor is configured to execute one or more computer programs stored in the memory to implement the steps of the deep learning method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores one or more computer programs executable by one or more processors to implement the steps of the deep learning method according to any one of claims 1-6.

Images