Neural network model training method and electronic equipment

Abstract

The invention provides a neural network model training method and electronic equipment. The training method executed by the first processor comprises the steps that parameter values of weight parameters of a neural network model are acquired for the tth batch of training samples, and the weight parameters of the neural network model comprise a first parameter set and a second parameter set based on gradient values; on the basis of the t-th batch of training samples, forward propagation and reverse propagation are carried out by using the obtained parameter values of the weight parameters, and gradient values of the weight parameters are determined; based on the determined gradient value of the weight parameter in the first parameter set, updating the parameter value of the weight parameter in the first parameter set; and sending the determined gradient value of the weight parameter in the second parameter set to a second processor configured to update the parameter value of the weight parameter in the second parameter set, the computing power of the first processor being higher than the computing power of the second processor.

Description

Technical Field

[0001] This disclosure generally relates to the field of neural network and processor technology, and more specifically, to a method for training a neural network model and an electronic device. Background Technology

[0002] In recent years, with the increasing complexity of neural network models, both the number of parameters in these models and the memory capacity of AI accelerators have grown dramatically. Consequently, the memory required for neural network model training has gradually exceeded the memory capacity of AI accelerators, making memory a major challenge in neural network model training. For example, the number of parameters in large Transformer models has increased exponentially by approximately 410 times every two years, while the memory capacity of a single parallel processing unit (such as a graphics processing unit (GPU)) has only doubled every two years. Summary of the Invention

[0003] The exemplary embodiments of this disclosure provide a method and electronic device for training a neural network model, which can reduce the memory requirements of the first processor for training the neural network model and improve training efficiency.

[0004] According to an example embodiment of this disclosure, a method for training a neural network model is provided. The training method is executed by a first processor, wherein the weight parameters of the neural network model include a first parameter set and a second parameter set based on the magnitude of gradient values. The training method includes: obtaining parameter values ​​of the weight parameters of the neural network model for a batch t of training samples; determining the gradient values ​​of the weight parameters by performing forward and backward propagation using the obtained parameter values ​​of the weight parameters based on the batch t of training samples; updating the parameter values ​​of the weight parameters in the first parameter set based on the determined gradient values ​​of the weight parameters in the first parameter set; and sending the determined gradient values ​​of the weight parameters in the second parameter set to a second processor configured to update the parameter values ​​of the weight parameters in the second parameter set, wherein the computing power of the first processor is higher than that of the second processor. The step of obtaining the parameter values ​​of the weight parameters of the neural network model includes: loading the parameter values ​​of the weight parameters in the first parameter set; receiving the updated parameter values ​​of the weight parameters in the second parameter set from the second processor; and loading the unupdated parameter values ​​of the weight parameters in the second parameter set.

[0005] The method may further include: determining the idle time during the first training process based on the (t-1)th batch of training samples when the first processor is idle. Based on a determined idle time Determine the parameter update threshold used in the second training process based on the t-th batch of training samples; send the determined parameter update threshold to the second processor, wherein the second processor is also configured to update the parameter values ​​of the weight parameters in the second parameter set based on the determined parameter update threshold.

[0006] The step of determining the parameter update threshold used in the second training process based on the t-th batch of training samples may include: determining the idle time. Is it greater than 0? Response time If the value is greater than 0, update the threshold and idle time based on the parameters used in the first training process. and free time Determine the parameter update threshold used in the second training process; respond to idle time. The value is equal to 0. Based on the parameter update threshold used in the first training process, the parameter update threshold used in the second training process is determined, where idle time is included. This represents the idle time during the third training process based on the (t-2)th batch of training samples, when the first processor is idle.

[0007] The method may further include: sorting the weight parameters in the first parameter set in ascending order of their gradient values ​​to obtain a first sorting result; sorting the weight parameters in the second parameter set in descending order of their gradient values ​​to obtain a second sorting result; comparing the gradient values ​​of the first N weight parameters in the first sorting result with the gradient values ​​of the first N weight parameters in the second sorting result; and updating the first parameter set and the second parameter set based on the comparison result.

[0008] The step of comparing the gradient values ​​of the first N weight parameters in the first sorting result with the gradient values ​​of the first N weight parameters in the second sorting result may include: comparing the gradient value of the i-th weight parameter in the first sorting result with the gradient value of the i-th weight parameter in the second sorting result, where the initial value of i is 1; wherein the step of updating the first parameter set and the second parameter set according to the comparison result includes: in response to the gradient value of the i-th weight parameter in the second sorting result being greater than the gradient value of the i-th weight parameter in the first sorting result, deleting the i-th weight parameter in the second sorting result from the second parameter set and adding the deleted i-th weight parameter in the second sorting result to the first parameter set, deleting the i-th weight parameter in the first sorting result from the first parameter set and adding the deleted i-th weight parameter in the first sorting result to the second parameter set, and determining whether i is equal to N; and in response to i not being equal to N, adding 1 to i, and repeating the step of comparing the gradient value of the i-th weight parameter in the first sorting result with the gradient value of the i-th weight parameter in the second sorting result until i is equal to N.

[0009] According to at least one embodiment of this disclosure, a method for training a neural network model is provided. The training method is executed by a second processor. The training method includes: receiving gradient values ​​of weight parameters in a second parameter set determined based on a batch t of training samples from a first processor, wherein the computing power of the first processor is higher than that of the second processor; obtaining a merged gradient value of each weight parameter in the second parameter set by merging a set of accumulated gradient values ​​with the received gradient values ​​of the weight parameters; updating the set of accumulated gradient values ​​when the merged gradient value is less than a parameter update threshold; updating the parameter value of each weight parameter in the second parameter set whose merged gradient value is greater than or equal to the parameter update threshold; and sending the updated parameter value of the weight parameters to the first processor.

[0010] The steps for obtaining the gradient merged value of each weight parameter in the second parameter set may include: for each weight parameter in the second parameter set, determining whether the gradient accumulation value set contains the gradient accumulation value of that weight parameter; if the gradient accumulation value set contains the gradient accumulation value of that weight parameter, then taking the larger value between the received gradient value of that weight parameter and the gradient accumulation value of that weight parameter as the gradient merged value of that weight parameter; if the gradient accumulation value set does not contain the gradient accumulation value of that weight parameter, then taking the received gradient value of that weight parameter as the gradient merged value of that weight parameter.

[0011] When the gradient merge value is less than the parameter update threshold, the step of updating the gradient accumulation set may include: for each weight parameter in the second parameter set whose gradient merge value is less than the parameter update threshold, determining whether the gradient accumulation set contains the gradient accumulation value of that weight parameter; if the gradient accumulation set contains the gradient accumulation value of that weight parameter, then using the sum of the gradient merge value of that weight parameter and the gradient accumulation value of that weight parameter as the updated gradient accumulation value of that weight parameter; if the gradient accumulation set does not contain the gradient accumulation value of that weight parameter, then adding the gradient merge value of that weight parameter as the gradient accumulation value of that weight parameter to the gradient accumulation set; and deleting the unupdated gradient accumulation value of the weight parameter from the gradient accumulation set.

[0012] The method may further include: before sending the updated parameter values, converting the data type of the updated weight parameter values ​​from a first data type to a second data type, wherein the precision of the first data type is higher than that of the second data type.

[0013] According to at least one embodiment of this disclosure, an electronic device is provided, the electronic device comprising: a first processor; a second processor; and at least one memory storing computer-executable instructions; wherein, when executed by the electronic device, the computer-executable instructions are configured to instruct the first processor to perform the following operations: for a batch t of training samples, obtaining parameter values ​​of weight parameters of a neural network model, wherein the weight parameters of the neural network model include a first parameter set and a second parameter set based on the magnitude of gradient values; determining gradient values ​​of weight parameters based on the batch t of training samples by performing forward and backward propagation using the obtained parameter values ​​of weight parameters; updating the parameter values ​​of weight parameters in the first parameter set based on the determined gradient values ​​of weight parameters in the first parameter set; sending the determined gradient values ​​of weight parameters in the second parameter set to the second processor, and wherein the computing power of the first processor is higher than that of the second processor, and wherein the parameter values ​​of weight parameters of the neural network model are obtained by: loading the parameter values ​​of weight parameters in the first parameter set; receiving updated parameter values ​​of weight parameters in the second parameter set from the second processor; and loading unupdated parameter values ​​of weight parameters in the second parameter set.

[0014] When executed by the electronic device, the computer-executable instructions can also be configured to instruct the first processor to perform the following operation: determine the idle time during the first training process based on the (t-1)th batch of training samples when the first processor is idle. Based on a determined idle time Determine the parameter update threshold used in the second training process based on the t-th batch of training samples; send the determined parameter update threshold to the second processor, wherein the second processor is also configured to update the parameter values ​​of the weight parameters in the second parameter set based on the determined parameter update threshold.

[0015] The parameter update threshold used in the second training process based on the t-th batch of training samples can be determined by the following steps: Determine the idle time. Is it greater than 0? Response time If the value is greater than 0, update the threshold and idle time based on the parameters used in the first training process. and free time Determine the parameter update threshold used in the second training process; respond to idle time. The value is equal to 0. Based on the parameter update threshold used in the first training process, the parameter update threshold used in the second training process is determined, where idle time is included. This represents the idle time during the third training process based on the (t-2)th batch of training samples, when the first processor is idle.

[0016] When executed by the electronic device, the computer-executable instructions can also be configured to instruct the first processor to perform the following operations: sorting the weight parameters in the first parameter set in ascending order of their gradient values ​​to obtain a first sorting result; sorting the weight parameters in the second parameter set in descending order of their gradient values ​​to obtain a second sorting result; comparing the gradient values ​​of the first N weight parameters in the first sorting result with the gradient values ​​of the first N weight parameters in the second sorting result; and updating the first parameter set and the second parameter set based on the comparison result.

[0017] The gradient values ​​of the first N weight parameters in the first sorting result can be compared with the gradient values ​​of the first N weight parameters in the second sorting result through the following operation: The gradient value of the i-th weight parameter in the first sorting result is compared with the gradient value of the i-th weight parameter in the second sorting result, where the initial value of i is 1; wherein the first and second parameter sets are updated according to the comparison result through the following operation: In response to the gradient value of the i-th weight parameter in the second sorting result being greater than the gradient value of the i-th weight parameter in the first sorting result, the i-th weight parameter in the second sorting result is deleted from the second parameter set and the deleted i-th weight parameter in the second sorting result is added to the first parameter set; the i-th weight parameter in the first sorting result is deleted from the first parameter set and the deleted i-th weight parameter in the first sorting result is added to the second parameter set, and it is determined whether i is equal to N; and in response to i not being equal to N, 1 is added to i, and the step of comparing the gradient value of the i-th weight parameter in the first sorting result with the gradient value of the i-th weight parameter in the second sorting result is repeated until i is equal to N.

[0018] When executed by the electronic device, the computer-executable instructions can instruct the second processor to perform the following operations: receive gradient values ​​of weight parameters in a determined second parameter set from the first processor; obtain a combined gradient value for each weight parameter in the second parameter set by merging the gradient accumulation value set with the received gradient values ​​of the weight parameters; update the gradient accumulation value set when the combined gradient value is less than a parameter update threshold; update the parameter value of each weight parameter in the second parameter set whose combined gradient value is greater than or equal to the parameter update threshold; and send the updated parameter values ​​of the weight parameters to the first processor.

[0019] The gradient merge value of each weight parameter in the second parameter set can be obtained through the following operations: For each weight parameter in the second parameter set, determine whether the gradient accumulation value set contains the gradient accumulation value of that weight parameter; if the gradient accumulation value set contains the gradient accumulation value of that weight parameter, then take the larger value between the received gradient value of that weight parameter and the gradient accumulation value of that weight parameter as the gradient merge value of that weight parameter; if the gradient accumulation value set does not contain the gradient accumulation value of that weight parameter, then take the received gradient value of that weight parameter as the gradient merge value of that weight parameter.

[0020] When the gradient merge value is less than the parameter update threshold, the gradient accumulation set can be updated as follows: For each weight parameter in the second parameter set whose gradient merge value is less than the parameter update threshold, determine whether the gradient accumulation set contains the gradient accumulation value of that weight parameter; if the gradient accumulation set contains the gradient accumulation value of that weight parameter, then the sum of the gradient merge value of that weight parameter and the gradient accumulation value of that weight parameter is used as the updated gradient accumulation value of that weight parameter; if the gradient accumulation set does not contain the gradient accumulation value of that weight parameter, then the gradient merge value of that weight parameter is added to the gradient accumulation set as the gradient accumulation value of that weight parameter; and finally, delete the unupdated gradient accumulation values ​​of the weight parameters from the gradient accumulation set.

[0021] When the computer-executable instructions are executed by the electronic device, they can also be configured to instruct the second processor to perform the following operation: before sending the updated parameter value, convert the data type of the updated weight parameter value from a first data type to a second data type, wherein the precision of the first data type is higher than that of the second data type.

[0022] According to at least one embodiment of the present disclosure, an electronic device is provided, the electronic device comprising: a first processor; a second processor; and at least one memory storing computer-executable instructions; wherein, when the computer-executable instructions are executed by the first processor and the second processor, they cause the first processor and the second processor to execute a training method for a neural network model as described above.

[0023] According to at least one embodiment of the present disclosure, a computer-readable storage medium is provided that stores instructions which, when executed by a first processor and a second processor of an electronic device, enable the electronic device to perform a training method for a neural network model as described above.

[0024] According to at least one embodiment of the present disclosure, a computer program product is provided, including computer-executable instructions, wherein when the computer-executable instructions are executed by at least one processor, they implement the training method of a neural network model executed by a first processor as described above and / or the training method of a neural network model executed by a second processor as described above.

[0025] The training method and electronic device for a neural network model according to an example embodiment of the present disclosure can reduce the access frequency of the first processor memory for training the neural network model, thereby improving training efficiency.

[0026] In the following description, some aspects and / or advantages of the general concept of this disclosure will be set forth, and other aspects and / or advantages will become apparent from the following description or from practice of the general concept of this disclosure. Attached Figure Description

[0027] These and / or other aspects and advantages of this application will become clearer and more readily understood from the following detailed description of embodiments of this application taken in conjunction with the accompanying drawings, wherein:

[0028] Figure 1 A flowchart illustrating a method for training a neural network model executed by a first processor according to at least one example embodiment of the present disclosure;

[0029] Figure 2 A flowchart illustrating a method for training a neural network model executed by a second processor according to at least one example embodiment of the present disclosure;

[0030] Figure 3 A flowchart illustrating a method for dynamically adjusting parameters to update a threshold according to at least one example embodiment of the present disclosure;

[0031] Figure 4 A flowchart illustrating a method for updating a first parameter set and a second parameter set according to at least one example embodiment of the present disclosure;

[0032] Figure 5 An example of dividing a first parameter set and a second parameter set according to at least one example embodiment of the present disclosure is shown;

[0033] Figure 6 A flowchart illustrating a training method for a neural network model according to at least one example embodiment of the present disclosure;

[0034] Figure 7 Examples of timing for each module according to at least one exemplary embodiment of this disclosure are shown;

[0035] Figure 8Examples of dynamically adjusting parameter update thresholds and updating a first parameter set and a second parameter set according to at least one example embodiment of the present disclosure are shown;

[0036] Figure 9 Examples of updating the first parameter set and the second parameter set according to at least one example embodiment of the present disclosure are shown;

[0037] Figure 10 Examples of gradient merging according to at least one exemplary embodiment of the present disclosure are shown;

[0038] Figure 11 An example of gradient accumulation according to at least one exemplary embodiment of this disclosure is shown;

[0039] Figure 12 Examples of gradient merging and gradient accumulation according to at least one exemplary embodiment of the present disclosure are shown;

[0040] Figure 13 A schematic diagram illustrating the effect of a training method for a neural network model according to at least one example embodiment of the present disclosure;

[0041] Figure 14 A structural block diagram of an electronic device according to at least one example embodiment of the present disclosure is shown. Detailed Implementation

[0042] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, examples of which are illustrated in the drawings, wherein the same reference numerals always refer to the same parts. The embodiments will now be described with reference to the accompanying drawings in order to explain this disclosure.

[0043] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following example embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0044] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "the entirety of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which includes the following three parallel cases: (1) performing step one; (2) performing step two; and (3) performing both step one and step two.

[0045] There are three main types of training schemes for large neural network models: parallel training, parameter offloading, and computation offloading. In parallel training schemes (e.g., multi-GPU training), each GPU stores only 1 / Z (where Z is the number of GPUs) of the model parameters, and communication between GPUs occurs through a broadcast mechanism. Parameter offloading schemes involve offloading the model parameters to a large-capacity storage device and completing training through prefetching. Computation offloading schemes, for example, offload the model parameters to a large-capacity storage device, while simple computations such as weight updates are handled by the CPU, and training is completed through step-by-step prefetching.

[0046] Parallel training schemes offer the advantage of sufficient bandwidth and throughput. However, broadcast mechanisms require complex software configurations, and the maintenance costs across multiple GPUs are high. Furthermore, GPUs are more expensive than CPUs and storage devices, which increases the cost of network training.

[0047] The parameter offloading scheme does not require complex software configuration and has low maintenance costs. However, the slow transfer speed between the GPU and storage devices, even when using prefetching, leads to high latency and low training efficiency.

[0048] Compute offloading schemes place some simple calculations on the CPU, using the CPU as an extended computing unit of the GPU, and accelerate computation through stepwise prefetching, which is more effective than parameter offloading schemes. However, the mismatch between CPU and GPU computing power can cause significant GPU latency. Furthermore, the large amount of data transfer between the CPU and storage devices typically leads to high latency.

[0049] In summary, the above solutions mainly suffer from two types of problems: input / output (I / O) bottlenecks and / or computational bottlenecks. The I / O bottleneck is primarily caused by the mismatch in data transfer speeds between the GPU and low-performance devices (e.g., CPUs) and storage devices; while the computational bottleneck is mainly caused by the mismatch in computational power between the GPU and low-performance devices. Therefore, this disclosure primarily addresses the I / O and computational bottlenecks to improve the training speed and efficiency of large neural network models. Specifically, the following solutions are proposed.

[0050] (1) Considering the sparsity of weight parameters, which is determined by the degree of gradient change, most weight parameters have small gradient values. This disclosure proposes that weight parameters be divided into hot and cold weight parameters. Hot weight parameters are defined as more active weight parameters (with larger gradient values), and other less active weight parameters are defined as cold weight parameters. The update operation of cold weight parameters can be offloaded from the first processor to the second processor.

[0051] (2) Considering that most weight parameters will remain in a hot / cold state over time, with only a few weight parameters alternating between hot and cold states, this disclosure proposes that the cost of exchanging hot and cold weight parameters is small, and they can be placed between the first processor and the second processor to reduce I / O costs.

[0052] (3) Considering that the mini-batch training method has been shown to have a negligible impact on training accuracy due to gradient accumulation and merging updates, this disclosure proposes that cold weight parameters be updated based on gradient accumulation. Cold weight parameters with larger merged gradients are updated immediately, while cold weight parameters with smaller merged gradients continue to accumulate gradients for later updates, thereby reducing computational cost.

[0053] The following will combine Figures 1 to 14 This disclosure describes the proposed method for accelerating neural network training based on weight sparsity.

[0054] Figure 1 A flowchart illustrating a training method for a neural network model executed by a first processor according to at least one example embodiment of the present disclosure; and Figure 2 A flowchart illustrating a method for training a neural network model executed by a second processor according to at least one example embodiment of the present disclosure.

[0055] The computing power of the first processor is higher than (or "greater than") that of the second processor. In at least one example embodiment, the first processor may be a parallel processor (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), etc.), and the second processor may be a near-memory processor, such as CXL-PNM (CMM-DC). PNM (Processing-near-Memory) technology based on the CXL (Compute Express Link) interface has proven to be an excellent solution for handling "high-capacity" AI models. By placing low-computing-power devices (e.g., central processing units (CPUs)) closer to memory, CMM-DC technology reduces data movement between the CPU and memory, thereby doubling the loading speed and increasing the capacity of artificial intelligence (AI) models by four times. Weight updates are a relatively simple task, but they involve a large amount of data movement (e.g., moving data between the CPU and memory via a bus). By offloading this computational task to CMM-DC, the frequency of data movement can be reduced, and the efficiency of training can be improved.

[0056] Neural network models can, for example, have trainable structures (e.g., using training data), such as artificial neural networks, decision trees, support vector machines, Bayesian networks, genetic algorithms, etc. Non-limiting examples of trainable structures can include convolutional neural networks (CNNs), generative adversarial networks (GANs), artificial neural networks (ANNs), region-based convolutional neural networks (R-CNNs), region proposal networks (RPNs), recurrent neural networks (RNNs), stacked deep neural networks (S-DNNs), state-space dynamic neural networks (S-SDNNs), deconvolutional networks, deep belief networks (DBNs), restricted Boltzmann machines (RBMs), fully convolutional networks, long short-term memory networks (LSTMs), classification networks, etc. In at least one example embodiment, the neural network model can be a large language model (LLM). A large language model is an AI system configured to understand and generate human language using deep learning methods by processing large amounts of unlabeled language data (such as text, audio, etc.).

[0057] Reference Figure 1 In step S101, the first processor obtains the parameter values ​​of the weight parameters of the neural network model for the t-th batch of training samples.

[0058] The weight parameters of a neural network model consist of a first set of parameters and a second set of parameters based on the magnitude of the gradient values. In other words, all weight parameters of a neural network model are divided into a first set of parameters and a second set of parameters according to the magnitude of their gradient values. Specifically, the weight parameters in the first set are the hot weight parameters with larger gradient values ​​(e.g., those accessed more frequently; and hereinafter also referred to as Hot Weights), and the weight parameters in the second set are the cold weight parameters with smaller gradient values ​​(e.g., those accessed less frequently; and hereinafter also referred to as Cold Weights).

[0059] In at least one example embodiment, the first processor is configured to initially partition the first and second parameter sets by sorting all weight parameters in descending order of gradient values; then, allocating the first M weight parameters to the first parameter set and the remaining weight parameters to the second parameter set. As an example, the specific value of M can be determined based on the computing power and memory of the first processor.

[0060] In at least one example embodiment, the first parameter set and the second parameter set are not fixed after the initial partitioning; the first processor can be configured to dynamically update the first parameter set and the second parameter set. This will be discussed in conjunction with... Figure 4 Provide a detailed description.

[0061] Therefore, in at least one example embodiment, step S101 includes: the first processor loading parameter values ​​of weight parameters in the first parameter set for the t-th batch of training samples; receiving updated parameter values ​​of weight parameters in the second parameter set from the second processor; and loading parameter values ​​of unupdated weight parameters in the second parameter set. Specifically, for the t-th batch of training samples, cold weight parameters (hereinafter referred to as Other Cold Weights) whose parameter values ​​have not been received from the second processor indicate that their parameter values ​​have not been updated, and their parameter values ​​are loaded from the memory of the first processor. Accordingly, the loaded parameter values ​​of Other Cold Weights are the parameter values ​​used for the (t-1)-th batch of training samples.

[0062] As an example, loading by the first processor can be loading from the memory of the first processor (e.g., the video memory of the GPU). As an example, receiving parameter values ​​of a portion of the weight parameters in the second parameter set from the second processor can be: prefetching parameter values ​​of a portion of the weight parameters in the second parameter set from the second processor.

[0063] In step S102, the first processor determines the gradient value of the weight parameters by performing forward propagation (FP) and backward propagation (BP) using the obtained weight parameter values ​​based on the t-th batch of training samples.

[0064] In step S103, the first processor updates the parameter values ​​of the weight parameters in the first parameter set based on the gradient values ​​of the weight parameters in the determined first parameter set. In other words, the first processor updates the parameter values ​​of the hot weight parameters based on the gradient values ​​of the determined hot weight parameters.

[0065] As an example, the first processor can update the parameter value of the hot weight parameter based on the gradient value of each hot weight parameter and the parameter values ​​of the optimizer parameters (e.g., Momentum and Variance).

[0066] In step S104, the first processor sends the gradient values ​​of the determined weight parameters in the second parameter set to the second processor, so that the second processor can update the parameter values ​​of the weight parameters in the second parameter set. In other words, the first processor sends the gradient values ​​of the determined cold weight parameters to the second processor, enabling the second processor to use the gradient values ​​of the determined cold weight parameters to update the parameter values ​​of the cold weight parameters.

[0067] Reference Figure 2 In step S201, the second processor receives from the first processor the gradient values ​​of the weight parameters in the second parameter set determined based on the t-th batch of training samples.

[0068] In step S202, the second processor merges the gradient accumulation set with the gradient values ​​of the received weight parameters to obtain the merged gradient values ​​of each weight parameter in the second parameter set.

[0069] In at least one example embodiment, step S202 may include: for each weight parameter in the second parameter set, determining whether the gradient accumulation value set contains the gradient accumulation value of the weight parameter; if the gradient accumulation value set contains the gradient accumulation value of the weight parameter, then taking the larger of the received gradient value of the weight parameter and the gradient accumulation value of the weight parameter as the gradient merged value of the weight parameter; if the gradient accumulation value set does not contain the gradient accumulation value of the weight parameter, then taking the received gradient value of the weight parameter as the gradient merged value of the weight parameter.

[0070] Combination Figure 10 and Figure 11 G Cold G represents the gradient values ​​of the weight parameters in the newly received second parameter set from the first processor (i.e., the gradient values ​​of the cold weight parameters). Acc Let G represent the set of gradient accumulation values. Mer This represents the gradient pooling value of the cold weight parameters. If G Acc If the gradient accumulation value includes any cold weight parameter (e.g., weight parameter 7 and weight parameter 8), then the larger of the gradient value of that cold weight parameter and the gradient accumulation value of that cold weight parameter is used as the gradient merged value of that cold weight parameter. For example, G... 8c and G 8a The larger value among them is used as the gradient merging value G of the weight parameter 8. 8m , will G 7c and G 7a The larger value among them is used as the gradient merging value G of the weight parameter 7. 7m If G Acc If the gradient accumulation value does not include any cold weight parameter (e.g., weight parameter 1, weight parameter 2, weight parameter 6), then the gradient value of that cold weight parameter is directly used as the combined gradient value of that cold weight parameter. For example, G... 6c The gradient merging value G, which is the weight parameter 6, is used as the gradient merging value. 6m G 2c The gradient merging value G, which is the weight parameter 2. 2m G 1c The gradient merging value G, which is the weight parameter 1 1m Furthermore, since weight parameters 3 and 5 were previously cold weight parameters, but are now non-cold weight parameters (so G...), AccIt also includes the cumulative gradient values ​​of weight parameter 3 and weight parameter 5, but the gradient values ​​of the currently received cold weight parameters do not include the gradient values ​​of weight parameter 3 and weight parameter 5. Therefore, there is no need to calculate the combined gradient values ​​of weight parameter 3 and weight parameter 5.

[0071] In step S203, the second processor updates the gradient accumulation set when the merged gradient value is less than the parameter update threshold. Specifically, the second processor can update the gradient accumulation set based on the gradient value being less than the parameter update threshold (which can also be expressed as τ in the following embodiments). t The gradient merged value is used to update the gradient accumulation set.

[0072] In at least one example embodiment, step S203 may include: for each weight parameter in the second parameter set whose gradient merge value is less than the parameter update threshold, determining whether the gradient accumulation value set contains the gradient accumulation value of the weight parameter; if the gradient accumulation value set contains the gradient accumulation value of the weight parameter, then using the sum of the gradient merge value of the weight parameter and the gradient accumulation value of the weight parameter as the updated gradient accumulation value of the weight parameter (to replace the original gradient accumulation value of the weight parameter in the gradient accumulation value set); if the gradient accumulation value set does not contain the gradient accumulation value of the weight parameter, then adding the gradient merge value of the weight parameter as the gradient accumulation value of the weight parameter to the gradient accumulation value set; and deleting the gradient accumulation value of the weight parameter that has not been updated in this step from the gradient accumulation value set.

[0073] Combination Figures 10 to 12 G ＜thr G represents the cold weight parameter whose gradient merging value is less than the parameter update threshold. Acc G represents the set of accumulated gradient values ​​before the update. Acc (new) represents the updated set of gradient accumulation values. If G Acc If the gradient accumulation value of any cold weight parameter (e.g., weight parameter 8) is less than the parameter update threshold, then the sum of the merged gradient value of that cold weight parameter and its cumulative gradient value is used as the updated cumulative gradient value of that cold weight parameter. For example, G... 8＜ and G 8a The sum of G 8an This serves as the new gradient accumulation value for the weight parameter 8. If G Acc If there is no gradient accumulation value for any cold weight parameter (e.g., weight parameter 1, weight parameter 2) that is less than the parameter update threshold, then the combined gradient value of that cold weight parameter is directly used as the gradient accumulation value of that cold weight parameter. For example, G... 2＜ The gradient accumulation value G, which is the weight parameter 2 2an Add to G Acc G 1＜ The gradient accumulation value G, which is the weight parameter 11an Add to G Acc Finally, from G Acc Delete the gradient accumulation values ​​of the weight parameters that were not updated in this operation (e.g., G). 3a G 7a G 5a ), thus obtaining G Acc (new). Specifically, on the one hand, since weight parameters 3 and 5 were previously cold weight parameters, but are now non-cold weight parameters (so G... Acc It also includes the cumulative gradient values ​​of weight parameters 3 and 5 (but the gradient values ​​of the currently received cold weight parameters do not include the gradient values ​​of weight parameters 3 and 5), therefore, from G Acc Delete the accumulated gradient values ​​of weight parameters 3 and 5; additionally, since the merged gradient value of weight parameter 7 is greater than or equal to the parameter update threshold, it is also removed from G. Acc Remove the gradient accumulation value of weight parameter 7.

[0074] As a preferred embodiment, the parameter update threshold is not fixed and can be dynamically adjusted. For example, the first processor can dynamically adjust the parameter update threshold based on the idle time it spends during training (specifically, idle due to waiting for data transfer with the second processor and / or the second processor processing data). This will be discussed in conjunction with... Figure 3 Provide a detailed description.

[0075] In step S204, the second processor updates the parameter value of each weight parameter in the second parameter set whose gradient merge value is greater than or equal to the parameter update threshold. Specifically, for each weight parameter in the second parameter set whose gradient merge value is greater than or equal to the parameter update threshold, the parameter value of the weight parameter can be updated based on the gradient merge value of the weight parameter.

[0076] As an example, the value of the weight parameter can be updated based on the gradient merging value of the weight parameter and the parameter values ​​of the optimizer parameters (e.g., Momentum and Variance).

[0077] Combination Figures 10 to 12 G ≥thr This represents the cold weight parameters whose gradient merge value is greater than or equal to the parameter update threshold. Since the gradient merge value of weight parameter 7 and weight parameter 6 is greater than or equal to the parameter update threshold, the gradient merge value G based on weight parameter 7 is... 7m Update the parameter value of weight parameter 7, and combine the gradients based on the gradient merge value G of weight parameter 6. 6m Update the value of weight parameter 6.

[0078] In step S205, the second processor sends the updated weight parameter values ​​to the first processor.

[0079] That is, the updated values ​​of the cold weight parameters (e.g., weight parameter 7 and weight parameter 6) will be sent to the first processor for example for the (t+1)th batch of training samples; while the values ​​of the cold weight parameters (i.e., Other Cold Weight) whose gradient merging values ​​are less than the parameter update threshold will not be updated.

[0080] According to some example embodiments of this disclosure, step S205 may include: converting the data type of the updated weight parameter value from a first data type to a second data type (e.g., Float2half), wherein the precision of the first data type is higher than that of the second data type; and then sending the weight parameter value with the converted data type to a first processor. As an example, the first data type may be float32 (e.g., a 32-bit floating-point number), and the second data type may be float16 (e.g., a 16-bit floating-point number).

[0081] According to some exemplary embodiments of this disclosure, such as Figure 5 As shown, the cold and hot weight parameters are divided according to the gradient value (the weight parameters in the first parameter set are the hot weight parameters W). h The weight parameters in the second parameter set are the cold weight parameters W. c Hot weight parameter updates are performed by the GPU; cold weight parameter updates are performed by CMM-DC. Gradient accumulation and gradient merging are performed on cold weight parameters, and only cold weight parameters whose gradient merging value exceeds a threshold are updated. The updated cold weight parameters are then sent to the GPU to complete forward propagation, thereby freeing up more I / O and computing resources.

[0082] Figure 6 A flowchart illustrating a training method for a neural network model according to at least one example embodiment of the present disclosure is provided. (Refer to...) Figure 6 See Table 1.

[0083] ① Prefetch Cold Weight 16 (e.g., the parameter value of the cold weight parameter with 16-bit floating-point numbers) into the GPU, and load Hot Weight 32 (e.g., the parameter value of the cold weight parameter with 32-bit floating-point numbers).

[0084] ② Calculate the Activation 16 of BP (e.g., the neuron activation value with 16-bit floating-point numbers).

[0085] ③Update the parameter value of the hot weight parameter with Hot Gradient 16 (e.g., the gradient value of the hot weight parameter with 16-bit floating-point numbers), and transmit Cold Gradient 16 (e.g., the gradient value of the cold weight parameter with 16-bit floating-point numbers) to CMM-DC.

[0086] ④Merged Cold Gradient 16 (e.g., the merged gradient value of the cold weight parameter with 16-bit floating-point numbers) is used for parameter update threshold judgment.

[0087] ⑤Update the cold weight with Merged Cold Gradient 16 (≥thr) (e.g., the merged gradient value greater than or equal to the parameter update threshold).

[0088] ⑥Calculate Cold Weight 16 with Cold Weight 32.

[0089] ⑦Perform gradient accumulation (e.g., update the set of gradient accumulation values) with Merged Cold Gradient 16 (<thr) (e.g., the merged gradient value less than the parameter update threshold). It should be understood that prefetching and gradient accumulation are performed simultaneously.

[0090] ⑧The gradient accumulation value is used to merge with the next batch of gradient values to obtain the merged gradient value. By delaying the calculation by one step, it is ensured that the cold weight parameter is updated first.

[0091] Table 1 Data Flow

[0092]

[0093]

[0094] Figure 3 The flowchart shows a method for dynamically adjusting a parameter update threshold (e.g., adopting an adaptive weight sparsity strategy to select an optimal parameter update threshold) according to at least one exemplary embodiment of the present disclosure. As an example, the method for dynamically adjusting the parameter update threshold can be executed by a first processor.

[0095] Refer to Figure 3 In step S301, determine the idle time when the first processor is in an idle state during the first training process based on the (t-1)-th batch of training samples Specifically, for the training process based on the (t-1)-th batch of training samples, determine the idle time when the first processor is in an idle state

[0096] In step S302, based on the determined idle time Determine the parameter update threshold used in the second training process based on the t-th batch of training samples. Specifically, based on a determined idle time. Determine the parameter update threshold used in the training process based on the t-th batch of training samples. That is, the parameter update threshold used by the second processor after merging the gradient values ​​of the cold weight parameters determined based on the t-th batch of training samples.

[0097] As at least one example embodiment, step S302 may include: determining idle time. Is it greater than 0? Response time If greater than 0, update the threshold and idle time based on the parameters used in the training process based on the (t-1)th batch of training samples (i.e., the first training process). Free time Determine the parameter update thresholds used in the training process based on the t-th batch of training samples (i.e., the second training process); in response to idle time The value is equal to 0. Based on the parameter update threshold used in the training process using the (t-1)th batch of training samples, the parameter update threshold used in the training process is determined; where idle time... This represents the idle time during the third training process based on the (t-2)th batch of training samples, where the first processor is in an idle state. Specifically, the idle time... This represents the idle time during which the first processor is in an idle state, as determined during the training process based on the (t-2)th batch of training samples.

[0098] In step S303, the determined parameter update threshold is sent to the second processor so that the second processor updates the parameter values ​​of the weight parameters in the second parameter set based on the received parameter update threshold.

[0099] Reference Figure 7 The following formula reflects the relationship between time variables:

[0100] T FP +T BP +T Idle =max(T) Trans T Update )

[0101] By transforming the formula, we get:

[0102] T Idle =max(T) Trans T Update )–(T FP +T BP )

[0103] The goal is to minimize T Idle :

[0104] min T Idle =min{max(T Trans T Update )–(T FP +T BP )},T Idle ≥0

[0105] Among them, T FP T represents the time of forward propagation; BP T represents the time of backpropagation. Trans T represents the larger of the time taken to transmit the gradient value of the cold weight parameter and the time taken to transmit the parameter value of the cold weight parameter; Update This represents the larger of the time taken to update the cold weight parameters and the time taken to update the cumulative gradient set; T Idle This indicates the idle time of the first processor.

[0106] As an example, an initial parameter for updating the threshold can be randomly selected, and then the parameter can be adaptively adjusted for updating the threshold during training using the following formula:

[0107]

[0108] Where, τ t This represents the parameter update threshold for generation t (i.e., the parameter update threshold used in the training process based on the training samples of batch t). This represents the ratio of GPU idle time between generations t-2 and t-1.

[0109] As an example, such as Figure 8 As shown, a timer can be set in the GPU to trigger the adjustment of the parameter update threshold, ensuring that the parameter update threshold is adaptively and dynamically adjusted. Furthermore, the hot / cold weight parameters can be swapped and iterated.

[0110] Figure 4 A flowchart illustrating a method (e.g., a hot / cold weight parameter exchange mechanism) for updating a first parameter set and a second parameter set according to at least one example embodiment of the present disclosure is shown. As an example, the method for updating the first parameter set and the second parameter set may be executed by a first processor.

[0111] Reference Figure 4 In step S401, the first sorting result is obtained by sorting the weight parameters in the first parameter set (i.e., sorting all thermal weight parameters) in ascending order of the gradient values ​​of the determined weight parameters.

[0112] In step S402, the second sorting result is obtained by sorting the weight parameters in the second parameter set (i.e., sorting all cold weight parameters) according to the gradient values ​​of the determined weight parameters from largest to smallest.

[0113] In step S403, the gradient values ​​of the first N weight parameters in the first sorting result are compared with the gradient values ​​of the first N weight parameters in the second sorting result, and the first parameter set and the second parameter set are updated according to the comparison result.

[0114] As at least one example embodiment, combined with Figure 9 Step S403 may include:

[0115] Step (a) compares the gradient value of the i-th weight parameter in the first sorting result with the gradient value of the i-th weight parameter in the second sorting result, where the initial value of i is 1.

[0116] In step (b), in response to the gradient value of the i-th weight parameter in the second sorting result being greater than the gradient value of the i-th weight parameter in the first sorting result, the i-th weight parameter in the second sorting result is removed from the second parameter set and added to the first parameter set, the i-th weight parameter in the first sorting result is removed from the first parameter set and added to the second parameter set, and it is determined whether i is equal to N.

[0117] In step (c), in response to i not being equal to N, let i = i + 1 (i.e., add 1 to i) and return to execute step (a).

[0118] Step (d) is to end the current exchange of hot and cold weight parameters in response to the gradient value of the i-th weight parameter in the second sorting result being less than or equal to the gradient value of the i-th weight parameter in the first sorting result.

[0119] In step (e), in response to i equaling N, the current exchange of hot and cold weight parameters ends.

[0120] It should be understood that Figure 9 The thermal parameters include not only thermal weight parameters, but also optimizer parameters for thermal weight parameters; Figure 9 The cold parameters in the algorithm include not only cold weight parameters, but also optimizer parameters for cold weight parameters.

[0121] like Figure 13 As shown, this disclosure offloads frequent near-data-side computations (weight updates) to the CMM-DC, while reducing computation and I / O, thereby making full use of the idle time of the GPU and CMM-DC. This disclosure achieves the following technical effects:

[0122] (1) Faster loading: CMM-DC offers faster loading speeds (twice as fast as CPU) and is well-suited for near-data operations such as weight updates.

[0123] (2) Less computation: Hot weight parameters are updated by the GPU, while in CMM-DC, only cold weight parameters whose gradient merging values ​​exceed the parameter update threshold are updated.

[0124] (3) Fewer I / O: Only the updated cold weight parameters are prefetched to the GPU for forward propagation. Other cold weight parameters (those not prefetched to the GPU) use their previously cached values ​​in the GPU.

[0125] Figure 14 A structural block diagram of an electronic device according to at least one example embodiment of the present disclosure is shown.

[0126] Reference Figure 14 The electronic device includes: a first processor 141, a second processor 142, and at least one memory 143. The at least one memory 143 stores a set of computer-executable instructions. When the set of computer-executable instructions is executed by the first processor 141 and the second processor 142, the training method of the neural network model as described in the above example embodiment is executed.

[0127] As an example, an electronic device can be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, the electronic device is not necessarily a single device; it can be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The electronic device can also be part of an integrated control system or system manager, or can be configured to interconnect locally or remotely (e.g., via wireless transmission) through an interface.

[0128] In an electronic device, the first processor 141 may be a graphics processing unit (GPU); the second processor 142 may be a near-memory processor (CMM-DC).

[0129] The first processor 141 and the second processor 142 can execute instructions or code stored in at least one memory 143, wherein the at least one memory 143 can also store data. The instructions and data can also be sent and received via a network through a network interface device, wherein the network interface device can employ any known transmission protocol.

[0130] In addition, electronic devices may include video displays (such as liquid crystal displays) and user interaction interfaces (such as keyboards, mice, touch input devices, etc.). All components of the electronic device may be interconnected via buses and / or networks.

[0131] According to at least one example embodiment of this disclosure, a first processor 141 and a second processor 142 that process at least one function or operation can be implemented using processing circuitry (such as hardware, software, or a combination of hardware and software). For example, more specifically, the processing circuitry may include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc.

[0132] According to at least one example embodiment of this disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by a first processor and a second processor, they cause the first and second processors to perform a training method for a neural network model as described in the above example embodiments. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray and / or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0133] According to exemplary embodiments of this disclosure, a computer program product may also be provided, wherein the instructions in the computer program product are executable by at least one processor to perform the training method of the neural network model as described in the exemplary embodiments above.

[0134] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered as examples only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0135] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for training a neural network model, wherein the training method is executed by a first processor, wherein, The weight parameters of the neural network model include a first parameter set and a second parameter set based on the gradient values. The training method includes: For the t-th batch of training samples, obtain the parameter values ​​of the weight parameters of the neural network model; Based on the t-th batch of training samples, the gradient values ​​of the weight parameters are determined by using the obtained weight parameter values ​​for forward and backward propagation. Based on the gradient values ​​of the weight parameters in the determined first parameter set, update the parameter values ​​of the weight parameters in the first parameter set; and The gradient values ​​of the weight parameters in the determined second parameter set are sent to a second processor configured to update the parameter values ​​of the weight parameters in the second parameter set, wherein the computing power of the first processor is higher than that of the second processor. The steps for obtaining the weight parameters of a neural network model include: Load the parameter values ​​of the weight parameters from the first parameter set; Receive the updated weight parameter values ​​from the second parameter set from the second processor; and Load the unupdated weight parameter values ​​from the second parameter set.

2. The training method according to claim 1 further includes: Determine the idle time during the first training process based on the (t-1)th batch of training samples when the first processor is idle. Based on a determined idle time Determine the parameter update threshold used in the second training process based on the t-th batch of training samples; The determined parameter update threshold is sent to the second processor. The second processor is also configured to update the parameter values ​​of the weight parameters in the second parameter set based on a determined parameter update threshold.

3. The training method according to claim 2, wherein, The steps for determining the parameter update threshold used in the second training process based on the t-th batch of training samples include: Determine free time Is it greater than 0? Response during idle time If the value is greater than 0, update the threshold and idle time based on the parameters used in the first training process. and free time Determine the parameter update thresholds used in the second training process; Response during idle time If the value is 0, the parameter update threshold used in the first training process is used to determine the parameter update threshold used in the second training process. Among them, free time This represents the idle time during the third training process based on the (t-2)th batch of training samples, when the first processor is idle.

4. The training method according to claim 1 further includes: The first sorting result is obtained by sorting the weight parameters in the first parameter set according to the gradient values ​​of the determined weight parameters in ascending order. The second sorting result is obtained by sorting the weight parameters in the second parameter set in descending order of the gradient values ​​of the determined weight parameters. Compare the gradient values ​​of the first N weight parameters in the first sorting result with the gradient values ​​of the first N weight parameters in the second sorting result; as well as Update the first and second parameter sets based on the comparison results.

5. The training method according to claim 4, wherein, The steps for comparing the gradient values ​​of the first N weight parameters in the first sorting result with the gradient values ​​of the first N weight parameters in the second sorting result include: Compare the gradient value of the i-th weight parameter in the first sorting result with the gradient value of the i-th weight parameter in the second sorting result, where the initial value of i is 1; The steps for updating the first and second parameter sets based on the comparison results include: In response to the gradient value of the i-th weight parameter in the second ranking result being greater than the gradient value of the i-th weight parameter in the first ranking result, the i-th weight parameter in the second ranking result is removed from the second parameter set and added to the first parameter set; the i-th weight parameter in the first ranking result is removed from the first parameter set and added to the second parameter set, and it is determined whether i is equal to N; and In response to i not being equal to N, add 1 to i and repeat the step of comparing the gradient value of the i-th weight parameter in the first sorting result with the gradient value of the i-th weight parameter in the second sorting result until i equals N.

6. A method for training a neural network model, the training method being executed by a second processor, the training method comprising: The first processor receives gradient values ​​of weight parameters from a second parameter set determined based on the t-th batch of training samples, wherein the computing power of the first processor is higher than that of the second processor. By merging the set of accumulated gradient values ​​with the gradient values ​​of the received weight parameters, the merged gradient values ​​of each weight parameter in the second parameter set are obtained. When the combined gradient value is less than the parameter update threshold, the set of accumulated gradient values ​​is updated. Update the parameter value of each weight parameter in the second parameter set whose gradient merge value is greater than or equal to the parameter update threshold; and Send the updated weight parameter values ​​to the first processor.

7. The training method according to claim 6, wherein, The steps to obtain the gradient merged values ​​of the weight parameters in the second parameter set include: For each weight parameter in the second parameter set, determine whether the set of gradient accumulation values ​​contains the gradient accumulation value of that weight parameter; If the set of gradient accumulation values ​​contains the gradient accumulation value of the weight parameter, then the larger of the received gradient value of the weight parameter and the gradient accumulation value of the weight parameter will be used as the gradient merge value of the weight parameter. If the gradient accumulation set does not contain the gradient accumulation value of the weight parameter, then the received gradient value of the weight parameter will be used as the merged gradient value of the weight parameter.

8. The training method according to claim 6, wherein, When the combined gradient value is less than the parameter update threshold, the steps for updating the gradient accumulation set include: For each weight parameter in the second parameter set whose gradient merge value is less than the parameter update threshold, determine whether the gradient accumulation value set contains the gradient accumulation value of that weight parameter. If the set of gradient accumulation values ​​contains the gradient accumulation value of the weight parameter, then the sum of the combined gradient value of the weight parameter and the gradient accumulation value of the weight parameter is used as the updated gradient accumulation value of the weight parameter. If the gradient accumulation set does not contain the gradient accumulation value for that weight parameter, then the combined gradient value of that weight parameter is added to the gradient accumulation set as the gradient accumulation value for that weight parameter; and Remove the gradient accumulation values ​​of the weight parameters that have not been updated from the gradient accumulation value set.

9. The training method according to claim 6, further comprising: Before sending the updated parameter values, the data type of the updated weight parameter values ​​is converted from the first data type to the second data type. The precision of the first data type is higher than that of the second data type.

10. An electronic device, wherein, The electronic device includes: First processor; Second processor; At least one memory that stores computer-executable instructions; Wherein, when the computer-executable instructions are executed by the electronic device, they are configured to instruct the first processor to perform the following operations: For the t-th batch of training samples, obtain the parameter values ​​of the weight parameters of the neural network model, where the weight parameters of the neural network model include a first parameter set and a second parameter set based on the magnitude of the gradient values; Based on the t-th batch of training samples, the gradient values ​​of the weight parameters are determined by using the obtained weight parameter values ​​for forward and backward propagation. Based on the gradient values ​​of the weight parameters in the determined first parameter set, update the parameter values ​​of the weight parameters in the first parameter set. The gradient values ​​of the weight parameters in the determined second parameter set are sent to the second processor, and The computing power of the first processor is higher than that of the second processor, and The weight parameters of the neural network model are obtained through the following operations: Load the parameter values ​​of the weight parameters from the first parameter set; Receive the updated weight parameter values ​​from the second parameter set from the second processor; and Load the unupdated weight parameter values ​​from the second parameter set.

Images