A training method and a training single machine of a deep learning model

Abstract

The application discloses a kind of training method and training single machine of deep learning model, it is related to deep learning technical field.The specific embodiment of the method includes: reading training sample, obtaining sparse parameter from training single machine local memory, obtaining dense parameter from the first trainer locally;First trainer and each second trainer on training single machine share sparse parameter in training single machine local memory;Loss value is calculated according to dense parameter and sparse parameter in the process of forward propagation, and dense parameter gradient and sparse parameter gradient are obtained in the process of back propagation;Synchronization is carried out with other trainer dense parameter gradient and sparse parameter gradient, and final dense parameter gradient and final sparse parameter gradient are calculated;The dense parameter of the first trainer locally is updated based on final dense parameter gradient, and the sparse parameter of training single machine local memory is updated based on final sparse parameter gradient.The embodiment can solve the technical problems of insufficient memory and parameter gradient failure.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a training method and a training machine for a deep learning model. Background Art

[0002] Deep learning boasts strong learning capabilities, wide coverage, strong adaptability, and good portability. It discovers distributed feature representations of data by combining low-level features to form more abstract, high-level representations of attribute categories or features. Current deep learning models are becoming increasingly complex, with large-scale and ultra-large-scale models gaining popularity. As these models become larger, the amount of data required must also increase. Deep learning algorithms cannot unbiasedly estimate patterns in the data if used solely within the current data-limited application scenarios. Therefore, achieving high accuracy requires extensive data support. Deep learning frameworks, particularly TensorFlow, face increasing challenges with sparse parameters.

[0003] To improve training efficiency, distributed training becomes essential. Distributed training strategies can be broadly categorized into data parallelism and model parallelism, depending on the parallelization approach. Data parallelism involves storing a copy of the model on each trainer (ranker), assigning different samples to different rankers for computation, and then merging the results from all the rankers to accelerate model training.

[0004] In the process of implementing the present invention, the inventors discovered that the prior art has at least the following problems:

[0005] For data-parallel synchronous distributed deep learning, each ranker in the training cluster needs to save the complete model, which has strict restrictions on the size of the model and cannot meet the training requirements of very large models.

[0006] For asynchronous training architectures with parameter servers, asynchronous updates of model parameters can lead to parameter gradient failure (after a ranker completes its parameter gradient calculation, it discovers that the model parameters in the parameter server have been updated. Since the current parameter gradient is calculated based on the parameters before the update, it has become invalid). This problem is exacerbated with the increase in the number of rankers, having a greater impact on model accuracy. Furthermore, the fully connected communication method between the rankers and parameter servers in the cluster significantly reduces network communication efficiency, severely limiting model training speed and preventing the cluster from achieving a good linear speedup ratio.

[0007] Therefore, in scenarios such as search, recommendation, and advertising, when training large-scale models with sparse parameters in the embedding layer on multiple machines and multiple graphics cards, there are problems such as insufficient memory and parameter gradient failure in the remote parameter server mode. Summary of the Invention

[0008] In view of this, an embodiment of the present invention provides a training method and a training machine for a deep learning model to solve the technical problems of insufficient memory and parameter gradient failure.

[0009] To achieve the above objectives, according to one aspect of an embodiment of the present invention, a deep learning model training method is provided, which is applied to a first trainer deployed on a training standalone machine, comprising:

[0010] Create an index;

[0011] Reading training samples, obtaining sparse parameters from the local memory of the training machine based on the index, and obtaining dense parameters locally from the first trainer; wherein each second trainer on the training machine shares the sparse parameters in the local memory of the training machine;

[0012] The loss value is calculated based on the dense parameters and sparse parameters during the forward propagation process, and the dense parameter gradient and sparse parameter gradient are obtained during the backward propagation process;

[0013] Synchronize dense parameter gradients and sparse parameter gradients with other trainers to calculate the final dense parameter gradients and the final sparse parameter gradients;

[0014] The local dense parameters of the first trainer are updated based on the final dense parameter gradient, and the sparse parameters in the local memory of the training machine are updated based on the final sparse parameter gradient.

[0015] Optionally, indexing includes:

[0016] Calling a local parameter server framework to perform the training stand-alone local memory mapping on the sparse parameters and creating an index based on the mapping result;

[0017] Synchronizing the index to other trainers on the training machine;

[0018] Among them, the primary key is the identifier of the sparse parameter, and the key value is the storage address of the sparse parameter in the local memory of the training stand-alone machine.

[0019] Optionally, obtaining sparse parameters from the local memory of the training machine based on the index includes:

[0020] Determining a sparse parameter identifier based on the training samples distributed to the first trainer;

[0021] The storage address of the sparse parameter corresponding to the sparse parameter identifier in the local memory of the training machine is determined based on the index, thereby obtaining the sparse parameter from the local memory of the training machine.

[0022] Optionally, during forward propagation, the loss value is calculated based on the dense parameters and the sparse parameters, and during backward propagation, the dense parameter gradient and the sparse parameter gradient are obtained, including:

[0023] The training samples are used to train a deep learning model using the dense parameters and the sparse parameters respectively, so that the loss value is calculated during the forward propagation process, and the dense parameter gradient and the sparse parameter gradient are obtained by derivation during the back propagation process.

[0024] Optionally, synchronize the dense parameter gradients and sparse parameter gradients with other trainers to calculate the final dense parameter gradients and the final sparse parameter gradients, including:

[0025] Based on the ring global reduction algorithm, synchronize dense parameter gradients and sparse parameter gradients with other trainers;

[0026] The average value of the dense parameter gradient on each trainer is calculated to obtain the final dense parameter gradient; at the same time, the final sparse parameter gradient is calculated based on the frequency of occurrence of sparse parameters on each trainer.

[0027] Optionally, the final sparse parameter gradient is calculated based on the frequency of occurrence of the sparse parameter on each trainer, including:

[0028] For each sparse parameter, calculate the frequency of occurrence of the sparse parameter on each trainer;

[0029] For each sparse parameter, the sparse parameter gradients on each trainer are summed and then divided by the frequency of occurrence of the sparse parameter on each trainer to obtain a final sparse parameter gradient.

[0030] Optionally, updating the sparse parameters in the local memory of the training machine based on the final sparse parameter gradient includes:

[0031] By calling the local parameter server framework, the final sparse parameter gradient is updated to the local memory of the training stand-alone machine.

[0032] In addition, according to another aspect of an embodiment of the present invention, a deep learning model training method is provided, which is applied to a second trainer, where the second trainer is deployed on a training standalone machine, including:

[0033] Reading training samples, obtaining sparse parameters from the local memory of the training machine, and obtaining dense parameters from the local memory of the second trainer; wherein the first trainer and each second trainer on the training machine share the sparse parameters in the local memory of the training machine;

[0034] The loss value is calculated based on the dense parameters and sparse parameters during the forward propagation process, and the dense parameter gradient and sparse parameter gradient are obtained during the backward propagation process;

[0035] Synchronize dense parameter gradients and sparse parameter gradients with other trainers to calculate the final dense parameter gradients;

[0036] Updating the local dense parameters of the second trainer based on the final dense parameter gradient;

[0037] The final sparse parameter gradient is calculated and updated by the first trainer on the training machine.

[0038] Optionally, read the training samples, calculate the loss value based on the dense parameters and sparse parameters during the forward propagation process, and obtain the dense parameter gradient and the sparse parameter gradient during the backward propagation process, including:

[0039] The training samples are used to train a deep learning model using the dense parameters and the sparse parameters respectively, so that the loss value is calculated during the forward propagation process, and the dense parameter gradient and the sparse parameter gradient are obtained by derivation during the back propagation process.

[0040] Optionally, obtaining sparse parameters from the local memory of the training machine includes:

[0041] Determining a sparse parameter identifier based on the training samples distributed to the second trainer;

[0042] Determine, based on the index, the storage address of the sparse parameter corresponding to the sparse parameter identifier in the local memory of the training stand-alone machine, thereby obtaining the sparse parameter from the local memory of the training stand-alone machine;

[0043] Among them, in the index, the primary key is the identifier of the sparse parameter, and the key value is the storage address of the sparse parameter in the local memory of the training stand-alone machine.

[0044] Optionally, synchronize dense and sparse parameter gradients with other trainers to calculate the final dense parameter gradients, including:

[0045] Based on the ring global reduction algorithm, synchronize dense parameter gradients and sparse parameter gradients with other trainers;

[0046] The average of the dense parameter gradients on each trainer is calculated to obtain the final dense parameter gradient.

[0047] In addition, according to another aspect of an embodiment of the present invention, a single training machine is provided, on which a first trainer and respective second trainers are deployed, sparse parameters are stored in a local memory of the single training machine, and dense parameters are stored in a local memory of the first trainer and a local memory of the second trainer.

[0048] The first trainer and each second trainer share the sparse parameters in the local memory of the training machine, and the sparse parameters are updated by the first trainer.

[0049] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, including:

[0050] one or more processors;

[0051] a storage device for storing one or more programs,

[0052] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of the above embodiments.

[0053] According to another aspect of the embodiments of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0054] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0055] One embodiment of the above invention has the following advantages or beneficial effects: because the sparse parameters are stored in the local memory of the training machine, each trainer shares the memory, and the local parameter update is performed after the parameter gradient synchronization is completed, the technical problems of insufficient memory and parameter gradient failure in the prior art are overcome. The embodiment of the present invention stores the sparse parameters in the local memory of the training machine, so that each trainer shares the memory, and adopts a mode in which dense parameters and sparse parameters are trained locally and independently, so that the dense parameters and sparse parameters adopt different storage and update modes in the trainer, which not only improves the memory usage and reduces the number of updates of sparse parameters, but also solves the problem of parameter gradient failure of the remote parameter server, ensures the accuracy of the model, and improves the training speed of the model.

[0056] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. Among them:

[0058] Figure 1 1 is a schematic diagram of the main process of a deep learning model training method according to one embodiment of the present invention;

[0059] Figure 2 2. This is a schematic diagram of the system architecture for multi-machine and multi-GPU distributed parallel training according to an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the main process of parameter synchronization according to an embodiment of the present invention;

[0061] Figure 4 1 is a schematic diagram of the main process of a deep learning model training method according to a reference embodiment of the present invention;

[0062] Figure 5 is a schematic diagram of the main process of a deep learning model training method according to another embodiment of the present invention;

[0063] Figure 6 1 is a schematic diagram of the main process of a training method for a deep learning model according to another reference embodiment of the present invention;

[0064] Figure 7 is a schematic diagram of main modules of a training apparatus for a deep learning model according to one embodiment of the present invention;

[0065] Figure 8 is a schematic diagram of main modules of a training device for a deep learning model according to another embodiment of the present invention;

[0066] Figure 9 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0067] Figure 10 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0069] In application scenarios such as product search, recommendation, and advertising, deep learning models are characterized by numerous training samples, a massive model embedding layer, and sparse parameters in the embedding layer, with each step updating only those parameters relevant to the current batch. For example, in a product search scenario, each product ID is a sparse feature, and 100 million products may require 100 million-dimensional features to represent them. Even if the model uses sparse embedding parameters to reduce the dimensionality of the products, the sparse parameters are still massive, and each training round involves relatively few products. For this scenario, using data synchronization and distributed training cannot meet the requirements of ultra-large-scale training due to insufficient memory. Furthermore, using an asynchronous training architecture with a parameter server can lead to parameter gradient failure, which in turn affects model accuracy. Furthermore, the fully connected communication between the ranker and parameter server reduces network communication efficiency.

[0070] In order to solve the technical problems existing in the prior art, the embodiments of the present invention adopt different storage and update strategies for sparse parameters and dense parameters based on the Horovod architecture.

[0071] Figure 1 Schematic diagram of the main process of the training method of the deep learning model according to an embodiment of the present invention. As an embodiment of the present invention, Figure 1 As shown, the training method of the deep learning model is applied to a first trainer, and the first trainer is deployed on a training standalone machine. The method may include the following steps:

[0072] Step 101: Create an index.

[0073] like Figure 2As shown, each training machine can be deployed with multiple trainers (ranker, card), such as ranker0, ranker1, ranker2, ranker3, etc. The embodiment of the present invention adopts a multi-machine multi-card distributed parallel training method to train deep learning models to improve the training efficiency of the model. Among them, the first trainer can be ranker0 deployed on the training machine, and the other rankers (ranker1, ranker2, ranker3, etc.) on the training machine can be called second trainers.

[0074] Optionally, step 101 may include: calling the local parameter server framework, mapping the sparse parameters to the local memory of the training machine and establishing an index based on the mapping result; synchronizing the index to other trainers on the training machine; wherein the primary key is the identifier of the sparse parameter, and the key value is the storage address of the sparse parameter in the local memory of the training machine. First, the first trainer (rank0) on each training machine initializes the sparse parameters, including: calling the local parameter server framework to parse the binary sparse parameter file, and then mapping the parsed sparse parameters to memory and establishing a key-value index, i.e., map<key,value_adddress> , where key = sparse parameter ID, value_address = the storage address of the sparse parameter in the local memory of the training machine.

[0075] The characteristics of deep learning model parameters for business scenarios such as search, advertising, and click-through rate of recommendation services are that the Embedding (embedding layer) is sparse and huge in number. Sparse parameters occupy the main part of the model scale, usually reaching the order of tens of GB to hundreds of GB. Since sparse parameters occupy the vast majority of the model, for training scenarios with multiple machines and multiple cards, it is difficult for each trainer to store a copy of the sparse parameters locally. On the contrary, the order of magnitude of dense parameters is relatively small, generally only between a few MB and 1GB, occupying only a small part of the model, and each trainer can store complete dense parameters locally. In view of the different characteristics of sparse parameters and dense parameters, the embodiment of the present invention adopts a hybrid training architecture for training, and adopts different storage modes for dense parameters and sparse parameters: for sparse parameters, shared memory can be used, and multiple trainers on each training machine can read the sparse parameters in the unified memory space through the local parameter server framework, and only one index map of sparse parameters is retained in each trainer.<key,value_adddress> This solves the problem of being unable to train larger models in scenarios with limited GPU memory. For dense parameters, each trainer independently stores a copy of the dense parameters. Logically, each trainer holds both a copy of the dense parameters and a copy of the sparse parameters.

[0076] Step 102: read the training sample, obtain the sparse parameters from the local memory of the training machine based on the index, and obtain the dense parameters locally from the first trainer; wherein the first trainer and each second trainer on the training machine share the sparse parameters in the local memory of the training machine.

[0077] Optionally, obtaining sparse parameters from the local memory of the training machine based on the index includes: determining a sparse parameter identifier based on the training samples distributed to the first trainer; determining the storage address of the sparse parameter corresponding to the sparse parameter identifier in the local memory of the training machine based on the index, thereby obtaining the sparse parameter from the local memory of the training machine. Since the sparse parameters are stored in the local memory of the training machine, and each trainer shares the sparse parameters in the local memory of the training machine, the first trainer and the second trainer can both determine the storage address of the sparse parameters in the local memory of the training machine based on the index, thereby obtaining the sparse parameters from the local memory of the training machine. Each trainer's local memory stores a copy of the dense parameters, so the first trainer can directly obtain the dense parameters from its local memory.

[0078] Step 103: Calculate the loss value according to the dense parameters and the sparse parameters during the forward propagation process, and obtain the dense parameter gradient and the sparse parameter gradient during the backward propagation process.

[0079] The first trainer (ranker0) reads training samples based on batch_size, then calculates the loss value based on the dense and sparse parameters during the forward propagation process, and obtains the dense and sparse parameter gradients during the backward propagation process. Specifically, the first trainer uses the dense and sparse parameters to complete the forward calculation of the deep learning model, and then calculates the parameter gradients (including the dense parameter gradient and the sparse parameter gradient) according to the backward propagation algorithm.

[0080] Optionally, step 103 may include: using the training samples and respectively using the dense parameters and the sparse parameters to train the deep learning model, thereby calculating the loss value in the forward propagation process, and obtaining the dense parameter gradient and the sparse parameter gradient by derivation in the backward propagation process. Figure 2 As shown, the embodiment of the present invention performs distributed training on multiple machines and multiple cards based on Horovod's TheRing Allreduce (ring global reduction) synchronous training architecture. In the distributed computing process, it is necessary to shard the computing task resources, usually by sharding the training samples, and then distributing the training samples to different trainers. After the first trainer receives the training samples, it adopts a training mode in which dense parameters and sparse parameters are trained independently. Specifically, dense parameters are obtained locally from the first trainer, and the training samples are used to train the deep learning model using the dense parameters, so that the loss value is calculated during the forward propagation process, and the dense parameter gradient is obtained by derivation during the back propagation process; at the same time, sparse parameters are obtained from the local memory of the training machine, and the training samples are used to train the deep learning model using the sparse parameters, so that the loss value is calculated during the forward propagation process, and the sparse parameter gradient is obtained by derivation during the back propagation process.

[0081] It should be pointed out that since the parameters of the Embedding layer are very sparse, each step only updates some parameters related to the current batch, and the training samples of each trainer are different, so each trainer can determine the sparse parameter identifiers that need to be obtained based on the training samples.

[0082] Step 104: synchronize the dense parameter gradient and the sparse parameter gradient with other trainers to calculate the final dense parameter gradient and the final sparse parameter gradient.

[0083] The first trainer obtains the dense parameter gradient and the sparse parameter gradient, and synchronizes the dense parameter gradient and the sparse parameter gradient with other trainers (including other trainers on this training machine and each trainer on other training machines). The synchronization process can be carried out simultaneously. After the synchronization is completed, the first trainer calculates the final dense parameter gradient and the final sparse parameter gradient based on the synchronization results.

[0084] Optionally, step 104 may include: synchronizing dense parameter gradients and sparse parameter gradients with other trainers based on a ring global reduction algorithm; calculating the average value of the dense parameter gradients on each trainer to obtain the final dense parameter gradient; and at the same time, calculating the final sparse parameter gradient based on the frequency of occurrence of the sparse parameters on each trainer. The embodiment of the present invention adopts a training mode in which dense parameters and sparse parameters are trained independently. In a multi-machine and multi-card training environment, different storage and update strategies are adopted for sparse parameters and dense parameters based on the Horovod architecture training mode. Figure 2 As shown in the figure, for sparse parameters, a shared memory approach is used to enable each trainer (ranker) on a training machine to share a copy of the Embedding parameters. Each ranker obtains and updates the sparse parameters through the local parameter server framework. After the All Gather operation is performed in each training machine to obtain the parameter gradients calculated by all global rankers, the first trainer (ranker0) on each training machine updates the sparse parameters in memory. A dual-loop logic structure is formed between each trainer to synchronize the dense parameter gradient and the sparse parameter gradient, as shown in the figure. Figure 3 After parameter gradient synchronization is complete, different calculation methods are used for dense and sparse parameters to calculate the final parameter gradients. For dense parameters, the first trainer averages the dense parameter gradients on each trainer to obtain the final dense parameter gradient. For sparse parameters, the first trainer calculates the final sparse parameter gradient based on the frequency of occurrence of the sparse parameter on each trainer.

[0085] Optionally, the final sparse parameter gradient is calculated based on the frequency of occurrence of the sparse parameter on each trainer, including: for each sparse parameter, calculating the frequency of occurrence of the sparse parameter on each trainer; for each sparse parameter, summing the sparse parameter gradients on each trainer and dividing the sum by the frequency of occurrence of the sparse parameter on each trainer to obtain the final sparse parameter gradient.

[0086] Due to the different storage modes of dense parameters and sparse parameters in multi-GPU training, different update strategies need to be adopted when updating model parameters. Figure 2-3As shown in , for dense parameters, each trainer (ranker0, ranker1, ranker2, ranker3, etc.) stores a full set of dense parameters. After the dense parameter gradients are synchronized, each trainer calculates the average value of the dense parameter gradients, and then each trainer updates its local dense parameters based on the gradients. For sparse parameters, after the parameter gradients are synchronized, each trainer also stores a copy of the gradient data. However, the number of sparse parameters and sparse parameters on each trainer may not be the same. Figure 3 As shown, the gradients of the three sparse parameters e, g, and i are on ranker0, the gradients of the three sparse parameters f, h, and i are on ranker 1, and the gradients of the three sparse parameters e, g, and i are on ranker 2. Therefore, the frequencies of occurrence of the five sparse parameters e, f, g, h, and i are 2, 1, 2, 1, and 3, respectively. The embodiment of the present invention calculates the gradients of different parameters by dividing the parameter gradients by the frequency of occurrence. For example, the gradient of the sparse parameter i is (i0+i1+i2) / 3, and the update of the gradient of the sparse parameter f is determined only by the gradient calculation of f on ranker1.

[0087] Step 105: Update the local dense parameters of the first trainer based on the final dense parameter gradient, and update the sparse parameters in the local memory of the training machine based on the final sparse parameter gradient.

[0088] Since each training machine and multiple trainers (ranker) share the same full set of sparse parameters, when updating the parameters, it is necessary to determine whether the ranker is ranker0 (i.e. the first trainer), and then only ranker0 will update the sparse parameters. Figure 2 As shown in Figure 1, ranker0 updates the final sparse parameter gradient to the local memory of the training machine so that each ranker on this training machine can share the latest sparse parameters.

[0089] Optionally, updating the sparse parameters of the local memory of the training machine based on the final sparse parameter gradient includes: updating the final sparse parameter gradient to the local memory of the training machine by calling the local parameter server framework.

[0090] In an embodiment of the present invention, both dense parameters and sparse parameters are updated synchronously. Compared with the sparse parameter update method of the remote parameter server mode, the embodiment of the present invention avoids the occurrence of gradient failure and ensures the accuracy of the model. At the same time, the local parameter update method is adopted to a certain extent to avoid the time consumption caused by network transmission, thereby improving the training speed of the model. When updating the sparse parameters, each training machine only updates the sparse parameters by calling the local parameter server framework on the GPU of the first trainer, reducing the frequency of operations for updating the sparse parameters. For example, in the case of three training machines with four trainers each, one round of training only requires updating the sparse parameters three times, reducing nine update operations.

[0091] Moreover, based on the sparse parameter storage mode, it is not necessary for each trainer on each training machine to completely save a full set of sparse parameters. Instead, through shared memory, a training machine only retains a full set of sparse parameters, and each trainer obtains the sparse parameters of the current trainer by accessing the index, which significantly improves memory utilization.

[0092] According to the various embodiments described above, it can be seen that the embodiments of the present invention solve the technical problems of insufficient memory and parameter gradient failure in the prior art by storing sparse parameters in the local memory of a single training machine so that each trainer shares the memory and completes parameter gradient synchronization and then performs local parameter update. The embodiments of the present invention store sparse parameters in the local memory of a single training machine so that each trainer shares the memory, and adopt a mode in which dense parameters and sparse parameters are trained independently locally, so that dense parameters and sparse parameters adopt different storage and update modes in the trainer, which not only improves the memory usage and reduces the number of updates of sparse parameters, but also solves the problem of parameter gradient failure of the remote parameter server, ensures the accuracy of the model, and improves the training speed of the model.

[0093] Figure 4 Schematic diagram of the main process of the training method of the deep learning model according to a reference embodiment of the present invention. As another embodiment of the present invention, Figure 4 As shown, the training method of the deep learning model is applied to a first trainer, and the first trainer is deployed on a training standalone machine, which may include:

[0094] Step 401: Call the local parameter server framework to parse the binary sparse parameter file to obtain sparse parameters.

[0095] like Figure 2 As shown in the figure, ranker0 on a single training machine calls the local parameter server framework to parse the binary sparse parameter file to obtain the full set of sparse parameters.

[0096] Step 402: Call the local parameter server framework to perform the training single-machine local memory mapping on the sparse parameters and create an index based on the mapping result.

[0097] After ranker0 obtains the sparse parameters, it calls the local parameter server framework to map and index the sparse parameters to the local memory of the training machine, so that each ranker on the training machine shares a full set of sparse parameters. The primary key is the identifier of the sparse parameter, and the key value is the storage address of the sparse parameter in the local memory of the training machine.

[0098] It should be noted that dense parameters are relatively small, typically only a few MB to 1 GB in size, and only occupy a small portion of the model. Each trainer can store the complete set of dense parameters locally. Therefore, the present invention uses different storage strategies for sparse and dense parameters.

[0099] Step 403: Synchronize the index to other trainers on the training machine.

[0100] Ranker0 synchronizes the index to other rankers on the training machine, so that these rankers can obtain sparse parameters from the local memory of the training machine through the index.

[0101] Step 404: Receive training samples distributed to the first trainer.

[0102] In the distributed computing process, computing task resources need to be sharded, usually by sharding the training samples and then distributing them to different trainers.

[0103] Step 405: Obtain dense parameters locally from the first trainer, adopt training samples and use the dense parameters to train the deep learning model, thereby calculating the loss value during the forward propagation process and obtaining the dense parameter gradient by derivation during the back propagation process.

[0104] Step 406: Determine a sparse parameter identifier based on the training samples distributed to the first trainer; determine the storage address of the sparse parameter corresponding to the sparse parameter identifier in the local memory of the training machine based on the index, thereby obtaining the sparse parameter from the local memory of the training machine; obtain the sparse parameter from the local memory of the training machine, use the training samples and the sparse parameters to train the deep learning model, thereby calculating the loss value during the forward propagation process, and obtaining the sparse parameter gradient by derivation during the back propagation process.

[0105] Step 407: Synchronize dense parameter gradients and sparse parameter gradients with other trainers based on the ring global reduction algorithm.

[0106] Step 408: Calculate the average value of the dense parameter gradients on each trainer to obtain the final dense parameter gradient; at the same time, calculate the final sparse parameter gradient based on the frequency of occurrence of the sparse parameters on each trainer.

[0107] like Figure 2 As shown in the figure, for sparse parameters, a shared memory approach is used to enable each trainer (ranker) on a training machine to share a copy of the Embedding parameters. Each ranker obtains and updates the sparse parameters through the local parameter server framework. After the All Gather operation is performed in each training machine to obtain the parameter gradients calculated by all global rankers, the first trainer (ranker0) on each training machine updates the sparse parameters in memory. A dual-loop logic structure is formed between each trainer to synchronize the dense parameter gradient and the sparse parameter gradient, as shown in the figure. Figure 3 shown.

[0108] After parameter gradient synchronization is complete, different calculation methods are used for dense and sparse parameters to calculate the final parameter gradients. For dense parameters, the first trainer averages the dense parameter gradients on each trainer to obtain the final dense parameter gradient. For sparse parameters, the first trainer calculates the final sparse parameter gradient based on the frequency of occurrence of the sparse parameter on each trainer.

[0109] Step 409: Update the local dense parameters of the first trainer based on the final dense parameter gradient.

[0110] Since each ranker stores a local copy of the dense parameters, after calculating the final dense parameter gradient, ranker0 directly updates the local dense parameters of ranker0. Similarly, after calculating the final dense parameter gradient, other rankers directly update their local dense parameters.

[0111] Step 410: Update the final sparse parameter gradient to the local memory of the training machine by calling the local parameter server framework.

[0112] Since each training machine and multiple trainers (ranker) share the same full set of sparse parameters, when updating parameters, it is necessary to determine whether the ranker is ranker0, and only ranker0 will update the sparse parameters. Figure 2 As shown in Figure 1, ranker0 updates the final sparse parameter gradient to the local memory of the training machine so that each ranker on this training machine can share the latest sparse parameters.

[0113] In view of the different characteristics of dense parameters and sparse parameters in application scenarios such as search, recommendation, and advertising, the embodiments of the present invention adopt two independent local parameter storage and update methods, which can significantly improve the utilization rate of training memory, enable the same machine configuration to train larger models, reduce the number of updates to sparse parameters, solve the problem of gradient failure when using remote parameter servers, ensure the accuracy of the model, and improve the training speed of the model. In the prior art, the environment configuration of five training machines with four trainers each used in the search and recommendation training scenario can save nearly 3 / 4 of the memory using the training method provided by the embodiments of the present invention.

[0114] In addition, the specific implementation content of the training method of the deep learning model in a reference embodiment of the present invention has been described in detail in the training method of the deep learning model described above, so the repeated content will not be described here.

[0115] Figure 5 Schematic diagram of the main process of the training method of the deep learning model according to another embodiment of the present invention. As another embodiment of the present invention, Figure 5 As shown, the training method of the deep learning model is applied to a second trainer, and the second trainer is deployed on a training standalone machine. The method may include the following steps:

[0116] Step 501: read the training sample, obtain the sparse parameters from the local memory of the training machine, and obtain the dense parameters from the local memory of the second trainer; wherein the first trainer and each second trainer on the training machine share the sparse parameters in the local memory of the training machine.

[0117] In this embodiment of the present invention, all the rankers except ranker0 on the training machine are called second trainers, so the second trainers can be ranker1, ranker2, ranker3, etc. Therefore, each second trainer locally stores a copy of the dense parameters and the index of the sparse parameters, and shares the sparse parameters in the local memory of the training machine.

[0118] Optionally, obtaining sparse parameters from the local memory of the training machine includes: determining a sparse parameter identifier based on the training samples distributed to the second trainer; determining the storage address of the sparse parameter corresponding to the sparse parameter identifier in the local memory of the training machine based on an index, thereby obtaining the sparse parameter from the local memory of the training machine; wherein, in the index, the primary key is the identifier of the sparse parameter, and the key value is the storage address of the sparse parameter in the local memory of the training machine. Since the sparse parameters are stored in the local memory of the training machine, and each trainer shares the sparse parameters in the local memory of the training machine, the second trainer can determine the storage address of the sparse parameter in the local memory of the training machine based on the index, thereby obtaining the sparse parameter from the local memory of the training machine.

[0119] Step 502: Calculate the loss value based on the dense parameters and the sparse parameters during the forward propagation process, and obtain the dense parameter gradient and the sparse parameter gradient during the backward propagation process.

[0120] Optionally, step 502 may include: using the training sample and respectively using the dense parameters and the sparse parameters to train the deep learning model, thereby calculating the loss value in the forward propagation process, and obtaining the dense parameter gradient and the sparse parameter gradient by derivation in the backward propagation process. Figure 2 As shown, the embodiment of the present invention performs distributed training on multiple machines and multiple cards based on Horovod's TheRing Allreduce synchronous training architecture. In the distributed computing process, it is necessary to shard the computing task resources, usually by sharding the training samples, and then distributing the training samples to different trainers. After the second trainer receives the training samples, it adopts a training mode in which dense parameters and sparse parameters are trained independently. Specifically, dense parameters are obtained locally from the second trainer, and the training samples are used to train the deep learning model using the dense parameters, so that the loss value is calculated during the forward propagation process, and the dense parameter gradient is obtained by derivation during the back propagation process; at the same time, sparse parameters are obtained from the local memory of the training machine, and the training samples are used to train the deep learning model using the sparse parameters, so that the loss value is calculated during the forward propagation process, and the sparse parameter gradient is obtained by derivation during the back propagation process.

[0121] It should be pointed out that since the parameters of the Embedding layer are very sparse, each step only updates some parameters related to the current batch, and the training samples of each trainer are different, so each trainer can determine the sparse parameter identifiers that need to be obtained based on the training samples.

[0122] Step 503: synchronize the dense parameter gradient and the sparse parameter gradient with other trainers to calculate the final dense parameter gradient.

[0123] The second trainer obtains the dense parameter gradient and the sparse parameter gradient, and synchronizes the dense parameter gradient and the sparse parameter gradient with other trainers (including other trainers on this training machine and each trainer on other training machines). The synchronization process can be carried out simultaneously. After the synchronization is completed, the second trainer calculates the final dense parameter gradient based on the synchronization results.

[0124] Optionally, step 503 may include: synchronizing dense parameter gradients and sparse parameter gradients with other trainers based on a ring global reduction algorithm; calculating the average value of the dense parameter gradients on each trainer to obtain the final dense parameter gradient. The embodiment of the present invention adopts a training mode in which dense parameters and sparse parameters are trained independently. In a multi-machine and multi-card training environment, different storage and update strategies are adopted for sparse parameters and dense parameters based on the Horovod architecture training mode. Figure 2 As shown in the figure, for sparse parameters, a shared memory approach is used to enable each trainer (ranker) on a training machine to share a copy of the Embedding parameters. Each ranker obtains and updates the sparse parameters through the local parameter server framework. After the All Gather operation is performed in each training machine to obtain the parameter gradients calculated by all global rankers, the first trainer (ranker0) on each training machine updates the sparse parameters in memory. A dual-loop logic structure is formed between each trainer to synchronize the dense parameter gradient and the sparse parameter gradient, as shown in the figure. Figure 3 After parameter gradient synchronization is complete, different calculation methods are used for dense and sparse parameters to calculate the final parameter gradients. For dense parameters, the second trainer averages the dense parameter gradients on each trainer to obtain the final dense parameter gradient. For sparse parameters, only the first trainer calculates the final sparse parameter gradient based on the frequency of occurrence of the sparse parameter on each trainer.

[0125] Step 504: Update the local dense parameters of the second trainer based on the final dense parameter gradient; wherein the final sparse parameter gradient is calculated and updated by the first trainer on the training machine.

[0126] After the second trainer calculates the final dense parameter gradient, it updates the local dense parameters of the second trainer based on the final dense parameter gradient.

[0127] It should be noted that since each training machine and multiple trainers (ranker) share the same full set of sparse parameters, when updating the parameters, it is necessary to determine whether the ranker is ranker0 (i.e. the first trainer), and then only ranker0 will update the sparse parameters. Figure 2 As shown in Figure 1, ranker0 updates the final sparse parameter gradient to the local memory of the training machine so that each ranker on this training machine can share the latest sparse parameters.

[0128] In addition, the specific implementation content of the training method of the deep learning model in another embodiment of the present invention has been described in detail in the training method of the deep learning model described above, so the repeated content will not be described here.

[0129] Figure 6 Schematic diagram of the main process of the training method of the deep learning model according to another reference embodiment of the present invention. As another embodiment of the present invention, Figure 6 As shown, the training method of the deep learning model is applied to a second trainer, and the second trainer is deployed on a training standalone machine, which may include:

[0130] Step 601: Receive training samples distributed to the second trainer.

[0131] Step 602: Obtain dense parameters locally from the second trainer, use training samples and the dense parameters to train the deep learning model, thereby calculating the loss value during the forward propagation process and obtaining the dense parameter gradient by derivation during the back propagation process.

[0132] Step 603: Determine a sparse parameter identifier based on the training samples distributed to the second trainer; determine the storage address of the sparse parameter corresponding to the sparse parameter identifier in the local memory of the training machine based on the index, thereby obtaining the sparse parameter from the local memory of the training machine; then use the training sample and the sparse parameter to train the deep learning model, thereby calculating the loss value during the forward propagation process and obtaining the sparse parameter gradient by derivation during the back propagation process.

[0133] Step 604: Synchronize dense parameter gradients and sparse parameter gradients with other trainers based on the ring global reduction algorithm.

[0134] Step 605: Calculate the average value of the dense parameter gradients on each trainer to obtain the final dense parameter gradient.

[0135] Step 606: Update the local dense parameters of the second trainer based on the final dense parameter gradient.

[0136] In addition, the specific implementation content of the training method of the deep learning model in another reference embodiment of the present invention has been described in detail in the training method of the deep learning model described above, so the repeated content will not be described here.

[0137] An embodiment of the present invention also provides a single training machine, on which a first trainer and respective second trainers are deployed, sparse parameters are stored in the local memory of the single training machine, and dense parameters are stored locally in the local memory of the first trainer and the local memory of the second trainer; wherein the first trainer and the respective second trainers share the sparse parameters in the local memory of the single training machine, and the sparse parameters are updated by the first trainer.

[0138] Due to the different storage modes of dense parameters and sparse parameters in multi-GPU training, different update strategies need to be adopted when updating model parameters. Figure 2-3 As shown, for dense parameters, each trainer (ranker0, ranker1, ranker2, ranker3, etc.) stores a full set of dense parameters. After synchronizing the dense parameter gradients, each trainer calculates the average value of the dense parameter gradients, and then each trainer updates its local dense parameters based on the gradients. For sparse parameters, since each training machine and multiple trainers (ranker) share the same full set of sparse parameters, when updating the parameters, it is necessary to determine whether the ranker is ranker0 (i.e., the first trainer), and only ranker0 will update the sparse parameters. Figure 2 As shown in Figure 1, ranker0 updates the final sparse parameter gradient to the local memory of the training machine so that each ranker on this training machine can share the latest sparse parameters.

[0139] The embodiment of the present invention is based on the storage mode of sparse parameters. It does not require each trainer on each training machine to completely save a full set of sparse parameters. Instead, through shared memory, a training machine only retains a full set of sparse parameters, and each trainer obtains the sparse parameters of the current trainer by accessing the index, which significantly improves memory utilization.

[0140] It can be seen that the embodiment of the present invention stores sparse parameters in the local memory of the training machine, so that each trainer shares the memory, and adopts a mode in which dense parameters and sparse parameters are trained independently locally, so that dense parameters and sparse parameters adopt different storage and update modes in the trainer, which not only improves the memory utilization rate and reduces the number of updates of sparse parameters, but also solves the problem of parameter gradient failure of the remote parameter server, ensures the accuracy of the model, and improves the training speed of the model.

[0141] Figure 7 Schematic diagram of the main modules of a training device for a deep learning model according to an embodiment of the present invention. Figure 7As shown, the training device 700 of the deep learning model is set in the first trainer, and the first trainer is deployed on the training stand. The device includes an index module 701, a first acquisition module 702, a first training module 703, a first calculation module 704 and a first update module 705; wherein the index module 701 is used to establish an index; the first acquisition module 702 is used to read the training sample, obtain the sparse parameters from the local memory of the training stand based on the index, and obtain the dense parameters from the local memory of the first trainer; wherein the first trainer and each second trainer on the training stand share the training The sparse parameters in the local memory of a single machine; the first training module 703 is used to calculate the loss value according to the dense parameters and the sparse parameters during the forward propagation process, and obtain the dense parameter gradient and the sparse parameter gradient during the back propagation process; the first calculation module 704 is used to synchronize the dense parameter gradient and the sparse parameter gradient with other trainers, and calculate the final dense parameter gradient and the final sparse parameter gradient; the first update module 705 is used to update the local dense parameters of the first trainer based on the final dense parameter gradient, and update the sparse parameters in the local memory of the training single machine based on the final sparse parameter gradient.

[0142] Optionally, the index module 701 is further configured to:

[0143] Calling a local parameter server framework to perform the training stand-alone local memory mapping on the sparse parameters and creating an index based on the mapping result;

[0144] Synchronizing the index to other trainers on the training machine;

[0145] Among them, the primary key is the identifier of the sparse parameter, and the key value is the storage address of the sparse parameter in the local memory of the training stand-alone machine.

[0146] Optionally, the first obtaining module 702 is further configured to:

[0147] Determining a sparse parameter identifier based on the training samples distributed to the first trainer;

[0148] The storage address of the sparse parameter corresponding to the sparse parameter identifier in the local memory of the training machine is determined based on the index, thereby obtaining the sparse parameter from the local memory of the training machine.

[0149] Optionally, the first training module 703 is further configured to:

[0150] The training samples are used to train a deep learning model using the dense parameters and the sparse parameters respectively, so that the loss value is calculated during the forward propagation process, and the dense parameter gradient and the sparse parameter gradient are obtained by derivation during the back propagation process.

[0151] Optionally, the first calculation module 704 is further configured to:

[0152] Based on the ring global reduction algorithm, synchronize dense parameter gradients and sparse parameter gradients with other trainers;

[0153] The average value of the dense parameter gradient on each trainer is calculated to obtain the final dense parameter gradient; at the same time, the final sparse parameter gradient is calculated based on the frequency of occurrence of sparse parameters on each trainer.

[0154] Optionally, the first calculation module 704 is further configured to:

[0155] For each sparse parameter, calculate the frequency of occurrence of the sparse parameter on each trainer;

[0156] For each sparse parameter, the sparse parameter gradients on each trainer are summed and then divided by the frequency of occurrence of the sparse parameter on each trainer to obtain a final sparse parameter gradient.

[0157] Optionally, the first updating module 705 is further configured to:

[0158] By calling the local parameter server framework, the final sparse parameter gradient is updated to the local memory of the training stand-alone machine.

[0159] It should be noted that the specific implementation content of the training device for the deep learning model described in the present invention has been described in detail in the training method for the deep learning model described above, so the repeated content will not be described here.

[0160] Figure 8 Schematic diagram of the main modules of a training device for a deep learning model according to another embodiment of the present invention. Figure 8As shown, the training device 800 of the deep learning model is set in the second trainer, and the second trainer is deployed on the training stand. The device includes a second acquisition module 801, a second training module 801802, a second calculation module 802803 and a second update module 803804; wherein the second acquisition module 801 is used to read the training sample, obtain the sparse parameters from the local memory of the training stand, and obtain the dense parameters from the local memory of the second trainer; wherein the first trainer and each second trainer on the training stand share the local memory of the training stand sparse parameters in; the second training module 802 is used to calculate the loss value according to the dense parameters and the sparse parameters during the forward propagation process, and obtain the dense parameter gradient and the sparse parameter gradient during the back propagation process; the second calculation module 803 is used to synchronize the dense parameter gradient and the sparse parameter gradient with other trainers to calculate the final dense parameter gradient; the second update module 804 is used to update the local dense parameters of the second trainer based on the final dense parameter gradient; wherein the final sparse parameter gradient is calculated and updated by the first trainer on the training machine.

[0161] Optionally, the second obtaining module 801 is further configured to:

[0162] Determining a sparse parameter identifier based on the training samples distributed to the second trainer;

[0163] Determine, based on the index, the storage address of the sparse parameter corresponding to the sparse parameter identifier in the local memory of the training stand-alone machine, thereby obtaining the sparse parameter from the local memory of the training stand-alone machine;

[0164] Among them, in the index, the primary key is the identifier of the sparse parameter, and the key value is the storage address of the sparse parameter in the local memory of the training stand-alone machine.

[0165] Optionally, the second training module 802 is further configured to:

[0166] The training samples are used to train a deep learning model using the dense parameters and the sparse parameters respectively, so that the loss value is calculated during the forward propagation process, and the dense parameter gradient and the sparse parameter gradient are obtained by derivation during the back propagation process.

[0167] Optionally, the second calculation module 803 is further configured to:

[0168] Based on the ring global reduction algorithm, synchronize dense parameter gradients and sparse parameter gradients with other trainers;

[0169] The average of the dense parameter gradients on each trainer is calculated to obtain the final dense parameter gradient.

[0170] It should be noted that the specific implementation content of the training device for the deep learning model described in the present invention has been described in detail in the training method for the deep learning model described above, so the repeated content will not be described here.

[0171] Figure 9 An exemplary system architecture 900 is shown to which a deep learning model training method or a deep learning model training apparatus according to an embodiment of the present invention may be applied.

[0172] like Figure 9 As shown, system architecture 900 may include terminal devices 901, 902, 903, a network 904, and a server 905. Network 904 is used to provide a medium for communication links between terminal devices 901, 902, 903 and server 905. Network 904 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0173] Users can use terminal devices 901, 902, and 903 to interact with server 905 via network 904 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 901, 902, and 903, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0174] The terminal devices 901 , 902 , and 903 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0175] Server 905 may be a server that provides various services, such as a backend management server (for example only) that supports shopping websites browsed by users using terminal devices 901, 902, and 903. The backend management server may analyze and process received data such as item information query requests, and feed back the processing results (for example, target push information and item information—for example only) to the terminal device.

[0176] It should be noted that the training method for the deep learning model provided in the embodiment of the present invention is generally executed by the server 905, and accordingly, the training device for the deep learning model is generally set in the server 905. The training method for the deep learning model provided in the embodiment of the present invention can also be executed by the terminal devices 901, 902, and 903, and accordingly, the training device for the deep learning model can be set in the terminal devices 901, 902, and 903.

[0177] It should be understood that Figure 9The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0178] Reference below Figure 10 , which shows a schematic structural diagram of a computer system 1000 of a terminal device suitable for implementing an embodiment of the present invention. Figure 10 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0179] like Figure 10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the system 1000 are also stored in the RAM 1003. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0180] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk; and a communication section 1009 including a network interface card such as a LAN card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read therefrom can be installed into the storage section 1008 as needed.

[0181] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009, and / or installed from a removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the above-mentioned functions defined in the system of the present invention are performed.

[0182] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer programs according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0184] The modules described in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, a processor may be described as comprising an indexing module, a first reading module, a first training module, a first computing module, and a first updating module. The names of these modules do not, in some cases, limit the modules themselves.

[0185] The modules described in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, a processor may be described as including a second reading module, a second training module, a second computing module, and a second updating module. The names of these modules do not, in some cases, limit the modules themselves.

[0186] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device implements the following method: obtaining sparse parameters and storing the sparse parameters in the local memory of the training unit; reading training samples, and obtaining dense parameter gradients and sparse parameter gradients using dense parameters and sparse parameters respectively; wherein the dense parameters are stored locally in the first trainer; synchronizing the dense parameter gradients and sparse parameter gradients with other trainers to calculate the final dense parameter gradients and the final sparse parameter gradients; updating the local dense parameters of the first trainer based on the final dense parameter gradients, and updating the sparse parameters in the local memory of the training unit based on the final sparse parameter gradients.

[0187] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device implements the following method: reading training samples, and obtaining dense parameter gradients and sparse parameter gradients using dense parameters and sparse parameters respectively; wherein the dense parameters are stored locally in the second trainer, and the sparse parameters are stored in the local memory of the training stand; synchronizing the dense parameter gradients and sparse parameter gradients with other trainers to calculate the final dense parameter gradients; and updating the local dense parameters of the second trainer based on the final dense parameter gradients.

[0188] As another aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which implements the method described in any of the above embodiments when executed by a processor.

[0189] According to the technical solution of the embodiment of the present invention, the technical problems of insufficient memory and parameter gradient failure in the prior art are overcome by adopting the technical means of storing sparse parameters in the local memory of the training stand-alone machine so that each trainer shares the memory and completes the parameter gradient synchronization and then performs the local parameter update. The embodiment of the present invention stores sparse parameters in the local memory of the training stand-alone machine so that each trainer shares the memory, and adopts a mode in which dense parameters and sparse parameters are trained independently locally, so that dense parameters and sparse parameters adopt different storage and update modes in the trainer, which not only improves the memory usage and reduces the number of updates of sparse parameters, but also solves the problem of parameter gradient failure of the remote parameter server, ensures the accuracy of the model, and improves the training speed of the model.

[0190] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A training method for a deep learning model, characterized in that: Applied to a first trainer, the first trainer is deployed on a single training machine, including: Establish an index; in the index, the primary key is the identifier of the sparse parameter, and the key value is the storage address of the sparse parameter in the local memory of the training stand-alone machine; Reading training samples, obtaining sparse parameters from the local memory of the training machine based on the index, and obtaining dense parameters locally from the first trainer; wherein the first trainer and each second trainer on the training machine share the sparse parameters in the local memory of the training machine; The loss value is calculated based on the dense parameters and sparse parameters during the forward propagation process, and the dense parameter gradient and sparse parameter gradient are obtained during the backward propagation process; Synchronize dense parameter gradients and sparse parameter gradients with other trainers to calculate the final dense parameter gradients and the final sparse parameter gradients; The local dense parameters of the first trainer are updated based on the final dense parameter gradient, and the sparse parameters in the local memory of the training machine are updated based on the final sparse parameter gradient.

2. The method according to claim 1, characterized in that Indexing includes: Calling a local parameter server framework to perform the training stand-alone local memory mapping on the sparse parameters and creating an index based on the mapping result; Synchronizing the index to other trainers on the training machine; Among them, the primary key is the identifier of the sparse parameter, and the key value is the storage address of the sparse parameter in the local memory of the training stand-alone machine.

3. The method according to claim 2, characterized in that Obtaining sparse parameters from the local memory of the training machine based on the index includes: Determining a sparse parameter identifier based on the training samples distributed to the first trainer; The storage address of the sparse parameter corresponding to the sparse parameter identifier in the local memory of the training machine is determined based on the index, thereby obtaining the sparse parameter from the local memory of the training machine.

4. The method according to claim 1, wherein During the forward propagation process, the loss value is calculated based on the dense parameters and sparse parameters. During the backward propagation process, the dense parameter gradient and the sparse parameter gradient are obtained, including: The training samples are used to train a deep learning model using the dense parameters and the sparse parameters respectively, so that the loss value is calculated during the forward propagation process, and the dense parameter gradient and the sparse parameter gradient are obtained by derivation during the back propagation process.

5. The method according to claim 1, wherein Synchronize dense parameter gradients and sparse parameter gradients with other trainers to calculate the final dense parameter gradients and the final sparse parameter gradients, including: Based on the ring global reduction algorithm, synchronize dense parameter gradients and sparse parameter gradients with other trainers; The average value of the dense parameter gradient on each trainer is calculated to obtain the final dense parameter gradient; at the same time, the final sparse parameter gradient is calculated based on the frequency of occurrence of sparse parameters on each trainer.

6. The method according to claim 5, characterized in that According to the frequency of occurrence of sparse parameters on each trainer, the final sparse parameter gradient is calculated, including: For each sparse parameter, calculate the frequency of occurrence of the sparse parameter on each trainer; For each sparse parameter, the sparse parameter gradients on each trainer are summed and then divided by the frequency of occurrence of the sparse parameter on each trainer to obtain a final sparse parameter gradient.

7. The method according to claim 1, characterized in that Updating the sparse parameters in the local memory of the training machine based on the final sparse parameter gradient includes: By calling the local parameter server framework, the final sparse parameter gradient is updated to the local memory of the training stand-alone machine.

8. A training method for a deep learning model, characterized in that: Applied to a second trainer, the second trainer is deployed on a single training machine, including: Reading training samples, obtaining sparse parameters from the local memory of the training machine, and obtaining dense parameters from the local memory of the second trainer; wherein the first trainer and each second trainer on the training machine share the sparse parameters in the local memory of the training machine; During the forward propagation process, a loss value is calculated based on the dense parameters and the sparse parameters, and during the backward propagation process, a dense parameter gradient and a sparse parameter gradient are obtained; wherein the dense parameters are stored locally in the second trainer, and the sparse parameters are stored in the local memory of the training machine, and each second trainer on the training machine shares the sparse parameters in the local memory of the training machine; Synchronize dense parameter gradients and sparse parameter gradients with other trainers to calculate the final dense parameter gradients; Updating the local dense parameters of the second trainer based on the final dense parameter gradient; The final sparse parameter gradient is calculated and updated by the first trainer on the training machine.

9. The method according to claim 8, characterized in that Obtaining sparse parameters from the local memory of the training machine includes: Determining a sparse parameter identifier based on the training samples distributed to the second trainer; Determine, based on the index, the storage address of the sparse parameter corresponding to the sparse parameter identifier in the local memory of the training stand-alone machine, thereby obtaining the sparse parameter from the local memory of the training stand-alone machine; Among them, in the index, the primary key is the identifier of the sparse parameter, and the key value is the storage address of the sparse parameter in the local memory of the training stand-alone machine.

10. The method according to claim 8, characterized in that During the forward propagation process, the loss value is calculated based on the dense parameters and sparse parameters. During the backward propagation process, the dense parameter gradient and the sparse parameter gradient are obtained, including: The training samples are used to train a deep learning model using the dense parameters and the sparse parameters respectively, so that the loss value is calculated during the forward propagation process, and the dense parameter gradient and the sparse parameter gradient are obtained by derivation during the back propagation process.

11. The method according to claim 8, characterized in that Synchronize dense parameter gradients and sparse parameter gradients with other trainers to calculate the final dense parameter gradients, including: Based on the ring global reduction algorithm, synchronize dense parameter gradients and sparse parameter gradients with other trainers; The average of the dense parameter gradients on each trainer is calculated to obtain the final dense parameter gradient.

12. A training stand-alone machine, characterized in that: A first trainer and each second trainer are deployed on the training machine, sparse parameters are stored in the local memory of the training machine, and dense parameters are stored in the local memory of the first trainer and the local memory of the second trainer; The first trainer and each second trainer share the sparse parameters in the local memory of the training machine, and the sparse parameters are updated by the first trainer using the training method of the deep learning model described in any one of claims 1-7.

13. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 11.

14. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images