Model training method and device

Abstract

The invention provides a model training method and device. The method comprises the following steps: a first training node obtains a first number of training samples from a training sample resource pool, and determines a gradient parameter corresponding to jth training of a first model according to the first number of training samples and a first model parameter during jth training of the first model, and determining a gradient descent learning rate corresponding to the jth training of the first model according to the number of times of training performed by each training node in the stored plurality of training nodes in the ith round of training process, and determining the jth training of the first model according to the gradient parameter and the gradient descent learning rate corresponding to the jth training of the first model, and updating a first model parameter of the first model during the jth training to obtain a second model parameter. The gradient descent learning rate of each training node is calculated according to the number of times of training performed by the multiple training nodes, it can be ensured that the gradient descent learning rates of the multiple training nodes at the same time are kept consistent, and therefore the model training effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method and device. Background Art

[0002] With the rapid development of artificial intelligence technology, artificial intelligence (AI) models are widely used in many fields, such as smart transportation, smart medical care, smart home, autonomous driving, finance and other fields. In order to improve the training efficiency of AI models, the training scheme adopted is usually to perform distributed training on AI models, that is, to train AI models in parallel using multiple devices (or training devices or training nodes). Among them, in the distributed training process of AI models, the main aspects involved are the allocation of training samples and the calculation of learning rate (or gradient descent learning rate).

[0003] Among them, in terms of training sample allocation, the industry has proposed some solutions, some of which are to evenly allocate the same number of training samples to multiple devices participating in AI model training. However, due to the different computing speeds of different devices, some devices will complete the calculation first while other devices are still calculating. In this case, if the calculation of all devices ends when the fastest device is finished, there will be a problem of wasting some training samples, thereby affecting the training accuracy of the AI ​​model; if the calculation of all devices ends only after the slowest device is finished, the faster device needs to wait for the slowest device to start the next round of AI model training, resulting in a waste of computing power and reducing the training speed of the AI ​​model. One solution is to manually allocate different numbers of training samples to multiple devices participating in AI model training when the AI ​​model starts each round of training. However, during the AI ​​model training process, the computing power of the device may be affected by various factors (such as temperature changes, resource changes, etc.) and is in a dynamic change process. Therefore, it is difficult to allocate a suitable number of training samples to multiple devices by manually allocating training samples, and there will also be problems of wasting training samples or waiting for devices.

[0004] In terms of learning rate calculation, one solution is to assign different learning rates to multiple devices participating in AI model training based on the computing speed of the devices. However, this solution will cause different learning rates for multiple devices at the same time, which will not only affect the training accuracy of the AI ​​model, but also make the learning rate unable to align with the synchronous training scenario. Therefore, multiple experiments are required to adjust the hyperparameters, which is time-consuming and laborious.

[0005] Therefore, the distributed training of AI models still needs further research. Summary of the invention

[0006] The present application provides a model training method and device to improve the model training effect.

[0007] In the first aspect, the present application provides a model training method, which can be performed by a model training device or a component (such as a chip, a chip system or a circuit, etc.) that can support the model training device to implement the functions required by the method. Exemplarily, the model training device can be any training node in a distributed training architecture, or it can also be a functional element (such as a plug-in, a component, a chip or a circuit, etc.) in any training node in the distributed training architecture, and the functional element has the function of implementing the model training method. Optionally, the distributed training architecture includes multiple training nodes, and the multiple training nodes can be used to train the first model in parallel, and the computing power of the multiple training nodes may also be different. For example, taking the model training device as the first training node included in the distributed training architecture, and the first training node executing the model training method as an example, the method may include the following steps:

[0008] The first training node may first obtain a first number of training samples from a training sample resource pool, wherein the first number of training samples are used for the first training node to perform the j-th training on the first model during the i-th round of training, and the first training node is any training node among the multiple training nodes. Afterwards, the first training node may determine the gradient parameters corresponding to the first model at the j-th training based on the first number of training samples and the first model parameters of the first model during the j-th training, and may determine the gradient descent learning rate corresponding to the first model at the j-th training based on the number of training times that each training node in the multiple training nodes saved during the i-th round of training. Then, the first training node may update the first model parameters of the first model during the j-th training based on the gradient parameters and the gradient descent learning rate corresponding to the first model at the j-th training, and obtain the second model parameters, wherein the second model parameters are used as the model parameters when the first training node performs the j+1-th training on the first model during the i-th round of training.

[0009] In this method, multiple training nodes are calculated by the gradient descent learning rate according to the number of training times of the first model in the i-th round of training according to the multiple training nodes, so that the gradient descent learning rates of multiple training nodes at the same time (or at the same moment) are kept consistent, thereby unifying the gradient descent learning rates of multiple training nodes, and there is no need to manually adjust the gradient descent learning rates of training nodes with fast or slow training speeds, thereby ensuring the convergence of model training, thereby reducing the model parameter debugging time and improving the model training effect. In addition, each training node obtains training samples from a training sample resource pool instead of manually allocating training samples to each training node, so that the problem of the total model training time becoming longer due to the manual allocation of training samples making the model training speed slower can be avoided, and the adaptive allocation of a suitable number of training samples to each training node can be achieved, effectively avoiding the problem of wasting training samples or empty training nodes.

[0010] In one possible design, a training sample resource pool may be located on a resource management node, and multiple training nodes may communicate with the resource management node respectively; a first training node obtains a first number of training samples from the training sample resource pool, including: the first training node may first send a first request to the resource management node, and then the first training node may receive the first number of training samples from the resource management node, wherein the first request is used to request the acquisition of the training samples required for the first training node to perform the j-th training of the first model during the i-th round of training.

[0011] In the above design, when any training node performs the jth training in the i-th round of training for the first model, it actively requests a certain number of training samples from the resource management node, and the resource management node dynamically allocates a certain number of training samples to each training node according to the request of each training node, so that the training samples in the training sample resource pool are adaptively allocated to each training node, thereby avoiding the problem that the training accuracy of the model becomes poor due to the waste of training samples caused by the average distribution of training samples or the manual distribution of training samples, and avoiding the problem that the total model training time becomes long due to the average distribution of training samples or the manual distribution of training samples, which makes the model training speed slower. Of course, it can also avoid wasting the computing power of the training nodes with fast training speed, thereby improving the model training accuracy and effectively improving the model training effect. It should be understood that the speed of training of the first model by the training node is related to the computing power of the training node.

[0012] In one possible design, the first training node determines the gradient parameters corresponding to the j-th training based on the first number of training samples and the first model parameters when the first model is trained for the j-th time, including: the first training node can first determine the loss function value corresponding to the j-th training of the first model based on the first number of training samples and the first model parameters, and then the first training node can use the gradient parameter calculation algorithm to perform a back propagation operation on the loss function value to determine the gradient parameters corresponding to the j-th training of the first model.

[0013] In the above design, by back-propagating the loss function value corresponding to the j-th training of the first model, the gradient parameters corresponding to the j-th training of the first model can be accurately obtained, which helps to effectively update the model parameters used by the first model in the j-th training.

[0014] In one possible design, the first training node determines the gradient descent learning rate corresponding to the j-th training according to the number of training times that each training node in the multiple training nodes saved in the i-th round of training, including: the first training node may first calculate the average or weighted average of the number of training times that the multiple training nodes have respectively performed in the i-th round of training, and then the first training node may calculate the gradient descent learning rate corresponding to the j-th training of the first model according to the initial gradient descent learning rate corresponding to the i-th round of training of the first model, and the average or weighted average.

[0015] Compared with the prior art scheme in which multiple training devices calculate the learning rate according to the number of training times performed by the training device or assign different learning rates to multiple training devices, the above design can ensure that the gradient descent learning rates of multiple training nodes are the same at the same time, thereby unifying the gradient descent learning rates of multiple training nodes, and under the same total training sample size, the gradient descent learning rate is also aligned with the synchronous training scenario, avoiding hyperparameter search, thereby effectively reducing the model parameter debugging time and improving the model training effect while ensuring the model training accuracy, and can realize dynamic adjustment of the gradient descent learning rate.

[0016] In one possible design, the method also includes: the first training node can send a second request to the second training node at a set time interval, wherein the second request is used to request the number of training times performed by the second training node in the i-th round of training, and the second training node is any training node among multiple training nodes except the first training node. Thereafter, after receiving the number of training times performed in the i-th round of training from the second training node, the first training node can save the number of training times performed by the second training node in the i-th round of training.

[0017] In the above design, multiple training nodes communicate with each other, so that each training node can obtain the number of training times that other training nodes have performed during the i-th round of training. This ensures that the gradient descent learning rate calculated by multiple training nodes at the same time is the same, and the gradient descent learning rate can be aligned with the synchronous training scenario, avoiding hyperparameter search.

[0018] In one possible design, when the i-th round of training of the first model by multiple training nodes begins, the training sample resource pool includes a second number of training samples, and when there are no training samples in the training sample resource pool, the i-th round of training of the first model by the multiple training nodes ends, wherein the second number is greater than the first number, and the training samples are used for the i-th round of training of the first model.

[0019] In the above design, multiple training nodes can obtain a certain number of training samples from the training sample resource pool to participate in the training of the first model during the i-th round of training of the first model by sharing the training sample resource pool, and the obtained training samples will be removed from the training sample resource pool. In this way, when the training sample resource pool is empty, multiple training nodes respectively end the i-th round of training of the first model, so that multiple training nodes can fully utilize all training samples in the training sample resource pool, and when all training samples in the training sample resource pool are fully utilized, multiple training nodes complete the i-th round of training for the first model at the same time, thereby ensuring the model training effect and reducing the model training time.

[0020] In one possible design, the first model may be a distributed neural network model.

[0021] In a possible design, the method also includes: after the first training node completes the jth training for the first model using the first number of training samples, it can continue to request a certain number of training samples from the training sample resource pool to participate in the j+1th training of the first model. When the training sample resource pool is not empty, the first training node can obtain a fourth number of training samples from the training sample resource pool, wherein the fourth number of training samples is used by the first training node to perform the j+1th training on the first model during the i-th round of training, and the fourth number is less than the second number. Afterwards, the first training node can determine the gradient parameters corresponding to the j+1th training based on the fourth number of training samples and the first model parameters when the first model is trained for the j+1th time, and can determine the gradient descent learning rate corresponding to the j+1th training based on the number of trainings that each training node in the saved multiple training nodes has performed during the i-th round of training. Then, the first training node can update the second model parameters based on the gradient descent learning rate and gradient parameters corresponding to the j+1th training to obtain fourth model parameters, and the fourth model parameters are used as model parameters when the first training node performs the j+2th training on the first model during the i-th round of training.

[0022] In the above design, after the first training node completes each training for the first model, it can continue to request a certain number of training samples from the training sample resource pool to participate in the next training of the first model. In this way, when the computing power of the first training node is relatively large, it can execute multiple requests to obtain a certain number of training samples from the training sample resource pool. In this way, the first training node can train the first model more times, and the training samples included in the training sample resource pool can be dynamically allocated to multiple training nodes based on the computing power of multiple training nodes, and the data load (or the number of training samples) can be matched with the computing power of the training node. Therefore, there will be no phenomenon that the training node with large computing power waits for the training node with small computing power, and the computing power of multiple training nodes can be fully utilized to achieve the effect of accelerating model training.

[0023] In the second aspect, the present application provides a possible model training device. Exemplarily, the model training device can be any training node in a distributed training architecture (such as a computing device or a computing device cluster), or it can also be a functional element in any training node in a distributed training architecture (such as a plug-in, component, chip or circuit, etc.), and the functional element has the function of implementing the model training method. For example, taking the example that the model training device is a chip arranged in a computing device, when the model training device is a chip, the model training device includes a communication interface and a processor, but does not include a memory. Among them, the communication interface exists as an input and output interface, and the input and output interface is used for the chip to implement the sending and receiving of the computing device. The input and output interface may include an input interface and / or an output interface, the input interface can implement the reception of the computing device, and the output interface can be used to implement the sending of the computing device. The processor is used to read and execute corresponding computer programs or instructions so that the corresponding functions of the model training device are implemented.

[0024] When the model training device is used to implement the method in any possible design of the first aspect, the beneficial effects can be found in the corresponding description of the first aspect, which will not be repeated here. The model training device has the function of implementing any possible design of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the model training device includes an acquisition module, a determination module and an update module. The acquisition module is used to acquire a first number of training samples from a training sample resource pool, wherein the first number of training samples is used for the first training node to perform the jth training of the first model during the i-th round of training, and the first training node is any training node among multiple training nodes. The determination module is used to determine the gradient parameters corresponding to the first model in the j-th training according to the first number of training samples and the first model parameters when the first model is trained for the j-th time. The determination module is also used to determine the gradient descent learning rate corresponding to the first model in the j-th training according to the number of training times that each training node in the multiple training nodes has performed in the i-th round of training. An updating module is used to update the first model parameters according to the gradient descent learning rate and gradient parameters corresponding to the first model in the jth training, and obtain the second model parameters, wherein the second model parameters are used as the model parameters when the first training node performs the j+1th training on the first model in the i-th round of training. These modules can perform the corresponding functions in any possible design of the first aspect above, and refer to the detailed description in the method example for details, which will not be repeated here.

[0025] In the third aspect, the present application provides a possible computing device, which has the function of implementing the behavior of the model training device in the method example of the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The computing device includes a communication interface, a memory, and a processor. Among them, the communication interface is used to communicate with other devices, the memory is used to store computer programs or instructions, and the processor is coupled to the memory and the communication interface. When the processor executes the computer program or instructions, the computing device executes the method in any possible design of the first aspect above.

[0026] In a fourth aspect, the present application provides a possible computing device cluster, which has the function of implementing the behavior of the model training device in the method example of the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The computing device cluster includes at least one computing device, any computing device includes a communication interface, a processor and a memory, and the processor in any computing device is configured to support the model training device to perform part or all of the functions of the method in any possible design of the first aspect above. The memory is coupled to the processor, which stores the necessary computer program instructions and data for the model training device.

[0027] In a fifth aspect, the present application provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method in any possible design of the first aspect above.

[0028] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute a method in any possible design of the first aspect described above.

[0029] In a seventh aspect, the present application further provides a computer chip, which is coupled to a memory, and is used to read a computer program stored in the memory to execute a method in any possible design of the first aspect above.

[0030] In an eighth aspect, the present application further provides a chip system, the chip system comprising a processor for supporting a computer device to implement the method in any possible design of the first aspect. In one possible design, the chip system further comprises a memory, the memory being used to store the necessary programs and data of the computer device. The chip system may be composed of a computer chip, or may include a computer chip and other discrete devices.

[0031] Based on the implementations provided in the above aspects, the present application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of a possible application scenario provided by an embodiment of the present application is exemplified;

[0033] Figure 2 A flowchart of a model training method provided in an embodiment of the present application is exemplified;

[0034] Figure 3 A schematic diagram exemplarily shows a training node obtaining a training sample from a training sample resource pool provided in an embodiment of the present application;

[0035] Figure 4 A schematic diagram showing, by way of example, a method provided by an embodiment of the present application in which each training node calculates a gradient descent learning rate corresponding to each training of a first model in an i-th round of training;

[0036] Figure 5 A flowchart of another model training method provided in an embodiment of the present application is exemplified;

[0037] Figure 6 A schematic diagram showing a possible structure of a model training device provided in an embodiment of the present application is exemplified;

[0038] Figure 7 A schematic diagram showing a possible structure of a computing device provided by an embodiment of the present application is exemplified;

[0039] Figure 8 A schematic diagram showing a possible computing device cluster structure provided by an embodiment of the present application is exemplified;

[0040] Fig. 9 A schematic diagram illustrating the structure of another possible computing device cluster provided in an embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0041] Before introducing the technical solution provided by the present application, some of the terms involved in the present application are first explained to facilitate understanding by those skilled in the art.

[0042] Computing power: refers to the computing power of a device. From mobile phones and personal computers (PCs) to supercomputers, without computing power, there will be no normal application of various software and hardware. For personal computers, the higher the central processing unit (CPU), graphics card, and memory configuration, the higher the computing power.

[0043] Optional, the unit of computing power is an indicator and benchmark for measuring the strength of computing power. There are currently many different measurement methods. Common measurement methods include Million Instructions Per Second (MIPS), Dhrystone Million Instructions executed Per Second (DMIPS), Operations Per Second (OPS), Floating-point Operations Per Second (FLOPS), Hash Per Second (Hash / s), etc.

[0044] For example, FLOPS has always been regarded as one of the main indicators for measuring computer computing speed. In terms of magnitude, the computing power of personal computers is at the GFLOPS level.

[0045] Optionally, computing power can be divided into two categories according to application areas:

[0046] General computing power: small amount of computing; conventional applications consume only a small amount of computing power.

[0047] High-performance computing (HPC) computing power: large amount of computing; a job requires a huge number of computing resources. HPC is a computer cluster system that connects multiple computer systems together through various interconnection technologies and uses the combined computing power of all connected systems to handle large computing problems, so it is often called a high-performance computing cluster.

[0048] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0049] The following is an introduction to possible application scenarios to which the model training method provided in this application is applicable. It should be noted that these introductions are for the convenience of understanding by those skilled in the art and do not constitute a limitation on the scope of protection claimed in this application.

[0050] Figure 1 The following is a schematic diagram showing a possible application scenario to which the present application embodiment is applicable. Figure 1As shown, the application scenario may include a terminal device 100 and a distributed training architecture 200. Exemplarily, the distributed training architecture 200 may include multiple training nodes, such as training node 201, training node 202, training node 203 and training node 204. Exemplarily, multiple training nodes can be used to train a certain model (such as the first model) in parallel, and the computing power of multiple training nodes may be different or the same. Optionally, any two of the multiple training nodes can be communicatively connected; the client terminal device 100 and any training node in the distributed training architecture 200 can be communicatively connected through one or more networks. The network can be a wired network or a wireless network. For example, the wireless network can be a wireless fidelity (WIREless-Fidelity, WIFI) network, or it can be a mobile cellular network, or it can be other forms of networks, which are not limited in the embodiments of the present application.

[0051] Among them, a target application for assisting in training the model is installed on the terminal device 100. Among them, the user can edit and submit a training request for a certain model (such as a certain neural network model) to one or more training nodes in the distributed training architecture 200 through the target application. Among them, the training request of the model may include the training information of the model, for example, the training information of the model may include the identification (or name) of the model or the model, etc. Exemplarily, the target application may be a client, a web application, or a small program embedded in other applications, etc. Optionally, the terminal device 100 may include a display for presenting the configuration interface provided by the distributed training architecture 200 to achieve human-computer interaction. Among them, the configuration interface displays information edited by the user or information provided to the user, etc.

[0052] Exemplarily, the terminal device 100 may be a smart phone, a tablet computer, a desktop computer, a computer with wireless transceiver function (such as a laptop computer, etc.), a PPD (Personal Digital Assistant), a mobile internet device (MID), a vehicle terminal, a wearable device with wireless communication function (such as a smart watch, a smart bracelet, smart glasses or a smart helmet, etc.), a virtual reality (VR) device, an augmented reality (AR) device or a smart home device (such as a smart speaker or a smart TV, etc.), etc. It should be understood that the present application does not limit the specific device form of the terminal device.

[0053] Any training node in the distributed training architecture 200 is a device (such as a server or computing device) used to provide services for users to perform corresponding operations on the target application installed on the terminal device 100. For example, any training node in the distributed training architecture 200 can be a model training device, or it can also be a computing device (such as a computer), a computing device cluster or other arbitrary device with a model training function, or it can also be a functional element (such as a chip or circuit with a model training function, etc.) set in a computing device, a computing device cluster or other arbitrary device. The model training device can be an independent physical server, or it can also be a server cluster or distributed system composed of multiple physical servers, or it can also be a cloud server for providing cloud services, cloud computing, cloud storage, cloud communications, network services, security services, big data and other cloud computing services. Exemplarily, the devices involved in the distributed training architecture 200 may include neural network computing devices (such as Huawei Ascend 910A 910B, Nvidia GPU (such as V100 A100, etc.), general-purpose CPU and other computing devices), servers that carry the above computing devices, and transmission media between devices (such as optical modules, Peripheral Component Interconnect Express (PCIE), network cables, etc.).

[0054] Exemplarily, the terminal device 100 is a desktop computer, the target application is a web browser installed on the desktop computer for assisting users in uploading model training information, the training node in the distributed training architecture 200 is a computing device with a model training function, and the model that a certain user (such as user A) needs to train is model A. User A can log in to the configuration interface provided by the distributed training architecture 200 through the web browser installed on the desktop computer used, and can edit the training information of model A on the configuration interface. In one example, after user A edits the training information of model A, he can send the training information of model A to a training node (such as training node 201) in the distributed training architecture 200. After obtaining the training information of model A, training node 201 can execute the training process for model A, and can synchronize the training information of model A to other training nodes (such as training node 202, training node 203 and training node 204, etc.), so that other training nodes can execute the training process for model A in parallel. In another example, after user A edits the training information of model A, he can send the training information of model A to multiple training nodes in the distributed training architecture 200. After the multiple training nodes respectively obtain the training information of model A, they can execute the training process for model A in parallel.

[0055] Optionally, the model training method provided in the embodiment of the present application can be used in a data-parallel distributed neural network training scenario, and can also be applied to other data-parallel distributed computing or other distributed tasks.

[0056] It should be noted that Figure 1 Only one possible application scenario is provided schematically, which is intended to more clearly illustrate the technical solution of the embodiment of the present application and does not limit the application scenario of the model training method provided in the present application. Figure 1 The form and quantity of each device in the application scenario shown are only for example and do not constitute a limitation of the present application. Figure 1 The names of the various devices in the application scenarios shown are only examples, and the names of the various devices in the specific implementation may also be other names, which are not specifically limited in this application. In addition, it is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0057] The following is based on Figure 1 The application scenario shown is used to introduce the specific implementation of the model training method in the embodiment of the present application in detail.

[0058] Figure 2 The following is a flow chart showing a model training method provided in an embodiment of the present application. The method can be but is not limited to being applicable to Figure 1 The application scenario shown. Figure 1 In the illustrated application, the distributed training architecture 200 includes multiple training nodes, which can be used to train a certain model (such as the first model) in parallel. The method flow can be executed by a model training device or a component (such as a chip, a chip system or a circuit, etc.) that can support the model training device to implement the functions required by the method. Exemplarily, the model training device can be Figure 1 Any training node in the distributed training architecture 200 shown, or Figure 1The functional element (such as a plug-in, component, chip or circuit, etc.) in any training node in the distributed training architecture 200 shown in the figure has the function of implementing the model training method. For example, any training node can be a computing device with a model training function or a computing device cluster with a model training function. In order to facilitate the introduction of the technical solution provided in the embodiment of the present application, the implementation process of the model training method executed by the first training node is introduced below, taking the model training device as the first training node included in the distributed training architecture 200 as an example. Among them, the first training node is any training node among the multiple training nodes included in the distributed training architecture 200. In one example, after a training node included in the distributed training architecture 200 obtains a training request submitted by a user through a terminal device, it can synchronize the training request to other training nodes. Among them, the training request is used to request training of the first model. In another example, the multiple training nodes included in the distributed training architecture 200 can respectively obtain the training requests submitted by the user through the terminal device. As Figure 2 As shown, the method includes:

[0059] Step 201: A first training node obtains a first number of training samples from a training sample resource pool.

[0060] Among them, the first number of training samples can be used for the first training node to perform the jth training on the first model during the i-th round of training. Exemplarily, the first model can be a distributed neural network model or other forms of neural network models (such as artificial intelligence (AI) large models), etc. It should be understood that any training node obtains a certain number of training samples from the training sample resource pool at least once during the i-th round of training for the first model, and uses a certain number of training samples obtained each time to train the first model. In other words, a training node performs one or more training on the first model during the i-th round of training of the first model. For example, taking a certain training node (such as training node A) as an example, at the beginning of the i-th round of training of the first model, training node A obtains a certain number of training samples (such as a number of training samples a0) from the training sample resource pool for the first time, and uses a certain number of training samples obtained for the first time to perform the first training for the first model. After completing the first training for the first model, if the training sample resource pool is not empty, training node A obtains a certain number of training samples (for example, a1 number of training samples) from the training sample resource pool for a second time, and uses the certain number of training samples obtained for the second time to perform the second training for the first model, and so on, until the training sample resource pool is empty. In this way, training node A has performed one or more trainings for the first model during the i-th round of training.

[0061] In the embodiment of the present application, the distributed training architecture (or distributed training system) maintains a common training sample resource pool (or data resource pool), and multiple training nodes included in the distributed training architecture share the training sample resource pool. Exemplarily, there are many implementation methods that can be used for the training sample resource pool, such as training sample queues, training sample sets, or other implementation methods.

[0062] In one example, at the beginning of each round of training of the first model, the training sample resource pool may include the same number of training samples. In another example, at the beginning of each round of training of the first model, the training sample resource pool may include different numbers of training samples. Optionally, at the beginning of each round of training of the first model, the types (or types) of training samples included in the training sample resource pool may be the same. For example, taking the first model as a recommendation model as an example, the training samples included in the training sample resource pool are business data of recommended services. Optionally, after each round of training of the first model, there may be no training samples in the training sample resource pool. For example, when the i-th round of training of the first model by multiple training nodes begins, the training sample resource pool includes a second number of training samples. When the second number of training samples included in the training sample resource pool does not exist, the i-th round of training of the first model by the multiple training nodes is also ended. Wherein, the second number is greater than the first number, and the second number of training samples are used to participate in the i-th round of training of the first model. For example, taking the training sample resource pool including 100 training samples as an example, the training sample resource pool includes the 100 training samples before the start of the i-th round of training of the first model, and the 100 training samples are used to participate in the i-th round of training of the first model. When each training node performs the i-th round of training for the first model, it obtains a certain number of training samples (such as training samples with a number of b0) from the training sample resource pool for the first time, and uses the certain number of training samples obtained for the first time to perform the first training for the first model. Exemplarily, the training samples with a number of b0 will be removed from the training sample resource pool after being obtained by the training node. After the first training for the first model is completed, if the training sample resource pool is not empty, the training node obtains a certain number of training samples (such as training samples with a number of b1) from the training sample resource pool for the second time, and uses the certain number of training samples obtained for the second time to perform the second training for the first model, and so on, until the training sample resource pool is empty. Exemplarily, the training samples with a number of b1 will be removed from the training sample resource pool after being obtained by the training node. When the training sample resource pool is empty (i.e., the 100 training samples do not exist in the training sample resource pool), the i-th round of training of the first model by the multiple training nodes is finished. Afterwards, the training sample resource pool will also include the 100 training samples before the i+1-th round of training of the first model begins, and the 100 training samples are used to participate in the i+1-th round of training of the first model.

[0063] It should be understood that what is saved (or stored) in the training sample resource pool may be training sample data, or training sample identifiers (or training sample indexes), or location information of training samples. Among them, the training sample identifier (or training sample index) is used to identify the training sample, and the location information of the training sample is used to indicate the location of the training sample. In one example, a certain training node is taken as the first training node, and the training sample identifier saved in the training sample resource pool is taken as an example. When the first training node performs a certain training (such as the qth training) in a certain round of training (such as the i-th round of training) on ​​the first model, the first training node obtains the first number of training sample identifiers required for the first model to perform the qth training from the training sample resource pool. According to the first number of training sample identifiers, the training sample data corresponding to the first number of training sample identifiers can be obtained from the storage area of ​​the local cache training sample data, or the training sample data corresponding to the first number of training sample identifiers can be obtained from other storage areas (such as databases). In another example, a certain training node is taken as the first training node, and the training sample location information saved in the training sample resource pool is taken as an example. When the first training node performs a certain training (such as the qth training) in a certain round of training (such as the i-th round of training) on ​​the first model, after obtaining the first number of training sample location information required for the qth training of the first model from the training sample resource pool, the first training node can obtain the training sample data corresponding to the first number of training sample location information based on the first number of training sample location information.

[0064] Figure 3 A schematic diagram of a training node obtaining a training sample from a training sample resource pool provided in an embodiment of the present application. Figure 3As shown, the distributed training architecture includes N training nodes (such as training node 1, training node 2, ..., training node N). Due to different computing power, the number of training times (or training steps) performed by the N training nodes in each round of parallel training of the first model may be different. For example, taking the first model in the i-th round of training as an example, when training node 1 performs the i-th round of training for the first model, it obtains a certain number of training samples (such as a number of c0 training samples) from the training sample resource pool for the first time, and uses the first number of training samples obtained to perform the first training for the first model. Exemplarily, the number of training samples c0 will be removed from the training sample resource pool after being obtained by training node 1. After the first training for the first model is completed, if the training sample resource pool is not empty, training node 1 obtains a certain number of training samples (such as a number of c1 training samples) from the training sample resource pool for the second time, and uses the second number of training samples obtained to perform the second training for the first model, and so on, until the training sample resource pool is empty. Exemplarily, after the training samples with a number of c1 are acquired by the training node 1, they will be removed from the training sample resource pool. In this way, when the training sample resource pool is empty, the training node 1 performs M training for the first model during the i-th round of training for the first model. When the training node 2 performs the i-th round of training for the first model, it acquires a certain number of training samples (such as a number of d0 training samples) from the training sample resource pool for the first time, and uses the certain number of training samples acquired for the first time to perform the first training for the first model. Exemplarily, after the training samples with a number of d0 are acquired by the training node, they will be removed from the training sample resource pool. After the first training for the first model is completed, if the training sample resource pool is not empty, the training node 2 acquires a certain number of training samples (such as a number of d1 training samples) from the training sample resource pool for the second time, and uses the certain number of training samples acquired for the second time to perform the second training for the first model, and so on, until the training sample resource pool is empty. Exemplarily, after the training samples with a number of d1 are acquired by the training node 2, they will be removed from the training sample resource pool. Thus, when the training sample resource pool is empty, the training node 2 performs K trainings on the first model during the i-th round of training for the first model. … When the training node N performs the i-th round of training for the first model, it obtains a certain number of training samples (e.g., a number of training samples s0) from the training sample resource pool for the first time, and performs the first training on the first model using the certain number of training samples obtained for the first time. Exemplarily, the number of training samples s0 will be removed from the training sample resource pool after being obtained by the training node.After the first training for the first model is completed, if the training sample resource pool is not empty, the training node N obtains a certain number of training samples (such as training samples of quantity s1) from the training sample resource pool for the second time, and uses the certain number of training samples obtained for the second time to perform the second training for the first model, and so on, until the training sample resource pool is empty. Exemplarily, after being obtained by the training node N, the training samples of quantity s1 will be removed from the training sample resource pool. In this way, when the training sample resource pool is empty, the training node N performs P training for the first model during the i-th round of training of the first model. In this way, the method can achieve the situation where no training samples are discarded (i.e., all training samples in the training sample resource pool are fully utilized), and can avoid the problem of poor model training accuracy caused by waste of training samples due to average distribution of training samples or manual distribution of training samples, and can avoid the problem of long total model training time caused by slow model training speed due to average distribution of training samples or manual distribution of training samples, and can ensure that multiple training nodes can quickly complete the i-th round of training for the first model, thereby shortening the model training time and improving the model training effect.

[0065] In a possible implementation, the training sample resource pool may be located on a resource management node, which is a node independent of the multiple training nodes, and the multiple training nodes may communicate with the resource management node respectively. Optionally, each of the multiple training nodes may be provided with a communication thread, and communicate with the resource management node or other training nodes through the communication thread.

[0066] For example, take the first model in the i-th round of training as an example. In one example, when the first training node starts the j-th training of the first model in the i-th round of training, the first training node can send a first request to the resource management node. The first request is used to request to obtain the training samples required for the first training node to perform the j-th training of the first model in the i-th round of training. After receiving the first request, the resource management node queries whether there are training samples in the training sample resource pool according to the first request. When there is at least one training sample in the training sample resource pool, the resource management node can select a first number of training samples from the training sample resource pool, and can send the first number of training samples to the first training node. For example, after sending the first number of training samples to the first training node, the resource management node can delete the first number of training samples in the training sample resource pool; or the resource management node can also select the first number of training samples from the training sample resource pool in the way of scooping up training samples, so that the first number of training samples does not exist in the training sample resource pool after being selected.

[0067] In another example, when a training node other than the first training node in the distributed training architecture (such as the second training node) starts the kth training of the first model in the i-th round of training, the second training node may send a training sample acquisition request to the resource management node. The training sample acquisition request is used to request the acquisition of the training samples required for the second training node to perform the kth training of the first model in the i-th round of training. After receiving the training sample acquisition request, the resource management node queries whether there is a training sample in the training sample resource pool according to the training sample acquisition request. When there is at least one training sample in the training sample resource pool, the resource management node may select a third number of training samples from the training sample resource pool, and may send the third number of training samples to the second training node. For example, after sending the third number of training samples to the second training node, the resource management node may delete the third number of training samples in the training sample resource pool; or the resource management node may also select the third number of training samples from the training sample resource pool in the manner of scooping up training samples, so that the third number of training samples does not exist in the training sample resource pool after being selected. Exemplarily, the second number may be determined by the resource management node according to the number of training samples currently existing in the training sample resource pool and / or the computing power of the second training node. When there are no training samples in the training sample resource pool, the resource management node may send an indication message to the second training node. The indication message is used to indicate that there are no training samples in the training sample resource pool. After receiving the indication message, the second training node ends or stops the i-th round of training of the first model.

[0068] For example, take two training nodes (such as training node A and training node B) as an example. When training node A performs the i-th round of training for the first model, it sends a training sample acquisition request to the resource management node. The resource management node can select a certain number of training samples (such as n0 training samples) from the training sample resource pool when the training sample resource pool is not empty according to the training sample acquisition request of training node A, and send the n0 training samples to the training node A. Optionally, after sending the n0 training samples to the training node A, the resource management node deletes the n0 training samples from the training sample resource pool. After obtaining the n0 training samples, the training node A can use the n0 training samples to perform the first training for the first model. After completing the first training for the first model, the training node A can continue to request the resource management node to obtain training samples, for example, the training node A sends a training sample acquisition request to the resource management node again. The resource management node may select a certain number of training samples (such as n1 training samples) from the training sample resource pool according to the training sample acquisition request of the training node A when the training sample resource pool is not empty, and send the n1 training samples to the training node A. Optionally, after sending the n1 training samples to the training node A, the resource management node may delete the n1 training samples from the training sample resource pool. After obtaining the n1 training samples, the training node A may use the n1 training samples to perform a second training for the first model, and so on, until the training sample resource pool is empty. Optionally, when the training sample resource pool is empty, the resource management node may send an indication message to the training node A. The indication message is used to indicate that the training sample resource pool is empty (or there are no training samples in the training sample resource pool). After receiving the indication message, the training node A ends or stops the i-th round of training for the first model.

[0069] When training node B performs the i-th round of training for the first model, it sends a training sample acquisition request to the resource management node. The resource management node can select a certain number of training samples (such as n0' number of training samples) from the training sample resource pool when the training sample resource pool is not empty according to the training sample acquisition request of training node B, and send the n0' number of training samples to training node B. Optionally, after sending the n0' number of training samples to training node B, the resource management node deletes the n0' number of training samples from the training sample resource pool. After obtaining the n0' number of training samples, training node B can use the n0' number of training samples to perform the first training for the first model. After completing the first training for the first model, training node B can continue to request the resource management node to obtain training samples, for example, training node B sends a training sample acquisition request to the resource management node again. The resource management node may obtain a training sample request from the training node B, and when the training sample resource pool is not empty, select a certain number of training samples (such as n1' training samples) from the training sample resource pool, and send n1' training samples to the training node B. Optionally, after sending n1' training samples to the training node B, the resource management node may delete the n1' training samples from the training sample resource pool. After obtaining n1' training samples, the training node B may use the n1' training samples to perform a second training on the first model, and so on, until the training sample resource pool is empty. Optionally, when the training sample resource pool is empty, the resource management node may send an indication message to the training node B. The indication message is used to indicate that the training sample resource pool is empty (or there are no training samples in the training sample resource pool). After receiving the indication message, the training node B ends or stops the i-th round of training on the first model.

[0070] It can be seen from the training process of the training node A and the training node B for the first model that after completing a training for the first model, the training node A and the training node B can respectively request the resource management node for the training samples required for the next training again. In this way, the training node with large computing power can quickly complete a training for the first model, and request to obtain the training samples again to perform the next training for the first model, while the training node with small computing power completes the training for the first model relatively slowly. Therefore, the training node with large computing power requests to obtain the training samples more times, and the number of trainings performed for the first model is also relatively large, and the training node with small computing power requests to obtain the training samples less times, and the number of trainings performed for the first model is also relatively small, so that the training samples included in the training sample resource pool can be dynamically allocated to multiple training nodes based on the computing power of multiple training nodes, and the effect of matching the data load with the computing power of the training node can be achieved, and the phenomenon of the training node with large computing power waiting for the training node with small computing power will not occur, and the computing power of multiple training nodes can be fully utilized to achieve the effect of accelerating model training. In addition, because training node A and training node B respectively end the i-th round of training for the first model when the training sample resource pool is empty, it is possible to avoid the problem of poor model training accuracy due to waste of training samples caused by evenly distributing training samples or manually distributing training samples without discarding any training samples (that is, fully utilizing all training samples in the training sample resource pool), and avoid the problem of long total model training time due to slow model training speed caused by evenly distributing training samples or manually distributing training samples, thereby improving the model training accuracy and effectively improving the model training effect.

[0071] In another possible implementation, the training sample resource pool may be located on a certain training node (such as a first training node), and the first training node may serve as a resource management node and have a management function of the resource management node for the training sample resource pool. Other training nodes except the first training node in the distributed training architecture may respectively request to obtain a corresponding number of training samples from the training sample resource pool by communicating with the first training node.

[0072] For example, let's continue to take the first model in the i-th round of training as an example. In one example, when the first training node starts the j-th training in the i-th round of training for the first model, the first training node can query whether there is a training sample in the training sample resource pool. When there is at least one training sample in the training sample resource pool, the first training node can select a first number of training samples from the training sample resource pool. For example, after selecting the first number of training samples from the training sample resource pool, the first training node can delete the first number of training samples in the training sample resource pool; or the first training node can also select the first number of training samples from the training sample resource pool in the manner of scooping up training samples, so that the first number of training samples does not exist in the training sample resource pool after being selected. Exemplarily, the first number can be determined by the first training node according to the number of training samples currently existing in the training sample resource pool and / or the computing power (or computing performance or computing speed) of the first training node. When there is no training sample in the training sample resource pool, the first training node can end or stop the i-th round of training for the first model.

[0073] In another example, when a training node other than the first training node in the distributed training architecture (such as the second training node) starts the kth training of the first model in the i-th round of training, the second training node may send a training sample acquisition request to the first training node. The training sample acquisition request is used to request the acquisition of the training samples required for the second training node to perform the kth training of the first model in the i-th round of training. After receiving the training sample acquisition request, the first training node queries whether there is a training sample in the training sample resource pool according to the training sample acquisition request. When there is at least one training sample in the training sample resource pool, the first training node may select a third number of training samples from the training sample resource pool, and may send the third number of training samples to the second training node. In one example, after sending the third number of training samples to the second training node, the first training node may delete the third number of training samples in the training sample resource pool. In another example, the first training node may select a third number of training samples from the training sample resource pool in the manner of scooping up training samples, so that the third number of training samples does not exist in the training sample resource pool after being selected.

[0074] For example, continue to take two training nodes (such as training node A and training node B) as an example, and assume that the training sample resource pool is located on training node A. When training node A performs the i-th round of training for the first model, it can query whether the training sample resource pool is empty (or whether there are training samples). When the training sample resource pool is not empty, training node A can select a certain number of training samples (such as n0 training samples) from the training sample resource pool. Optionally, after selecting n0 training samples from the training sample resource pool, training node A can delete the n0 training samples from the training sample resource pool. After obtaining n0 training samples, training node A can use n0 training samples to perform the first training for the first model. After completing the first training for the first model, training node A can query again whether the training sample resource pool is empty. When the training sample resource pool is not empty, training node A again selects a certain number of training samples (such as n1 training samples) from the training sample resource pool. Optionally, after selecting n1 training samples from the training sample resource pool, training node A may delete the n1 training samples from the training sample resource pool. After obtaining n1 training samples, training node A may use n1 training samples to perform a second training on the first model, and so on, until the training sample resource pool is empty. Optionally, when the training sample resource pool is empty, training node A may end or stop the i-th round of training on the first model.

[0075] When training node B performs the i-th round of training for the first model, it sends a training sample acquisition request to training node A. Training node A can select a certain number of training samples (such as n0' number of training samples) from the training sample resource pool when the training sample resource pool is not empty based on the training sample acquisition request of training node B, and send n0' number of training samples to training node B. Optionally, after sending n0' number of training samples to training node B, training node A deletes n0' number of training samples from the training sample resource pool. After obtaining n0' number of training samples, training node B uses n0' number of training samples to perform the first training for the first model. After completing the first training for the first model, training node B can continue to request training node A to obtain training samples, for example, training node B sends a training sample acquisition request to training node A again. Training node A can obtain a training sample request from training node B, and when the training sample resource pool is not empty, select a certain number of training samples (such as n1' number of training samples) from the training sample resource pool, and send n1' number of training samples to training node B. After obtaining n1' number of training samples, training node B uses n1' number of training samples to perform a second training for the first model, and so on, until the training sample resource pool is empty. Optionally, when the training sample resource pool is empty, training node A can send an indication message to training node B. The indication message is used to indicate that the training sample resource pool is empty (or there are no training samples in the training sample resource pool). After receiving the indication message, training node B ends or stops the i-th round of training for the first model.

[0076] Step 202: The first training node determines a gradient parameter corresponding to the j-th training according to a first number of training samples and a first model parameter of the first model during the j-th training.

[0077] In an embodiment of the present application, after obtaining the first number of training samples for participating in the jth training of the first model in the i-th round of training, the first training node can calculate the loss function value corresponding to the jth training of the first model according to the first number of training samples for the jth training of the first model in the i-th round of training and the first model parameters used by the first model in the j-th training of the i-th round of training. Afterwards, the first training node can use a gradient parameter calculation algorithm to perform a back propagation operation on the loss function value corresponding to the j-th training of the first model to obtain the gradient parameters corresponding to the j-th training of the first model. Exemplarily, the gradient parameter calculation algorithm may include but is not limited to a stochastic gradient descent method, a batch gradient descent method, or a mini-batch gradient descent method.

[0078] For example, the first training node is training node 1, the first model is a distributed neural network model, there are 5 training samples participating in the j-th training of the distributed neural network model in the i-th round of training, and the gradient parameter calculation algorithm is the stochastic gradient descent method. Training node 1 can calculate the loss function value corresponding to the j-th training of the distributed neural network model based on the 5 training samples and the model parameters used by the distributed neural network model in the j-th training in the i-th round of training. Afterwards, training node 1 can use the stochastic gradient descent method to perform a back-propagation operation on the loss function value to obtain the stochastic gradient corresponding to the j-th training of the distributed neural network model.

[0079] Step 203: The first training node determines a gradient descent learning rate corresponding to the jth training according to the number of training times that each training node in the saved multiple training nodes has performed in the i-th round of training.

[0080] In an embodiment of the present application, the multiple training nodes included in the distributed training architecture will also communicate with each other during each round of training of the first model. During the mutual communication process, the multiple training nodes can obtain the number of training times that the other party has performed on the first model during the current round of training. Optionally, at the beginning of each round of training of the first model, the multiple training nodes can respectively maintain a training number record table locally, which is used to record in real time the number of training times that have been performed on the first model during each round of training.

[0081] Exemplarily, take the first training node included in multiple training nodes as an example. The first training node can send a second request to other training nodes in the multiple training nodes except the first training node at a set time interval, or the first training node can also send a second request to other training nodes in the multiple training nodes except the first training node in real time. Among them, the second request is used to request to obtain the number of trainings that other training nodes have performed in the i-th round of training. After receiving the second request, the other training nodes can send the number of trainings that have been performed on the first model in the i-th round of training to the first training node. After receiving the number of trainings that have been performed on the first model in the i-th round of training from other training nodes, the first training node can save the number of trainings that have been performed on the first model by other training nodes in the i-th round of training. It should be understood that the first training node can save the number of trainings that have been performed on the first model in the i-th round of training in real time.

[0082] For example, the first training node is training node 1, the second training node included in the other training nodes is training node 2, and training node 1 sends a second request to training node 2 at a set time interval. The second training node is any training node among the other training nodes. Training node 1 can send a second request to training node 2 at a set time interval (for example, every 20 seconds). The second request is used to request the number of trainings that training node 2 has performed during the i-th round of training. After receiving the second request, training node 2 can send the number of trainings that have been performed on the first model during the i-th round of training (for example, the number of trainings is 3) to training node 1. After receiving the number of trainings that have been performed on the first model during the i-th round of training from training node 2, training node 1 can save the number of trainings that training node 2 has performed on the first model during the i-th round of training.

[0083] In an embodiment of the present application, the first training node may determine the gradient descent learning rate corresponding to the j-th training of the first model by, but not limited to, the following implementation method:

[0084] Method 1: The first training node may first calculate the average number of training times that the multiple training nodes have performed in the i-th round of training. Afterwards, the first training node may determine the gradient descent learning rate corresponding to the j-th training of the first model based on the average value and the initial gradient descent learning rate corresponding to the i-th round of training of the first model.

[0085] Method 2: The first training node may first calculate the weighted average of the number of training times that the multiple training nodes have performed in the i-th round of training. Afterwards, the first training node may determine the gradient descent learning rate corresponding to the j-th training of the first model based on the weighted average and the initial gradient descent learning rate corresponding to the i-th round of training of the first model.

[0086] Optionally, after the first training node calculates the gradient descent learning rate corresponding to the j-th training of the first model, if the first training node can continue to obtain a certain number of training samples from the training sample resource pool during the i-th round of training of the first model, the first training node can calculate the gradient descent learning rate corresponding to the j+1-th training of the first model, and so on, until the first training node cannot obtain training samples from the training sample resource pool during the i-th round of training of the first model.

[0087] Exemplarily, the gradient descent learning rate corresponding to the j-th training of the first model satisfies the following formula:

[0088] lr=scheduler(init_lr,average,other)

[0089] Among them, lr (Learning Rate) is used to represent the gradient descent learning rate corresponding to the jth training of the first model, init_lr is used to represent the initial gradient descent learning rate corresponding to the i-th round of training of the first model, average is used to represent the average or weighted average of the number of training times that multiple training nodes have performed in the i-th round of training, and other is used to represent other parameters.

[0090] For example, when average is used to represent the average number of training times that multiple training nodes have performed in the i-th round of training, average satisfies the following form:

[0091] average=mean(local_step1,local_step2,…,local_stepN)

[0092] Among them, local_step1, local_step2, ..., local_stepN are respectively used to indicate the number of times training node 1, training node 2, ..., training node N has been trained in the i-th round of training, or can also be respectively used to indicate the step (or number of) training that training node 1, training node 2, ..., training node N has executed in the i-th round of training, or can also be respectively used to indicate how many times (or steps) training node 1, training node 2, ..., training node N has executed in the i-th round of training.

[0093] When used to represent the weighted average of the number of training times that multiple training nodes have performed in the i-th round of training, average satisfies the following form:

[0094] or

[0095] Among them, N is used to represent the total number of training nodes, local_step j It is used to indicate the number of times the jth training node has been trained in the i-th round of training. j The weight used to represent the number of times the j-th training node has been trained in the i-th round of training.

[0096] Figure 4 The schematic diagram of an embodiment of the present application is exemplarily shown in which each training node calculates the gradient descent learning rate corresponding to each training of the first model in the i-th round of training. Wherein, any training node is configured with at least the following two threads, namely, a computing thread and a communication thread. Figure 4As shown, when the computing thread of each training node performs the first training in the i-th round of training for the first model (i.e., local_step=0), it obtains a certain number of training samples from the training sample resource pool for the first time. Moreover, during the i-th round of training for the first model, the communication thread of the training node can periodically (or at a set time interval) obtain the number of trainings that other training nodes have performed for the first model from other training nodes, and can calculate the average (i.e., average or weighted average) of the number of trainings that the multiple training nodes obtained each time have performed in the i-th round of training for the first model (e.g., average1). For example, after the communication thread of the training node obtains the number of trainings that other training nodes have performed for the first model from other training nodes at time a, the average number of trainings that the multiple training nodes obtained at time a have performed in the i-th round of training of the first model can be calculated (e.g., average1'). After a certain period of time (e.g., period b), i.e., time (a+b), after the communication thread of the training node obtains the number of trainings that other training nodes have performed for the first model from other training nodes at time (a+b), the average number of trainings that the multiple training nodes obtained at time (a+b) have performed in the i-th round of training of the first model can be calculated (e.g., average2'), and so on. Afterwards, the computing thread of the training node uses a certain number of training samples obtained by the communication thread to perform forward and backward calculations on the first model (i.e., the first model training) to obtain the gradient parameters corresponding to the first model in the first training. Then, the computing thread of the training node can calculate the gradient descent learning rate lr (such as lr1) corresponding to the first model in the first training according to the average (such as average1) currently calculated (or currently provided) by the communication thread and the initial gradient descent learning rate corresponding to the first model in the i-th round of training. Finally, the computing thread of the training node can update the model parameters of the first model in the first training according to the gradient parameters and gradient descent learning rate corresponding to the first model in the first training to obtain the second model parameters. The second model parameters are used as the model parameters of the training node when the first model is trained for the second time in the i-th round of training.

[0097] After calculating and obtaining the second model parameters, the computing thread of the training node can perform the second training (i.e., local_step=1) in the i-th round of training for the first model. At this time, when the training sample resource pool is not empty, the computing thread of the training node obtains a certain number of training samples from the training sample resource pool for the second time. Afterwards, the computing thread of the training node uses a certain number of training samples obtained by the communication thread to perform forward and backward calculations (i.e., second model training) for the first model to obtain the gradient parameters corresponding to the first model in the second training. Then, the computing thread of the training node can calculate the gradient descent learning rate lr (e.g., lr2) corresponding to the first model in the second training according to the average (e.g., average2) currently calculated (or currently provided) by the communication thread and the initial gradient descent learning rate corresponding to the first model in the i-th round of training. Finally, the computing thread of the training node can update the model parameters of the first model in the second training according to the gradient parameters and gradient descent learning rate corresponding to the first model in the second training, and obtain the third model parameters. Among them, the third model parameters are used as the model parameters of the training node when the training node performs the third training on the first model in the i-th round of training. And so on, until the training sample resource pool is empty.

[0098] In addition, the communication thread of each training node can periodically (or at a set time interval) update the model parameters of the local first model. For example, taking a certain training node (such as training node A) as an example, the communication thread of training node A can obtain the model parameters of the first model from other training nodes at set time intervals. Afterwards, training node A can aggregate the model parameters of the first model in training node A and the model parameters of the first model in other training nodes (such as calculating the average or weighted average) to obtain the aggregated model parameters. Then, the communication thread of training node A can use the aggregated model parameters to update the current model parameters of the local first model.

[0099] It should be understood that Figure 4 The position where the computing thread calculates the gradient descent learning rate lr and the position where local_step+=1 (i.e., local_step is incremented) are only examples, for the convenience of explaining the technical solution in the embodiment of the present application, and do not constitute a limitation on the technical solution in the embodiment of the present application. Optionally, the computing thread calculating the gradient descent learning rate lr and local_step+=1 can be placed at any position of the computing thread. Figure 4The position where the communication thread calculates the average is only an example, for the convenience of explaining the technical solution in the embodiment of the present application, and does not constitute a limitation on the technical solution in the embodiment of the present application. Optionally, the position where the average is calculated can be at any position in the communication thread, not necessarily at the beginning of the communication thread.

[0100] Step 204: The first training node updates the first model parameters according to the gradient descent learning rate and the gradient parameters to obtain the second model parameters.

[0101] After calculating the gradient descent learning rate and gradient parameters corresponding to the j-th training of the first model in the i-th round of training, the first training node can update the first model parameters (or first model parameter weights) used in the j-th training of the first model according to the gradient descent learning rate and the gradient parameters to obtain second model parameters (or second model parameter weights).

[0102] In one example, when the first training node cannot obtain training samples from the training sample resource pool after training the first model for the jth time, the first training node ends or stops the i-th round of training of the first model. At this time, if the first model still has the i+1th round of training, the second model parameters are used as the model parameters of the first training node during the first training of the first model in the i+1th round of training. If the first model does not have the i+1th round of training, the second model parameters are used as the final model parameters of the first training node training the first model.

[0103] In another example, when the first training node can obtain training samples from the training sample resource pool after training the first model for the jth time, the second model parameters are used as model parameters for the first training node when performing the j+1th training of the first model during the i-th round of training.

[0104] It can be seen from the above steps 201 to 204 that the multiple training nodes are all calculated by the gradient descent learning rate according to the number of training times of the first model in the i-th round of training according to the multiple training nodes, so that the gradient descent learning rates of the multiple training nodes at the same time (or at the same moment) are kept consistent, so as to unify the gradient descent learning rates of the multiple training nodes, and there is no need to manually adjust the gradient descent learning rates of the training nodes with fast or slow training speeds, so as to ensure the convergence of the model training, thereby reducing the model parameter debugging time and improving the model training effect. In addition, each training node obtains (or requests to obtain) training samples from the training sample resource pool instead of manually allocating training samples to each training node, so that the problem of the total model training time becoming longer due to the manual allocation of training samples making the model training speed slower can be avoided, and the problem of adaptively allocating a suitable number of training samples to each training node can be achieved, effectively avoiding the waste of training samples or the problem of empty training nodes.

[0105] Exemplarily, the following takes the first training node as training node 1, the training sample resource pool is located on the resource management node, and the training sample resource pool stores training sample data as an example, and introduces the implementation process of training node 1 obtaining a first amount (such as min-batch) of training sample data from the training sample resource pool to participate in the jth training of the first model in the i-th round of training.

[0106] Figure 5 The following is a flow chart showing another model training method provided by an embodiment of the present application. Figure 5 As shown, the method includes:

[0107] Step 501: When executing the jth training of the first model in the i-th training round, the training node 1 sends a first request to the resource management node. The resource management node receives the first request.

[0108] The first request is used to request to obtain the training samples required for the jth training of the first model during the i-th round of training of the first model.

[0109] Step 502: The resource management node determines whether the training sample resource pool is empty according to the first request. If so, execute step 506; if not, execute step 503.

[0110] Step 503: The resource management node obtains min-batch training sample data from the training sample resource pool.

[0111] Optionally, when determining that the training sample resource pool is not empty, the resource management node may select a min-batch of training sample data from the training sample resource pool.

[0112] Step 504: The resource management node sends the min-batch training sample data to the training node 1. The training node 1 receives the min-batch training sample data.

[0113] Step 505: Training node 1 uses the min-batch training sample data to perform the j-th training in the i-th round of training on the first model.

[0114] After training the first model for the jth time using the min-batch training sample data, training node 1 can return to execute step 501. At this time, the request sent to the resource management node in step 501 is used to request to obtain the training samples required for training the first model for the j+1th time during the i-th round of training of the first model.

[0115] Step 506: The resource management node sends indication information to the training node 1. The training node 1 receives the indication information.

[0116] The indication information is used to indicate that there is no training sample in the training sample resource pool.

[0117] Step 507: Training node 1 ends the i-th round of training for the first model according to the instruction information.

[0118] It can be seen from the above steps 501 to 507 that when each training node performs the jth training in the i-th round of training for the first model, it actively requests a certain number of training samples from the resource management node, and the resource management node dynamically allocates a certain number of training samples to each training node according to the request of each training node, so that the training samples in the training sample resource pool are adaptively allocated to each training node. Therefore, the problem of poor model training accuracy caused by waste of training samples due to average distribution of training samples or manual distribution of training samples can be avoided, and the problem of long total model training time caused by slow model training speed due to average distribution of training samples or manual distribution of training samples can be avoided. Of course, the computing power of training nodes with fast training speed can be avoided, thereby improving the model training accuracy and effectively improving the model training effect.

[0119] It should be noted that in the description of the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there may be three relationships, for example, A and / or B, which can represent: the situation where A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are an "or" relationship. "At least one of the following (individuals)" or its similar expression refers to any combination of these items, including any combination of single items (individuals) or plural items (individuals). For example, "at least one of A, B and C" includes A, B, C, AB, AC, BC or ABC. And, unless otherwise specified, the ordinal numbers such as "first", "second", and "third" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects. In addition, the terms "including", "comprising", "having" and their variations appearing in the present application all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0120] In addition, it should be noted that each step involved in the above embodiments can be performed by a corresponding device, or by a chip, processor, or chip system in the device, and the embodiments of the present application do not limit them. The above embodiments are only described by taking the corresponding device as an example.

[0121] It should be noted that in the above embodiments, some steps can be selected for implementation, and the order of the steps in the diagram can be adjusted for implementation, and this application does not limit this. It should be understood that executing some steps in the diagram, adjusting the order of the steps, or combining them for specific implementation all fall within the scope of protection of this application.

[0122] It is understandable that, in order to implement the functions in the above embodiments, the various devices involved in the above embodiments include hardware structures and / or software modules corresponding to the execution of the various functions. It should be easily appreciated by those skilled in the art that, in combination with the units and method steps of the various examples described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0123] It should be noted that the "steps" in the embodiments of the present application are only for illustration, and are a method of expression used to better understand the embodiments, and do not constitute a substantial limitation on the execution of the scheme of the present application. For example, the "steps" can also be understood as "features". In addition, the steps do not constitute any limitation on the execution order of the scheme of the present application. Any changes in the order of steps, step merging, or step splitting made on this basis that do not affect the implementation of the overall scheme, and the resulting new technical solutions are also within the scope of the disclosure of the present application.

[0124] Based on the same concept, the present application embodiment also provides a possible model training device, which can be but is not limited to being applicable to Figure 1 In the application scenario shown in FIG. 1 , the model training device can be Figure 1 Any training node in the distributed training architecture 200 shown, or Figure 1 The functional element (such as a plug-in, component, chip or circuit, etc.) in any training node in the distributed training architecture 200 shown in the figure has the function of implementing the model training method. Optionally, the model training device is used to implement the model training method provided in the above embodiment, or the module (such as a chip) of the model training device is used to implement the model training method provided in the above embodiment, so the beneficial effects possessed by the above embodiment can also be achieved. For example, taking the model training device as a chip arranged in a computing device as an example, when the model training device is a chip, the model training device includes a communication interface and a processor, but does not include a memory. Among them, the communication interface exists as an input and output interface, and the input and output interface is used for the chip to implement the sending and receiving of the computing device. The input and output interface may include an input interface and / or an output interface, the input interface can realize the reception of the computing device, and the output interface can be used to realize the sending of the computing device. The processor is used to read and execute corresponding computer programs or instructions so that the corresponding functions of the model training device are realized. Optionally, when the chip implements the corresponding functions of the model training device in the above embodiment, the input and output interface can implement the transceiver operation performed by the model training device in the above embodiment; the processor can implement other operations except the transceiver operation performed by the model training device in the above embodiment. For specific related descriptions, please refer to the related descriptions in the above embodiment, which will not be described in detail here.

[0125] See also Figure 6 The model training device 600 includes an acquisition module 601 (or may be called a transceiver module or a transceiver unit, used to send and receive data), a determination module 602 and an update module 603.

[0126] Optionally, the acquisition module 601 may include a receiving module and / or a sending module. The receiving module may be used for the model training device 600 to receive signals (information or data, etc.); the sending module may be used for the model training device 600 to send signals (information or data, etc.).

[0127] When the model training device 600 is used to implement Figure 2 or Figure 5 The functions of the model training device in the method embodiment shown are as follows: an acquisition module 601 is used to acquire a first number of training samples from a training sample resource pool, wherein the first number of training samples is used for the first training node to perform the j-th training on the first model during the i-th round of training, and the first training node is any training node among multiple training nodes. A determination module 602 is used to determine the gradient parameters corresponding to the first model in the j-th training according to the first number of training samples and the first model parameters when the first model is trained for the j-th time. The determination module 602 is also used to determine the gradient descent learning rate corresponding to the first model in the j-th training according to the number of training times each training node in the multiple training nodes has performed in the i-th round of training. An update module 603 is used to update the first model parameters according to the gradient descent learning rate and gradient parameters corresponding to the first model in the j-th training to obtain the second model parameters, wherein the second model parameters are used as the model parameters when the first training node performs the j+1-th training on the first model in the i-th round of training.

[0128] Wherein, when the model training device 600 is used to implement Figure 2 or Figure 5 For a more detailed description of the acquisition module 601, the determination module 602 and the update module 603, please refer to the above Figure 2 or Figure 5 The relevant description about the model training device in the method embodiment shown will not be repeated here.

[0129] Optional, above Figure 6 The acquisition module, determination module and update module shown in the figure can all be implemented by software, or can all be implemented by hardware. Exemplarily, the implementation of the determination module is introduced below by taking the determination module as an example. Similarly, the implementation of the acquisition module and the update module can refer to the implementation of the determination module, which will not be repeated here.

[0130] When implemented by software, the determination module is used as an example of a software functional unit, and the determination module may be an application or a code block running on a computer device. Among them, the computer device may be at least one of a physical host, a virtual machine, a container and other computing devices. Furthermore, the above-mentioned computer device may be one or more. For example, the determination module may be an application running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same availability zone (AZ) or in different AZs. The multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. Among them, usually a region may include multiple AZs.

[0131] Similarly, multiple hosts / virtual machines / containers for running the application can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Typically, a region can include multiple VPCs, and a VPC can include multiple AZs.

[0132] When implemented by hardware, the determination module is taken as an example of a hardware functional unit, and the determination module may include at least one computing device, such as a server, etc. Alternatively, the determination module may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0133] The multiple computing devices included in the determination module can be distributed in the same AZ or in different AZs. The multiple computing devices included in the determination module can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the determination module can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0134] It should be noted that the steps that the acquisition module, determination module and update module are responsible for implementing can be specified as needed, and the full functions of the model training device can be realized by respectively implementing different execution steps of the model training method through the acquisition module, determination module and update module.

[0135] In addition, it should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The functional modules in the embodiments of the present application can be integrated into one module, or each module can exist physically separately, or two or more modules can be integrated into one module. For example, the acquisition module, the determination module and the update module can be integrated into one module, or the acquisition module, the determination module and the update module are the same module. For another example, the determination module and the update module can be integrated into one module, or the determination module and the update module are the same module. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0136] Based on the same concept, the present application embodiment also provides a possible computing device for executing the model training method shown in the above method embodiment. The relevant features can be found in the above method embodiment and will not be repeated here. Figure 7 As shown, the computing device 700 includes: a bus 701, a processor 702 and a memory 704. Optionally, the computing device 700 may also include a communication interface 703. The processor 702, the communication interface 703 and the memory 704 communicate through the bus 701. The computing device 700 may be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 700.

[0137] The bus 701 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus 701 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 701 may include a path for transmitting information between various components of the computing device 700 (for example, the memory 704, the processor 702, and the communication interface 703).

[0138] The processor 702 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0139] The memory 704 may include a volatile memory, such as a random access memory (RAM). The processor 702 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0140] The memory 704 stores executable program codes, and the processor 702 executes the executable program codes to respectively implement the functions of the aforementioned model training device (such as the acquisition module, the determination module, and the update module), thereby implementing the model training method provided in the embodiment of the present application. That is, the memory 704 stores computer program instructions for executing the model training method.

[0141] Alternatively, the memory 704 stores executable code, and the processor 702 executes the executable code to implement the functions of the aforementioned model training device (such as the acquisition module, the determination module, and the update module), thereby implementing the model training method provided in the embodiment of the present application. That is, the memory 704 stores computer program instructions for the model training device to execute the model training method provided in the embodiment of the present application.

[0142] The communication interface 703 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 700 and other devices or a communication network.

[0143] Based on the same concept, the embodiment of the present application also provides a possible computing device cluster for executing the method shown in the above method embodiment. The relevant features can be found in the above method embodiment, which will not be repeated here. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop, a laptop, or a smart phone.

[0144] like Figure 8 As shown, the computing device cluster 800 includes at least one computing device 700. The memory 704 in one or more computing devices 700 in the computing device cluster 800 may store the same instructions for executing the model training method provided in the embodiment of the present application.

[0145] In some possible implementations, the memory 704 of one or more computing devices 700 in the computing device cluster 800 may also respectively store some instructions for executing the model training method. In other words, the combination of one or more computing devices 700 can jointly execute the instructions for executing the model training method.

[0146] It should be noted that the memory 704 in different computing devices 700 in the computing device cluster 800 can store different instructions, which are respectively used to execute part of the functions of the model training device. That is, the instructions stored in the memory 704 in different computing devices 700 can implement the functions of one or more modules in the acquisition module, the determination module or the update module.

[0147] In some possible implementations, one or more computing devices in the computing device cluster 800 may be connected via a network, which may be a wide area network or a local area network. Fig. 9 A possible implementation is shown. Fig. 9 As shown, two computing devices 700A and 700B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 704 in the computing device 700A stores instructions for executing the functions of the acquisition module 601. At the same time, the memory 704 in the computing device 700B stores instructions for executing the functions of the determination module 602 and the update module 603. In some possible implementations, the memory 704 of one or more computing devices 700 in the computing device cluster may also respectively store some instructions for executing the model training method. In other words, a combination of one or more computing devices 700 can jointly execute instructions for executing the model training method.

[0148] It should be understood that Fig. 9 The functions of the computing device 700A shown in FIG. 7 may also be completed by multiple computing devices 700. Similarly, the functions of the computing device 700B may also be completed by multiple computing devices 700.

[0149] Based on the same concept, the embodiment of the present application also provides a computer program product comprising instructions. The computer program product may be a software or program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the model training method provided in the embodiment of the present application.

[0150] Based on the same concept, an embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the model training method provided in the embodiment of the present application.

[0151] Based on the same concept, an embodiment of the present application also provides a computer chip, which is coupled to a memory and is used to read a computer program stored in the memory to execute the model training method provided in an embodiment of the present application.

[0152] Based on the same concept, an embodiment of the present application also provides a chip system, which includes a processor for supporting a computer device to implement the model training method provided in an embodiment of the present application. In one possible design, the chip system also includes a memory, which is used to store the necessary programs and data for the computer device. The chip system can be composed of a computer chip, or it can include a computer chip and other discrete devices.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: The method is applied to a distributed training architecture, the distributed training architecture includes a plurality of training nodes, the plurality of training nodes are used to train a first model in parallel, the plurality of training nodes have different computing powers, and the method includes: The first training node obtains a first number of training samples from a training sample resource pool, where the first number of training samples is used by the first training node to perform a j-th training on the first model during an i-th round of training, and the first training node is any training node among the multiple training nodes; The first training node determines, according to the first number of training samples and the first model parameters of the first model when performing the j-th training, a gradient parameter corresponding to the j-th training; The first training node determines the gradient descent learning rate corresponding to the j-th training according to the number of training times that each training node in the plurality of training nodes has performed during the i-th round of training; The first training node updates the first model parameters according to the gradient descent learning rate and the gradient parameters to obtain second model parameters, and the second model parameters are used as model parameters when the first training node performs the j+1th training on the first model during the i-th round of training.

2. The method according to claim 1, characterized in that The training sample resource pool is located on a resource management node, and the multiple training nodes communicate with the resource management node respectively; The first training node obtains a first number of training samples from a training sample resource pool, including: The first training node sends a first request to the resource management node, where the first request is used to request to obtain training samples required for the first model to perform the jth training during the i-th round of training; The first training node receives the first number of training samples from the resource management node.

3. The method according to claim 1 or 2, characterized in that The first training node determines the gradient parameter corresponding to the j-th training according to the first number of training samples and the first model parameters of the first model when performing the j-th training, including: The first training node determines a loss function value corresponding to the j-th training according to the first number of training samples and the first model parameters; The first training node uses a gradient parameter calculation algorithm to perform a back propagation operation on the loss function value to determine the gradient parameter corresponding to the j-th training.

4. The method according to any one of claims 1 to 3, characterized in that: The first training node determines the gradient descent learning rate corresponding to the j-th training according to the number of training times that each training node in the plurality of training nodes has performed in the i-th round of training, including: The first training node determines an average or weighted average of the number of training times performed by the multiple training nodes in the i-th round of training; The first training node determines the gradient descent learning rate corresponding to the j-th training according to the initial gradient descent learning rate corresponding to the i-th round of training and the average value or the weighted average value.

5. The method according to claim 4, characterized in that The method further comprises: The first training node sends a second request to a second training node at a set time interval, where the second request is used to request to obtain the number of trainings performed by the second training node during the i-th round of training, and the second training node is any training node among the multiple training nodes except the first training node; The first training node receives the number of trainings performed by the second training node during the i-th round of training from the second training node, and saves the number of trainings performed by the second training node during the i-th round of training.

6. The method according to any one of claims 1 to 5, characterized in that: When the i-th round of training of the first model performed by the multiple training nodes starts, the training sample resource pool includes a second number of training samples; when there are no training samples in the training sample resource pool, the i-th round of training of the first model performed by the multiple training nodes ends, the second number is greater than the first number, and the training samples are used for the i-th round of training of the first model.

7. The method according to any one of claims 1 to 6, characterized in that: The first model is a distributed neural network model.

8. A model training device, characterized in that: The device is applied to a distributed training architecture, the distributed training architecture includes a plurality of training nodes, the plurality of training nodes are used to train a first model in parallel, the plurality of training nodes have different computing powers, and the device includes: an acquisition module, configured to acquire a first number of training samples from a training sample resource pool, wherein the first number of training samples is used for a first training node to perform a j-th training on the first model during an i-th round of training, and the first training node is any training node among the multiple training nodes; a determination module, configured to determine a gradient parameter corresponding to the j-th training according to the first number of training samples and the first model parameters of the first model when the j-th training is performed; The determination module is further configured to determine the gradient descent learning rate corresponding to the j-th training according to the number of training times that each training node in the plurality of training nodes has performed during the i-th round of training; An updating module is used to update the first model parameters according to the gradient descent learning rate and the gradient parameters to obtain second model parameters, wherein the second model parameters are used as model parameters when the first training node performs the j+1th training on the first model during the i-th round of training.

9. The device according to claim 8, characterized in that The training sample resource pool is located on a resource management node, and the multiple training nodes communicate with the resource management node respectively; When the acquisition module acquires a first number of training samples from the training sample resource pool, it is specifically used to: Sending a first request to the resource management node, where the first request is used to request to obtain training samples required for the first model to perform the jth training during the i-th round of training; The first number of training samples is received from the resource management node.

10. The device according to claim 8 or 9, characterized in that When the determination module determines the gradient parameter corresponding to the j-th training according to the first number of training samples and the first model parameter of the first model during the j-th training, the determination module is specifically used to: Determining a loss function value corresponding to the j-th training according to the first number of training samples and the first model parameters; A gradient parameter calculation algorithm is used to perform a back propagation operation on the loss function value to determine the gradient parameter corresponding to the j-th training.

11. The device according to any one of claims 8 to 10, characterized in that: When the determination module determines the gradient descent learning rate corresponding to the j-th training according to the number of training times that each training node in the plurality of training nodes has performed in the i-th round of training, the determination module is specifically used to: Determine an average or weighted average of the number of training times that the multiple training nodes have performed during the i-th round of training; Determine the gradient descent learning rate corresponding to the j-th training according to the initial gradient descent learning rate corresponding to the i-th round of training and the average value or the weighted average value.

12. The device according to claim 11, characterized in that The acquisition module is also used for: Sending a second request to a second training node at a set time interval, where the second request is used to request obtaining the number of training times performed by the second training node during the i-th round of training, where the second training node is any training node among the multiple training nodes except the first training node; The number of trainings performed by the second training node during the i-th round of training is received from the second training node, and the number of trainings performed by the second training node during the i-th round of training is saved.

13. The device according to any one of claims 8 to 12, characterized in that: When the i-th round of training of the first model performed by the multiple training nodes starts, the training sample resource pool includes a second number of training samples; when there are no training samples in the training sample resource pool, the i-th round of training of the first model performed by the multiple training nodes ends, the second number is greater than the first number, and the training samples are used for the i-th round of training of the first model.

14. The device according to any one of claims 8 to 13, characterized in that The first model is a distributed neural network model.

15. A computing device, characterized in that: The computing device includes a communication interface, a processor, and a memory; The communication interface is used to receive and send data; The memory is used to store computer program instructions; The processor calls the computer program instructions stored in the memory to execute the method according to any one of claims 1 to 7.

16. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, each computing device includes a processor and a memory; the memory in the at least one computing device is used to store computer program instructions; The processor in the at least one computing device calls the computer program instructions stored in the memory to execute the method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the method according to any one of claims 1 to 7.

18. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device, the computing device is caused to perform the method as claimed in any one of claims 1 to 7.

Images