Method, apparatus and system for optimizing machine learning model training

Abstract

This specification provides methods, apparatus, and a distributed machine learning model training system for optimizing machine learning model training. In response to obtaining gradient information of model parameters after completing local model computation, each member processing device slices the gradient information into a first number of gradient information slices; and together with all other member processing devices of its respective distributed training node, performs data merging processing using its respective gradient information slices. Each numbered member processing device obtains a first gradient information slice merging result after data merging processing using the gradient information slices with the corresponding number. Subsequently, each member processing device, together with all other member processing devices of other distributed training nodes with the same number, performs data merging processing using its respective first gradient information slice merging result to obtain a second gradient information slice merging result; and stores the second gradient information slice merging result in the storage space allocated to that member processing device.

Description

Technical Field

[0001] The embodiments in this specification generally relate to the field of artificial intelligence, and more particularly to methods, apparatus and distributed machine learning model training systems for optimizing machine learning model training. Background Technology

[0002] With the development of artificial intelligence technology, machine learning models are increasingly being applied to various scenarios, such as item classification, information feed push, crowd recognition, and risk identification. In some scenarios, training sample data used for machine learning model training is stored on different data nodes. These multiple data nodes form a distributed machine learning model training system to train the machine learning model. During the distributed machine learning model training process, multiple different data nodes generate different model state data at different stages of the training process. The generated model state data needs to be stored on each data node and used collaboratively by different data nodes in the distributed machine learning model training system. This results in storage space consumption for data nodes and data communication overhead between data nodes. The storage space consumption of data nodes and data communication overhead affect the system performance of the distributed machine learning model training system. Summary of the Invention

[0003] In view of the above, embodiments of this specification provide a method, apparatus, and distributed machine learning model training system for optimizing machine learning model training. By using this method and apparatus for optimizing machine learning model training, the storage space occupied at the training nodes of the distributed machine learning training system and the data communication overhead can be reduced, thereby improving the system performance of the distributed machine learning model training system.

[0004] According to one aspect of the embodiments of this specification, a method for optimizing machine learning model training is provided, the method being performed by a member processing device in one of at least two distributed training nodes participating in machine learning model training, the method comprising: in response to completing local model computation to obtain gradient information of model parameters, slicing the gradient information into a first number of gradient information slices, the first number being the number of member processing devices in the distributed training node to which the member processing device is located; together with all other member processing devices in the distributed training node to which the member processing device belongs, performing data merging processing using their respective gradient information slices, wherein each member processing device in the distributed training node is assigned a number, and each numbered member processing device obtains a first gradient information slice merging result after data merging processing using the gradient information slice with the corresponding number; together with all other member processing devices with the same number in other distributed training nodes, performing data merging processing using their respective first gradient information slice merging results to obtain a second gradient information slice merging result; and storing the second gradient information slice merging result in storage space allocated to the member processing device.

[0005] Optionally, in one example of the above aspects, in response to completing the local model calculation to obtain the gradient information of the model parameters, slicing the gradient information into a first number of gradient information slices may include: in response to completing the local model calculation to obtain the gradient information of the model parameters, obtaining the available storage space of each member processing device in the distributed training node to which the member processing device belongs; and slicing the gradient information into a first number of gradient information slices according to the available storage space of each member processing device.

[0006] Optionally, in one example of the above aspects, slicing the gradient information into a first number of gradient information slices based on the available storage space of each member processing device may include: slicing the gradient information into a first number of gradient information slices based on the available storage space of each member processing device, wherein each of the sliced ​​gradient information slices has a slice size that matches the proportion of the available storage space of the corresponding numbered member processing device.

[0007] Optionally, in one example of the above aspects, in response to obtaining the gradient information of model parameters after completing local model computation, obtaining the available storage space of each member processing device in the distributed training node to which the member processing device belongs may include: in response to obtaining the gradient information of model parameters after completing local model computation, obtaining the available storage space and available processing capacity of each member processing device in the distributed training node to which the member processing device belongs. Accordingly, slicing the gradient information into a first number of gradient information slices according to the available storage space of each member processing device may include: slicing the gradient information into a first number of gradient information slices according to the available storage space and available processing capacity of each member processing device, wherein each slice has a slice size that matches the proportion of available storage space and available processing capacity of the corresponding numbered member processing device.

[0008] Optionally, in one example of the above aspects, the method may further include: updating the model parameters of the machine learning model to be trained using the second gradient information slice merging result in response to obtaining the second gradient information slice merging result; and performing model parameter concatenation using their respective updated model parameters together with all other member processing devices of the distributed training node to which the member processing device belongs to update the model parameters of the machine learning model.

[0009] Optionally, in one example of the above aspects, the method may further include: in response to initiating a local model training process, performing model parameter concatenation using partial model parameters of the machine learning model stored in each member processing device, together with all other member processing devices of the distributed training node to which the member processing device belongs, to generate a complete machine learning model.

[0010] Optionally, in one example of the above aspects, member processing devices within the same distributed training node are linked by a first communication link, and member devices with the same number on different distributed training nodes are linked by a second communication link, wherein the data transmission bandwidth of the first communication link is higher than that of the second communication link.

[0011] Alternatively, in one example of the above aspects, the member processing device includes a GPU device, and the storage space includes video memory.

[0012] According to another aspect of the embodiments of this specification, an apparatus for optimizing machine learning model training is provided. The apparatus is applied to a member processing device in one of at least two distributed training nodes participating in machine learning model training. The apparatus includes: a data slicing unit, which, in response to gradient information of model parameters obtained from local model computation, slices the gradient information into a first number of gradient information slices, the first number being the number of member processing devices in the distributed training node to which the member processing device is located; an intra-node data processing unit, which, together with all other member processing devices in the distributed training node to which the member processing device belongs, performs data merging processing using their respective gradient information slices, wherein each member processing device in the distributed training node is assigned a number, and each numbered member processing device obtains a first gradient information slice merging result after data merging processing using the gradient information slice with the corresponding number; an inter-node data processing unit, which, together with all other member processing devices with the same number in other distributed training nodes, performs data merging processing using their respective first gradient information slice merging results to obtain a second gradient information slice merging result; and a data storage unit, which stores the second gradient information slice merging result in storage space allocated to the member processing device.

[0013] Optionally, in one example of the above aspects, the data slicing unit includes: an available storage space acquisition module, which, in response to completing local model calculation to obtain gradient information of model parameters, acquires the available storage space of each member processing device in the distributed training node to which the member processing device belongs; and a data slicing module, which slices the gradient information into a first number of gradient information slices according to the available storage space of each member processing device.

[0014] Optionally, in one example of the above aspects, the data slicing module slices the gradient information into a first number of gradient information slices based on the available storage space of each member processing device, and each of the sliced ​​gradient information slices has a slice size that matches the proportion of the available storage space of the corresponding numbered member processing device.

[0015] Optionally, in one example of the above aspects, the data slicing unit further includes: an available processing capacity acquisition module, which acquires the available processing capacity of each member processing device in the distributed training node to which the member processing device belongs. The data slicing module slices the gradient information into a first number of gradient information slices based on the available storage space and available processing capacity of each member processing device, wherein each slice has a slice size that matches the proportion of available storage space and available processing capacity of the corresponding numbered member processing device.

[0016] Optionally, in one example of the above aspects, the apparatus may further include: a local model parameter update unit, which updates the model parameters of the machine learning model to be trained using the merged result of the second gradient information slice in response to obtaining the second gradient information slice aggregation result; and a model update unit, which, together with all other member processing devices of the distributed training node to which the member processing device belongs, performs model parameter concatenation using their respective updated model parameters to update the model parameters of the machine learning model.

[0017] Optionally, in one example of the above aspects, the apparatus may further include: a model generation unit, which, in response to initiating a local model training process, together with all other member processing devices of the distributed training node to which the member processing device belongs, performs model parameter concatenation using partial model parameters of the machine learning model stored in each of them to generate a complete machine learning model.

[0018] According to another aspect of the embodiments of this specification, a distributed machine learning model training system is provided, comprising: at least two distributed training nodes, each training node including a storage device and at least two member processing devices, each member processing device including means for optimizing machine learning model training as described above.

[0019] According to another aspect of the embodiments of this specification, an apparatus for optimizing machine learning model training is provided, comprising: at least one processor, a memory coupled to the at least one processor, and a computer program stored in the memory, wherein the at least one processor executes the computer program to implement the method for optimizing machine learning model training as described above.

[0020] According to another aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores executable instructions, which, when executed, cause a processor to perform the method for optimizing the training of a machine learning model as described above.

[0021] According to another aspect of the embodiments of this specification, a computer program product is provided, including a computer program that is executed by a processor to implement the method for optimizing machine learning model training as described above. Attached Figure Description

[0022] A further understanding of the nature and advantages of this specification can be achieved by referring to the following figures. In the figures, similar components or features may have the same reference numerals.

[0023] Figure 1 A block diagram of a distributed machine learning training system according to an embodiment of this specification is shown.

[0024] Figure 2 An example flowchart of a distributed machine learning model training process according to an embodiment of this specification is shown.

[0025] Figure 3 An example flowchart of a method for optimizing the training of a machine learning model according to an embodiment of this specification is shown.

[0026] Figure 4 A schematic diagram illustrating an example of full gather processing between member processing devices within the same training node according to an embodiment of this specification is shown.

[0027] Figure 5 An example schematic diagram of Reduce processing between member processing devices within the same training node according to an embodiment of this specification is shown.

[0028] Figure 6 A schematic diagram illustrating an example of full Reduce processing between member processing devices of different training nodes according to an embodiment of this specification is shown.

[0029] Figure 7 A schematic diagram illustrating an example of data transmission between member devices of a training node according to an embodiment of this specification is shown.

[0030] Figure 8 An example block diagram of a model training optimization apparatus according to an embodiment of this specification is shown.

[0031] Figure 9 An example block diagram of a data slicing unit according to an embodiment of this specification is shown.

[0032] Figure 10 An example schematic diagram of a model training and optimization apparatus based on a computer system according to an embodiment of this specification is shown. Detailed Implementation

[0033] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the various examples. For example, the described methods may be performed in a different order than described, and steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in other examples.

[0034] As used herein, the term "comprising" and its variations are open terms meaning "including but not limited to". The term "based on" means "at least partially based on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below. Unless explicitly indicated by the context, the definition of a term shall remain consistent throughout the specification.

[0035] In distributed machine learning model training, multiple training nodes generate different model state data at different stages of the training process. This model state data can include, for example, model parameters generated when defining the model structure, gradient information generated during backpropagation after model computation, and optimizer state data generated when updating model parameters. This generated model state data needs to be stored on each training node for subsequent collaborative use by different training nodes in the distributed machine learning model training system. In conventional distributed machine learning model training schemes, each training node needs to store all model state data generated during model training and transmit the required model state data between or within training nodes, thus consuming a large amount of storage space and incurring significant data communication overhead, thereby greatly reducing the system performance of the distributed machine learning model training system.

[0036] Therefore, embodiments of this specification provide a method, apparatus, and distributed machine learning model training system for optimizing machine learning model training. Using this method and apparatus for optimizing machine learning model training, the gradient information generated by each member processing device within a training node is divided into multiple gradient information slices. Each member processing device within the same training node performs data merging processing using its respective gradient information slice, thereby obtaining a first gradient information slice merging result after data merging processing using its corresponding gradient information slice. Subsequently, member processing devices with the same number in different training nodes perform data merging processing again using their respective first gradient information slice merging results, thereby obtaining a second gradient information slice merging result, which is then stored locally. This reduces the storage space occupied at the training nodes of the distributed machine learning training system. Furthermore, during model updates, each member processing device uses the locally stored second gradient information slice merging result to update its local model parameters, and uses its local model update result with other member processing devices within the same training node to perform model parameter concatenation, thereby completing the local model update. This approach transmits only the first gradient information slice merging result between different training nodes, while other model state data is transmitted within the same training node. This greatly reduces data communication overhead and improves the system performance of the distributed machine learning model training system.

[0037] In this specification, the term "machine learning model" can refer to a machine learning model applied in a business scenario to provide business prediction services, such as a machine learning model used for classification prediction, business risk prediction, etc. Examples of machine learning models may include, but are not limited to: linear regression models, logistic regression models, neural network models, decision tree models, support vector machines, etc. Examples of neural network models may include, but are not limited to: deep neural network (DNN) models, convolutional neural network (CNN) models, backpropagation (BP) neural networks, etc.

[0038] The specific implementation of a machine learning model can depend on the business scenario in which it is applied. For example, in an application scenario where a machine learning model is used to classify users, it is implemented as a user classification model. Accordingly, this model can be used to predict user classification based on user characteristic data of the users to be classified. In an application scenario where a machine learning model is used to predict business risks of business transactions occurring on a business system, it is implemented as a business risk prediction model. Accordingly, this model can be used to predict business risks based on the business transaction characteristic data of that business transaction.

[0039] The following description, with reference to the accompanying drawings, describes a method, apparatus, and distributed machine learning model training system for optimizing machine learning model training according to embodiments of this specification.

[0040] Figure 1 A block diagram of a distributed machine learning model training system 100 according to an embodiment of this specification is shown.

[0041] like Figure 1 As shown, the distributed machine learning model training system 100 includes at least two training nodes 110. For example... Figure 1 As shown, the distributed machine learning model training system 100 includes four training nodes 110, namely, training nodes 110-1, 110-2, 110-3, and 110-4. It should be noted that in other embodiments, the distributed machine learning model training system 100 may include more or fewer training nodes 110. The training nodes 110 can communicate with each other via a network 120 to share data. In some embodiments, the network 120 can be any one or more of a wired network or a wireless network. Examples of the network 120 may include, but are not limited to, cable networks, fiber optic networks, telecommunications networks, corporate intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigZee networks, near field communication (NFC), device internal buses, device internal lines, etc., or any combination thereof. In some embodiments, the training nodes 110 can communicate directly with each other without the need for the network 120.

[0042] Each training node 110 may have at least two member processing devices 111 and a storage device 112. Each training node also has local training data and provides the local training data to the at least two member processing devices 111 to perform local model training of the machine learning model. For example, each member processing device may perform model calculations during forward propagation, gradient information calculations during backpropagation, local model parameter updates, etc., and store the resulting model state data in the storage space of the storage device allocated to that member processing device.

[0043] In some embodiments, examples of member processing devices at each training node may include, but are not limited to, GPUs, GNU processors, or MPUs. Each training node has the same number of member processing devices, and each member processing device at each training node is assigned a different number. Each numbered member processing device stores its corresponding model state data in its allocated storage space. Member processing devices with the same number at different training nodes can form a correspondence.

[0044] In some embodiments, member processing devices within the same distributed training node can be linked via a first communication link, and member devices with the same ID on different distributed training nodes can be linked via a second communication link. The data transmission bandwidth of the first communication link is higher than that of the second communication link. For example, when the member processing device is a GPU, member processing devices within the same distributed training node can be linked via a high-speed NVLINK link. The first communication link can also be referred to as an internal communication link, and the second communication link can also be referred to as an external communication link.

[0045] Figure 2 An example flowchart of a distributed machine learning model training process 200 according to an embodiment of this specification is shown. Figure 2 In the example, the distributed machine learning model training system includes two training nodes, Node0 and Node1, and each training node has two GPU devices, for example, two GPU cards. Each GPU device stores a portion of the model parameters of the machine learning model to be trained, and the model parameters stored by the two GPU devices within the same training node constitute the complete model parameters of the machine learning model.

[0046] like Figure 2 As shown in section 210, in response to initiating the machine learning model training process, each member processing device within the same training node collaboratively generates a complete model of the machine learning model to be trained using its own stored partial model parameters. For example, two GPU devices within each training node can transmit their respective partial model parameters to other GPU devices via an internal communication link. Subsequently, each GPU device concatenates the received partial model parameters with its stored partial model parameters to generate the complete model parameters of the machine learning model to be trained, thereby obtaining a complete machine learning model.

[0047] At 220, each member processing device uses local training data to perform local model computations, thereby obtaining gradient information for updating model parameters.

[0048] At 230, each member processing device stores the obtained gradient information in the storage space of the storage device allocated to each member device.

[0049] Next, at 240, each member processing device uses the gradient information stored in its own storage space to perform local model parameter updates, and each member processing device within the same training node uses its own local model parameter update results to collaboratively complete the model update of the machine learning model.

[0050] exist Figure 2The distributed machine learning model training process shown herein, according to embodiments of this specification, involves steps 210, 230, and 240. These will now be described in detail with reference to the accompanying drawings.

[0051] Figure 3 An example flowchart of a method 300 for optimizing the training of a machine learning model according to an embodiment of this specification is shown. Figure 3 In the example, the distributed machine learning model training system includes two training nodes, Node0 and Node1, and each training node has two GPU devices. Each GPU device stores a portion of the model parameters of the machine learning model to be trained, and the model parameters stored by the two GPU devices within the same training node together constitute the complete model parameters of the machine learning model. Furthermore, each member processing device in each training node is assigned a different number. For example... Figure 3 As shown, the two GPU devices in training nodes Node0 and Node1 are assigned the numbers 0 and 1, namely Node0.GPU0, Node0.GPU1, Node1.GPU0 and Node1.GPU1.

[0052] like Figure 3 As shown, at 310, in response to initiating the local model training process, each member device, together with all other member processing devices on the distributed training node to which it belongs, performs model parameter concatenation using partial model parameters of their respective stored machine learning models to generate a complete machine learning model. In some embodiments, model parameter concatenation can be implemented using a full gather process.

[0053] Figure 4 A schematic diagram illustrating an example of full gather processing between member processing devices within the same training node according to an embodiment of this specification is shown.

[0054] like Figure 4 As shown, Node0.GPU0 within training node Node0 initially stores some model parameters P00, and Node0.GPU1 initially stores some model parameters P01. Model parameters P00 and P01 together constitute the complete model parameters of the machine learning module. Node1.GPU0 within training node Node1 initially stores some model parameters P10, and Node1.GPU1 initially stores some model parameters P11. Model parameters P10 and P11 together constitute the complete model parameters of the machine learning module.

[0055] In response to initiating the local model training process, Node0.GPU0 transmits model parameters P00 to Node0.GPU1 via an internal communication link, and Node0.GPU1 transmits model parameters P00 to Node0.GPU0 via an internal communication link. Node1.GPU0 transmits model parameters P10 to Node1.GPU1 via an internal communication link, and Node1.GPU1 transmits model parameters P11 to Node1.GPU0 via an internal communication link.

[0056] Subsequently, each GPU device uses the received partial model parameters and its existing partial model parameters to perform model parameter concatenation to obtain complete machine learning models. For example, Node0.GPU0 can concatenate the received model parameter P01 after the stored model parameter P00 to obtain the complete model parameters of the machine learning model to be trained. Node0.GPU1 can concatenate the received model parameter P00 before the stored model parameter P01 to obtain the complete model parameters of the machine learning model to be trained. Node1.GPU0 can concatenate the received model parameter P11 after the stored model parameter P10 to obtain the complete model parameters of the machine learning model to be trained. Node1.GPU1 can concatenate the received model parameter P10 before the stored model parameter P11 to obtain the complete model parameters of the machine learning model to be trained. In some embodiments, each model parameter of the machine learning model may have a unique parameter position in the complete model parameters, and each GPU device can fill each model parameter into its respective parameter position to complete the model parameter concatenation process.

[0057] At 320, in response to completing the local model calculation and obtaining the gradient information of the model parameters, each member processing device slices the obtained gradient information into a first number of gradient information slices, where the first number is the number of member processing devices in the distributed training node where the member processing device is located. For example, in Figure 3 In the example, the number of member processing devices in each distributed training node is 2, so each member processing device will slice the obtained gradient information into 2 gradient information slices.

[0058] In some embodiments, each member processing device can divide the obtained gradient information slice into a first number of gradient information slices. In some embodiments, after each member processing device completes local model calculation to obtain the gradient information of the model parameters, the available storage space of all member processing devices in the distributed training node to which the member processing device belongs can be obtained. Then, according to the available storage space of each member processing device, the obtained gradient information slice is divided into a first number of gradient information slices, and each divided gradient information slice has a slice size that matches the proportion of the available storage space of the corresponding numbered member processing device.

[0059] In some embodiments, in response to each member processing device completing local model computation and obtaining gradient information of model parameters, the available storage space and available processing capacity of each member processing device in the distributed training node to which the member processing device belongs can be obtained. Subsequently, based on the available storage space and available processing capacity of each member processing device, the obtained gradient information is sliced ​​into a first number of gradient information slices, and each slice has a slice size that matches the proportion of available storage space and available processing capacity of the corresponding numbered member processing device.

[0060] At step 330, each member processing device, together with all other member processing devices on the distributed training node to which it belongs, performs data merging processing using its respective gradient information slices. Each numbered member processing device obtains a first gradient information slice merged result after data merging processing using the gradient information slice with the corresponding number. It should be noted that in the above data merging processing, the elements in each corresponding gradient information slice are merged accordingly, so that the storage space occupied by the merged gradient information slice result remains unchanged. For example, assuming that the storage space occupied by the corresponding numbered gradient information slice of a member processing device is k bits, then the storage space occupied by the merged gradient information slice result is also k bits. In some embodiments, the data merging processing can be implemented using Reduce processing.

[0061] Figure 5 An example schematic diagram of Reduce processing between member processing devices within the same training node according to an embodiment of this specification is shown. Figure 5 The diagram shows the Reduce process executed together by Node0.GPU0 and Node0.GPU1.

[0062] like Figure 5As shown, Node0.GPU0 divides the obtained gradient information into gradient information slices t00 and t01, and Node0.GPU1 divides the obtained gradient information into gradient information slices t10 and t11. Then, Node0.GPU0 synchronizes gradient information slice t01 to Node0.GPU1, and Node0.GPU1 synchronizes gradient information slice t10 to Node0.GPU0. Next, Node0.GPU0 performs data merging on the received gradient information slice t10 and the retained gradient information slice t00 to obtain the first gradient information slice merging result T00, i.e., T00 = t00 + t10. Node0.GPU1 performs data merging on the received gradient information slice t01 and the retained gradient information slice t11 to obtain the first gradient information slice merging result T01, i.e., T01 = t01 + t11. The gradient information slice t00 and the first gradient information slice merging result T00 at Node0.GPU0 occupy the same amount of storage space. The gradient information slice t11 at Node0.GPU1 and the merged result T01 of the first gradient information slice occupy the same amount of storage space. It should be noted that... Figure 5 The diagram only shows the Reduce process of Node0.GPU0.

[0063] Back Figure 3 After each member processing device obtains its own first gradient information slice merging result, at point 340, each member processing device, together with all other member processing devices with the same ID on other distributed training nodes, performs data merging processing using their respective first gradient information slice merging results to obtain the second gradient information slice merging result. It should be noted that in the data merging processing at point 340, member processing devices with the same ID on different distributed training nodes collaborate, and each participating member processing device performs the corresponding data merging processing, and the data merging processing operations performed are identical. In this specification, this data merging processing at point 340 can also be referred to as "complete data merging processing." In some embodiments, the above-mentioned data merging processing at point 340 can be implemented using "complete reduce processing."

[0064] Figure 6 A schematic diagram illustrating an example of full Reduce processing between member processing devices of different training nodes according to an embodiment of this specification is shown.

[0065] like Figure 6As shown, after Node0.GPU0, Node0.GPU1, Node1.GPU0, and Node1.GPU1 obtain their respective first gradient information slice merging results T00, T01, T10, and T11, Node0.GPU0 synchronizes the first gradient information slice merging result T00 to Node1.GPU0, and Node1.GPU0 synchronizes the first gradient information slice merging result T10 to Node0.GPU0. Node0.GPU1 synchronizes the first gradient information slice merging result T01 to Node1.GPU1, and Node1.GPU1 synchronizes the first gradient information slice merging result T11 to Node0.GPU1. Subsequently, each member processing device performs data merging using the received first gradient information slice merging result and its own obtained first gradient information slice merging result, thereby obtaining the second gradient information slice merging result. If Figure 6 As shown, Node0.GPU0 and Node1.GPU0 add the corresponding elements in the first gradient information slice merging result T00 and the first gradient information slice merging result T10, respectively, to obtain the second gradient information slice merging result T0. Node0.GPU1 and Node1.GPU1 add the corresponding elements in the first gradient information slice merging result T01 and the first gradient information slice merging result T11, respectively, to obtain the second gradient information slice merging result T1.

[0066] At 350, each member processing device stores the merged result of its respective second gradient information slice in the storage space allocated to that member processing device, thereby completing the gradient information storage processing.

[0067] Optionally, at 360, in response to obtaining the merged result of the second gradient information slices, each member processing device uses its own merged result of the second gradient information slices to locally update the model parameters of the machine learning model to be trained.

[0068] At point 370, each member processing device, together with all other member processing devices on the distributed training node to which it belongs, uses its updated model parameters to perform model parameter concatenation to update the model parameters of the machine learning model, thereby completing the current model training process. Here, the model parameter concatenation process used to update the model parameters of the machine learning model can employ... Figure 4 The process of splicing model parameters shown in the figure will not be described in detail here.

[0069] The above describes the machine learning model training process according to an embodiment of this specification. Figure 7 A schematic diagram illustrating an example of data transmission between member processing devices of training nodes during gradient information storage, according to an embodiment of this specification, is shown.

[0070] like Figure 7 As shown, during gradient information storage, the data slices after data segmentation are transmitted between the member processing devices within the same training node via internal communication links. Data transmission between member processing devices across training nodes is limited to the transmission of the merged result of the first gradient information slices between member processing devices with the same ID via external communication links.

[0071] Furthermore, during the generation of complete model parameters for the machine learning model to be trained, some model parameters that need to be transmitted are transmitted via internal communication links among the member processing devices within the same training node. Additionally, during local model updates, each member processing device uses its own second gradient information slice merging result to perform local model parameter updates locally, and transmits the obtained local model parameter update results to other member processing devices within the same training node via internal communication links, thereby achieving local model updates. In this process, only intra-node communication via internal communication links is required, without data transmission across training nodes, thus significantly reducing the data communication overhead caused by data transmission between training nodes.

[0072] Using the aforementioned method and apparatus for optimizing machine learning model training, the gradient information generated by each member processing device within a training node is divided into multiple gradient information slices. Each member processing device within the same training node performs data merging processing using its respective gradient information slice, thereby obtaining a first gradient information slice merging result after data merging processing using the corresponding gradient information slice. Subsequently, member processing devices with the same number in different training nodes perform data merging again using their respective first gradient information slice merging results, thereby obtaining a second gradient information slice merging result and storing it locally. This reduces the storage space occupied at the training nodes of the distributed machine learning training system.

[0073] The above description, with reference to the accompanying drawings, outlines a method for optimizing the training of a machine learning model according to embodiments of this specification.

[0074] Figure 8 An example block diagram of a model training optimization apparatus 800 according to an embodiment of this specification is shown. Figure 8 As shown, the model training optimization device 800 includes a model generation unit 810, a data splitting unit 820, an intra-node data processing unit 830, an inter-node data processing unit 840, a data storage unit 850, a local model parameter update unit 860, and a model update unit 870. The model training optimization device 800 is applied to a member processing device of a distributed training node participating in the training of a distributed machine learning model.

[0075] The model generation unit 810 is configured to, in response to initiating a local model training process, work with all other member processing devices on the distributed training node to which it belongs, to perform model parameter concatenation using partial model parameters of their respective stored machine learning models to generate a complete machine learning model. The operation of the model generation unit 810 can be referenced above. Figure 3 The operation described in 310.

[0076] The data slicing unit 820 is configured to slice the gradient information obtained from the local model calculation into a first number of gradient information slices in response to the gradient information obtained from the local model calculation. The first number is the number of member processing devices in the distributed training node to which the member processing device belongs. The operation of the data slicing unit 820 can be referred to the above reference. Figure 3 The operation described in 320.

[0077] The intra-node data processing unit 830 is configured to perform data merging processing together with all other member processing devices in the distributed training node to which it belongs, using their respective gradient information slices. Each member processing device in the distributed training node is assigned a number, and each numbered member processing device receives the first gradient information slice merging result after data merging processing using the gradient information slice of the corresponding number. The operation of the intra-node data processing unit 830 can be referenced above. Figure 3 The operation described in 330.

[0078] The inter-node data processing unit 840 is configured to perform data merging processing together with all other member processing devices of the same number as other distributed training nodes, using their respective first gradient information slice merging results to obtain the second gradient information slice merging result. The operation of the inter-node data processing unit 840 can be referenced above. Figure 3 The operation described in 340.

[0079] The data storage unit 850 is configured to store the merged result of the second gradient information slices in the storage space allocated to the member processing device. The operation of the data storage unit 850 can be referenced above. Figure 3 The operation described in 350.

[0080] The local model parameter update unit 860 is configured to update the model parameters of the machine learning model to be trained using the merged result of the second gradient information slices in response to obtaining the merged result. The operation of the local model parameter update unit 860 can be referenced above. Figure 3 The operation described in 360.

[0081] The model update unit 870 is configured to work with all other member processing devices on the distributed training node to which it belongs, using their respective updated model parameters to perform model parameter concatenation to update the model parameters of the machine learning model. The operation of the model update unit 870 can be referenced above. Figure 3 The operation described in 370.

[0082] Figure 9 An example block diagram of a data slicing unit 900 according to an embodiment of this specification is shown. Figure 9 As shown, the data slicing unit 900 includes an available storage space acquisition module 910 and a data slicing module 930.

[0083] The available storage space acquisition module 910 is configured to acquire the available storage space of each member processing device in the distributed training node to which the member processing device belongs in response to the gradient information of the model parameters obtained after completing the local model calculation.

[0084] The data slicing module 930 is configured to slice gradient information into a first number of gradient information slices based on the available storage space of each member processing device.

[0085] In some embodiments, the data slicing module 930 can slice the gradient information into a first number of gradient information slices according to the available storage space of each member processing device, and each of the sliced ​​gradient information slices has a slice size that matches the proportion of the available storage space of the corresponding member processing device.

[0086] In some embodiments, the data slicing unit 900 may further include an available processing capacity acquisition module (not shown). The available processing capacity acquisition module is configured to acquire the available processing capacity of each member processing device in the distributed training node to which the member processing device belongs. Accordingly, the data slicing module 930 slices the gradient information into a first number of gradient information slices based on the available storage space and available processing capacity of each member processing device. Each slice has a slice size that matches the proportion of available storage space and available processing capacity of the corresponding numbered member processing device.

[0087] In some embodiments, the model training optimization apparatus 800 may not include the model generation unit 810, or the local model parameter update unit 860 and the model update unit 870, or the three component units mentioned above.

[0088] As referred above Figures 1 to 9This specification describes a method and apparatus for optimizing machine learning model training according to embodiments thereof. The model training optimization apparatus described above can be implemented in hardware, software, or a combination of both.

[0089] Figure 10 A schematic diagram illustrating an example of a computer-based model training and optimization apparatus 1000 implemented according to an embodiment of this specification is shown. Figure 10 As shown, the model training optimization apparatus 1000 may include at least one processor 1010, a memory (e.g., non-volatile memory) 1020, a main memory 1030, and a communication interface 1040, and the at least one processor 1010, memory 1020, main memory 1030, and communication interface 1040 are connected together via a bus 1060. At least one processor 1010 executes at least one computer-readable instruction (i.e., the elements implemented in software described above) stored or encoded in the memory.

[0090] In one embodiment, computer-executable instructions are stored in memory, which, when executed, cause at least one processor 1010 to: slice the gradient information into a first number of gradient information slices in response to completing local model computation to obtain gradient information of model parameters, the first number being the number of member processing devices in the distributed training node to which the member processing device belongs; perform data merging processing using their respective gradient information slices together with all other member processing devices in the distributed training node to which the member processing device belongs, wherein each member processing device in the distributed training node is assigned a number, and each numbered member processing device obtains a first gradient information slice merging result after data merging processing using the gradient information slice with the corresponding number; perform data merging processing using their respective first gradient information slice merging results together with all other member processing devices with the same number in other distributed training nodes to obtain a second gradient information slice merging result; and store the second gradient information slice merging result in storage space allocated to the member processing device.

[0091] It should be understood that the computer-executable instructions stored in memory, when executed, cause at least one processor 1010 to perform the above-described combinations in the various embodiments of this specification. Figures 1-9 The description includes various operations and functions.

[0092] According to one embodiment, a program product, such as a machine-readable medium (e.g., a non-transitory machine-readable medium), is provided. The machine-readable medium may have instructions (i.e., the elements implemented in software as described above), which, when executed by a machine, cause the machine to perform the above-described combinations of the various embodiments of this specification. Figures 1-9The various operations and functions described. Specifically, a system or apparatus equipped with a readable storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer or processor of the system or apparatus to read and execute the instructions stored in the readable storage medium.

[0093] In this case, the program code itself, which can be read from the readable medium, can perform the functions of any of the above embodiments. Therefore, the machine-readable code and the readable storage medium storing the machine-readable code constitute part of the present invention.

[0094] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0095] According to one embodiment, a computer program product is provided, the computer program product including a computer program, which, when executed by a processor, causes the processor to perform the above-described combinations of the various embodiments of this specification. Figures 1-9 The description includes various operations and functions.

[0096] Those skilled in the art will understand that the various embodiments disclosed above can be modified and varied without departing from the spirit of the invention. Therefore, the scope of protection of this invention should be defined by the appended claims.

[0097] It should be noted that not all steps and units in the above process and system structure diagrams are mandatory; some steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure; that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0098] In the above embodiments, the hardware units or modules can be implemented mechanically or electrically. For example, a hardware unit, module, or processor may include permanent dedicated circuitry or logic (such as a dedicated processor, FPGA, or ASIC) to perform the corresponding operation. The hardware unit or processor may also include programmable logic or circuitry (such as a general-purpose processor or other programmable processor), which can be temporarily configured by software to perform the corresponding operation. The specific implementation method (mechanical, dedicated permanent circuitry, or temporarily configured circuitry) can be determined based on cost and time considerations.

[0099] The specific embodiments described above with reference to the accompanying drawings are exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of the claims. The term "exemplary" as used throughout this specification means "serving as an example, instance, or illustration" and does not imply that it is "preferred" or "advantageous" compared to other embodiments. Specific details are included to provide an understanding of the described techniques. However, these techniques can be practiced without these specific details. In some instances, well-known structures and apparatuses are shown in block diagram form to avoid obscuring the concepts of the described embodiments.

[0100] The foregoing description of this disclosure is provided to enable any person skilled in the art to implement or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but is consistent with the widest scope of the principles and novel features disclosed herein.

Claims

1. A method for optimizing machine learning model training, the method being performed by a member processing device in one of at least two distributed training nodes participating in machine learning model training, the method comprising: In response to the gradient information of the model parameters obtained by completing the local model calculation, the gradient information is sliced ​​into a first number of gradient information slices, where the first number is the number of member processing devices in the distributed training node where the member processing device is located. Together with all other member processing devices of the distributed training node to which the member processing device belongs, data merging processing is performed using their respective gradient information slices. Each member processing device in the distributed training node is assigned a number, and each numbered member processing device obtains the first gradient information slice merging result after data merging processing using the gradient information slice with the corresponding number. Together with all other member processing devices with the same number in other distributed training nodes, data merging processing is performed using their respective first gradient information slice merging results to obtain the second gradient information slice merging results. as well as The merged result of the second gradient information slice is stored in the storage space allocated to the member processing device. The method further includes: In response to obtaining the merged result of the second gradient information slices, the model parameters of the machine learning model to be trained are updated using the merged result of the second gradient information slices; and Together with all other member processing devices on the distributed training node to which the member processing device belongs, the model parameters of the machine learning model are updated by performing model parameter concatenation using their respective updated model parameters.

2. The method as described in claim 1, wherein, In response to obtaining gradient information of model parameters after completing local model computation, slicing the gradient information into a first number of gradient information slices includes: In response to obtaining gradient information of model parameters after completing local model computation, the available storage space of each member processing device in the distributed training node to which the member processing device belongs is obtained; and Based on the available storage space of each member processing device, the gradient information is sliced ​​into a first number of gradient information slices.

3. The method as described in claim 2, wherein, Based on the available storage space of each member processing device, slicing the gradient information into a first number of gradient information slices includes: Based on the available storage space of each member processing device, the gradient information is sliced ​​into a first number of gradient information slices, and each of the slices has a slice size that matches the proportion of the available storage space of the corresponding member processing device.

4. The method of claim 2, wherein, In response to obtaining gradient information of model parameters after completing local model computation, the available storage space of each member processing device in the distributed training node to which the member processing device belongs includes: In response to obtaining gradient information of model parameters after completing local model computation, the available storage space and available processing capacity of each member processing device in the distributed training node to which the member processing device belongs are obtained, and Based on the available storage space of each member processing device, slicing the gradient information into a first number of gradient information slices includes: Based on the available storage space and available processing capacity of each member processing device, the gradient information is sliced ​​into a first number of gradient information slices, and each of the slices has a slice size that matches the proportion of available storage space and available processing capacity of the corresponding member processing device.

5. The method of claim 1, further comprising: In response to initiating the local model training process, together with all other member processing devices on the distributed training node to which the member processing device belongs, the model parameter concatenation is performed using the partial model parameters of the machine learning model stored in each device to generate a complete machine learning model.

6. The method of claim 1, wherein, Member processing devices within the same distributed training node are linked by a first communication link, and member devices with the same number on different distributed training nodes are linked by a second communication link. The data transmission bandwidth of the first communication link is higher than that of the second communication link.

7. The method of claim 1, wherein, The member processing device includes a GPU device, and the storage space includes video memory.

8. An apparatus for optimizing machine learning model training, the apparatus being applied to a member processing device in one of at least two distributed training nodes participating in machine learning model training, the apparatus comprising: The data slicing unit, in response to completing the local model calculation to obtain the gradient information of the model parameters, slices the gradient information into a first number of gradient information slices, where the first number is the number of member processing devices in the distributed training node where the member processing device is located. The data processing unit within the node, together with all other member processing devices of the distributed training node to which the member processing device belongs, performs data merging processing using their respective gradient information slices. Each member processing device within the distributed training node is assigned a number, and each numbered member processing device obtains the first gradient information slice merging result after performing data merging processing using the gradient information slice with the corresponding number. The data processing unit between nodes, together with all other member processing devices with the same number as other distributed training nodes, performs data merging processing using their respective first gradient information slice merging results to obtain the second gradient information slice merging results. as well as The data storage unit stores the merged result of the second gradient information slice in the storage space allocated to the member processing device. The device further includes: The local model parameter update unit, in response to obtaining the merged result of the second gradient information slice, updates the model parameters of the machine learning model to be trained using the merged result of the second gradient information slice; and The model update unit, together with all other member processing devices of the distributed training node to which the member processing device belongs, performs a complete model parameter concatenation using their respective updated model parameters to update the model parameters of the machine learning model.

9. The apparatus of claim 8, wherein, The data slicing unit includes: The available storage space acquisition module, in response to obtaining gradient information of model parameters after completing local model calculation, acquires the available storage space of each member processing device in the distributed training node to which the member processing device belongs; and The data slicing module slices the gradient information into a first number of gradient information slices based on the available storage space of each member processing device.

10. The apparatus of claim 9, wherein, The data slicing module slices the gradient information into a first number of gradient information slices based on the available storage space of each member processing device. Each of the slices has a slice size that matches the proportion of the available storage space of the corresponding member processing device.

11. The apparatus of claim 9, wherein, The data slicing unit further includes: The available processing capacity acquisition module acquires the available processing capacity of each member processing device in the distributed training node to which the member processing device belongs. The data slicing module divides the gradient information into a first number of gradient information slices based on the available storage space and available processing capacity of each member processing device. Each gradient information slice has a slice size that matches the ratio of available storage space and available processing capacity of the corresponding member processing device.

12. The apparatus of claim 8, further comprising: The model generation unit, in response to initiating a local model training process, together with all other member processing devices of the distributed training node to which the member processing device belongs, uses partial model parameters of the machine learning model stored in each device to perform model parameter concatenation to generate a complete machine learning model.

13. A distributed machine learning model training system, comprising: At least two distributed training nodes, each training node including a storage device and at least two member processing devices, each member processing device including means for optimizing machine learning model training as described in any one of claims 8 to 12.

14. An apparatus for optimizing the training of a machine learning model, comprising: At least one processor, Memory coupled to the at least one processor, and A computer program stored in the memory, which is executed by the at least one processor to implement the method for optimizing machine learning model training as described in any one of claims 1 to 7.

15. A computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the method for optimizing the training of a machine learning model as described in any one of claims 1 to 7.

16. A computer program product comprising a computer program executed by a processor to implement the method for optimizing machine learning model training as described in any one of claims 1 to 7.

Images