Distributed Systems and Methods for Performing Multiple Machine Learning Tasks

Abstract

A distributed system and method for performing multiple machine learning tasks are provided. The distributed system includes multiple computing devices configured to acquire different portions of a specified dataset and jointly execute multiple machine learning tasks; wherein each computing device is configured to execute the multiple machine learning tasks in parallel based on its acquired portion of data, wherein the multiple machine learning tasks are multiple model training tasks or multiple model prediction tasks. According to the distributed system and method, the time required to complete multiple machine learning tasks can be effectively shortened.

Description

[0001] This application is a divisional application of patent application No. 201910759163.0, filed on August 16, 2019, entitled "Distributed System and Method Thereof for Performing Multiple Machine Learning Tasks". Technical Field

[0002] This invention generally relates to the field of artificial intelligence, and more specifically, to a distributed system and method for performing multiple machine learning tasks. Background Technology

[0003] The outcome of a machine learning training task is typically determined by the values ​​of a large number (e.g., dozens) of configuration parameters (i.e., hyperparameters). In scenarios such as automated machine learning, in order to evaluate the outcome of machine learning training tasks under different configurations, it is often necessary to try different values ​​and combinations of configuration parameters, and to calculate and evaluate the machine learning training tasks under different configurations.

[0004] In the process of exploring the best machine learning model, the number of machine learning training tasks to be performed increases exponentially with the number of configuration parameters and the number of possible values ​​for each configuration parameter. For example, for a machine learning training task that only has 10 configuration parameters to configure, if each configuration parameter has 3 possible values, then these 10 configuration parameters will generate a total of 59,049 possible combinations, which will correspond to 59,049 machine learning training tasks.

[0005] Existing technologies typically execute machine learning training tasks under different configurations independently, but there are significant task execution efficiency issues, which often prevent automated machine learning tasks and other tasks that require training multiple machine learning models from being completed within a reasonable time. Summary of the Invention

[0006] An exemplary embodiment of the present invention provides a distributed system and method for performing multiple machine learning tasks, which can solve the problem that multiple machine learning tasks cannot be completed within a reasonable time in the prior art.

[0007] According to an exemplary embodiment of the present invention, a distributed system for performing multiple machine learning tasks is provided, comprising: a plurality of computing devices configured to acquire different portions of data from a specified dataset and jointly perform multiple machine learning tasks; wherein each computing device is configured to perform the multiple machine learning tasks in parallel based on its acquired portions of data, wherein the multiple machine learning tasks are multiple model training tasks or multiple model prediction tasks.

[0008] Optionally, each computing device is configured to: on the one hand, request data from the specified dataset from the data source, preprocess the requested data and save it locally; on the other hand, read the locally saved data and perform the multiple machine learning tasks based on the read data.

[0009] Optionally, each computing device is configured to execute, in parallel, the machine learning tasks that require the use of each piece of data, based on each piece of data read.

[0010] Optionally, each computing device is configured to: based on each piece of data read, execute in parallel the machine learning tasks that require the use of that data among the multiple machine learning tasks using vectorized instructions.

[0011] Optionally, the distributed system further includes: a parameter server configured to maintain parameters of multiple machine learning models involved in the multiple machine learning tasks, wherein the parameters of the machine learning models are in the form of key-value pairs, wherein the parameter server is configured to: merge the parameters of the multiple machine learning models with the same key in the form of a single key corresponding to multiple values ​​and then save them; or, merge the parameters of the multiple machine learning models with the same key in the form of a single key corresponding to multiple values ​​and then compress the merged result according to a first compression method and then save it.

[0012] Optionally, when the plurality of machine learning tasks are the plurality of model training tasks, each computing device is configured to: provide the merged results of training the plurality of machine learning models to a parameter server, or provide the merged results of training the plurality of machine learning models, which are also compressed according to a second compression method, to the parameter server so that the parameter server updates the parameters of the plurality of machine learning models, wherein the results are merged in the form of a single key corresponding to multiple values.

[0013] Optionally, the parameter server is configured to provide each computing device with the parameters of the plurality of machine learning models required by each computing device that has been merged, or to provide each computing device with the parameters of the plurality of machine learning models required by each computing device that has been merged and compressed according to a third compression method.

[0014] Optionally, when the plurality of machine learning tasks are the plurality of model training tasks, the parameter server is configured to: during the execution of the plurality of machine learning tasks by each computing device, save the intermediate computation results generated by the computing device when training a machine learning model that can be used for other machine learning models, so as to use the intermediate computation results for the other machine learning models.

[0015] Optionally, the hyperparameters corresponding to the one machine learning model and the other machine learning models differ only in the number of training epochs, wherein the number of training epochs corresponding to the one machine learning model is greater than the number of training epochs corresponding to the other machine learning models, and the parameter server is configured to use the parameters of the one machine learning model obtained by the computing device during the training of the one machine learning model when the number of training epochs reaches the number of training epochs corresponding to the other machine learning models as the parameters of the other machine learning models.

[0016] Optionally, each computing device is configured to: set the network configuration used by the plurality of machine learning tasks; and / or, the parameter server is configured to: set the network configuration for the plurality of machine learning tasks.

[0017] Optionally, each computing device is configured to: use zero-copy technology for network transmission for the plurality of machine learning tasks; and / or set the size of the maximum transmission unit in network transmission for the plurality of machine learning tasks; and / or, the parameter server is configured to: use zero-copy technology for network transmission for the plurality of machine learning tasks; and / or set the size of the maximum transmission unit in network transmission for the plurality of machine learning tasks.

[0018] Optionally, each computing device is configured to: configure the memory used by the plurality of machine learning tasks; and / or, the parameter server is configured to: configure the memory for the plurality of machine learning tasks.

[0019] Optionally, each computing device is configured to: bind the plurality of machine learning tasks to a set of central processing units (CPUs) such that the set of CPUs uses adjacent memory to execute the plurality of machine learning tasks; and / or, configure a memory management unit for the plurality of machine learning tasks such that the operating system and CPU of the computing device manage the memory used by the plurality of machine learning tasks using the configured memory management unit; and / or, the parameter server is configured to: configure a memory management unit for the plurality of machine learning tasks such that the operating system and CPU of the parameter server manage the memory used by tasks related to the plurality of machine learning tasks using the configured memory management unit; and / or, bind tasks related to the plurality of machine learning tasks to a set of central processing units (CPUs) such that the set of CPUs uses adjacent memory to execute tasks related to the plurality of machine learning tasks.

[0020] According to another exemplary embodiment of the present invention, a method for performing multiple machine learning tasks using a distributed system is provided, wherein the distributed system includes multiple computing devices, and the method includes: the multiple computing devices respectively acquiring different portions of data from a specified dataset; the multiple computing devices jointly performing multiple machine learning tasks based on the acquired portions of data, wherein each computing device performs the multiple machine learning tasks in parallel based on its own acquired portions of data, wherein the multiple machine learning tasks are multiple model training tasks or multiple model prediction tasks.

[0021] Optionally, the step of the multiple computing devices acquiring different portions of the specified dataset includes: each computing device requesting data from the specified dataset from the data source; each computing device preprocessing the requested data and storing it locally; wherein the step of each computing device executing the multiple machine learning tasks in parallel based on its acquired portions includes: each computing device reading the locally stored data and executing the multiple machine learning tasks based on the read data.

[0022] Optionally, the step of each computing device performing the plurality of machine learning tasks based on the read data includes: each computing device performing, in parallel, the machine learning task that requires the use of each piece of data among the plurality of machine learning tasks, based on each piece of data read.

[0023] Optionally, the step of each computing device performing the plurality of machine learning tasks based on the read data includes: each computing device performing, in parallel, the machine learning tasks that require the use of each piece of data among the plurality of machine learning tasks, using vectorized instructions, based on each piece of data read.

[0024] Optionally, the distributed system further includes a parameter server, wherein the method further includes: the parameter server maintaining parameters of multiple machine learning models involved in the multiple machine learning tasks, wherein the parameters of the machine learning models are in the form of key-value pairs, wherein the parameter server merges the parameters of the multiple machine learning models with the same key in the form of a single key corresponding to multiple values ​​and saves them, or the parameter server merges the parameters of the multiple machine learning models with the same key in the form of a single key corresponding to multiple values, and compresses the merged result according to a first compression method and saves it.

[0025] Optionally, the method further includes: when the plurality of machine learning tasks are the plurality of model training tasks, each computing device provides the merged results of training the plurality of machine learning models to the parameter server, or each computing device provides the merged results of training the plurality of machine learning models, which are compressed according to a second compression method, to the parameter server to update the parameters of the plurality of machine learning models, wherein the results are merged in the form of a single key corresponding to multiple values.

[0026] Optionally, the method further includes: the parameter server providing each computing device with the parameters of the plurality of machine learning models required by each computing device, which are merged and compressed according to a third compression method; or, the parameter server providing each computing device with the parameters of the plurality of machine learning models required by each computing device, which are merged and compressed according to a third compression method.

[0027] Optionally, the method further includes: when the plurality of machine learning tasks are the plurality of model training tasks, the parameter server saves intermediate calculation results generated by the computing device when training a machine learning model, which can be used for other machine learning models, during the execution of the plurality of machine learning tasks by each computing device, so as to use the intermediate calculation results for the other machine learning models.

[0028] Optionally, the hyperparameters corresponding to the one machine learning model and the other machine learning models differ only in the number of training epochs, wherein the number of training epochs corresponding to the one machine learning model is greater than the number of training epochs corresponding to the other machine learning models, and the parameter server uses the parameters of the one machine learning model obtained by the computing device during the training of the one machine learning model when the number of training epochs reaches the number of training epochs corresponding to the other machine learning models as the parameters of the other machine learning models.

[0029] Optionally, the method further includes: each computing device setting the network configuration used by the plurality of machine learning tasks; and / or, a parameter server setting the network configuration for the plurality of machine learning tasks.

[0030] Optionally, the step of each computing device setting the network configuration used by the plurality of machine learning tasks includes: each computing device using zero-copy technology for network transmission for the plurality of machine learning tasks; and / or setting the size of the maximum transmission unit in network transmission for the plurality of machine learning tasks; wherein, the step of the parameter server setting the network configuration for the plurality of machine learning tasks includes: the parameter server using zero-copy technology for network transmission for the plurality of machine learning tasks; and / or setting the size of the maximum transmission unit in network transmission for the plurality of machine learning tasks.

[0031] Optionally, the method further includes: configuring memory used by each computing device for the plurality of machine learning tasks; and / or, configuring memory for the plurality of machine learning tasks by a parameter server.

[0032] Optionally, the step of configuring the memory used by each computing device for the plurality of machine learning tasks includes: each computing device binding the plurality of machine learning tasks to a set of central processing units (CPUs) so that the set of CPUs uses adjacent memory to execute the plurality of machine learning tasks; and / or configuring a memory management unit for the plurality of machine learning tasks so that the operating system and CPU of the computing device manage the memory used by the plurality of machine learning tasks using the configured memory management unit; wherein, the step of configuring the memory for the plurality of machine learning tasks by the parameter server includes: the parameter server configuring a memory management unit for the plurality of machine learning tasks so that the operating system and CPU of the parameter server manage the memory used by tasks related to the plurality of machine learning tasks using the configured memory management unit; and / or, the parameter server binding tasks related to the plurality of machine learning tasks to a set of central processing units (CPUs) so that the set of CPUs uses adjacent memory to execute tasks related to the plurality of machine learning tasks.

[0033] The distributed system and method for performing multiple machine learning tasks according to exemplary embodiments of the present invention can effectively shorten the time required to complete multiple machine learning tasks, enabling multiple machine learning tasks to be completed within a reasonable time.

[0034] Further aspects and / or advantages of the general concept of the invention will be set forth in part in the description which follows, and in part will be obvious from the description or may be learned by practice of the general concept of the invention. Attached Figure Description

[0035] The above and other objects and features of exemplary embodiments of the present invention will become clearer from the following description taken in conjunction with the accompanying drawings, which illustrate exemplary embodiments, wherein:

[0036] Figure 1 A block diagram of a distributed system for performing multiple machine learning tasks according to an exemplary embodiment of the present invention is shown.

[0037] Figure 2 An example of a distributed system performing multiple machine learning tasks according to an exemplary embodiment of the present invention is shown;

[0038] Figure 3 A block diagram of a distributed system for performing multiple machine learning tasks according to another exemplary embodiment of the present invention is shown;

[0039] Figure 4 An example is shown of a parameter server storing parameters of multiple machine learning models according to an exemplary embodiment of the present invention;

[0040] Figure 5 An example of parameter transmission for a multi-machine learning model according to an exemplary embodiment of the present invention is shown;

[0041] Figure 6 An example of parallel execution of multiple machine learning tasks according to an exemplary embodiment of the present invention is shown;

[0042] Figure 7 A flowchart illustrating a method for performing multiple machine learning tasks using a distributed system according to an exemplary embodiment of the present invention is shown. Detailed Implementation

[0043] The present invention will now be described in detail with reference to embodiments thereof, examples of which are illustrated in the accompanying drawings, wherein the same reference numerals refer to the same parts throughout. The embodiments will be described below with reference to the accompanying drawings in order to explain the present invention.

[0044] Figure 1 A block diagram of a distributed system for performing multiple machine learning tasks according to an exemplary embodiment of the present invention is shown. Figure 1 As shown, a distributed system for performing multiple machine learning tasks according to an exemplary embodiment of the present invention includes a plurality of computing devices 1000 (e.g., 1000-1, 1000-2, ..., 1000-n (where n is an integer greater than 1)).

[0045] Specifically, multiple computing devices 1000 are configured to acquire different portions of a specified dataset and jointly execute multiple machine learning tasks. Each computing device 1000 is configured to execute the multiple machine learning tasks in parallel based on its acquired portion of data. In other words, different computing devices 1000 jointly execute the same multiple machine learning tasks for different data, and the same computing device 1000 executes these multiple machine learning tasks in parallel.

[0046] Here, the multiple machine learning tasks are either multiple model training tasks or multiple model prediction tasks. A model training task is a task used to train a machine learning model, and the computing device 1000 can update the parameters of the corresponding machine learning model by executing the model training task; a model prediction task is a task used to make predictions using the machine learning model, and the computing device 1000 can obtain prediction results using the corresponding machine learning model by executing the model prediction task.

[0047] As an example, the multiple model training tasks can be: multiple model training tasks using the same machine learning algorithm but with different training configurations (e.g., hyperparameter configurations); or multiple model training tasks using different machine learning algorithms. These different machine learning algorithms can be different machine learning algorithms of the same type (e.g., machine learning algorithms belonging to the same neural network type but with different specific structures (e.g., the depth of the neural network), or they can be different machine learning algorithms of different types. For example, the types of machine learning algorithms can include, but are not limited to: linear regression algorithms, neural network algorithms, and FM algorithms. In other words, the multiple machine learning models trained by the multiple model training tasks can be machine learning models of the same type and the same structure, or they can be machine learning models of the same type but with different structures, or they can be machine learning models of different types.

[0048] Each computing device 1000 is configured to acquire a portion of the specified dataset. The data acquired by different computing devices 1000 have no overlap, and the sum of the data acquired by different computing devices 1000 is exactly the specified dataset. As an example, each computing device 1000 may be configured to: on the one hand, request data from the specified dataset from a data source, preprocess the requested data, and store it locally; on the other hand, read the locally stored data and perform the multiple machine learning tasks based on the read data.

[0049] In existing technologies, each computing device typically executes a machine learning task independently. When multiple computing devices execute multiple machine learning tasks simultaneously, and these tasks share the same dataset, each computing device needs to read the entire dataset separately; that is, the entire dataset will be read multiple times. However, according to an exemplary embodiment of the present invention, each computing device 1000 only needs to read a portion of the dataset, not the entire dataset, and each piece of data in the dataset is read only once, without being read repeatedly. This significantly saves the time that the computing device 1000 spends reading data from the data source and subsequently preprocessing the read data.

[0050] As an example, in each computing device 1000, the task of requesting data from the data source and the plurality of machine learning tasks may be executed by two (or two groups of) threads respectively, that is, the thread used by the task of requesting data from the data source is different from the thread used by the plurality of machine learning tasks.

[0051] As an example, when a data source receives a request from each computing device 1000, it can allocate data from a specified dataset to that device (e.g., one data point or a data block containing multiple data points at a time) until all data in the specified dataset has been allocated. In this way, each data point in the specified dataset will only be read by one computing device 1000, meaning each data point will only be read once. For example, the computing devices 1000 can compete for data in the specified dataset, with the computing device 1000 with greater processing power acquiring more data.

[0052] Figure 2 An example of a distributed system performing multiple machine learning tasks according to an exemplary embodiment of the present invention is shown. Figure 2 As shown, when the distributed system according to an exemplary embodiment of the present invention includes four computing devices 1000 (i.e., 1000-1, 1000-2, 1000-3, and 1000-4), each of the four computing devices 1000 obtains a portion of data from a data source (e.g., a data warehouse). For example, each of the four computing devices 1000 obtains 1 / 4 of the dataset D, and the sum of the data obtained by each of the four computing devices 1000 is the entire dataset D, i.e., D1 + D2 + D3 + D4 = D. Furthermore, the four computing devices 1000 jointly execute multiple model training tasks (i.e., training machine learning model A, training machine learning model B, and training machine learning model C) based on the data obtained from the data source. Specifically, each computing device 1000 trains model A, model B, and model C in parallel based on its own obtained 1 / 4 of the dataset D. In other words, for the training of each model (e.g., model A), all four computing devices 1000 participated based on their own acquired data, and each computing device 1000 trained multiple machine learning models in parallel based on its own acquired data.

[0053] As an example, the locally stored data used by the multiple machine learning tasks may be exactly the same or partially the same.

[0054] As an example, each computing device 1000 can be configured to execute, in parallel, the machine learning tasks that require the use of each piece of data read, based on each piece of data read. When the locally stored data used by the multiple machine learning tasks is exactly the same, the multiple machine learning tasks are executed in parallel for each piece of data read. For example, when the multiple machine learning tasks are multiple model training tasks, and the multiple model training tasks need to use the same dataset to train their respective machine learning models, multiple machine learning models can be trained in parallel for each piece of data read. According to an exemplary embodiment of the present invention, on the one hand, a piece of data read from local storage at one time can be used for multiple machine learning tasks, avoiding repeated reading of data and improving the efficiency of data reading; on the other hand, executing multiple machine learning tasks in parallel for the same piece of data can effectively shorten the execution time of multiple machine learning tasks.

[0055] As an example, each computing device 1000 can be configured to execute, in parallel, the machine learning tasks that require the use of each piece of data in the plurality of machine learning tasks, using vectorized instructions, based on each piece of data read. For example, when the plurality of machine learning tasks are multiple model training tasks, the update amount of multiple machine learning models can be calculated using the same vectorized instruction based on each piece of data read, thereby reducing both the computational load and the computation time for updating the multiple machine learning models.

[0056] Here, a vectorized (SIMD, Single Instruction Multiple Data) instruction refers to an instruction in the central processing unit that can operate on multiple data items simultaneously with a single instruction. Such an instruction can complete more computations in the same amount of time compared to a regular instruction.

[0057] Furthermore, it should be understood that other appropriate methods can also be used to execute machine learning tasks that require the same data in parallel based on each piece of data read, thereby accelerating the execution of multiple machine learning tasks. As an example, multiple or multiple sets of threads (or hardware hyper-threading) can be used to execute machine learning tasks that require the same data in parallel, i.e., different machine learning tasks use different threads. Additionally, using a processor with higher parallelism (e.g., a graphics processing unit (GPU) or a field-programmable gate array (FPGA)) in the computing device 1000 can also be used to execute machine learning tasks that require the same data in parallel, achieving better parallel execution results and accelerating the execution of multiple machine learning tasks.

[0058] Figure 3A block diagram of a distributed system for performing multiple machine learning tasks according to another exemplary embodiment of the present invention is shown. Figure 3 As shown, a distributed system for performing multiple machine learning tasks according to another exemplary embodiment of the present invention may consist of multiple computing devices 1000 and parameter servers 2000.

[0059] Specifically, the parameter server 2000 is configured to maintain parameters of multiple machine learning models involved in the multiple machine learning tasks, wherein the parameters of the machine learning models are in the form of key-value pairs.

[0060] It should be noted that both the computing device 1000 and / or the parameter server 2000 are defined by the processes they perform or the functions they implement, and can refer to either physical or virtual entities. For example, the computing device 1000 can refer to an actual computing machine or a logical entity deployed on that machine. Similarly, the parameter server 2000 can refer to an actual computing machine or act as one or more logical entities deployed on the same and / or different computing machines as the computing device 1000. As an example, the parameter server 2000 can be deployed on a single computing machine; or, the parameter server 2000 can be deployed on multiple computing machines simultaneously.

[0061] As an example, parameter server 2000 can be configured to store multiple key-value pairs with the same key in the parameters of the multiple machine learning models in the form of a single key corresponding to multiple values, so as to avoid storing a large amount of duplicate information in parameter server 2000.

[0062] Furthermore, as an example, the parameter server 2000 can be configured to: merge the parameters of the multiple machine learning models with the same key in the form of a single key corresponding to multiple values, and compress the merged result according to a first compression method before saving. That is, the parameters of the multiple machine learning models after merging with the same key are compressed again; in other words, the duplicate information in the parameters of the multiple machine learning models is merged, and the non-duplicate information is compressed, so as to further reduce the storage overhead of the parameter server 2000.

[0063] Figure 4 An example is shown of a parameter server storing parameters for multiple machine learning models according to an exemplary embodiment of the present invention. Figure 4 As shown, each machine learning model corresponds to a set of key-value pairs. In this set of key-value pairs, the keys of different key-value pairs are all different, and each key corresponds to its own value. For example, the set of key-value pairs corresponding to machine learning model 1 includes at least the keys k1, k2, k3, ..., k m , respectively corresponding to the value v m1,1 v m1,2v m1,3 ..., v m1,m Machine learning model 2 corresponds to another set of key-value pairs, which includes at least the keys k1, k2, k3, ..., k m , respectively corresponding to the value v m2,1 v m2,2 v m2,3 ..., v m2,m Where m is an integer greater than 1, it can be seen that at least some of the key-value pairs in the two sets of key-value pairs have completely identical keys. According to an exemplary embodiment of the present invention, when saving parameters of multiple machine learning models, the parameter server 2000 merges and saves key-value pairs with the same key corresponding to different machine learning models into a single key corresponding to multiple values, for example, saving it as key k1 corresponding to value v. m1,1 v m2,1 v m3,1 ..., v mn,1 In addition, the parameters after merging can be further compressed, for example, by using the h compression function, thus avoiding the linear increase in storage overhead when storing parameters of multiple machine learning models at the same time.

[0064] As an example, the parameter server 2000 may be configured to provide each computing device 1000 with parameters of multiple machine learning models that each computing device 1000 needs to read in order to perform the multiple model training tasks, so that each computing device 1000 can train the multiple machine learning models using the read parameters, and the parameter server 2000 updates the parameters of the multiple machine learning models according to the results obtained by each computing device 1000 in training the multiple machine learning models (e.g., model update amount); or, the parameter server 2000 may provide each computing device 1000 with parameters of multiple machine learning models that each computing device 1000 needs to read in order to perform the multiple model prediction tasks, so that each computing device 1000 can use the read parameters to make predictions using the multiple machine learning models.

[0065] As an example, when the multiple machine learning tasks are multiple model training tasks, each computing device 1000 can be configured to provide the results obtained from training the multiple machine learning models to the parameter server 2000 in the form of a single key corresponding to multiple values, so that the parameter server 2000 updates the parameters of the multiple machine learning models. Furthermore, the results obtained from training the multiple machine learning models with the same key can be further compressed using a second compression method before being provided to the parameter server 2000; that is, the merged and compressed results obtained from training the multiple machine learning models are provided to the parameter server 2000. This effectively avoids the transmission of duplicate information, reduces the amount of data that needs to be transmitted between the computing device 1000 and the parameter server 2000, and effectively reduces the network overhead between the computing device 1000 and the parameter server 2000.

[0066] As an example, parameter server 2000 may be configured to provide each computing device 1000 with multiple key-value pairs having the same key from the parameters of the plurality of machine learning models required by each computing device 1000, in the form of a single key corresponding to multiple values. Furthermore, the parameters of the plurality of machine learning models required by each computing device, which are combined and compressed according to a third compression method, may also be provided to each computing device.

[0067] It should be understood that the first compression method, the second compression method, and the third compression method may be the same as or different from each other. For example, the first compression method and the third compression method may be the same or different. When the first compression method and the third compression method are different, when the parameter server 2000 sends the parameters of the multiple machine learning models it needs to the computing device 1000, it may first decompress the parameters stored in the parameter server 2000 that have been compressed according to the first compression method, and then recompress them according to the third compression method before sending them to the computing device 1000; or, the parameter server 2000 may recompress the parameters stored therein that have been compressed according to the first compression method according to the third compression method before sending them to the computing device 1000, thereby reducing the network overhead between the computing device 1000 and the parameter server 2000.

[0068] Figure 5 An example of parameter transfer for a multi-machine learning model according to an exemplary embodiment of the present invention is shown. Figure 5 As shown, when the parameter server 2000 provides parameters of multiple machine learning models (e.g., key-value pairs about key k1) to the computing device 1000, it can arrange the parameters of multiple machine learning models (e.g., k1: [v...]) in the form of a single key corresponding to multiple values. m1,1 v m2,1 v m3,1 ..., vmn,1 The parameters are provided to the computing device 1000 and can be further compressed after merging the same keys (e.g., f(k1, [v]). m1,1 v m2,1 v m3,1 ..., v mn,1 The parameters of the multiple machine learning models required by the computing device 1000, which are to be merged and compressed, are provided to the computing device 1000, such as... Figure 5 As shown, function f is a compression function. It should be understood that function h and function f can be the same compression function or different compression functions. According to an exemplary embodiment of the present invention, by merging duplicate information in the data to be transmitted and compressing non-duplicate information, the network transmission overhead between the computing device 1000 and the parameter server 2000 is effectively reduced, thereby lowering the transmission cost.

[0069] Figure 6 An example of parallel execution of multiple machine learning tasks according to an exemplary embodiment of the present invention is shown. In the prior art, it is necessary to use a corresponding instruction for each machine learning model to obtain the model update amount (i.e., the model training result). However, according to an exemplary embodiment of the present invention, the same vectorized instruction can be used for multiple machine learning models to obtain the model update amount of multiple machine learning models. Further, as an example, the computing device 1000 can upload the merged and compressed model training results (e.g., the variation of at least one parameter of the multiple machine learning models (e.g., the parameter corresponding to key k1)) to the parameter server 2000. When the parameter server 2000 receives the merged and compressed model training results uploaded by the computing device 1000, it can decompress the received model training results, decompress the parameter of the multiple machine learning models stored in the parameter server 2000, and based on the variation (k1: [△]) in the form of a single key corresponding to multiple values, the update amount is obtained. m1,1 , △ m2,1 , △ m3,1 、…、△ mn,1 ]), for the parameter k1 obtained after decompression: [v m1,1 v m2,1 v m3,1 ..., v mn,1 The parameters of the multiple machine learning models are then updated and compressed and saved.

[0070] As an example, when the multiple machine learning tasks are multiple model training tasks, the parameter server 2000 can be configured to: during the execution of the multiple machine learning tasks by each computing device 1000, save intermediate computation results generated when the computing device 1000 trains a machine learning model that can be used for other machine learning models, so as to use the intermediate computation results for the other machine learning models. This avoids the repeated computation of reusable information by multiple machine learning tasks and improves the efficiency of computing resource utilization.

[0071] As an example, the hyperparameters of the one machine learning model and the other machine learning models differ only in the number of training epochs, wherein the number of training epochs for the one machine learning model is greater than the number of training epochs for the other machine learning models. The parameter server 2000 can be configured to use the parameters of the one machine learning model obtained by the computing device 1000 during the training of the one machine learning model, when the number of training epochs reaches the number of training epochs corresponding to the other machine learning models, as the parameters of the other machine learning models. For example, if the hyperparameters of model training task 1, model training task 2, and model training task 3 differ only in the number of training epochs, and all other hyperparameters are the same, wherein model training task 1 has 30 training epochs, model training task 2 has 20 training epochs, and model training task 3 has 10 training epochs, then the parameters of the machine learning model obtained when the number of training epochs for model training task 1 reaches 10 epochs can be used as the parameters of the machine learning model corresponding to model training task 3; and the parameters of the machine learning model obtained when the number of training epochs for model training task 1 reaches 20 epochs can be used as the parameters of the machine learning model corresponding to model training task 2.

[0072] Considering that in the exemplary embodiment of the present invention, when the computing device 1000 and the parameter server 2000 transmit information related to multiple machine learning models simultaneously, the volume of a single transmission is significantly larger than the volume of a message when each machine learning task is executed independently. Therefore, according to the exemplary embodiment of the present invention, the network configuration can be adaptively configured according to the task type to meet the unique network transmission requirements when multiple machine learning tasks are executed in parallel. Specifically, the network transmission parameters can be automatically adjusted for the multiple machine learning tasks, so that the network can transmit large-volume messages more efficiently, thereby improving network utilization and task completion efficiency.

[0073] As an example, each computing device 1000 may be configured to set the network configuration used by the plurality of machine learning tasks.

[0074] As an example, each computing device 1000 can be configured to use zero-copy technology for network transmission for the plurality of machine learning tasks. Zero-copy technology allows network hardware (e.g., a network interface card) to directly transmit data in memory without copying the data from memory to the network hardware's cache before transmission. This transmission method offers better acceleration for multi-machine learning task scenarios where individual data packets are larger.

[0075] As an example, each computing device 1000 can automatically use zero-copy technology when receiving data from or sending data to the parameter server 2000 for the plurality of machine learning tasks.

[0076] As another example, each computing device 1000 can be configured to set the size of the Maximum Transmission Unit (MTU) in network transmission for the plurality of machine learning tasks. That is, by setting a larger MTU, large data packets can be split into fewer, larger network transmission packets at the network layer, enabling the network to transmit data at a higher transmission rate.

[0077] As an example, parameter server 2000 can be configured to set network configurations for the multiple machine learning tasks.

[0078] As an example, parameter server 2000 may be configured to: use zero-copy technology for network transmission for the plurality of machine learning tasks; and / or, set the size of the maximum transmission unit in the network transmission for the plurality of machine learning tasks.

[0079] Considering that in the exemplary embodiments of the present invention, during the execution of multiple machine learning tasks, it is necessary to store the relevant information of multiple machine learning models simultaneously. Compared to the independent execution of each machine learning task, although the overall memory required is less due to model information merging and compression, the memory volume required for a single allocation is larger. Therefore, according to the exemplary embodiments of the present invention, memory can be adaptively configured according to task type to meet the unique memory management requirements when multiple machine learning tasks are executed in parallel. Specifically, memory allocation parameters can be automatically adjusted for the multiple machine learning tasks to improve memory allocation efficiency and memory utilization efficiency.

[0080] As an example, each computing device 1000 may be configured to configure the memory used by the plurality of machine learning tasks.

[0081] In a multi-CPU architecture, the speed at which the CPU allocates and accesses memory is related to the physical location of the memory. In scenarios where multiple machine learning tasks are executed in parallel, the amount of memory accessed per process is higher, leading to greater bandwidth requirements for cross-CPU memory access.

[0082] As an example, each computing device 1000 can be configured to bind the plurality of machine learning tasks to a set of CPUs within it, so that the set of CPUs uses memory adjacent to them to execute the plurality of machine learning tasks. This achieves proximity-based memory allocation and access, improving memory allocation and access efficiency. For example, a non-uniform memory access architecture (NUMA) can be used to bind the plurality of machine learning tasks to a set of CPUs (i.e., to a specific NUMA region of the computing device 1000).

[0083] As another example, each computing device 1000 can be configured to configure a memory management unit for the plurality of machine learning tasks, so that the operating system and CPU of the computing device 1000 manage the memory used by the plurality of machine learning tasks using the configured memory management unit. That is, by setting a larger memory management unit (i.e., page) to manage the memory used by the plurality of machine learning tasks, the memory allocator can allocate a small number of large blocks of memory more efficiently, thereby reducing the number of pages that the operating system and CPU need to manage and increasing management efficiency.

[0084] As an example, parameter server 2000 can be configured to allocate memory for the plurality of machine learning tasks.

[0085] As an example, parameter server 2000 may be configured to bind tasks related to the plurality of machine learning tasks to a set of central processing unit CPUs, such that the set of CPUs uses the memory adjacent to them to execute tasks related to the plurality of machine learning tasks.

[0086] As an example, parameter server 2000 can be configured to configure memory management units for the plurality of machine learning tasks, so that the operating system and CPU of parameter server 2000 manage the memory used by the tasks related to the plurality of machine learning tasks in the configured memory management units.

[0087] Figure 7 A flowchart illustrating a method for performing multiple machine learning tasks using a distributed system according to an exemplary embodiment of the present invention is shown. The distributed system includes multiple computing devices.

[0088] like Figure 7As shown, in step S10, the plurality of computing devices respectively acquire different portions of the specified dataset.

[0089] As an example, each computing device may request data from the specified dataset from the data source, and store the requested data locally after preprocessing it.

[0090] In step S20, the plurality of computing devices jointly execute a plurality of machine learning tasks based on the acquired partial data, wherein each computing device executes the plurality of machine learning tasks in parallel based on its own acquired partial data, wherein the plurality of machine learning tasks are a plurality of model training tasks or a plurality of model prediction tasks.

[0091] As an example, each computing device can read locally stored data and perform the multiple machine learning tasks based on the read data.

[0092] As an example, step S10 can be executed first, followed by step S20. That is, multiple machine learning tasks are executed only after the computing device has saved all the data of the corresponding specified dataset locally. As another example, steps S10 and S20 can be executed simultaneously. That is, the computing device can acquire data and save it locally while executing multiple machine learning tasks based on the locally saved data.

[0093] As an example, each computing device can execute, in parallel, the machine learning tasks that require the use of each piece of data, from among the multiple machine learning tasks, based on each piece of data read.

[0094] As an example, each computing device can use vectorized instructions to execute in parallel the machine learning tasks that require the use of each piece of data, based on each piece of data read.

[0095] As an example, the distributed system may further include a parameter server, wherein the method for performing multiple machine learning tasks using a distributed system according to an exemplary embodiment of the present invention may further include: the parameter server maintaining parameters of multiple machine learning models involved in the multiple machine learning tasks, wherein the parameters of the machine learning models have key-value pairs, wherein the parameter server merges the parameters of the multiple machine learning models with the same key in the form of a single key corresponding to multiple values ​​and saves them, or the parameter server merges the parameters of the multiple machine learning models with the same key in the form of a single key corresponding to multiple values, and compresses the merged result according to a first compression method and saves it.

[0096] As an example, a method for performing multiple machine learning tasks using a distributed system according to an exemplary embodiment of the present invention may further include: when the multiple machine learning tasks are multiple model training tasks, each computing device provides the merged results of training the multiple machine learning models to a parameter server, or each computing device provides the merged results of training the multiple machine learning models, which are compressed according to a second compression method, to the parameter server so that the parameter server updates the parameters of the multiple machine learning models, wherein the results are merged in the form of a single key corresponding to multiple values.

[0097] As an example, a method for performing multiple machine learning tasks using a distributed system according to an exemplary embodiment of the present invention may further include: a parameter server providing each computing device with parameters of the plurality of machine learning models required by each computing device, which are merged; or, the parameter server providing each computing device with parameters of the plurality of machine learning models required by each computing device, which are merged and compressed according to a third compression method.

[0098] As an example, the method for performing multiple machine learning tasks using a distributed system according to an exemplary embodiment of the present invention may further include: when the multiple machine learning tasks are multiple model training tasks, the parameter server saves intermediate computation results generated by the computing device when training a machine learning model, which can be used for other machine learning models, during the execution of the multiple machine learning tasks by each computing device, so as to use the intermediate computation results for the other machine learning models.

[0099] As an example, the hyperparameters corresponding to the one machine learning model and the other machine learning models differ only in the number of training epochs. The number of training epochs corresponding to the one machine learning model is greater than the number of training epochs corresponding to the other machine learning models. The parameter server can use the parameters of the one machine learning model obtained by the computing device during the training of the one machine learning model, when the number of training epochs reaches the number of training epochs corresponding to the other machine learning models, as the parameters of the other machine learning models.

[0100] As an example, a method for performing multiple machine learning tasks using a distributed system according to an exemplary embodiment of the present invention may further include: each computing device setting the network configuration used by the plurality of machine learning tasks; and / or, a parameter server setting the network configuration for the plurality of machine learning tasks.

[0101] As an example, each computing device may use zero-copy technology for network transmission for the plurality of machine learning tasks; and / or, set the size of the maximum transmission unit in the network transmission for the plurality of machine learning tasks.

[0102] As an example, the parameter server may use zero-copy technology for network transmission for the multiple machine learning tasks; and / or, set the size of the maximum transmission unit in the network transmission for the multiple machine learning tasks.

[0103] As an example, a method for performing multiple machine learning tasks using a distributed system according to an exemplary embodiment of the present invention may further include: each computing device configuring the memory used by the plurality of machine learning tasks; and / or, a parameter server configuring the memory for the plurality of machine learning tasks.

[0104] As an example, each computing device may bind the plurality of machine learning tasks to a set of central processing units (CPUs) so that the set of CPUs uses the memory adjacent to them to execute the plurality of machine learning tasks; and / or, configure a memory management unit for the plurality of machine learning tasks so that the operating system and CPU of the computing device manage the memory used by the plurality of machine learning tasks in the configured memory management unit.

[0105] As an example, the parameter server may configure a memory management unit for the plurality of machine learning tasks so that the parameter server's operating system and CPU manage the memory used by the tasks associated with the plurality of machine learning tasks using the configured memory management unit; and / or, bind the tasks associated with the plurality of machine learning tasks to a set of central processing units (CPUs) so that the set of CPUs uses the memory adjacent to them to execute the tasks associated with the plurality of machine learning tasks.

[0106] It should be understood that the steps involved in the above method can be performed by the computing device 1000 and / or the parameter server 2000 in the previously described distributed system, and the operations involved in the above steps have been referred to Figures 1 to 6 A detailed description has been provided, and the relevant details will not be repeated here.

[0107] It should be understood that components such as parameter servers, computing devices, or devices or units constituting them in a distributed system according to exemplary embodiments of the present invention can be configured respectively as software, hardware, firmware, or any combination thereof to perform specific functions. For example, these components may correspond to dedicated integrated circuits, pure software code, or modules combining software and hardware. When they are implemented as software, firmware, middleware, or microcode, program code or code segments for performing corresponding operations can be stored in a computer-readable medium such as a storage medium, so that a processor can perform corresponding operations by reading and running the corresponding program code or code segments. Furthermore, one or more functions implemented by these components can also be uniformly performed by components in a physical entity device (e.g., a computing machine).

[0108] It should be noted that the distributed system according to the exemplary embodiments of the present invention can rely entirely on the operation of computer programs to realize the corresponding functions. That is, each component corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., a lib library) to realize the corresponding functions.

[0109] The foregoing has described various exemplary embodiments of the present invention. It should be understood that the above description is merely exemplary and not exhaustive, and the present invention is not limited to the disclosed exemplary embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A distributed system for performing multiple machine learning tasks, comprising: Multiple computing devices are configured to acquire different portions of a specified dataset and jointly execute multiple machine learning tasks. Each computing device is configured to execute multiple machine learning tasks in parallel using multiple machine learning models based on a portion of the data it acquires, wherein the multiple machine learning tasks are multiple model training tasks or multiple model prediction tasks. Each of the computing devices is configured to, based on each piece of data, execute in parallel, the machine learning tasks that require the use of that data among the multiple machine learning tasks through multiple machine learning models.

2. The distributed system as described in claim 1, wherein, Each computing device is configured to: on the one hand, request data from the specified dataset from the data source, preprocess the requested data and save it locally; on the other hand, read the locally saved data and execute the multiple machine learning tasks based on the read data.

3. The distributed system as described in claim 1, wherein, Each computing device is configured to execute, in parallel, the machine learning tasks that require the use of each piece of data, using vectorized instructions, based on each piece of data read.

4. The distributed system as described in claim 1, further comprising: A parameter server is configured to maintain parameters for multiple machine learning models involved in the multiple machine learning tasks, wherein the parameters of the machine learning models are in key-value pair form. The parameter server is configured to: merge the parameters of the multiple machine learning models with the same key in the form of a single key corresponding to multiple values ​​and then save them; or, merge the parameters of the multiple machine learning models with the same key in the form of a single key corresponding to multiple values ​​and then compress the merged result according to a first compression method and then save it.

5. The distributed system as described in claim 4, wherein, When the plurality of machine learning tasks are the plurality of model training tasks. Each computing device is configured to either provide the merged results of training the plurality of machine learning models to a parameter server, or provide the merged results of training the plurality of machine learning models, which are also compressed according to a second compression method, to the parameter server to update the parameters of the plurality of machine learning models, wherein the results are merged in the form of a single key corresponding to multiple values.

6. The distributed system as described in claim 4, wherein, The parameter server is configured to either provide each computing device with the parameters of the plurality of machine learning models required by each computing device that has been merged, or to provide each computing device with the parameters of the plurality of machine learning models required by each computing device that has been merged and compressed according to a third compression method.

7. The distributed system as described in claim 4, wherein, When the plurality of machine learning tasks are the plurality of model training tasks. The parameter server is configured to: during the execution of the plurality of machine learning tasks by each computing device, save intermediate computation results generated when the computing device trains a machine learning model that can be used for other machine learning models, so as to use the intermediate computation results for the other machine learning models.

8. The distributed system as described in claim 7, wherein, The hyperparameters of the one machine learning model and the other machine learning models differ only in the number of training epochs, wherein the number of training epochs for the one machine learning model is greater than the number of training epochs for the other machine learning models. The parameter server is configured to use the parameters of the one machine learning model obtained by the computing device during the training of the one machine learning model, when the number of training rounds reaches the number of training rounds corresponding to the other machine learning models, as the parameters of the other machine learning models.

9. The distributed system as described in claim 4, wherein, Each computing device is configured to: set the network configuration used by the plurality of machine learning tasks; And / or, the parameter server is configured to: set the network configuration for the multiple machine learning tasks.

10. The distributed system as described in claim 9, wherein, Each computing device is configured to: use zero-copy technology for network transmission for the plurality of machine learning tasks; and / or, set the size of the maximum transmission unit in the network transmission for the plurality of machine learning tasks. And / or, the parameter server is configured to: use zero-copy technology for network transmission for the plurality of machine learning tasks; and / or, set the size of the maximum transmission unit in network transmission for the plurality of machine learning tasks.

11. The distributed system as described in claim 4, wherein, Each computing device is configured to: allocate the memory used by the plurality of machine learning tasks; And / or, the parameter server is configured to: configure memory for the plurality of machine learning tasks.

12. The distributed system as described in claim 11, wherein, Each computing device is configured to: bind the plurality of machine learning tasks to a set of central processing units (CPUs) such that the set of CPUs uses adjacent memory to execute the plurality of machine learning tasks; and / or configure a memory management unit for the plurality of machine learning tasks such that the operating system and CPU of the computing device manage the memory used by the plurality of machine learning tasks using the configured memory management unit. And / or, the parameter server is configured to: configure a memory management unit for the plurality of machine learning tasks, such that the operating system and CPU of the parameter server manage the memory used by the tasks associated with the plurality of machine learning tasks using the configured memory management unit; and / or, bind the tasks associated with the plurality of machine learning tasks to a set of central processing units (CPUs), such that the set of CPUs uses the memory adjacent to them to execute the tasks associated with the plurality of machine learning tasks.

13. A method for performing multiple machine learning tasks using a distributed system, wherein, The distributed system includes multiple computing devices, wherein the method includes: The multiple computing devices respectively acquire different portions of the specified dataset; The multiple computing devices jointly execute multiple machine learning tasks based on the acquired partial data. Each computing device executes these multiple machine learning tasks in parallel using multiple machine learning models, based on its own acquired partial data. The multiple machine learning tasks are either multiple model training tasks or multiple model prediction tasks. Each computing device, based on each piece of data, executes in parallel the machine learning tasks that require the use of that data among the multiple machine learning tasks through multiple machine learning models.

14. The method of claim 13, wherein, The steps of having the multiple computing devices acquire different portions of the specified dataset include: Each computing device requests data from the specified dataset from the data source; Each computing device preprocesses the requested data and stores it locally. The step of each computing device performing the multiple machine learning tasks in parallel based on a portion of the data it acquires includes: Each computing device reads locally stored data and performs the multiple machine learning tasks based on the read data.

15. The method of claim 13, wherein, Each computing device performs the multiple machine learning tasks based on the read data, including the following steps: Each computing device, based on each piece of data read, uses vectorized instructions to execute in parallel the machine learning tasks that require that piece of data among the multiple machine learning tasks.

16. The method of claim 13, wherein, The distributed system further includes a parameter server, and the method further includes: The parameter server maintains the parameters of multiple machine learning models involved in the multiple machine learning tasks. The parameters of the machine learning models are in the form of key-value pairs. The parameter server merges the parameters of the multiple machine learning models with the same key in the form of a single key corresponding to multiple values ​​and saves them. Alternatively, the parameter server merges the parameters of the multiple machine learning models with the same key in the form of a single key corresponding to multiple values ​​and compresses the merged result according to a first compression method before saving it.

17. The method of claim 16, further comprising: When the multiple machine learning tasks are used to train the multiple models, each computing device provides the parameter server with the combined results of training the multiple machine learning models, or each computing device provides the parameter server with the combined results of training the multiple machine learning models, compressed according to a second compression method, so that the parameter server updates the parameters of the multiple machine learning models. The results are merged in such a way that a single key corresponds to multiple values.

18. The method of claim 16, further comprising: The parameter server provides each computing device with the parameters of the multiple machine learning models required by each computing device, which are merged and compressed according to a third compression method.

19. The method of claim 16, further comprising: When the plurality of machine learning tasks are the plurality of model training tasks, the parameter server saves the intermediate computation results generated by the computing device when training a machine learning model, which can be used for other machine learning models, during the execution of the plurality of machine learning tasks by each computing device, so as to use the intermediate computation results for the other machine learning models.

20. The method of claim 19, wherein, The hyperparameters of the one machine learning model and the other machine learning models differ only in the number of training epochs, wherein the number of training epochs for the one machine learning model is greater than the number of training epochs for the other machine learning models. Specifically, the parameter server uses the parameters of the machine learning model obtained by the computing device during the training of the machine learning model, when the number of training rounds reaches the number of training rounds corresponding to the other machine learning models, as the parameters of the other machine learning models.

21. The method of claim 16, further comprising: Each computing device is configured with the network settings used by the plurality of machine learning tasks; And / or, the parameter server sets the network configuration for the multiple machine learning tasks.

22. The method of claim 21, wherein, The step of setting the network configuration used by each computing device for the plurality of machine learning tasks includes: each computing device using zero-copy technology for network transmission for the plurality of machine learning tasks; and / or, setting the size of the maximum transmission unit in the network transmission for the plurality of machine learning tasks; The step of setting the network configuration for the plurality of machine learning tasks by the parameter server includes: the parameter server using zero-copy technology for network transmission for the plurality of machine learning tasks; and / or setting the size of the maximum transmission unit in network transmission for the plurality of machine learning tasks.

23. The method of claim 16, further comprising: Each computing device is configured with the memory used by the plurality of machine learning tasks; And / or, the parameter server configures the memory for the multiple machine learning tasks.

24. The method of claim 23, wherein, The step of configuring the memory used by the plurality of machine learning tasks in each computing device includes: each computing device binding the plurality of machine learning tasks to a set of central processing units (CPUs) such that the set of CPUs uses memory adjacent to them to execute the plurality of machine learning tasks; and / or configuring a memory management unit for the plurality of machine learning tasks such that the operating system and CPU of the computing device manage the memory used by the plurality of machine learning tasks in the configured memory management unit. The step of configuring memory for the plurality of machine learning tasks in the parameter server includes: configuring a memory management unit for the plurality of machine learning tasks in the parameter server so that the operating system and CPU of the parameter server manage the memory used by the tasks related to the plurality of machine learning tasks using the configured memory management unit; and / or binding the tasks related to the plurality of machine learning tasks to a group of central processing units (CPUs) so that the group of CPUs uses the memory adjacent to them to execute the tasks related to the plurality of machine learning tasks.

Images