Performance collaborative transfer learning between cloud storage and cloud computing

Abstract

A computing device (200) comprising a client (201) and a server (202) is described. The computing device (200) is configured to: obtain a machine learning code (204); segmenting the machine learning code (204) into a first portion (204a) and a second portion (204b); executing the first portion (204a) of the machine learning code (204) on the server (202); executing the second portion (204b) of the machine learning code (204) on the client (201); a result (306) of the machine learning code (204) is output. In this manner, the machine learning code (204) can be segmented and executed on the client (201) and the server (202) in an efficient manner.

Description

Technical Field

[0001] The present invention relates to a machine learning apparatus and method for executing machine learning codes on a client and a server. Background Art

[0002] The computing power of classified cloud object stores (COS) is increasing. Users may no longer be restricted to running a limited subset of structured query language (SQL) alongside storage. Users may perform complex tasks such as image processing inside COS, and some providers have even begun to offer their COS with graphical processing units (GPUs). However, some fundamental limitations remain, such as: (i) the connection between the cloud network and the computing layer may cause bottlenecks, and (ii) despite upgrades, COS computing resources may be expensive because they are only intended to supplement the computing layer, not replace it.

[0003] US10,649,988B1, US10,904,298B2, US2020 / 0401886 A1, and US2019 / 0250998 A1 describe building a machine learning (ML) system when a storage layer is involved.

[0004] US10,649,988B1 provides very high-level guidance on optimization modules (e.g., resource-based admission control) and on monitoring modules (e.g., identifying bottlenecks and reconfiguring resource allocation accordingly). US10,904,298B2 and US2019 / 0250998 A1 focus on different contexts and do not focus on improving transfer learning TL processing. Meanwhile, US2020 / 0401886 A1 may not provide a solution to reduce network traffic between the storage layer and the computing layer in the cloud.

[0005] In addition, US11,003,988B2, WO 2016 / 118257 A1 and US2020 / 0210834 A1 describe distributed ML systems and opportunities for partitioning ML computations. US11,003,988B2 can improve deep learning medical systems by using TL and other deep learning techniques. WO 2016 / 118257A1 describes a method for compressing ML networks, such as neural networks. US2020 / 0210834 A1 describes a method for partitioning DNNs on heterogeneous LAN devices based on DNN representation to alleviate LAN network bottlenecks.

[0006] Figure 1An exemplary computing device of the prior art is schematically shown. The computing device 100 may include a computing layer 101 and an object storage 102. The computing layer 101 may include a processor 103, which may execute a machine learning code 104. The object storage 102 may include a memory 105. The object storage 102 may transfer a large amount of data 106 to the computing layer 106.

[0007] Existing techniques may not be able to reduce the bottleneck between the client and the server. Existing techniques may also not be able to cope with concurrent inputs to machine learning code.

[0008] It is desirable to develop an apparatus and method that overcomes the above-mentioned problems. Summary of the invention

[0009] According to a first aspect, a computing device is provided, the computing device comprising a client and a server; the client and / or the server comprising one or more processors and a memory, the memory storing data in a non-transient form, the data defining a program code executable by the one or more processors, wherein the program code, when executed by the one or more processors, enables the computing device to: obtain a machine learning code; split the machine learning code into a first part and a second part; execute the first part of the machine learning code on the server; execute the second part of the machine learning code on the client; and output a result of the machine learning code. By splitting the machine learning code into the first part and the second part, the machine learning code can be executed on the client and the server in an efficient manner.

[0010] In some implementations, the first portion of the machine learning code includes at least a portion of the inference portion of the machine learning code. In this way, more required portions of the machine learning code can be executed on the server, which may be more efficient.

[0011] In some implementations, the first portion of the machine learning code includes all of the reasoning portions of the machine learning code. In this way, all of the required portions of the machine learning code can be executed on the server, which may be more efficient.

[0012] In some implementations, the second portion of the machine learning code includes all of the training portions of the machine learning code. In this way, data transfer between the server and the client can be reduced, which can reduce the load on the network bandwidth.

[0013] In some implementations, the machine learning code is a transfer learning code. In this way, the device can use the knowledge learned from one training context and reuse it in a related training context. In the related context, there may be reasoning and retraining.

[0014] In some implementations, the apparatus is configured to segment the machine learning code according to one or more features of the machine learning code. In this way, the machine learning code, for example, can effectively segment the memory requirements between the client and the server.

[0015] In some implementations, the apparatus is configured to execute a first portion of the machine learning code to generate a sample output for a synthetic data sample, and to split the machine learning code according to the sample output. In this way, the apparatus can test the output requirements of the machine learning code so that the machine learning code can be effectively split between the client and the server.

[0016] In some implementations, the device is configured to partition the machine learning code based on one or more characteristics of the computing device. In this way, the capacity of the computing device, such as memory, can be taken into account to effectively partition the machine learning code between the client and the server.

[0017] In some implementations, the apparatus is configured to partition the machine learning code based on an evaluation of the sample output and the bandwidth of a network connecting the client and the server. In this way, the capacity of the bandwidth and the output requirements of the machine learning code can be considered to effectively partition the machine learning code between the client and the server.

[0018] In some implementations, the apparatus is used to control a batch size of a first portion of the machine learning code.

[0019] In some implementations, the apparatus is configured to control a batch size of the first portion of the machine learning code based on one or more of: memory of the server to be occupied by inputs and outputs of the first portion of the machine learning code; and memory of the server to be occupied by weights of the machine learning code.

[0020] In some implementations, the device is used to obtain one or more subsequent machine learning codes. In this way, multiple users can submit machine learning codes sequentially and / or simultaneously for execution by the computing device.

[0021] In some implementations, the device is used to individually control the batch size of the first portion of each machine learning code. In this way, a computing device, particularly a server, may be able to prioritize and / or optimize the execution of multiple machine learning codes sequentially and / or concurrently.

[0022] In some implementations, the apparatus is configured to individually control the batch size of the first portion of each machine learning code based on one or more of: a memory of the server to be occupied by inputs and outputs of the first portion of each machine learning code; and a memory of the server to be occupied by weights of each machine learning code.

[0023] In some implementations, the machine learning code is obtained from the user, and / or the results of the machine learning code are output to the user. In this way, the user can submit the machine learning code to be executed by the computing device and receive the output of the machine learning code from the computing device.

[0024] According to a second aspect, a method for executing a machine learning code is provided, the method comprising the steps of: obtaining a machine learning code; dividing the machine learning code into a first part and a second part; executing the first part of the machine learning code on a server; executing the second part of the machine learning code on a client; and outputting a result of the machine learning code. By dividing the machine learning code into the first part and the second part, the machine learning code can be executed on the client and the server in an efficient manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0026] Figure 1 An exemplary computing device of the prior art is schematically shown.

[0027] Figure 2 An exemplary computing device of the first embodiment is schematically shown.

[0028] Figure 3A An exemplary computing device of the second embodiment is schematically shown.

[0029] Figure 3B An exemplary computing device of the third embodiment is schematically shown.

[0030] Figure 3C An exemplary computing device of the fourth embodiment is schematically shown.

[0031] Figure 4 An exemplary computing device of the fourth embodiment is schematically shown.

[0032] Figure 5 An example of a computer-implemented method for executing machine learning code is shown.

[0033] Figure 6 An example of an apparatus for performing the methods described herein is shown. DETAILED DESCRIPTION

[0034] The apparatus and methods described herein relate to executing machine learning code.

[0035] Embodiments of the present invention can solve one or more of the aforementioned problems by splitting a machine learning code into a first part and a second part, executing the first part of the machine learning code on a server, and executing the second part of the machine learning code on a client. This can enable the machine learning code to be split and executed on the client and the server in an efficient manner.

[0036] The system can leverage the unique structure of transfer learning (TL), a combination of feature extraction (also known as inference) and training, to flexibly bypass the above issues and improve client- and operator-centric metrics. The system can provide methods and techniques for TL across the computing layer and the COS layer, and can achieve significant improvements while remaining completely transparent to users.

[0037] The system can provide a mechanism to process TL computations faster in a cloud or data center environment. Specifically, ML training can be split between computation and storage. This may reduce the amount of data moving between computation and storage.

[0038] Figure 2 An exemplary computing device of the first embodiment is schematically shown. Figure 3A An exemplary computing device of the second embodiment is schematically shown. Figure 3B An exemplary computing device of the third embodiment is schematically shown. Figure 3C An exemplary computing device of the third embodiment is schematically shown. Figure 4 An exemplary computing device of a fourth embodiment is schematically shown.Each embodiment shares common features as well as different features, details of which are described herein.

[0039] like Figures 3A to 3C and Figure 4 As shown in , the computing device 200 can be used to obtain the machine learning code 204. In other words, the computing device 200 can be used to request and / or receive the machine learning code 204. The computing device 200 can be used to obtain the machine learning 204 from the user 301. The user 301 can be a user of the computing device 200. The user 301 can ask the computing device 200 to execute the machine learning code 204. The user 301 can submit the machine learning code 301. The user 301 can be away from the computing device 200. The user 301 can communicate with the computing device 200 through a network such as the Internet. The computing device 200 can also obtain information about the user 301 and the machine learning code 204. In this way, the computing device 200 can know who or what the user 301 is.

[0040] The computing device 200 may be used to obtain one or more sets of subsequent machine learning codes 204. The computing device 200 may be used to sequentially split and / or execute multiple sets of machine learning codes 204. The computing device 200 may be used to simultaneously obtain one or more sets of machine learning codes 204 and the first set of machine learning codes 204. The computing device 200 may be used to simultaneously split and / or execute multiple sets of machine learning codes 204. The multiple machine learning codes 204 may be provided by the same user 301. Alternatively, the multiple machine learning codes 204 may be provided by different users 301. In this way, the computing device 200 may be able to handle multiple requests for executing machine learning codes 204.

[0041] The machine learning code 204 may include a neural network (NN). Specifically, the machine learning code 204 may include a deep neural network (DNN). The machine learning code 204 may include multiple NN layers, so that the machine learning code 204 is considered to be a DNN. The machine learning code 204 may include Python code. The machine learning code 204 may include transfer learning code. The machine learning code 204 may include an inference portion of the machine learning code 204. The machine learning code 204 may include a training portion of the machine learning 204. The inference portion of the machine learning code 204 may be in an earlier layer of the NN than the training portion of the machine learning code 204. Specifically, the inference portion of the machine learning code 204 may be a NN layer before the layer of the training portion of the machine learning 204. The inference portion of the machine learning code 204 may include weights that are fixed or frozen during execution. The training portion of the machine learning code 204b may include weights that are variable during execution.

[0042] The computing device 200 may include a client 201. Figure 2 , Figures 3A to 3C as well as Figure 4 As shown in , client 201 can be a computing layer. Client 201 can be used to obtain machine learning code 204. User 301 can be far away from client 201. User 301 can communicate with client 201 through a network such as the Internet. Alternatively, user 301 can be located at client 201.

[0043] The computing device 200 may include a server 202. Figure 2 , Figures 3A to 3C as well as Figure 4 As shown in , the server 202 may be a COS layer. The server 202 may be remote from the client 201. Figure 4As shown in , server 202 can communicate with client 201 via network 401. Network 401 can be a data center network. In this case, client 201 and server 202 can be part of the same data center. Alternatively, network 401 can be the Internet. In this case, client 201 and server 202 can be located in different locations. Network 401 can have limited bandwidth.

[0044] like Figure 2 As shown in , client 201 may include one or more GPUs 203b. Figure 2 Two GPUs 203b are illustratively shown. However, the client 201 may also include a different number of GPUs 203b. The GPUs 203b may be used to execute machine learning code.

[0045] like Figures 3A to 3C and Figure 4 As shown in FIG. 3 , the client 201 may include a computing analysis module 303 . One or more of the GPUs 203 b may be used to execute the process of the computing analysis module 303 .

[0046] The computation analysis module 303 may be used to obtain the machine learning code 204. The computation analysis module 303 may be used to analyze the machine learning code 204. The computation analysis module 303 may be used to analyze the machine learning code 204 to determine features about the machine learning code 204. For example, the computation analysis module 303 may be used to analyze the machine learning code 204 to understand the workload of the machine learning code 204. The computation analysis module 303 may analyze the number of layers in the NN. The computation analysis module 303 may analyze the amount of memory to be used by the machine learning code 204. The computation analysis module 303 may analyze the amount of memory to be used by the weights of the machine learning code 204. The computation analysis module 303 may analyze the memory to be used by the input and output of the machine learning code 204. The computation analysis module 303 may also obtain a configuration from the user 301 without analysis. In this case, the computation analysis module 303 may obtain a frozen index, such as the last layer of the inference part. The computation analysis module 303 may also obtain a training batch size from the user 301.

[0047] like Figures 3A to 3C and Figure 4 As shown in , the client 201 may include a model segmentation module 304. The model segmentation module 304 may be executed on a central processing unit (CPU) of the client 201. Alternatively, one or more of the GPUs 203b may be used to execute the process of the model segmentation module 304.

[0048] The computing device 200 may be used to split the machine learning code 204 into a first portion 204a and a second portion 204b. Specifically, the model splitting module 304 may be used to split the machine learning code 204 into a first portion 204a and a second portion 204b. Once the machine learning code 204 is split, the machine learning code may be saved to a corresponding portion of the computing device 200, i.e., the client 201 or the server 202, according to the splitting.

[0049] The model segmentation module 304 can be used to segment the machine learning code 204 according to whether the first part 204a and the second part 204b include the inference part of the machine learning 204 and / or the training part of the machine learning code 204.

[0050] The first portion 204a of the machine learning code may include at least a portion of the inference portion of the machine learning code 204. The first portion 204a of the machine learning code may include all of the inference portion of the machine learning code 204. For example, part or all of the inference layer of the NN may be in the first portion 204a of the machine learning code. Executing more inference portions of the machine learning code on the server 201 may optimize the computing device 200.

[0051] Preferably, the second portion 204b of the machine learning code includes all training portions of the machine learning code 204. For example, all training layers of the NN may be in the second portion 204b of the machine learning code. In some implementations, it may be required that all training layers of the NN are in the second portion 204b of the machine learning code. Executing all training portions of the machine learning code on the client 201 may optimize the computing device 200. Specifically, executing all training portions of the machine learning code 204 on the client 201 may transfer more memory-intensive outputs to the client 201, which may reduce data transfer between the server 202 and the client 201. This is because the training portion of the machine learning code 204 is typically a cheaper portion. In addition, even if this is not the case, transferring more memory-intensive outputs to the client 202 may be associated with greater data transfer. The reason why the entire training may be on the client may be related to the runtime of the application. Split training may require back-transfer splitting between the server and the client, which may be inefficient.

[0052] Typically, the inference portion of the machine learning code 204, also referred to as the feature extraction phase (i.e., the first few layers of the DNN), is more demanding than the training phase (the latter few layers of the DNN) in terms of (i) execution time, (ii) GPU memory, and (iii) output size. Therefore, while running training on the computational layers, pushing feature extraction partially or fully down to the server 202 (also referred to as the COS) reduces the amount of data passed over the network. In addition, splitting the TL computation can decouple the batch size of feature extraction from the batch size of training. This decoupling can reduce the memory requirements of the TL computation and help more efficiently manage the scarce and expensive GPU memory of the COS, thereby allowing concurrent users to better share the GPU of the COS.

[0053] In theory, it is possible to push both stages of TL, feature extraction and training, next to storage. However, this may have limitations. Doing so can reduce network traffic to the minimum possible, i.e., no data needs to be passed during training; only at the end, the user may want to download the trained model from COS. However, this solution may not be able to decouple the batch size for feature extraction from the batch size for training, leading to a choice between unsatisfactory options: limiting concurrency on COS, resulting in poor throughput, or risking out-of-memory (OOM) errors.

[0054] The model segmentation module 304 can calculate the optimal segmentation of the machine learning code 204, or a segmentation that is as close to the optimal segmentation as possible. For example, the model segmentation module 304 can segment the layers of the NN so that all training layers will be executed on the client 201. The model segmentation module 304 can also segment the layers of the NN so that as many inference layers as possible will be executed on the server 202. In some cases, the segmentation in the machine learning code 204 may preferably be between the last inference layer and the first training layer to reduce the traffic between the client 201 and the server 202. Although in other cases, this may cause the memory requirements on the server 202 to be too high, or exceed the limit. In these other cases, it may be preferable to segment the machine learning code 304 between the inference layers of the NN. In any case, the model segmentation module 304 will be used to segment the machine learning code 204 in an efficient manner so that the requirements for traffic between the client 201 and the server 202 and the memory usage of the server 202 are balanced.

[0055] The model segmentation module 304 may include a module segmentation algorithm. The model segmentation algorithm may include two stages: (i) candidate selection, which may be guided only by the machine learning code 204 or model, properties; and (ii) winner selection, which selects one of the candidate layers and is guided by the environment, i.e., the properties of the computing device 200, i.e., the network bandwidth of the server 202.

[0056] Candidate selection may be based on intermediate output size estimation by computation analysis module 303. Client 201 may select a layer whose output size is smaller than the input size (scaled by the batch size). The main goal may be to reduce network traffic compared to sending the entire application input to client 201.

[0057] The model segmentation module 304 may be used to segment the machine learning code 204 according to one or more features of the machine learning code 204. Specifically, the model segmentation module 304 may obtain features about the machine learning code 204 from the computing and analysis module 303. Alternatively, the model segmentation module 304 may obtain features from other sources other than the computing and analysis module 303, such as the user 301.

[0058] The model segmentation module 304 can use the features of the machine learning code 204 to calculate the optimal segmentation of the machine learning code 204, or a segmentation as close to the optimal segmentation as possible, as described herein. For example, the model segmentation module 304 can segment the machine learning code 204 using the workload of the machine learning code 204. The model segmentation module 304 can segment the machine learning code 204 using the number of layers in the NN. The model segmentation module 304 can segment the machine learning code 204 using the amount of memory to be used by the machine learning code 204. The model segmentation module 304 can segment the machine learning code 204 using the amount of memory to be used by the weights of the machine learning code 204. The model segmentation module 304 can segment the machine learning code 204 using the memory to be used by the inputs and outputs of the machine learning code 204.

[0059] Winner selection may be a dynamic approach to balancing two potentially opposing demands: (i) pushing down as few layers as possible to save storage resources while (ii) reducing the time spent on network communication to improve user-perceived latency. The key to the algorithm's success is based on the observation that layer output size generally decreases through the machine learning code 204, but not monotonically. Therefore, layers with relatively small output sizes can be found early in the DNN. To best navigate this tradeoff, the algorithm can select the earliest candidate layer with an output size below C, where C is a function of the network bandwidth, essentially weighing the optimal split point relative to network transmission to reduce pushdown on COS.

[0060] The model segmentation module 304 can be used to segment the machine learning code 204 according to one or more features about the computing device 200. Specifically, the model segmentation module 304 can obtain features about the computing device 200 from the computing device 200. The model segmentation module 304 can use the features of the computing device 200 to calculate the best segmentation of the machine learning code 204, or a segmentation as close to the best as possible, as described herein. For example, the model segmentation module 304 can use features about the bandwidth in the network 401 between the client 201 and the server 202 to segment the machine learning code 204. In this way, the bandwidth of the server can limit the amount of traffic between the client 201 and the server 202, which in turn may require more inference layers of the NN on the server 202.

[0061] The model segmentation module 304 can be used to inform the server 202 how to segment the machine learning code 204. The server 301 can be used to extract the data set from the storage device 205.

[0062] like Figures 3A to 3C and Figure 4 As shown in FIG. 1 , the client 201 may include a training module 302. One or more of the GPUs 203b may be used to execute the process of the training module 302.

[0063] The computing device 200 may be used to execute the second portion 204b of the machine learning code 204. Specifically, the training module 302 may be used to execute the second portion 204b of the machine learning code 204. In other words, the training module 302 may be used to run the second portion 204b of the machine learning code 204.

[0064] As described herein, the second portion 204b of the machine learning code 204 may include a training portion of the machine learning code 204. Thus, the training module 204 may be used to execute a training layer of the NN of the machine learning code 204. Additionally, as described herein, the second portion 204b of the machine learning code 204 may include portions of an inference portion of the machine learning code 204. Thus, in these cases, the training module 204 may also be used to execute portions of an inference layer of the NN of the machine learning code 204.

[0065] The computing device 200 may be used to execute the first portion 204a of the machine learning code 204 on the synthetic data sample. Specifically, the reasoning module 305 may be used to execute the first portion 204a of the machine learning code on the synthetic data sample. Figure 4As shown in , the reasoning module 305 can also be referred to as the feature extraction module 305. Alternatively or in addition, the training module 302 can be used to execute the first portion 204a of the machine learning code against the synthetic data sample. The reasoning module 305 and / or the training module 302 can be used to generate sample outputs based on the synthetic data sample. For example, when compared to the batch size of the real data of the machine learning code 204, the synthetic data sample can have a very small batch size. In this way, the reasoning module 305 and / or the training module 302 can run the first portion 204a of the machine learning code 204 against a significantly smaller amount of data to sample the output data that may come from the first portion 204a of the machine learning code 204.

[0066] The model segmentation module 304 can be used to segment the machine learning code 204 according to the sample output. Figure 4 As shown in , the synthetic data samples can be provided from the model segmentation module 304 to the training module 302 to generate sample outputs. The model segmentation module 304 can then use the sample outputs to provide guidance on the segmentation of the machine learning code 204. The sample outputs can provide prediction outputs of the training module 302, which can be used in evaluating the segmentation of the machine learning code 204.

[0067] Specifically, the computation analysis module 303 may perform an analysis process to estimate the output size of each deep learning layer. In order to obtain the analysis data, the computation analysis module 303 may use the inference model and track the memory consumption and intermediate output size of each layer, and run the forward method using a synthetic data sample (i.e., with the same dimensions as the input data). One data sample (i.e., a batch size of 1) may usually be sufficient, so this process may be lightweight in terms of latency (lower microseconds compared to a full sample) and memory consumption (lower MB compared to a full sample). Based on this metadata, the model segmentation module 304 may split the ML calculation into two parts: a part 204b to be executed on the computation layer 201 (training calculation), and another part 204a to be executed on the COS layer 202 (feature extraction calculation).

[0068] Additionally, the model segmentation module 304 can be used to segment the machine learning code 204 based on an evaluation between the sample output and the bandwidth of the network 401 connecting the client 201 and the server 202. In this way, the model segmentation module 304 can be used to segment the machine learning code 204 in an efficient manner so that the requirements for the traffic between the client 201 and the server 202 and the memory usage of the client 201 are balanced.

[0069] like Figure 2 As shown in FIG. 2 , server 202 may include one or more GPUs 203 a. Figure 2One GPU 203a is schematically shown. However, the server 202 may also include a different number of GPUs 203a. The GPU 203a can be used to execute machine learning code.

[0070] like Figures 3A to 3C and Figure 4 As shown in FIG. 3 , server 202 may include inference module 305 . One or more GPUs 203 a may be used to execute the process of inference module 305 . Figure 4 The reasoning module 305 is referred to as the feature extraction module 305, and these two terms can be used interchangeably.

[0071] The computing device 200 may be used to execute the first portion 204a of the machine learning code 204. Specifically, the reasoning module 305 may be used to execute the first portion 204a of the machine learning code 204. In other words, the reasoning module 305 may be used to run the first portion 204a of the machine learning code 204.

[0072] As described herein, the first portion 204a of the machine learning code 204 may include an inference portion of the machine learning code 204. Thus, the inference module 305 may be used to execute an inference layer of the NN of the machine learning code 204. Alternatively, as described herein, the second portion 204b of the machine learning code 204 may include portions of the inference portion of the machine learning code 204. Thus, in these cases, the training module 302 may also be used to execute portions of the inference layer of the NN of the machine learning code 204.

[0073] The computing device 200 may be used to execute the first portion 204a of the machine learning code before the second portion 204b of the machine learning code. Specifically, the reasoning module 305 may execute the first portion 204a of the machine learning code 204 before the training module 302 executes the second portion 204b of the machine learning code 204. The computing device 200 may be used to send the intermediate output 206 from the server 202 to the client 201. The computing device 200 may be used to send the intermediate output 206 from the server 202 to the client 201 via the network 401.

[0074] like Figure 3C and Figure 4 As shown in FIG. 3 , the server 202 may include a batch adaptation module 307 . One or more of the GPUs 203 a may be used to perform the process of the batch adaptation module 307 .

[0075] A possible problem with executing code on server 202 is that the amount of memory 205 on server 202 may be limited, but there are many requests that can benefit from server-side pushdown. Computing device 200 can be used to control the batch size (ML computation granularity) in order to control the memory consumed by each request. The goal of batch adaptation may be to fit multiple client requests simultaneously in server memory.

[0076] The computing device 200 may be used to control the batch size of the first portion 204a of the machine learning code 204. Specifically, the batch adaptation module 307 may be used to control the batch size of the first portion 204a of the machine learning code 204. Controlling the batch size of the first portion 204a of the machine learning code 204 may enable the number of samples to be processed to be changed. This in turn may allow the amount of memory used by the first portion 204a of the machine learning code 204 to be changed.

[0077] The batch adaptation module 307 can be used to control the batch size of the first portion 204a of the machine learning code 204 based on the memory 205 of the server 202 that will be occupied by the input and output 206 of the first portion 204a of the machine learning code 204. The batch adaptation module 307 can estimate the memory requirements of the input and output 206 of the first portion 204a of the machine learning code 204 and use this information to control the batch size. For example, if the input / output memory requirements are low, then the batch size can be increased. Alternatively, if the input / output memory requirements are high, such as above the memory 205 limit, then the batch size can be reduced.

[0078] The batch adaptation module 307 can be used to control the batch size of the first portion 204a of the machine learning code 204 based on the memory 205 of the server 202 that will be occupied by the weights of the machine learning code 204. The batch adaptation module 307 can estimate the memory requirements of the weights of the machine learning code 204 and use this information to control the batch size. For example, if the weight memory requirement is low, then the batch size can be increased. Alternatively, if the weight memory requirement is high, such as above the memory 205 limit, then the batch size can be reduced.

[0079] The batch adaptation module 307 may also be used to control the batch size of the first portion 204a of the machine learning code 204 based on the memory 205 of the server 202 that will be occupied by the inputs and outputs 206 of the first portion 204a of the machine learning code 204 and the weights of the machine learning code 204. In this way, the batch adaptation module 307 may change the batch size based on all memory requirements.

[0080] As described herein, the computing device 200 can be used to obtain one or more subsequent and / or concurrent machine learning codes 204. The computing device 200 can be used to control the batch size of the first portion 204a of each machine learning code 204. Specifically, the batch adaptation module 307 can be used to control the batch size of the first portion 204a of each machine learning code 204. Controlling the batch size of the first portion 204a of each machine learning code 204 can enable the number of samples to be processed to be changed for each machine learning code 204. This in turn can allow the amount of memory used by each first portion 204a of the machine learning code 204 to be changed. This can enable the batch adaptation module 307 to prioritize certain machine learning codes 204 over other machine learning codes, and / or allow more machine learning codes 204 to be executed simultaneously.

[0081] The batch adaptation module 307 can be used to control the batch size of the first portion 204a of each machine learning code 204 according to the memory 205 of the server 202 that will be occupied by the input and output 206 of the first portion 204a of each machine learning code 204. The batch adaptation module 307 can estimate the memory requirements of the input and output 206 of the first portion 204a of each machine learning code 204 and use this information to control the batch size of each machine learning code 204. For example, if the input / output memory requirements of a particular machine learning code 204 are low, then the batch size can be increased and / or another machine learning code 204 can be executed simultaneously. Alternatively, if the input / output memory requirements of a particular machine learning code 204 are high, such as above the memory 205 limit, then the batch size can be reduced, which can enable another machine learning code 204 to be executed simultaneously.

[0082] The batch adaptation module 307 can be used to control the batch size of the first portion 204a of each machine learning code 204 according to the memory 205 of the server 202 that will be occupied by the weights of each machine learning code 204. The batch adaptation module 307 can estimate the memory requirements of the weights of each machine learning code 204 and use this information to control the batch size. For example, if the memory requirements of the weights of a particular machine learning code 204 are low, then the batch size can be increased and / or another machine learning code 204 can be executed simultaneously. Alternatively, if the memory requirements of the weights of a particular machine learning code 204 are high, such as above the memory 205 limit, then the batch size can be reduced, which can enable another machine learning code 204 to be executed simultaneously.

[0083] The batch adaptation module 307 may also be used to control the batch size of the first portion 204a of each machine learning code 204 according to the memory 205 of the server 202 to be occupied by the input and output 206 of the first portion 204a of each machine learning code 204 and the weights of each machine learning code 204. In this way, the batch adaptation module 307 may change the batch size of each machine learning code 204 according to all memory requirements.

[0084] The batch adaptation module 307 may include a batch adaptation algorithm. The batch adaptation algorithm may be run repeatedly on the server. A new run of the algorithm may be triggered when two conditions are met: (i) there is available GPU memory 205 for new requests; and (ii) there is at least one queued request that has not been considered in the previous run of the algorithm.

[0085] Server 202 may receive multiple requests in quick succession. Therefore, server 202 may wait for new requests for a short time, which is a fraction of the time required to service one request. This approach can navigate the following tradeoffs. If the server delays the start of the algorithm for too long, this may delay requests unnecessarily. However, if the server does not wait enough time, then arriving requests may have to wait for the current batch to finish processing (when memory is insufficient).

[0086] The batch adaptation algorithm can take into account already running requests (i.e., when applying the algorithm), but not future requests. The goal of the algorithm is to maximize GPU memory utilization for existing requests while fitting as many requests as possible in GPU memory. The output of the algorithm is the batch size to use for each request, i.e., the storage batch size.

[0087] To select the batch size for each request, the server 201, and in particular the batch adaptation module 307, may solve the optimization problem in Equation 1:

[0088]

[0089] is the set of requests in the queue, b r is the batch size used to request r (i.e. the decision variable of the optimization problem), is the amount of memory occupied by the inputs and intermediate outputs of the DNN model for request r, is the amount of memory occupied by the DNN model weights for request r, is the total amount of GPU memory, is the amount of memory occupied by other already running requests in addition to the estimated reserved memory for CUDA and ML frameworks, and and are the minimum and maximum bounds allowed on the batch size. is set by the client when sending a request (usually the same as the training batch size), while Set by the operator.

[0090] The computing device 200 may be used to output the result 306 of the machine learning code 204. Specifically, the client 201 may be used to output the result 306 of the machine learning code 204. The training module 302 of the client 201 may be used to output the result 306 of the machine learning code 204. The result may come from the end of the second part 204b of the machine learning code 204. The training module 302 may be used to output the result 306 of the machine learning code 204 to the user 301. The training module 302 outputs the result 306 of the machine learning code 204 to the same user 301 who provided the machine learning code 204. In this way, the training module 302 may return the result 306 to the user 301.

[0091] As described herein, the computing device 200 can be used to obtain one or more subsequent and / or concurrent machine learning codes 204, which can be obtained from the same or different users 301. The training module 302 can be used to output the results 306 of each machine learning code 204 to the corresponding user 301 that provided the machine learning code 204.

[0092] The computing device 200 may provide a segmentation algorithm to push part of the computation (the reasoning part of TL, such as feature extraction) down to the storage layer 202 to achieve low latency and alleviate network bottlenecks. The segmentation point in the ML model may depend on the model 204 and the environment. The proposed algorithm may automatically detect the segmentation point. The segmentation algorithm includes two stages: (i) candidate selection: the layer for which segmentation will be beneficial may be selected based entirely on the training model; and (ii) winner selection: one of the candidate layers to be segmented may be selected based on the environment properties.

[0093] The computing device 200 may provide a batch size adaptation mechanism that may efficiently use a limited amount of storage side memory 205 while also improving storage side request concurrency. The proposed mechanism may allow the computing device 200 to serve multiple client requests with limited storage side memory (GPU memory 205). The mechanism may select a batch size to be used in each request.

[0094] Figure 5Summarizing an example of a method 500 for executing machine learning code. At step 501, the method 500 includes obtaining machine learning code. At step 502, the method 500 includes splitting the machine learning code into a first portion and a second portion. At step 503, the method 500 includes executing the first portion of the machine learning code on a server. At step 504, the method 500 includes executing the second portion of the machine learning code on a client. At step 505, the method 500 includes outputting a result of the machine learning code.

[0095] Figure 6 Schematically illustrates an example of an apparatus 600 for implementing method 500. Computing device 200 may include apparatus 600. Specifically, client 201 and / or server 202 may include apparatus 600. Apparatus 600 may be implemented on an electronic device such as a laptop, a tablet computer, a smart phone, or a television.

[0096] The apparatus 600 includes a processor 601 for processing a data set in the manner described herein. For example, the processor 601 may be implemented as a computer program running on a programmable device such as a central processing unit (CPU). The apparatus 600 includes a memory 602 arranged to communicate with the processor 601. The memory 602 may be a non-volatile memory. The processor 601 may also include a cache ( Figure 6 602). The apparatus may include more than one processor and more than one memory. The memory may store data executable by the processor. The processor may be configured to operate according to a computer program stored in a non-transitory form on a machine-readable storage medium. The computer program may store instructions for causing the processor to perform its method in the manner described herein.

[0097] Applicants hereby disclose separately each individual feature described herein and any combination of two or more such features. In this sense, given the common sense of a person skilled in the art, such features or combinations can be implemented as a whole according to this specification, without considering whether such features or combinations of features can solve any problem disclosed herein, and without limiting the scope of the claims. Applicants indicate that various aspects of the present invention can be composed of any such individual features or combinations of features. In view of the above description, it will be obvious to a person skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. A computing device (200), characterized in that: The computing device (200) comprises a client (201) and a server (202); the client (201) and / or the server (202) comprises one or more processors (601) and a memory (602), wherein the memory stores data in a non-transitory form, wherein the data defines a program code executable by the one or more processors (601), wherein the program code, when executed by the one or more processors (601), causes the computing device (200) to: Get machine learning code (204); Splitting the machine learning code (204) into a first part (204a) and a second part (204b); executing the first portion (204a) of the machine learning code (204) on the server (202); executing the second portion (204b) of the machine learning code (204) on the client (201); The results (306) of the machine learning code (204) are output.

2. The computing device (200) according to claim 1, characterized in that The first portion (204a) of the machine learning code (204) includes at least a portion of an inference portion of the machine learning code (204).

3. The computing device (200) according to claim 2, characterized in that The first portion (204a) of the machine learning code (204) includes all of the inference portion of the machine learning code (204).

4. The computing device (200) according to any one of the preceding claims, characterized in that The second portion (204b) of the machine learning code (204) includes all training portions of the machine learning code (204).

5. The computing device (200) according to any one of the preceding claims, characterized in that The machine learning code (204) is a transfer learning code.

6. The computing device (200) according to any one of the preceding claims, characterized in that The apparatus (200) is used to segment the machine learning code (204) according to one or more features of the machine learning code (204).

7. The computing device (200) according to claim 6, characterized in that The apparatus (200) is configured to execute the first portion (204a) of the machine learning code (204) to generate a sample output for a synthetic data sample, and to segment the machine learning code (204) according to the sample output.

8. The computing device (200) according to any one of the preceding claims, characterized in that The device (200) is used to segment the machine learning code (204) according to one or more characteristics of the computing device (200).

9. The computing device (200) according to claim 8 as appended to claim 7, characterized in that The apparatus (200) is used to partition the machine learning code (204) based on an evaluation between the sample output and a bandwidth of a network (206) connecting the client (201) and the server (202).

10. The computing device (200) according to any one of the preceding claims, characterized in that The apparatus (200) is used to control a batch size of the first portion (204a) of the machine learning code (204).

11. The computing device (200) according to claim 10, characterized in that The apparatus (200) is configured to control the batch size of the first portion (204a) of the machine learning code (204) according to one or more of the following: memory (205) of the server (202) to be occupied by the input and output (206) of the first portion (204a) of the machine learning code (204); The server (202) has memory (205) to be occupied by the weights of the machine learning code (204).

12. The computing device (200) according to any one of the preceding claims, characterized in that The device (200) is used to obtain one or more subsequent machine learning codes (204).

13. The computing device (200) according to claim 12, characterized in that The apparatus (200) is used to individually control the batch size of each of the first parts (204a) of the machine learning code (204).

14. The computing device (200) according to claim 13, characterized in that The apparatus (200) is configured to individually control the batch size of each of the first portions (204a) of the machine learning code (204) according to one or more of the following: the memory (205) of the server (202) to be occupied by the input and the output (206) of each of the first portions (204a) of the machine learning code (204); The memory (205) of the server (202) to be occupied by the weights of each of the machine learning codes (204).

15. The computing device (200) according to any one of the preceding claims, characterized in that The machine learning code (204) is obtained from a user (301), and / or the result (306) of the machine learning code (204) is output to the user (301).

16. A method (500) for executing a machine learning code, characterized in that: The method (500) comprises the following steps: Get machine learning code (501); Splitting the machine learning code into a first part and a second part (502); executing the first portion of the machine learning code on a server (503); executing the second portion of the machine learning code on a client (504); The results of the machine learning code (504) are output.

Images