Computer implementation methods, computer programs, computer systems (optimization of machine learning workload deployment)

Abstract

To provide a system for optimizing deployment of machine learning workloads.SOLUTION: A computer device receives information pertaining to a machine learning workload to be processed for a client device. The computer device determines a machine learning model for the workload and a processing location for the workload based, at least in part, on the information. The computer device generates a request to process the workload at the determined processing location utilizing the determined machine learning model.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This invention generally relates to the field of machine learning, and more specifically to the optimization of machine learning workload deployment between client devices and cloud environments. [Background technology]

[0002] Generally speaking, the field of cloud computing provides on-demand access over the internet to computing resources hosted by remote data centers. Cloud service providers (CSPs) offer a variety of models, including software-as-a-service, infrastructure-as-a-service, and platform-as-a-service. [Overview of the Initiative] [Problems that the invention aims to solve]

[0003] Various embodiments of the present invention provide optimization for workload deployment across server-side and client-side environments. [Means for solving the problem]

[0004] Various embodiments of the present invention provide methods, systems, and program products for performing the following operations (not necessarily in this order): (i) receiving information about a machine learning workload to be processed for a client device; (ii) at least partially determining a machine learning model for the workload and a processing location for the workload based on such information; and (iii) generating a request to process the workload at the determined processing location using the determined machine learning model. [Brief explanation of the drawing]

[0005] [Figure 1] A functional block diagram showing a computing environment in which a system optimizes the deployment of a machine learning model according to an exemplary embodiment of the present invention.

[0006] [Figure 2] A flowchart showing an operation process for executing a system to determine an optimal workload on a computing device within the environment of FIG. 1 according to an exemplary embodiment of the present invention.

[0007] [Figure 3] A flowchart showing an operation process for executing a system to train a machine learning model for an autonomous vehicle on a computing device within the environment of FIG. 1 according to an exemplary embodiment of the present invention.

[0008] [Figure 4] A table showing parameters related to a model and preprocessing according to an exemplary embodiment of the present invention.

[0009] [Figure 5] A diagram showing an equation for determining specific variables utilized by a machine learning model within the computing environment shown in FIG. 1 according to an exemplary embodiment of the present invention.

[0010] [Figure 6] A table showing values related to the determination of which machine learning model should be utilized to process a workload according to an exemplary embodiment of the present invention.

[0011] [Figure 7] A diagram showing a cloud computing environment according to at least one embodiment of the present invention.

[0012] [Figure 8]This figure shows an abstraction model layer according to at least one embodiment of the present invention.

[0013] [Figure 9] This is a block diagram of components of one or more computing devices in the computing environment shown in Figure 1, according to an exemplary embodiment of the present invention. [Modes for carrying out the invention]

[0014] Detailed embodiments of the present invention are disclosed herein with reference to the accompanying drawings. It should be understood that the disclosed embodiments are merely illustrative of possible embodiments of the invention and may take various forms. Furthermore, each of the examples given in relation to the various embodiments is intended to be illustrative and not limiting. Moreover, the drawings are not necessarily to scale, and some features may be exaggerated to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting but merely representative for teaching to those skilled in the art who utilize the invention in various ways.

[0015] References in this specification to "one embodiment," "one example," "exemplary embodiment," etc., indicate that the embodiments described may include certain features, structures, or characteristics, but not all embodiments necessarily include such features, structures, or characteristics. Furthermore, such language does not necessarily refer to the same embodiment. Moreover, if certain features, structures, or characteristics are described in relation to one embodiment, it is likely that the influence of such features, structures, or characteristics in relation to other embodiments, whether explicitly described or not, is within the knowledge of those skilled in the art.

[0016] Various embodiments of the present invention optimize the deployment of workloads (e.g., data processing) across a cloud server (i.e., server side) and one or more client devices such as autonomous vehicles (i.e., client side). In various cases, for example, when processing results are urgently needed for vehicle operation, the optimal workload is deployed on the client side. In other cases, for example, when processing is resource-intensive but not time-constrained, or when processing may benefit multiple vehicles, the optimal workload is deployed on the server side.

[0017] Various embodiments of the present invention provide a method for optimizing workload deployment across server-side and client-side environments. Generally speaking, a workload is data typically processed by machine learning models to perform vehicle actions (e.g., object detection, machine vision, acceleration, etc.). In various embodiments, optimizing workload deployment involves using a trained neural network to determine whether to deploy the workload to the server side, the client side, or a combination of both. In addition, the size and type of the neural network used for workload processing are also optimized based on the deployment parameters. For example, in various embodiments, if a limited number of autonomous vehicles are traveling on the road, the workload can be deployed on the server side; alternatively, if a larger number of autonomous vehicles are traveling on the road, the workload can be deployed on the client side to prevent data processing bottlenecks.

[0018] The present invention will now be described in detail with reference to the drawings.

[0019] Figure 1 is a functional block diagram showing a computing environment, generally designated as 100, according to one embodiment of the present invention. The computing environment 100 comprises a computer system 120 and a client device 130. The computer system 120 comprises a deployment optimization program 122, a computer interface 124, a server workload program 126, and a database 128. The client device 130 comprises a client program 132, a client interface 134, a client workload program 136, and a sensor 138.

[0020] The computer system 120 may include one or more computing devices, servers, server clusters, web servers, databases, and storage devices. The computer system 120 operates to communicate with client device 130 and various other client devices (not shown) via a network such as network 110. For example, the computer system 120 communicates with client program 132 to transfer data between database 128 and various other databases (not shown) connected to network 110. In general, the computer system 120 can be any computing device or combination of devices that are communicably connected to a network consisting of various computing devices, including, but not limited to, client device 130, for providing the functions described herein. The computer system 120 may include internal and external hardware components as described with respect to Figure 9. Various embodiments of the present invention recognize that the system shown in Figure 1 may include any number of computing devices, servers, databases, or storage devices, or combinations thereof, and the present invention is not limited to what is shown in Figure 1. Therefore, in some embodiments, some or all of the features and functions of the computer system 120 are included as part of the client device 130 or another computing device, or both. Similarly, in some embodiments, some of the features and functions of the client device 130 are included as part of the computer system 120 or another computing device, or both.

[0021] In addition, in some embodiments, the computer system 120 represents a cloud computing platform. Cloud computing is a service delivery model for enabling convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with service providers. Cloud models may include characteristics such as on-demand self-service, broad network access, resource pooling, rapid resilience, and measurable services, and can be represented by service models including Platform-as-a-Service (PaaS) models, Infrastructure-as-a-Service (IaaS) models, and Software-as-a-Service (SaaS) models, and can be implemented as various deployment models, including private clouds, community clouds, public clouds, and hybrid clouds.

[0022] In various embodiments, the computer system 120 represents an autonomous vehicle network server. In the embodiment shown in Figure 1, the deployment optimization program 122, the computer interface 124, and the server workload program 126 are stored on and executed by the computer system 120, respectively. In other embodiments, the database 128 can store and / or execute a different total number of applications without departing from the scope of the present invention. Generally, the deployment optimization program 122, which runs on the computer system 120, receives deployment parameters and optimization target data from the server workload program 126 and the client workload program 136, respectively, as described herein. In addition, the deployment optimization program 122 operates to notify client devices 130 and various other computing devices (not shown) via the network 110 of the workload conditions and / or assignments that may occur within the computer system 120. In one example, the deployment optimization program 122 takes the form of a machine learning application that utilizes elements of the computer system 120 to analyze deployment parameters and optimization target data sent from the server workload program 126 and the client workload program 136 with respect to the deployment of workloads for data processing between the client and server sides. This example will be referenced in the discussion of various embodiments herein to illustrate various aspects of the invention, but the invention should not be construed as being limited to such embodiments. In some embodiments, an IoT application running on the computer system 120 may also include analysis logic, as described herein, that analyzes data from one or more client devices (e.g., client device 130) to facilitate the optimization of device configuration rules for deploying workload configurations.

[0023] Computer system 120 is shown in Figure 1 for illustrative purposes. However, it should be understood that in various embodiments, computer system 120 may include any number of databases managed according to the functionality of the deployment optimization program 122. Generally, database 128 represents data, and computer interface 124 manages the ability to view said data. In other embodiments, deployment optimization program 122 represents code that provides the ability to take specific actions on another physical or virtual resource, and deployment optimization program 122 manages the ability to use and modify the data. Client program 132 may also represent any combination of the above features, and deployment optimization program 122 has access to the data stored on database 128. To illustrate various aspects of the present invention, examples of deployment optimization program 122 are presented, in which deployment optimization program 122 represents one or more of a local IoT network and machine learning models for optimizing the deployment of a workload for data processing.

[0024] In this exemplary embodiment, the deployment optimization program 122, the computer interface 124, and the server workload program 126 are stored on the computer system 120. However, in other embodiments, the deployment optimization program 122, the computer interface 124, and the server workload program 126 may be stored externally and accessed through a communication network such as network 110. Network 110 can be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the two, and may include wired, wireless, or any other connection known in the art. Generally, network 110 can be any combination of connections and protocols that support communication between the computer system 120 and the client device 130, according to a desired embodiment of the present invention.

[0025] The deployment optimization program 122 is shown in Figure 1 for illustrative simplicity. In various embodiments of the present invention, the deployment optimization program 122 represents logical operations performed on a computer system 120, where a computer interface 124 manages the ability to view these logical operations and their results on the computer system 120. The computer system 120 can include any number of logical operations that are managed and performed according to the deployment optimization program 122. In some embodiments, the deployment optimization program 122 represents an administrator that analyzes input and output (I / O) data. In addition, when the deployment optimization program 122 performs a workload data processing configuration for an autonomous vehicle, it operates to monitor the analyzed I / O data and determines which workloads are deployed on one or more of the following: (i) a server workload program 126, (ii) a client workload program 136, or (iii) a combination of the server workload program 126 and the client workload program 136. In some embodiments, the deployment optimization program 122 determines whether a particular action is likely to be performed, generates a workload optimization request, and communicates a notification to the client device 130 indicating that a modification or reconfiguration of workload data processing has been requested for the client device 130.

[0026] The computer system 120 includes a computer interface 124. The computer interface 124 provides an interface between the computer system 120 and a client device 130. In some embodiments, the computer interface 124 may be a graphical user interface (GUI), a web user interface (WUI), or an image projector, and may display text, documents, a web browser, windows, user options, application interfaces, operation commands, images, and a holographic display, and may include information that a program presents to the user (e.g., graphics, text, and sound) and control sequences that the user uses to control the program. In some embodiments, the computer system 120 accesses data communicated from the client device 130 via a client-based application running on the computer system 120. For example, the computer system 120 may include mobile application software that provides an interface between the computer system 120 and the client device 130.

[0027] In various embodiments of the present invention, the client device 130 represents an autonomous vehicle capable of executing machine-readable program instructions and communicating with the computer system 120. In various embodiments, the computing environment 100 comprises various additional other client devices (e.g., various other autonomous vehicles) not shown. In another embodiment, the client device 130 and the various other client devices represent a computing system that utilizes clustered computers and components, acting as a single pool of seamless resources with access to the computer system 120 and the network 110. The client device 130 may comprise internal and external hardware components as illustrated and described in more detail with respect to Figure 9.

[0028] Sensor 138 is shown in Figure 1 for illustrative simplicity. In various embodiments of the present invention, sensor 138 represents various computing devices running on client device 130, where sensor 138 communicates data to deployment optimization program 122 and client program 132, respectively. In various embodiments, sensor 138 represents a computing device running on client device 130 that monitors and determines data processing of electrical components. In addition, in various embodiments, sensor 138 determines the locations of various other client devices (e.g., autonomous vehicles) to determine the data processing speed of a group of autonomous vehicles (i.e., client device 130 and various other client devices). In various embodiments, sensor 138 determines the data processing speed of each autonomous vehicle and generates data to be communicated to deployment optimization program 122, which determines the respective workload, generates a workload optimization request, and communicates the workload optimization request to client device 130.

[0029] Various embodiments of the present invention recognize that in some cases, workload preprocessing may be required, desired, or both. Preprocessing generally involves collecting data for machine learning models such as neural networks. Some examples of preprocessing operations include, but are not limited to, (i) data cleaning, (ii) instance selection, (iii) normalization, (iv) transformation, (v) feature extraction, and (vi) selection. Generally, data quality is a priority prior to performing analysis or training a neural network, or both. Various embodiments of the present invention provide that data preprocessing enables more effective knowledge discovery during the training phase of a neural network by removing irrelevant and redundant information or noisy and unreliable data.

[0030] In various embodiments, the deployment optimization program 122 includes a convolutional neural network (CNN), which includes I / O data and a number of hidden layers (e.g., RELU layers) consisting of neurons. In various embodiments, the CNN receives an input that includes, but is not limited to, (i) deployment parameters and (ii) optimization target data, which is placed in neurons in the first layer of the CNN. In various embodiments, the output of the first layer travels through one or more hidden layers, where the output data of the first layer is placed in the input of the second layer, and so on, to form a fully connected neural network. The deployment optimization program 122 analyzes (i) the output data of the multilayer perceptual neural network (MLP) and (ii) the changes in the weight outputs. Various embodiments of the present invention recognize that the output data represents the optimal deployment of a workload. In various embodiments, the deployment optimization program 122 receives the optimal deployment for a workload as output from the CNN, where the optimal deployment for the workload may represent, for example, deploying the workload on the server side when a limited number of autonomous vehicles are traveling on a roadway. Alternatively, the optimal deployment for this workload may be to deploy the workload on the client side to prevent data processing bottlenecks when there are more autonomous vehicles on the road than a threshold.

[0031] Various embodiments of the present invention provide a deployment optimization program 122, which runs as a neural network, that determines the optimal deployment of a workload for data processing. In various embodiments, the deployment optimization program 122 determines the optimal deployment of a workload for data processing across one or more autonomous vehicles (i.e., client devices 130). In various embodiments, the deployment optimization program 122 determines the optimal deployment of a workload for data processing across a cloud server (i.e., computer system 120). In various embodiments, the deployment optimization program 122, which runs as a neural network, receives as input (i) deployment parameters, including preprocessing performed on the client side, preprocessing performed on the server side, the workload on the client side, and the workload on the server side, and (ii) optimization target data, including, but not limited to, time, data processing speed, memory, and accuracy. In various embodiments, the client-side preprocessing and server-side preprocessing represent filtering to remove irrelevant and redundant information, or data containing noise and unreliable, or both, from the data sent to the neural network. In various embodiments, the workload executed on the client side, the server side, or both represents data processing using a set of program instructions that execute autonomous vehicle operations. In addition, in various embodiments, the deployment optimization program 122 determines whether to deploy the workload on the server workload program 126 when a limited number of autonomous vehicles are traveling on the roadway, and whether to deploy the workload on the client workload program 136 to prevent data processing bottlenecks when more autonomous vehicles than a threshold are traveling on the roadway.

[0032] Various embodiments of the present invention recognize that various deployment parameters and optimization target data may be utilized by a deployment optimization program 122, which is executed as a neural network to predict the optimal deployment of the workload. Various embodiments of the present invention further recognize that, when a network connection is used between a computer system 120 and a client device 130, data transfer latency and bandwidth are deployment parameters that are considered by the neural network (i.e., the deployment optimization program 122) when predicting the optimal deployment of the workload. In various embodiments, it is optimal to deploy the workload on the client workload program 136 when there is a high number (e.g., more than a threshold) of images or data to be processed, or when there is a high number (e.g., more than a threshold) of autonomous vehicles on the roadway, or both. In various embodiments, it is optimal to deploy the workload on the server workload program 126 when there is a small number of images or data to be processed, or when there is a limited number of autonomous vehicles on the roadway, or both.

[0033] Various embodiments of the present invention provide a neural network that predicts the respective workloads for data processing deployed across autonomous vehicles (i.e., client devices 130) and cloud networks (i.e., computer systems 120). Various embodiments of the present invention further provide that the deployed workload achieves high accuracy and low response time within the boundaries and constraints of deployment parameters, including, for example, (i) available hardware, (ii) available power, (iii) available bandwidth and latency, (iv) available memory, and (v) available storage. In various embodiments, the deployment optimization program 122 predicts the optimal end-to-end solution based on the optimization target. In various embodiments, the deployment optimization program 122 determines, at least in part, a model for handling the workload based on the status of the ecosystem (e.g., computing network), in which case the optimal workload is determined across the hardware or software stack, or both (e.g., server-side or client-side). In addition, in various embodiments, the optimization target is used in different ways, for example, without machine learning models / neural networks. For example, in various embodiments, instead of being used in training a machine learning model / neural network, the expansion parameters and optimization targets are initialized as variables, and the expansion optimization program 122 uses an integer linear programming model to compute an optimization function that generates the optimal workload expansion to achieve the desired performance and accuracy. In other embodiments, a naive maximization function may be used. In these embodiments, various combinations of workload expansions can be computed, and the workload expansion that yields the maximum output can be used.

[0034] Figure 2 is a flowchart illustrating an exemplary embodiment of the present invention for determining the optimal workload within a computing environment 100. More specifically, Figure 2 shows the combined overall operation 200 of a deployment optimization program 122. In some embodiments, operation 200 represents the logical operations of the deployment optimization program 122, which represents the interaction between a logical computing device communicating with a computer system 120 and various other computing devices connected to a network 110. It should be understood that Figure 2 provides an example of one implementation and does not imply any limitation with respect to an environment in which different embodiments may be implemented. Many modifications may be made to the environment shown. In one embodiment, the sequence of operations in flowchart 200 can terminate at any operation. In addition to the features described above, any operation in flowchart 200 can be resumed at any point in time.

[0035] In operation 202, the deployment optimization program 122 receives deployment parameters and optimization target data. In various embodiments, the deployment optimization program 122 includes a neural network running on the computer system 120 (i.e., the cloud computing network). In various embodiments, the deployment optimization program 122 identifies deployment parameters (i.e., deployment variables) for the workload to be deployed for data processing. In various embodiments, the parameter y represents where the workload should be deployed (e.g., client-side, server-side, or a combination of client-side and server-side). In addition, in various embodiments, y represents a binary variable which is true when the inference is performed on the cloud network (i.e., the computer system 120) and false when it is performed on the client device (i.e., the client device 130). In various embodiments, processing the workload on the client device 130 allows for the elimination of transmission delays between the computer system 120 and the client device 130. Various embodiments of the present invention recognize that processing the workload on the client device 130 requires additional battery and memory consumption on the client device 130, which operates with limited battery and memory availability compared to the cloud network (i.e., computer system 120). In various embodiments, z represents a binary variable which is true when data preprocessing (e.g., images for object detection) is performed on the cloud network (i.e., computer system 120). In various embodiments, m represents an optical model with lower accuracy, however m has lower requirements for memory usage, battery usage, and inference time. In various embodiments, s represents the size of the data, where having a larger data file corresponds to higher accuracy, because a larger data file can carry more information, however s has higher requirements for inference time and memory.In various embodiments, the deployment optimization program 122 further identifies deployment parameters, including (i) resources required for the workload to be processed on the client device 130 and for preprocessing the remainder of the workload, and (ii) resources required for the workload to be processed on the computer system 120 (i.e., the cloud network) and for preprocessing the remainder of the workload, although these do not identify and limit the values ​​of the variables as discussed above. In various embodiments, definitions of various parameters and variables are shown in Figure 4, which will be discussed in more detail below. In various embodiments, the deployment optimization program 122 receives optimization target data, including, but not limited to, (i) memory capacity, (ii) data processing speed, (iii) battery level, and (iv) vehicle operations to be performed (e.g., object detection, acceleration, etc.).

[0036] Various embodiments of the present invention provide, in some cases, that a workload is run on each client device (i.e., client device 130) to estimate the deployment parameters shown in Figure 4, and that a client program 132 monitors the data obtained from the running workload and communicates the performance and resources available to the deployment optimization program 122. In various embodiments, this estimation of the deployment parameters is highly accurate, however, this estimation requires one or more runs of the workload and availability to test each individual client device one or more times. Alternatively, various embodiments of the present invention provide that if each individual client device is not available to be tested one or more times using a workload that is a reported result from a standardized dataset, the deployment optimization program 122 scales the parameters of the dataset using the floating-point operations per second (FLOPS), bandwidth, and power of the client device and retests the client device. However, scaling the parameters can result in a potentially inaccurate model because the performance of the model rarely scales linearly with respect to the data of the client device. Various embodiments of the present invention monitor the workload running on each client device and determine the performance of the resources used by the deployment optimization program 122.

[0037] In operation 204, the deployment optimization program 122 determines one or more optimization targets. In various embodiments of the present invention, one or more optimization targets (e.g., workload execution time on the client and server sides, available energy budget on the client and server sides, total memory on the client and server sides, hardware resources on the client and server sides, and desired accuracy of the workload; these are also referred to as time, power, memory, and accuracy) represent the values ​​to be determined (e.g., optimization target data). In various embodiments, the deployment optimization program 122 determines the optimization targets, which are (i) minimizing the total predicted t, (ii) minimizing the total power p drawn from the client device 130, and (iii) the total memory usage mem on the device equals the maximum free device memory mem max This includes ensuring that it does not exceed (iv) the model accuracy acc. Various embodiments of the present invention provide that some examples of equations used by the unfolding optimization program 122 to solve the optimization targets mentioned above are shown in Figure 5, which are discussed in more detail in the following paragraph and further below.

[0038] Various embodiments of the present invention provide a deployment optimization program 122 that determines the value of the optimization target. Equation 500A (see Figure 5) represents the sum of the transmission times for data processed between the computer system 120 and the client device 130. In various embodiments, the deployment optimization program 122 determines the value of the total time t as shown in equation 500A. Various embodiments of the present invention provide a threshold t max We recognize that reducing the prediction time beyond t does not yield any beneficial value in optimizing workload processing. In one exemplary embodiment, the dataset represents streaming video from a camera, where there is no benefit in having a processing speed faster than the frame rate of the streaming video, and therefore the constraint is t ≤ t maxIt is recognized as follows. Equation 500B (see Figure 5) represents the total power consumed for data processed between the computer system 120 and the client device 130. In various embodiments, the unfolding optimization program 122 determines the value of the total power p as shown in equation 500B. Various embodiments of the present invention provide that the power consumed by the memory optimization target is negligible compared to the power consumed through the use of the graphical processing unit (GPU) and central processing unit (CPU) through the execution of the neural network model. In addition, the constraint of the memory optimization target is mem ≤ mem max It is expressed as follows. Equation 500C (see Figure 5) represents the total memory for data processed between the computer system 120 and the client device 130.

[0039] In various embodiments, the unfolding optimization program 122 determines the accuracy of the global optimization target for the optimization target specified above, as shown in equation 500D (see Figure 5). Various embodiments of the present invention provide that the accuracy function for determining the global optimization target is a nonlinear function. Various embodiments of the present invention further present that the weights applied to increase the accuracy of the global optimization target should be greater when lower accuracy is presented as being compared to higher accuracy. In one exemplary embodiment, increasing the accuracy of the global optimization target from 80% to 90% should receive a lower weight than increasing the accuracy from 30% to 40%.

[0040] In various embodiments, the unfolding optimization program 122 determines the optimization function by solving equation 500E (see Figure 5), where the determination of the optimization target in equations 500A, 500B, 500C, and 500D (hereafter, equations 500A to 500D) is utilized in the formulation of equation 500E. Various embodiments of the present invention provide that α used in the calculation of f(acc) specifies an accuracy-power trade-off. Various embodiments of the present invention further provide that the value of α varies depending on the accuracy requirements of the particular workload at use (e.g., (i) client-side, (ii) server-side, or (iii) a combination of client-side and server-side).

[0041] In operation 206, the deployment optimization program 122 uses a neural network to determine the optimal workload based on the deployment parameters and optimization target data. In various embodiments, the deployment optimization program 122 utilizes the deployment parameters and optimization target data relating to the workload to be performed and places this data in neurons of the first layer of a machine learning model, where the deployment optimization program 122, running as a machine learning model, passes the neurons through multiple hidden layers to the output layer of the machine learning model. In various embodiments, the deployment optimization program 122 generates an optimal deployment for the workload to process the data, at least partially based on the deployment parameters and the optimization target data calculated from equations 500A to 500E. In various embodiments, the deployment optimization program 122 predicts that the optimal deployment for the workload to process the data should be client-side (e.g., client workload program 136). In various embodiments, the deployment optimization program 122 predicts that the optimal deployment for the workload to process the data should be server-side (e.g., server workload program 126). In various embodiments, the deployment optimization program 122 predicts that the optimal deployment for a workload to process data should be a combination of server-side and client-side deployments, examples of which are discussed in further detail below.

[0042] In operation 208, the deployment optimization program 122 generates a workload optimization request. In various embodiments, the deployment optimization program 122 generates a workload optimization request based at least in part on a prediction of the optimal deployment of the workload for processing the data. In various embodiments, the deployment optimization program 122 generates a workload optimization request based at least in part on the optimal deployment of the workload to be deployed on the client side. In various embodiments, the deployment optimization program 122 generates a program instruction that commands the client workload program 136 to deploy the optimal workload for processing data relating to operational functions, which are, for example, (i) image processing, (ii) radar positioning of various other vehicles (e.g., autonomous vehicles or non-autonomous vehicles, or both), (iii) lidar sensors for measuring the distance of light pulses, (iv) spatial detection of roadways and lane boundaries, (v) motion and object detection sensors to assist in automatic parking, and (vi) navigation operations, including, but not limited to, operations utilizing compasses and geographic maps. In various embodiments, the deployment optimization program 122 communicates a workload optimization request to the client workload program 136, along with program instructions that command the client workload program 136 to deploy a workload for data processing and to perform actions to complete the requested data processing.

[0043] In various embodiments, the deployment optimization program 122 generates workload optimization requests based at least partially on the optimal deployment of the workload deployed on the server side. In various embodiments, the deployment optimization program 122 communicates to the server workload program 126 a set of program instructions for deploying a workload for data processing on the computer system 120 relating to operational functions, which include (i) image processing, (ii) radar positioning of various other vehicles (e.g., autonomous vehicles or non-autonomous vehicles, or both), (iii) lidar sensors for measuring the distance of pulses of light, (iv) spatial detection of roadways and lane boundaries, (v) motion and object detection sensors to assist in automatic parking, and (vi) navigation operations, including, but not limited to, operations utilizing compasses and geographic maps. In various embodiments, the deployment optimization program 122 communicates to the client program 132 program instructions instructing the server workload program 126 to execute a workload for data and communicate data required by the client program 132 to execute functions for operating the client device 130 (i.e., an autonomous vehicle). In addition, in various embodiments, the deployment optimization program 122 communicates to the client program 132 a set of program instructions that instruct the client program 132 to communicate all data relating to the operational functions of the client device 130.

[0044] In various embodiments, the deployment optimization program 122 generates a workload optimization request based at least partially on the optimal deployment of the workload to be deployed in a combination of server-side and client-side operations. In various embodiments, the deployment optimization program 122 executes program instructions to deploy the workload for data processing on the computer system 120 and the client program 132 relating to the operational function. In various embodiments, the deployment optimization program 122 further determines that the workload has a higher likelihood of more efficient execution when the workload associated with the operational function is delegated between a combination of server-side and client-side operations. In one exemplary embodiment, the deployment optimization program 122 determines that it is more efficient to deploy the workload on the server side and communicates a set of program instructions to the server workload program 126 to execute the program instructions to deploy the workload, which includes, but is not limited to, (i) determining the radar position of various other vehicles (e.g., autonomous vehicles or non-autonomous vehicles, or both), (ii) navigation operations, and (iii) image processing. In addition, in this exemplary embodiment, the deployment optimization program 122 determines that it is more effective to deploy the workload on the client device and communicates a set of program instructions to the client workload program 136 to execute the program instructions that deploy the workload, including, but not limited to, (i) a lidar sensor measuring the distance of light pulses, (ii) road and lane boundary detection, and (iii) sensor assistance in automatic parking. In various embodiments, the deployment optimization program 122 identifies a first subset of workloads for data processing for operational functions to be deployed on the server side (i.e., the server workload program 126) and a second subset of workloads for data processing for operational functions to be deployed on the client side (i.e., the client workload program 136).In various embodiments, the server workload program 126 executes a workload for data processing for a first subset of operational functions and communicates data to the client program 132 along with program instructions for executing the operational functions for the first subset on the client device 130 (i.e., the autonomous vehicle). In various embodiments, the deployment optimization program 122 communicates program instructions to the client workload program 136 instructing it to deploy a workload for data processing for a second subset of operational functions. In addition, in various embodiments, the deployment optimization program 122 communicates program instructions to the client workload program 136 instructing it to execute a second subset of operational functions on the client device 130.

[0045] Figure 3 shows a flowchart 300 illustrating the operation of a deployment optimization program 122 in a computing environment 100 according to an exemplary embodiment of the present invention. More specifically, Figure 3 shows the combined overall operation 300 of the deployment optimization program 122 running on a computer system 120. In some embodiments, operation 300 represents the logical operations of the deployment optimization program 122, and the interaction between the deployment optimization program 122, the client program 132, and the sensor 138 represents the logical units running on the computer system 120. It should be understood that Figure 3 provides an example of one implementation and does not imply any limitation with respect to environments in which different embodiments may be implemented. Many modifications may be made to the environment shown. In one embodiment, the sequence of operations in flowchart 300 can be executed simultaneously. In addition, the sequence of operations in flowchart 300 can be terminated at any operation. In addition to the features described above, any operation in flowchart 300 can be resumed at any point in time.

[0046] In operation 302, the deployment optimization program 122 receives training deployment parameters and training optimization target data. In various embodiments, the deployment optimization program 122 receives as training data (i) deployment parameters including client-side preprocessing, server-side preprocessing, client-side workload, and server-side workload, and (ii) optimization target data including client-side and server-side workload execution time, available energy budget on the client-side and server-side, total memory on the client-side and server-side, client-side and server-side hardware resources, and desired accuracy of the workload. Various embodiments of the present invention provide that any combination of deployment parameters and optimization target data may exist in the training data to incorporate a wide range of possible variations between the deployment parameters and the optimization target data in order to predict the optimal workload for processing the data to be deployed.

[0047] In operation 304, the deployment optimization program 122 trains a neural network. In various embodiments, the training data is at least partially based on (i) deployment parameters, (ii) optimization target data, and (iii) an optimal deployment workload associated with the deployment parameters and the optimization target data (based on either user input / labelling or a separate optimization function). Various embodiments of the present invention provide that the training data is required to train a neural network to learn and predict an optimal workload for subsequent predictions. In one exemplary embodiment, the deployment optimization program 122 receives values for optimization target data including, but not limited to, (i) workload execution times (predicted time values are less than or equal to a threshold maximum value for predicted time) on the client side and the server side, (ii) power capabilities of the client side and server side hardware resources, (iii) total memory on the client side and server side (total memory values are equal to or less than the maximum capabilities of the GPU and CPU), and a percentage value of the desired accuracy of the workload, and the deployment optimization program 122 receives deployment parameters including, but not limited to, (i) a workload deployed on the client side (e.g., p e,m , t e,m , r e,m , y = 0), (ii) a workload deployed on the server side (e.g., p c,m , t c,m , r c,m , y = 1), (iii) preprocessing data on the client side (e.g., p e,pr , t e,pr , r e,pr , z = 0), and (iv) preprocessing data on the server side (e.g., p c,pr , t c,pr , r c,pr , z = 1). In this exemplary embodiment, the deployment optimization program 122 utilizes the deployment parameters and the optimization targets to determine an optimal workload to be deployed on either the client side or the server side, as discussed above.

[0048] In the first exemplary embodiment, the deployment optimization program 122 is trained using inputs that include deployment parameters, optimization target data, and an optimal workload to be deployed to complete three subtasks (e.g., vehicle operation). The vehicle operation to be completed by the workload includes object detection, object recognition, and navigation operation. The optimization target data includes, but is not limited to, time and memory. The maximum threshold for workload execution time is 3 seconds, the client side can perform an execution time of 1 second, the server side can perform an execution time of 0.5 seconds, and the total transmission time between the server side and the client side is 1 second. Total memory cannot exceed the maximum capacity of the GPU and CPU operating on the client side and the server side. Total free memory on the client side is equal to or less than 1.15 megabytes (MB), and total free memory on the server side is equal to or less than 2.9 terabytes (TB). The deployment parameters are (i) the workload to be deployed on the client side (e.g., p e,m t e,m , r e,m (ii) workload deployed on the server side (e.g., p c,m t c,m , r c,m (iii) Preprocessed data on the client side (e.g., p e,pr t e,pr , r e,pr (iv) Preprocessed data on the server side (e.g., p c,pr t c,pr , r c,prThis includes z=1). The optimal workload deployed to complete the three vehicle operations includes (i) deploying the object detection vehicle operation workload on the client side (because the transmission time required for data to be communicated to the server side to detect objects and then back to the client side to perform the vehicle operation is excessively long), (ii) deploying the object recognition vehicle operation workload on the server side (because the total amount of free memory required to complete the object recognition and object detection vehicle operations on the client side exceeds 1.15 MB of free memory on the client side), and (iii) deploying the navigation vehicle operation workload on the server side (because the total free memory required to complete the navigation vehicle operation exceeds the total free memory on the client side (e.g., 1.15 MB)). In this exemplary embodiment, the deployment optimization program 122 determines the respective optimal deployment for each vehicle operation (e.g., object detection, object recognition, and navigation operations) across either the client side or the server side. In this exemplary embodiment, the deployment optimization program 122 determines that the optimal deployment for completing object detection is to deploy the workload on the client side, the deployment optimization program 122 determines that the optimal workload for completing object recognition is to deploy the workload on the server side, and the deployment optimization program 122 determines that the optimal workload for completing navigation operations is to deploy the workload on the server side. Various embodiments of the present invention provide, as discussed above, that any combination of values ​​for optimization target data, including time, power, memory, and accuracy, and any combination of deployment parameters can be provided as input to a neural network (e.g., the deployment optimization program 122) to determine the optimal deployment of a workload (e.g., data processing) to complete vehicle operations on the server side, the client side, or a combination of both the server and client sides.

[0049] In at least the second exemplary embodiment, the deployment optimization program 122 is a trained neural network. In this second exemplary embodiment, the deployment optimization program 122 determines the optimal workload to deploy for three vehicle operations (e.g., object detection, object recognition, and navigation operations). At the first time point (e.g., T=0), the deployment optimization program 122 receives input including deployment parameters and optimization target data. At T=0, the deployment optimization program 122 (i) determines the workload to be deployed on the client side (e.g., p e,m t e,m , r e,m (ii) workload deployed on the server side (e.g., p c,m t c,m , r c,m (iii) Preprocessed data on the client side (e.g., p e,pr t e,pr , r e,pr (iv) Preprocessed data on the server side (e.g., p c,pr t c,pr , r c,prThe system receives expansion parameters including z=1). In this exemplary embodiment, at T=0, the optimization target data includes, but is not limited to, time and memory. The optimization target data for time and memory at T=0 includes: a maximum threshold for workload execution time of 3 seconds, the client side being able to execute for 1 second, the server side being able to execute for 0.5 seconds, and the total transmission time between the server and client sides being 0.7 seconds. Total memory cannot exceed the maximum capacity of the GPUs and CPUs operating on the client and server sides. Total free memory on the client side is equal to or less than 2.15 megabytes (MB), and total free memory on the server side is equal to or less than 7.6 terabytes (TB). At T=0, the deployment optimization program 122 determines that the optimal workload for the deployment of the three vehicle operations includes deploying the workload for the object detection vehicle operation on the server side (because the amount of total free memory and low threshold transmission time make it possible to process object detection on the server side), deploying the workload for the object recognition vehicle operation on the server side (because the amount of total free memory and low threshold transmission time make it possible to process object recognition on the server side), and deploying the workload for the navigation vehicle operation on the server side. Various embodiments of the present invention provide a system that continuously identifies updated optimization target data and updated deployment parameters and uses these updates to determine the optimal workload to be deployed across the server side, client side, or a combination of server and client sides using a neural network.

[0050] In this second exemplary embodiment, at a second time point (e.g., T=1) that is later than T=0, the optimization target data includes, but is not limited to, time and memory. The optimization target data for time and memory at T=1 includes: a maximum threshold for workload execution time of 3 seconds, the client side being able to perform an execution time of 1 second, the server side being able to perform an execution time of 0.5 seconds, and the total transmission time between the server and client sides being 1.2 seconds. Total memory cannot exceed the maximum capacity of the GPUs and CPUs operating on the client and server sides. Total free memory on the client side is equal to or less than 1.9 megabytes (MB), and total free memory on the server side is equal to or less than 2.6 terabytes (TB). The deployment optimization program 122 receives the optimization target data and deployment parameters at T=1 as input to the neural network, as discussed above. The deployment optimization program 122 determines the optimal workload to be deployed for the three vehicle operations at T=1. At T=1, the deployment optimization program 122 determines that the optimal workload for deploying the three vehicle operations includes deploying the workload for the object detection vehicle operation on the client side (because the transmission time for communicating a set of program instructions so that data is communicated to the server side to detect objects and then back to the client side to execute the vehicle operation is excessively long), deploying the workload for the object recognition vehicle operation on the server side (because the total amount of free memory required to complete the object recognition vehicle operation and the object detection vehicle operation on the client side exceeds the 1.9MB of free memory on the client side), and deploying the workload for the navigation vehicle operation on the server side (because the total free memory required to complete the navigation vehicle operation exceeds the total free memory on the client side (e.g., 1.9MB)).

[0051] In various embodiments, training data is provided to neurons in the input layer of the neural network. In various embodiments, the deployment optimization program 122 learns from the training data within the neurons as the neurons' signals travel through multiple hidden layers in the neural network. In various embodiments, the deployment optimization program 122 learns as the training data travels through the edges of multiple hidden layers, where the edge weights are adjusted as the deployment optimization program 122 continues to learn from the training data. In various embodiments, the edge weights increase or decrease the intensity of the signal at the edge. In various embodiments, the deployment optimization program 122 receives output data from the output layer of the neural network based at least in part on an optimal workload deployed on the server side, the client side, or a combination of both server and client sides.

[0052] In various embodiments, the optimal workload to be deployed relates to autonomous vehicle operations, and further, the optimal workload includes, but is not limited to, different operations performed on the autonomous vehicle. In a third exemplary embodiment, traffic may be light during the early morning hours, and the network itself may not be as congested as during rush hour. In addition, the electric autonomous vehicle may have sufficient battery power to complete various vehicle operations. In this exemplary embodiment, the neural network receives input (e.g., deployment parameters and optimization target data) including, but is not limited to, this data (e.g., network traffic, available battery power level, amount of vehicle operations to be completed), and the neural network determines the optimal deployment of the workload. In addition, in this exemplary embodiment, the deployment optimization program 122 determines that object detection-related operations are optimally deployed on the autonomous vehicle, and that the object recognition task is optimally deployed on the backend server side based on low network utilization, at least in part.

[0053] In the fourth exemplary embodiment, the network becomes congested during rush hour because a large number of vehicles are traveling on the roadway. In this exemplary embodiment, the optimal workload must be updated to take into account high thresholds of vehicle and network congestion. In this exemplary embodiment, the neural network receives input (e.g., deployment parameters and optimization target data) including this data (e.g., high network traffic congestion, available battery power level, amount of vehicle operations to be completed, etc.), and the neural network determines the optimal deployment of the workload. In addition, in this exemplary embodiment, the deployment optimization program 122 determines that the optimal deployment of the workload for object detection and object recognition is, at least in part, based on high network utilization and extends to the autonomous vehicle.

[0054] In various embodiments, once the neural network training in operation 304 is complete, the trained neural network can be used to predict one or more optimal deployment workloads, as further discussed in operation 206. In various embodiments, the deployment optimization program 122 at least partially determines the optimal workload based on the output data and communicates the optimal workload to be deployed on the server workload program 126, the client workload program 136, or a combination of the server workload program 126 and the client workload program 136.

[0055] Figure 4 shows Table 400. Table 400 defines parameters related to performance and cost for the deployment of workloads for data processing and preprocessing on various devices (e.g., client-side and server-side). In various embodiments, the deployment parameters define the resources required by the client-side workload for preprocessing and data processing during the workload on the client device 130. In addition, in various embodiments, the deployment parameters define the resources required by the server-side workload for preprocessing and data processing during the workload on the computer system 120.

[0056] In various embodiments, with respect to storage resources, p store (s) represents the power required on the client side to store an image of size s. In various embodiments, m image (s) represents the memory requested by the client to cache an image of size s. In various embodiments, for the transmission resource, p trans (s) and t trans (s) represents the power and time required to transmit an image of size s from the client to the server, respectively. Various embodiments of the present invention provide that the output bounding box can be transmitted via a text file, so that the output bounding box has no transmission cost. Finally, γ represents the average compression achieved by preprocessing an image of size s. Various embodiments of the present invention provide that the transmission resources between the client and the server are directly proportional to the size of the image, where the time required to transmit the preprocessed image is γt trans It is presented as (s).

[0057] In Figure 5, (i) equation 500A represents at least the target optimization time, (ii) equation 500B represents at least the target optimization power, (iii) equation 500C represents at least the target optimization memory, (iv) equation 500D represents at least the target optimization accuracy, and (v) equation 500E represents at least the optimized value. Generally, the optimization targets—time, power, memory, and accuracy—are calculated according to at least (i) equation 500A, (ii) equation 500B, (iii) equation 500C, and (iv) equation 500D, respectively, to solve for the optimized value of equation 500E in Figure 5.

[0058] In equation 500A, the value of the total time t is determined. In equation 500A, the first term of the time equation is: yz(t c,pr +t c,m +t trans (s)) represents both preprocessing and workload processing performed on the cloud network, where the first term represents the sum of the transmission time between the server and client, and the time required on the cloud network for preprocessing and workload processing, respectively. In equation 500A, the second term of the time equation:

number

number

[0059] In equation 500B, the value of the total power p is determined. In equation 500B, the first term is: yz(p c,pr +p c,m +p trans (s)) represents both pre-processing and workload processing that occur on the server side, and is the sum of the power consumption for server-side pre-processing and the power consumption for workload prediction, respectively. In equation 500B, the second term:

number

number

[0060] In equation 500C, the constraints of the memory optimization target are determined. In equation 500C, term 1:

number

number

number

[0061] Equation 500D determines the accuracy of the overall optimization target. Equation 500D is a nonlinear function where α and β are parameters that specify the weights of accuracy in the overall optimization target.

[0062] In equation 500E, the optimized value is determined. In equation 500E, the maximum value applicable to the optimization target is calculated, where α used in the calculation of f(acc) specifies the accuracy-power trade-off.

[0063] In various embodiments, in addition to determining the optimal location (e.g., client or server) for deploying the workload, the deployment optimization program 122 also determines the optimal machine learning model for handling the workload. For example, in the exemplary embodiment, eight different machine learning models are considered for deployment. In this exemplary embodiment, the client device is an autonomous vehicle, and the workload is an object detection workload. The autonomous vehicle processes the workload using an embedded low-power computing board, while the server processes the workload using a high-power GPU. The transmission power for transferring images from the autonomous vehicle to the server is negligible compared to running the model, with an average transmission rate of 1.0 MBps. Furthermore, for this exemplary embodiment, it is assumed that only one image size is used per model, and that preprocessing and processing (e.g., model inference) occur on the same device. In addition, this embodiment utilizes an exemplary object detection dataset containing multiple exemplary images with bounding boxes for annotation.

[0064] Further reference to the exemplary embodiment, Figure 6 shows a table 600 containing various values ​​for determining which machine learning model should be used to handle the workload. As shown in Figure 6, the horizontal axis shows the machine learning model names of the eight machine learning models, row 0 shows the image size, and row 1 shows the machine learning model accuracy measured in terms of mean average precision (mAP). The calculated power, memory, and inference time for each model on the client autonomous vehicle are shown in rows 2-4, respectively. The time for transmitting images from the autonomous vehicle to the server, the inference time on the server (e.g., processing time), and the total time on the server are shown in rows 5-7, respectively. The calculated accuracy function is then shown in row 8, where α=5 and β=1, and the calculated optimization function for the autonomous vehicle is shown in row 9 (where the optimization function for the server is the same as in row 8, as the server's power is assumed to be zero).

[0065] Referring further to the exemplary embodiment shown in Figure 6, the memory cutoff (mem max The available free memory on the autonomous vehicle is 2000MB. Furthermore, the deployment parameters specify that it should be possible to process at least 4 images per second (250 milliseconds per image). Thus, many of the models are eliminated from consideration based on their values ​​in line 7 (for the cloud) and line 2 (for the client). For the client-side models, only the four fastest models (TinyYOLOv3, TinyYOLOv4, YOLOv4, and YOLOv5) are retained, and for the server-side models, only two (TinyYOLOv3 and TinyYOLOv4) are retained. For these models, the optimization functions for the server (line 8) and the client (line 9) are compared, and as a result, model TinyYOLOv4 is selected for execution on the server because it has the highest value.

[0066] Various embodiments of the present invention provide a computer implementation method comprising the steps of: receiving information about a machine learning workload to be processed for a client device; determining, at least in part, a machine learning model for the workload and a processing location for the workload based on the information; and generating a request to process the workload at the determined processing location using the determined machine learning model.

[0067] In various embodiments, the determined processing location is the client device, and in various embodiments, the determined processing location is a server. In various embodiments, the step of determining the processing location for the workload includes the step of selecting the server from a set of available servers.

[0068] In various embodiments, the workload includes an input image. In various embodiments, the method further comprises, at least in part, deciding to preprocess the input image before processing the workload based on the information, and at least in part, determining a preprocessing location based on the information. In various embodiments, the determined processing location is a server, and the determined preprocessing location is the client device. In various embodiments, the method further comprises, at least in part, determining an output image size for the preprocessing of the input image based on the information.

[0069] In various embodiments, the information identifies (i) one or more processing time requirements, and (ii) available processing resources on the client device. In various embodiments, the information further identifies one or more machine learning model accuracy requirements. In various embodiments, the information further identifies (i) that the workload is located on the client device, and (ii) the transmission rate for sending the workload between the client device and the server.

[0070] In various embodiments, the step of determining the machine learning model for the workload and the processing location for the workload includes the step of computing an optimization function using integer linear programming.

[0071] In various embodiments, the client device is an autonomous vehicle, and the workload is selected from the group consisting of (i) object detection workloads, (ii) object recognition workloads, (iii) radar positioning workloads, (iv) lidar (light detection and ranging) workloads, (v) spatial detection workloads, (vi) automatic parking workloads, and (vii) navigation workloads.

[0072] While this disclosure includes a detailed description of cloud computing, it should be understood in advance that the implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in combination with any other type of computing environment that is currently known or may be developed in the future.

[0073] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and deployed with minimal administrative effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0074] The characteristics are as follows:

[0075] On-demand self-service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers.

[0076] Broad network access: This capability is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0077] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. While consumers generally have no control or knowledge of the exact location of the resources provided, there is location independence in that they may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0078] Rapid resilience: This capability allows for rapid and elastic provisioning, sometimes automatically, enabling quick scaling out and rapid release and rapid scaling in. To consumers, the capacity available for provisioning often appears unlimited and can be purchased in any quantity at any time.

[0079] Services measured: Cloud systems automatically control and optimize resource usage by leveraging metric capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, thereby providing transparency to both service providers and consumers.

[0080] The service model is as follows:

[0081] Software as a Service (SaaS): The ability provided to consumers is the use of a provider's applications running on cloud infrastructure. These applications are accessible from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0082] Platform as a Service (PaaS): The ability provided to consumers is to deploy applications they have created or acquired, written using programming languages ​​and tools supported by the provider, on cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they do control the deployed applications and, in some cases, the configuration of the application hosting environment.

[0083] Infrastructure as a Service (IaaS): The ability provided to consumers is to provision processing, storage, networking, and other basic computing resources, where consumers can deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do control the operating system, storage, and deployed applications, and in some cases have limited control over selected networking components (e.g., host firewalls).

[0084] The deployment model is as follows:

[0085] Private Cloud: This cloud infrastructure operates solely for a specific organization. The private cloud may be managed by that organization or a third party and may reside on-premises or off-premises.

[0086] Community Cloud: This cloud infrastructure is shared by several organizations and supports a specific community that shares common interests (e.g., mission, security requirements, policies, and compliance considerations). The community cloud may be managed by those organizations or third parties and may reside on-premises or off-premises.

[0087] Public Cloud: This cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0088] Hybrid Cloud: This cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain independent entities but are joined together by standard or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0089] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing lies an infrastructure that includes a network of interconnected nodes.

[0090] Referring now to Figure 7, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 comprises one or more cloud computing nodes 10 that can communicate with local computing devices used by cloud consumers, such as personal digital assistants (PDAs) or mobile phones 54A, desktop computers 54B, laptop computers 54C, or automotive computer systems 54N, or a combination thereof. The nodes 10 may communicate with each other. The nodes 10 may be physically or virtually grouped within one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or a combination thereof, as described above in this specification (not shown). This enables the cloud computing environment 50 to provide infrastructure, platforms, or software, or a combination thereof, as a service that does not require cloud consumers to maintain resources on their local computing devices for that purpose. The types of computing devices 54A-N shown in Figure 7 are for illustrative purposes only, and it should be understood that the computing node 10 and the cloud computing environment 50 can communicate with any type of computerized device via any type of network, or a network addressable connection, or both (for example, using a web browser).

[0091] Referring now to Figure 8, a set of functional abstraction layers provided by the cloud computing environment 50 (Figure 7) is shown. The components, layers, and functionalities shown in Figure 8 are intended to be illustrative only, and it should be understood in advance that embodiments of the present invention are not limited to them. As shown, the following layers and corresponding functionalities are provided:

[0092] The hardware and software layer 60 comprises hardware components and software components. Examples of hardware components include a mainframe 61, a RISC (Reduced Instruction Set Computer) architecture-based server 62, a server 63, a blade server 64, a storage device 65, and network and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0093] The virtualization layer 70 provides an abstraction layer that may provide examples of virtual entities, including virtual servers 71, virtual storage 72, virtual networks 73 including virtual private networks, virtual applications and operating systems 74, and virtual clients 75.

[0094] In one example, the management layer 80 may provide the functions described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 82 provides cost tracking as resources are used within the cloud computing environment and charges or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides cloud computing resource allocation and management to ensure that required service levels are met. Service level agreement (SLA) planning and execution 85 provides pre-arrangements and procurement of cloud computing resources where future requirements are expected to conform to the SLA.

[0095] The workload layer 90 provides examples of functions that can be utilized in a cloud computing environment. Examples of workloads and functions that can be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analysis processing 94, transaction processing 95, and providing soothing output 96.

[0096] Figure 9 shows a block diagram 900 of the components of a computer system 120 and a client device 130 according to an exemplary embodiment of the present invention. It should be understood that Figure 9 merely provides an example of one implementation and does not imply any limitation with respect to environments in which different embodiments may be implemented. Many modifications may be made to the environment shown.

[0097] The computer system 120 and client device 130 include a communication fabric 902 that provides communication between a computer processor 904, memory 906, persistent storage 908, a communication unit 910, and an input / output (I / O) interface 912. The communication fabric 902 can be implemented in any architecture designed to pass data or control information, or both, between processors (e.g., microprocessors, communication and network processors, etc.), system memory, peripheral devices, and any other hardware components in the system. For example, the communication fabric 902 can be implemented using one or more buses.

[0098] The memory 906 and persistent storage 908 are computer-readable storage media. In this embodiment, the memory 906 includes random access memory (RAM) 914 and cache memory 916. Generally, the memory 906 can include any suitable volatile or non-volatile computer-readable storage media.

[0099] The deployment optimization program 122, computer interface 124, server workload program 126, database 128, client program 132, client interface 134, client workload program 136, and sensor 138 are stored in persistent storage 908 for execution or access, or both, by one or more of the respective computer processors 904 via one or more of the memories 906. In this embodiment, persistent storage 908 includes a magnetic hard disk drive. As an alternative to, or in addition to, a magnetic hard disk drive, persistent storage 908 may include a solid-state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0100] The media used by persistent storage 908 may be removable. For example, a removable hard drive may be used for persistent storage 908. Other examples include optical and magnetic disks, thumb drives, and smart cards inserted into the drive for transfer to another computer-readable storage medium, which is also part of persistent storage 908.

[0101] In these examples, the communication unit 910 provides communication with other data processing systems or devices (including resources of the network 110). In these examples, the communication unit 910 includes one or more network interface cards. The communication unit 910 may provide communication through the use of either or both physical communication links and wireless communication links. The deployment optimization program 122, computer interface 124, server workload program 126, database 128, client program 132, client interface 134, client workload program 136, and sensor 138 may be downloaded to persistent storage 908 via the communication unit 910.

[0102] The I / O interface 912 enables the input and output of data to and from other devices that may be connected to the computer system 120 and the client device 130. For example, the I / O interface 912 may provide a connection to an external device 918 such as a keyboard, keypad, touchscreen, or any other suitable input device or combination thereof. The external device 918 may also include portable computer-readable storage media such as thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to carry out embodiments of the present invention, such as the deployment optimization program 122, computer interface 124, server workload program 126, database 128, client program 132, client interface 134, client workload program 136, and sensor 138, can be stored on such portable computer-readable storage media and can be loaded onto persistent storage 908 via the I / O interface 912. The I / O interface 912 is also connected to the display 920.

[0103] The display 920 provides a mechanism for displaying data to the user, and may be, for example, a computer monitor or a television screen.

[0104] The present invention may be a system, method, or computer program product, or a combination thereof. The computer program product may include one or more computer-readable storage media having computer-readable program instructions that cause a processor to execute an aspect of the present invention.

[0105] A computer-readable storage medium can be a tangible device capable of holding and storing instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of those described above. A non-exhaustive list of more specific examples of computer-readable storage media includes, namely, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multipurpose disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved raised structures recording instructions, and any suitable combination of those described above. When used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0106] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers such computer-readable program instructions for storage in a computer-readable storage medium within each computing / processing device.

[0107] The computer-readable program instructions that perform the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, where one or more programming languages ​​include object-oriented programming languages ​​such as Smalltalk®, C++, etc., and conventional procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or fully on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), and the connection may be to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized by executing computer-readable program instructions using state information of computer-readable program instructions in order to perform an aspect of the present invention.

[0108] Aspects of the present invention are described herein with reference to flowcharts or block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or both, and any combination of blocks in a flowchart or block diagram, or both, can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions may be provided to the processor of a general-purpose computer, a dedicated computer, or another programmable data processing device to generate a machine, thereby creating means for implementing functions / operations specified in one or more blocks of a flowchart or block diagram, or both, through the instructions executed via the processor of the computer or other programmable data processing device. Furthermore, these computer-readable program instructions may be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing device, or another device, or a combination thereof, to function in a specific manner, thereby including a product containing instructions that implement modes of functions / operations specified in one or more blocks of a flowchart or block diagram, or a combination thereof.

[0110] Furthermore, computer-readable program instructions may be loaded into a computer, other programmable data processing device, or other device to execute a series of operational steps on the computer, other programmable device, or other device, thereby generating a computer implementation process in which the instructions executed on the computer, other programmable device, or other device implement functions / operations specified by one or more blocks of a flowchart or block diagram, or a combination thereof.

[0111] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions that implement a specified logical function. In some alternative implementations, the functions described in a block may be performed in an order different from the order shown in the drawings. For example, two blocks shown consecutively may actually be executed substantially simultaneously, and blocks may be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, or both, and any combination of blocks in a block diagram or flowchart, or both, may be implemented by a dedicated hardware-based system that performs a specified function or operation, or a combination of dedicated hardware and computer instructions.

[0112] The programs described herein are identified based on the applications in which they are implemented in specific embodiments of the present invention. However, any particular program names used herein are for convenience only, and it should be understood that the present invention should not be limited to use only in any specific application identified, implied, or both by such names.

[0113] For example, terms such as "Smalltalk" may be trademarked in various jurisdictions around the world, and it should be noted that here they are used only in reference to products or services that are appropriately named by such trademarks, to the extent that such trademarks exist.

Claims

1. A step of receiving information including deployment parameters and optimization target data relating to a machine learning workload processed for the client device by one or more processors, including a client-side processor provided in the client device and a server-side processor provided in the server. A step in which one or more processors, including the client-side processor and the server-side processor, determine, at least partially, a machine learning model for the machine learning workload and a processing location for the machine learning workload based on the information, wherein the processing location is at least one of the client-side processor and the server-side processor, A step of generating a request to process the machine learning workload at the determined processing location using the determined machine learning model, using one or more processors, including the client-side processor and the server-side processor. A computer implementation method comprising the following features.

2. The aforementioned machine learning workload includes an input image, The aforementioned computer implementation method is A step of deciding, at least partially, to preprocess the input image before processing the machine learning workload based on the information, using one or more processors, including the client-side processor and the server-side processor; A step in which one or more processors, including the client-side processor and the server-side processor, determine a preprocessing location based on the information, where the preprocessing location is at least one of the client-side processor and the server-side processor. The computer implementation method according to claim 1, further comprising the following:

3. The computer implementation method according to claim 2, wherein the determined processing location is the client device.

4. The computer implementation method according to claim 2, wherein the determined processing location is a server.

5. The computer implementation method according to claim 4, wherein the step of determining the processing location for the machine learning workload includes the step of selecting the server from a set of available servers.

6. The processing location determined above is a server, The computer implementation method according to claim 2, wherein the determined preprocessing location is the client device.

7. A step of determining, at least partially, the output image size for the preprocessing of the input image based on the information, using one or more processors, including the client-side processor and the server-side processor. The computer implementation method according to claim 2, further comprising the above.

8. The computer implementation method according to claim 2, wherein the information identifies (i) one or more processing time requirements, and (ii) available processing resources on the client device.

9. The computer implementation method according to claim 8, wherein the information further identifies one or more machine learning model accuracy requirements.

10. A step of receiving information including deployment parameters and optimization target data relating to a machine learning workload processed for the client device by one or more processors, including a client-side processor provided in the client device and a server-side processor provided in the server. A step of determining, at least partially, a machine learning model for the machine learning workload and a processing location for the machine learning workload based on the information, using one or more processors, including the client-side processor and the server-side processor; A step of generating a request to process the machine learning workload at the determined processing location using the determined machine learning model, using one or more processors, including the client-side processor and the server-side processor. Equipped with, A computer implementation method that identifies (i) one or more processing time requirements, (ii) available processing resources on the client device, (iii) the machine learning workload being located on the client device, and (iv) the transmission rate for sending the machine learning workload between the client device and the server.

11. A step of receiving information including deployment parameters and optimization target data relating to a machine learning workload processed for the client device by one or more processors, including a client-side processor provided in the client device and a server-side processor provided in the server. A step of determining, at least partially, a machine learning model for the machine learning workload and a processing location for the machine learning workload based on the information, using one or more processors, including the client-side processor and the server-side processor; A step of generating a request to process the machine learning workload at the determined processing location using the determined machine learning model, using one or more processors, including the client-side processor and the server-side processor. Equipped with, A computer implementation method comprising the step of determining the machine learning model for the machine learning workload and the processing location for the machine learning workload, the step of calculating an optimization function using integer linear programming.

12. The computer implementation method according to any one of claims 1 to 11, wherein the client device is an autonomous vehicle.

13. The computer implementation method according to claim 12, wherein the machine learning workload is selected from the group consisting of (i) object detection workload, (ii) object recognition workload, (iii) radar positioning workload, (iv) lidar (light detection and ranging) workload, (v) spatial detection workload, (vi) automatic parking workload, and (vii) navigation workload.

14. A computer program comprising program instructions, wherein the program instructions are: A program instruction that receives information including deployment parameters and optimization target data for a machine learning workload processed for a client device, A program instruction that determines, at least in part, a machine learning model for the machine learning workload and a processing location for the machine learning workload based on the information, wherein the processing location is at least one of a client-side processor located on the client device and a server-side processor located on the server side. A program instruction that generates a request to process the machine learning workload at the determined processing location using the determined machine learning model. A computer program that has [a certain characteristic].

15. The machine learning workload includes an input image, The aforementioned computer program, A program instruction that, at least in part, determines to preprocess the input image before processing the machine learning workload based on the information, The computer program according to claim 14, further comprising, at least in part, a program instruction for determining a preprocessing location based on the aforementioned information, wherein the preprocessing location is at least one of the client-side processor and the server-side processor.

16. A computer program comprising program instructions, wherein the program instructions are: A program instruction that receives information including deployment parameters and optimization target data for a machine learning workload processed for a client device, Program instructions that determine, at least in part, a machine learning model for the machine learning workload and a processing location for the machine learning workload based on the information, A program instruction that generates a request to process the machine learning workload at the determined processing location using the determined machine learning model. It has, The information identifies (i) one or more processing time requirements, (ii) available processing resources on the client device, (iii) the machine learning workload located on the client device, and (iv) the transmission rate for sending the machine learning workload between the client device and the server. Computer program.

17. A computer program comprising program instructions, wherein the program instructions are: A program instruction that receives information including deployment parameters and optimization target data for a machine learning workload processed for a client device, Program instructions that determine, at least in part, a machine learning model for the machine learning workload and a processing location for the machine learning workload based on the information, A program instruction that generates a request to process the machine learning workload at the determined processing location using the determined machine learning model. It has, The program instructions for determining the machine learning model for the machine learning workload and the processing location for the machine learning workload include program instructions for calculating an optimization function using integer linear programming. Computer program.

18. The computer program according to any one of claims 14 to 17, wherein determining the processing location for the machine learning workload includes determining whether or not to process the machine learning workload on the client device or on a server.

19. The computer program according to claim 18, wherein the client device is an autonomous vehicle.

20. A plurality of processors including a client-side processor provided in a client device and a server-side processor provided on the server side, One or more computer-readable storage media, A program instruction stored on one or more computer-readable storage media, which is executed by at least one of a plurality of processors, including the client-side processor and the server-side processor, wherein the stored program instruction is A program instruction that receives information including deployment parameters and optimization target data for a machine learning workload processed for a client device, A program instruction that determines, at least in part, a machine learning model for the machine learning workload and a processing location for the machine learning workload based on the information, wherein the processing location is at least one of the client-side processor and the server-side processor, A program instruction that generates a request to process the machine learning workload at the determined processing location using the determined machine learning model. Program instructions and A computer system equipped with the following features.

21. The machine learning workload includes an input image, The stored program instructions are A program instruction that, at least in part, determines to preprocess the input image before processing the machine learning workload based on the information, A program instruction that determines a preprocessing location based at least partially on the aforementioned information, wherein the preprocessing location is at least one of the client-side processor and the server-side processor. The computer system according to claim 20, further comprising:

22. A plurality of processors including a client-side processor provided in a client device and a server-side processor provided on the server side, One or more computer-readable storage media, A program instruction stored on one or more computer-readable storage media, which is executed by at least one of a plurality of processors, including the client-side processor and the server-side processor, wherein the stored program instruction is A program instruction that receives information including deployment parameters and optimization target data for a machine learning workload processed for a client device, Program instructions that determine, at least in part, a machine learning model for the machine learning workload and a processing location for the machine learning workload based on the information, A program instruction that generates a request to process the machine learning workload at the determined processing location using the determined machine learning model. Program instructions and Equipped with, The information identifies (i) one or more processing time requirements, (ii) available processing resources on the client device, (iii) the machine learning workload located on the client device, and (iv) the transmission rate for sending the machine learning workload between the client device and the server. Computer system.

23. A plurality of processors including a client-side processor provided in the client device and a server-side processor provided on the server side, One or more computer-readable storage media, A program instruction stored on one or more computer-readable storage media, which is executed by at least one of a plurality of processors, including the client-side processor and the server-side processor, wherein the stored program instruction is A program instruction that receives information including deployment parameters and optimization target data for a machine learning workload processed for a client device, Program instructions that determine, at least in part, a machine learning model for the machine learning workload and a processing location for the machine learning workload based on the information, A program instruction that generates a request to process the machine learning workload at the determined processing location using the determined machine learning model. Program instructions and Equipped with, The program instructions for determining the machine learning model for the machine learning workload and the processing location for the machine learning workload include program instructions for calculating an optimization function using integer linear programming. Computer system.

24. The computer system according to any one of claims 20 to 23, wherein determining the processing location for the machine learning workload includes determining whether or not to process the machine learning workload on the client device or on a server.

25. The computer system according to claim 24, wherein the client device is an autonomous vehicle.

Images