Control the performance of deep learning models deployed on edge devices via predictive models
The processor receives data and predicts model performance, adjusts control inputs to optimize deep learning models, solves the problems of resource limitation and difficulty in monitoring in edge devices, and realizes efficient and low-cost model performance management.
Patent Information
- Application Number
- CN202010861510.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-08-26
- Filing Date
- 2020-08-25
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-08-25
AI Technical Summary
In the Internet of Things and smart edge devices, the deployment of deep learning models faces the problems of resource limitation and difficulty in monitoring, especially in the offline state, which cannot effectively monitor and optimize model performance, resulting in poor performance and high computing costs.
The processor receives data input and predicts the performance of the model, and performs model prediction control based on rolling time domain technology, adjusts control inputs to optimize the performance of the deep learning model, and realizes real-time monitoring and performance improvements of the deployed models on edge devices.
Effectively monitor and optimize the performance of deep learning models on edge devices, ensure efficient operation in resource-constrained environments, reduce computing costs, and maintain model performance offline.
Smart Images

Figure CN112433733B_ABST
Abstract
Description
Technical Field
[0001] The present technology relates to deployed deep learning (DL) models. More specifically, the technology relates to controlling deployed DL models. Background Art
[0002] With the introduction of the Internet of Things (IoT) and smart edge devices, full lifecycle planning for machine learning (ML) models may include data collection and training, as well as deployment and monitoring. Summary of the invention
[0003] According to embodiments described herein, a system may include a processor for receiving data input and predicted performance of a DL model deployed on a resource-constrained edge device from a prediction model. The processor may further modify a control input of the DL model based on the data input or the predicted performance of the DL model. The processor may also send a control input to the deployed DL model to modify the performance of the DL model.
[0004] According to another embodiment described herein, a method may include monitoring, via a processor, input data and a predicted performance of a DL model from a predictive model. The method may also include modifying, via the processor, a control input based on the input data and the predicted performance of the DL model. The method may further include sending, via the processor, the modified control input to the DL model.
[0005] According to another embodiment described herein, a computer program product for controlling a deployed DL model may include a computer-readable storage medium having program code therein. The computer-readable storage medium is not the transient signal itself. The program code is executable by a processor to cause the processor to monitor input data and the predicted performance of the DL model from the prediction model. The program code may also cause the processor to modify the control input based on the input data and the predicted performance of the DL model. The program code may also cause the processor to send the modified control input to the DL model. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] Figure 1 is a block diagram of an example system for controlling a deployed DL model via a predictive model;
[0007] Figure 2 is a block diagram of an example system for training a predictive model to be used to control a deployed DL model;
[0008] Figure 3 is a block diagram of an example method by which a predictive model may be trained to control a deployed DL model;
[0009] Figure 4is a block diagram of an example method that can control a deployed DL model via a predictive model;
[0010] Figure 5A is a block diagram of an example computing device that can control a DL learning model via a predictive model;
[0011] Figure 5B is a block diagram of an example computing device that may be controlled using a DL learning model via a predictive model;
[0012] Figure 6 is a process flow diagram of an example cloud computing environment according to embodiments described herein;
[0013] Figure 7 is a process flow diagram of an example abstract model layer according to embodiments described herein;
[0014] Fig. 8A A tangible, non-transitory computer-readable medium that is an example of a DL model that can be deployed via predictive model control; and
[0015] Figure 8B is a tangible, non-transitory computer-readable medium that can store an example of a DL model to be deployed via predictive model control. DETAILED DESCRIPTION
[0016] With the introduction of the Internet of Things (IoT) and smart edge devices, the entire lifecycle planning of machine learning models (ML) may include data collection and training, as well as deployment and monitoring. Deploying ML models outside the cloud directly into mobile applications may face the additional challenge of supporting multiple platforms, hundreds of chipsets, and billions of installations. In addition, in the case of deep learning models (DL), higher performance is achieved at the expense of higher computational cost or memory consumption at runtime compared to other ML models.
[0017] Furthermore, when a DL model is deployed into a mobile device, the conditions in which the mobile application will be used can be detected and the training data can be augmented accordingly. For example, a model may achieve high accuracy detection on bright images from the training dataset, but may perform poorly in some real-world low-light settings encountered by smartphone users. Applying different data augmentation techniques in response to new situations can improve the performance of DL models in production.
[0018] After the application is deployed, the system can therefore continue to collect data, capture the inputs and outputs of the model running on the device, and monitor accuracy to improve the model over time. However, unlike cloud environments that feature robust logging infrastructure, deploying models on mobile devices may prevent developers from monitoring the performance of the model. For example, collecting data and monitoring accuracy may include measuring runtime performance, memory usage, battery consumption, accuracy, and all of these across heterogeneous hardware to ensure that the DL model runs efficiently and does not degrade the expected model performance. However, mobile devices may be resource-constrained and therefore cannot perform all of these resource-intensive analyses of DL model performance after deployment. In addition, in some cases, the mobile device may be offline, so models deployed on offline mobile devices may not be directly monitored by cloud services.
[0019] According to an embodiment of the present disclosure, a processor may receive data input and predicted performance of a DL model deployed on a resource-constrained edge device from a prediction model. The processor may modify the control input of the DL model based on the data input and the predicted performance. The processor may then send the control input to the deployed DL model to modify the performance of the DL model. Therefore, the technology described herein is able to effectively monitor DL models deployed on resource-constrained edge devices by using model predictive control based on receding horizon technology. For example, the prediction model may predict the model response deployed within a specified time range. Therefore, the technology provides a trained prediction model that is used as a controller for monitoring and changing the performance of the deployed model.
[0020] Reference now Figure 1 , a block diagram illustrating an example system for controlling a deployed DL model via a predictive model. The example system 100 may be used Figure 5A The computing device 500A and Figure 5B The computing device 500B or the processor 802 of the computer readable medium 800A and 800B, using Figure 4 The method 400 is implemented. Figure 1A fog layer 102 is included that is communicatively coupled to an edge layer 104. For example, the fog layer 102 can be a cloud layer that includes one or more servers for coordinating multiple computing devices in the edge layer 104. For example, the edge layer 104 can include multiple edge devices, such as mobile devices. An example edge device is shown in the computing device 500B of Figure 5B. The system 100 includes an input 106 received at the fog layer 102. For example, the input can be any suitable data input, such as an image or other form of data to be processed. The fog layer 102 includes an optimization unit 108 coupled to a data buffer 110 and a decision function unit 112. The edge layer 104 includes a prediction model 114 that is intermittently coupled to a deployment model 116.
[0021] The fog layer 102 also includes a first switch 118 that can be coupled to the nodes 120A, 120B, and 120C. As shown by arrow 122, the first switch 118 can switch the coupling between the nodes 120A, 120B, and 120C. Figure 1 In the example of FIG. 1 , the first switch 118 is configured to be coupled to the node 120B. The edge layer 104 further includes a second switch 124. The second switch 124 can be coupled to the nodes 126A and 126B. The second switch 124 can switch the coupling between the nodes 126A and 126B, as shown by arrow 128. For example, when the first switch 118 is switched to the node 120A, the second switch 124 can be switched to the node 126A. When the first switch 118 is switched to the node 120B, the second switch 124 can be switched to the node 126B.
[0022] exist Figure 1 In the example of FIG. 1 , the performance control system 100 includes a deployed model 116 to be controlled, a prediction model 114, an optimization unit 108, and a decision function unit 112. The prediction model 114 models the performance of the deployed model 116. The optimization unit 108 determines the input required to produce the target performance for the deployed model 116. The decision function unit 112 decides to switch to the deployed model 116 when the optimization unit 108 has solved the best input, or to switch to the prediction model 114 between samples. The optimization unit 108 uses the prediction model 114 to calculate the next control input based on the prediction of the prediction model 114. Once the objective function is minimized, the input is passed to the deployed model 116.
[0023] Still reference Figure 1, the system 100 may begin with an input signal 106in(n), which is submitted to an optimization unit 108. The optimization unit 108 generates an output signal d(n), which is submitted to a decision function unit 112. The decision function unit 112 generates an output, which is used as an input to a prediction model 114 via a node 120A, or as an input to a deployed model 116 via a node 120B, or as an input to a data buffer 110 via a node 120C. For example, when the optimization unit 108 has solved for an optimal control input that optimizes a predefined cost function, the first switch 118 may be set to the deployed model 116. In various examples, between samples, the first switch 118 may be set to the prediction model 114. For example, the optimization unit 108 may use the output of the prediction model 114 to calculate the next control input based on the prediction (pred(n)) of the deployed model 116. Once the optimization unit 108 minimizes the cost function, the control input may be passed to the deployed model 116. For example, the control input may be a number of parameter settings for the deployment model 116 .
[0024] As an example use case scenario, if the deployed model 116 is to analyze the video input 106 in real time, the system 100 can be used to ensure that the deployed model 116 runs at least 30 frames per second on all target mobile devices in the edge layer 104. In some examples, the system 100 can also set an alarm that notifies the developer when there is a significant change in the input data, a prediction that may indicate a failure, or a change in the way people use their applications. In this case, the input 106 is a set of frames. The processing of each frame may take a different amount of time and resources. For example, each frame may use a different amount of memory or battery resources. As an example, depending on the batch size of the input 106 being processed and other image parameters, the processing may be more efficient, but may use more resources such as memory. The optimization unit 108 can then determine the best batch size to use so as to maximize the total throughput of inference within a specified range. The specified range can be a predefined amount of time. In various examples, the system 100 can perform optimization, followed by prediction, and then decision making. For example, the system 100 can execute the following algorithm:
[0025] Initialize j
[0026] pred(n)=M_p(In(n),s_i)
[0027] While J(N,pred(n))}>ζdo
[0028] Repeat
[0029] D←In(n)
[0030] j=j+1
[0031] pred(n+j)=M_p(In(n+j),s_i){i∈R:1≤i≤n}
[0032] u(n+j)←arg min{J(N,pred(n+j))}
[0033] End do
[0034] Return
[0035] For s∈L
[0036] Y(u)={M_d(u)}
[0037] s_i(u)=K(M_d(u),Y(u))
[0038] Train M_p(u,s_i(u))
[0039] end
[0040] Among them, M_d is the deployed model, M_p is the prediction model, In is the new input data, s_i is a set of edge-related performance indicators, D is the data buffer, J is the cost function, N is the cost range, u is the control signal, ζ is the threshold, and K is the performance estimator.
[0041] It should be understood that Figure 1 The block diagram is not intended to indicate that system 100 will include Figure 1 Instead, system 100 may include Figure 1 Fewer or additional components not shown (e.g., additional inputs, layers, models or additional data buffers, units, etc.).
[0042] Figure 2 is a block diagram illustrating an example system for training a predictive model for use in controlling a deployed DL model. The example system 200 may be used Figure 5A A computing device 500A or a processor 802 having a computer readable medium 800A, using Figure 3 The method 300 is implemented. Figure 2 The deployment model 202 is coupled to a performance estimator 204. For example, the deployment model 202 may be Figure 1The deployed model 116 of the embodiment of the present invention is further coupled to the prediction model 206 and can be used to estimate the performance of the deployed model 202 given the training input 208. For example, the prediction model 206 can be the prediction model 114 of FIG. 1. The deployed model 202 can generate an output 210 based on the training input 208. For example, the output 210 can be a detection or classification of the training input 208. The prediction model 206 is coupled to a prediction error calculator 212.
[0043] exist Figure 2 In the example of using the prediction model 206 as Figure 1 Prior to the controller, the prediction model 206 can be trained to learn the forward dynamics of the deployed model 202. For example, by applying a series of inputs 208 to the deployed model 202, a corresponding output 210 can be obtained. The system 200 can then select a set of training patterns based on the output 210. Once the prediction model 206 is trained with the collected data, the prediction model 206 can be used to predict the future performance of the deployed model 202 based on edge performance-related metrics. For example, edge performance-related metrics can include runtime performance, or memory usage, or battery consumption, or accuracy, and other performance-related metrics. In various examples, the predictions from the prediction model 206 can then be used by the optimization algorithm to determine a control input that minimizes a predefined performance criterion within a specified time frame, such as Figure 1 As described in.
[0044] like Figure 2As shown, the prediction model and the deployment model 202 receive the same input 208In(n). In various examples, the performance estimator 204 calculates selected edge-related metrics, such as inference time, memory usage, battery consumption, etc. The prediction model 206 may also have additional inputs from the performance estimator 204. In various examples, the additional inputs help the prediction model capture the dynamic characteristics of the deployment model 202. For example, if the deployment model 202 is a deep neural network (DNN) model, the dynamic characteristics of the deployment model 202 may include the time it takes for the DNN model to perform inference on the training data 208 including a given set of input images. The prediction error calculator 212 may calculate the prediction error between the response pred(n) of the network and the output perf(n) of the performance estimator. The prediction error may then be used to update the structure of the prediction model 206. For example, multiple parameters of the prediction model 206 may be modified based on the prediction error. In some examples, the process is repeated until the prediction error is reduced to a predefined level that is insignificant. For example, the process may be repeated until the prediction error does not exceed a predetermined error threshold. As an example, if the prediction model 206 is a neural network, the training data 208 may cover the entire range of operations in which the model is deployed, since non-linear neural networks may not be able to extrapolate accurately.
[0045] It should be understood that Figure 2 The block diagram is not intended to indicate that system 200 will include Figure 2 Instead, system 200 may include Figure 2 Fewer or additional components not shown (e.g., additional training inputs, deployed models, outputs, models or additional performance estimators, etc.).
[0046] Figure 3 is a process flow diagram of an example method by which a predictive model can be trained to control a deployed DL model. The method 300 can be implemented using any suitable computing device, such as Figure 5A The computing device 500A and the reference Figure 2 For example, the method described below may be performed by Figure 5A The processor 502 or Fig. 8A The method 300 may be implemented by the processor 802 of the system. The method 300 may be used to train a prediction model based on a prediction error between a performance estimate and a predicted performance of the prediction model.
[0047] At block 302, training data is received. For example, the training data may be a set of images or video frames, as well as other types of suitable training data.
[0048] At block 304, an output of the deployment model is generated based on the training data. For example, the output may be a classification of a detected object or image. In some examples, the output may also include runtime performance information, such as a timestamp or processing duration for detection or classification, memory usage statistics, battery usage statistics, and accuracy information.
[0049] At block 306, a performance estimate is generated based on the training data and the output of the deployed model. For example, the performance estimate can be an estimated runtime performance, an estimated memory usage, an estimated battery usage, or an estimated accuracy. In some examples, the performance estimate can be a measurement that provides a performance estimate within a specified time frame.
[0050] At block 308, a predicted performance is generated based on the training data and the performance estimate. For example, the predicted performance may be the predicted performance of the deployed model given input data within a specified time range.
[0051] At block 310, a prediction error is calculated based on the prediction and the performance estimate. For example, the prediction error may be the difference between the prediction and the performance estimate.
[0052] At block 312, the prediction model is updated based on the calculated prediction error. For example, any number of parameters of the prediction model may be updated to reduce the prediction error.
[0053] Figure 3 The process flow diagram of the method 300 is not intended to indicate that all operations of the method 300 are included in every case. In addition, the method 300 may include any suitable number of additional operations. For example, additional iterations of the method 300 may be performed so that the prediction model is modified based on the prediction error until the prediction error does not exceed the threshold error.
[0054] Figure 4 is a process flow diagram of an example method that can control a deployed DL model via a predictive model. The method 400 can be implemented using any suitable computing device, such as Figure 5A and Figure 5B The computing devices 500A and 500B, and reference Figure 1 For example, the method described below may be performed by Figure 5A and Figure 5B The processor 502 or Fig. 8A and Figure 8B The processor 802 is implemented.
[0055] At block 402, input data and prediction performance of a DL model from a prediction model are monitored. For example, the input data may include images or video frames. The prediction performance may be a measure of runtime performance, memory usage, battery consumption, or accuracy.
[0056] At block 404, the control input is modified based on the predicted performance of the DL model or the input data from the predictive model. For example, the control input may be modified in response to detecting a change in the input data exceeding a threshold. In some examples, the control input may be modified in response to detecting a failure of the DL model. In various examples, the control input is modified to optimize a predefined performance criterion within a specified time frame.
[0057] At block 406, the modified control input is sent to the DL model. For example, the modified control input can be sent to the DL model deployed on the edge device. In various examples, the modified control input can also be received and processed by the prediction model on the edge device.
[0058] Figure 4 The process flow diagram of the method 400 is not intended to indicate that all operations of the method 400 are included in every case. In addition, the method 400 may include any suitable number of additional operations. For example, the method 400 may include generating an alarm in response to detecting a change in the predictive performance of the DL model or a failure of the DL model.
[0059] In various scenarios, the techniques described herein can be implemented in a cloud computing environment. As discussed in more detail below with reference to at least FIGS. 5-8 , a computing device configured to control a deployed DL model via a predictive model can be implemented in a cloud computing environment. It is understood in advance that although the present disclosure may include descriptions about cloud computing, the implementation of the teachings recorded herein is not limited to a cloud computing environment. On the contrary, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.
[0060] Cloud computing is a service delivery model for convenient, on-demand network access to a shared pool of configurable computing resources. Configurable computing resources are resources that can be quickly deployed and released with minimal management cost or interaction with the service provider, such as networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.
[0061] Features include:
[0062] On-demand self-service: Cloud consumers can unilaterally and automatically deploy computing capabilities such as server time and network storage on demand without human interaction with the service provider.
[0063] Broad network access: Computing power can be accessed over the network through standard mechanisms that facilitate the use of the cloud through different types of thin-client or thick-client platforms (e.g., mobile phones, laptops, personal digital assistants (PDAs)).
[0064] Resource pool: The computing resources of the provider are grouped into resource pools and serve multiple consumers through a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated on demand. Generally, consumers cannot control or even know the exact location of the provided resources, but can specify the location (such as country, state or data center) at a higher level of abstraction, so it is location-independent.
[0065] Rapid elasticity: The ability to quickly and elastically (sometimes automatically) deploy computing power to scale up quickly, and quickly release it to scale down quickly. To consumers, the computing power available for deployment often appears to be unlimited, and any amount of computing power can be accessed at any time.
[0066] Measurable services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.
[0067] The service model is as follows:
[0068] Software as a Service (SaaS): The capability provided to consumers is to use the provider's applications running on a cloud infrastructure. Applications can be accessed from a variety of client devices through a thin client interface such as a web browser (e.g., web-based email). Aside from limited user-specific application configuration settings, consumers neither manage nor control the underlying cloud infrastructure including networks, servers, operating systems, storage, or even individual application capabilities.
[0069] Platform as a Service (PaaS): The capability provided to consumers is to deploy applications created or acquired by consumers on the cloud infrastructure, which are created using programming languages and tools supported by the provider. Consumers neither manage nor control the underlying cloud infrastructure including networks, servers, operating systems or storage, but have control over the applications they deploy and may also have control over the configuration of the application hosting environment.
[0070] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, network, and other basic computing resources on which consumers can deploy and run arbitrary software including operating systems and applications. Consumers neither manage nor control the underlying cloud infrastructure, but have control over the operating system, storage, and applications they deploy, and may have limited control over selected network components (such as host firewalls).
[0071] The deployment model is as follows:
[0072] Private Cloud: Cloud infrastructure is run solely for an organization. The cloud infrastructure can be managed by the organization or a third party and can exist inside or outside the organization.
[0073] Community Cloud: A cloud infrastructure is shared by several organizations and supports a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). A community cloud can be managed by multiple organizations within the community or by a third party and can exist inside or outside the community.
[0074] Public cloud: Cloud infrastructure is provided to the public or to large industry groups and is owned by an organization that sells cloud services.
[0075] Hybrid cloud: A cloud infrastructure consisting of two or more clouds (private, community or public) in a deployment model that remain distinct entities but are bound together by standardized or proprietary technologies (such as cloud burst sharing for load balancing between clouds) that enable data and application portability.
[0076] The cloud computing environment is service-oriented, with characteristics centered on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is the infrastructure that includes a network of interconnected nodes.
[0077] Figure 5A 500A is a block diagram of an example computing device that can control a DL model deployed via a predictive model. The computing device 500A can be, for example, a server, a desktop computer, a laptop computer, or a tablet computer. In some examples, the computing device 500A can be a cloud computing node. The computing device 500A can be described in the general context of computer system executable instructions such as program modules executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. The computing device 500A can be practiced in a distributed cloud computing environment, in which tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located in local and remote computer system storage media, including memory storage devices.
[0078] The computing device 500A may include a processor 502 for executing stored instructions, and a memory device 504 for providing temporary memory space for the operation of the instructions during operation. The processor may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory device 504 may include random access memory (RAM), read-only memory, flash memory, or any other suitable memory system.
[0079] Processor 502 may be connected via system interconnect 506 (e.g., The computing device 500A is connected to an input / output (I / O) device interface 508, which is adapted to connect the computing device 500A to one or more I / O devices 510. The I / O devices 510 may include, for example, a keyboard and a pointing device, wherein the pointing device may include a touch pad or a touch screen, etc. The I / O devices 510 may be built-in components of the computing device 500A, or may be devices externally connected to the computing device 500A.
[0080] The processor 502 may also be linked to a display interface 512 adapted to connect the computing device 500A to a display device 514 via a system interconnect 506. The display device 514 may include a display screen that is a built-in component of the computing device 500A. The display device 514 may also include a computer monitor, a television, or a projector, etc., externally connected to the computing device 500A. In addition, a network interface controller (NIC) 516 may be adapted to connect the computing device 500A to a network 518 via the system interconnect 506. In some embodiments, the NIC 516 may transmit data using any suitable interface or protocol, such as an Internet small computer system interface, etc. The network 518 may be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, etc. An external computing device 520 may be connected to the computing device 500A via the network 518. In some examples, the external computing device 520 may be an external network server 520. In some examples, the external computing device 520 may be a cloud computing node.
[0081] The processor 502 may also be linked to a storage device 522 via a system interconnect 506, and the storage device 522 may include a hard drive, an optical drive, a USB flash drive, a drive array, or any combination thereof. In some examples, the storage device 522 may include a receiver module 524, a training module 526, and a prediction controller module 528. The receiver module 524 may receive data inputs and predicted performance of a DL model deployed on an edge device from a prediction model. For example, the data input may be an image or a video frame. In various examples, the predicted performance may include runtime performance, memory usage, battery consumption, accuracy, or any combination thereof. The training module 526 may train the prediction model based on a prediction error between a performance estimate and an output of the prediction model. For example, the output of the prediction model is based on the training input, and the performance estimate is based on the training input and the output of the DL model. The prediction controller module 528 may modify the control input of the DL model based on the data input and the predicted performance. For example, the prediction controller module 528 may be such as Figure 1The predictive model and optimization unit described above. In some examples, the predictive controller module 528 can modify the control input based on the predicted performance in response to detecting a change in the input data exceeding a threshold. In various examples, the predictive controller module 528 can modify the control input based on the predicted performance in response to detecting a failure of the DL model. The predictive controller module 528 can modify the control input to optimize a predefined performance criterion within a specified time frame. The predictive controller module 528 can then send the control input to the deployed DL model to modify the performance of the DL model.
[0082] It should be understood that Figure 5A The block diagram is not intended to indicate that computing device 500A will include Figure 5A Instead, computing device 500A may include Figure 5A 502). Fewer or additional components not shown in the embodiment (e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.) may be included. For example, the processor may switch the receipt of control inputs between a data buffer, a deployment model, and a prediction model. In addition, any functionality of the receiver module 524, the training module 526, and the controller module 528 may be implemented in part or in whole in hardware and / or the processor 502. For example, the functionality may be implemented using a dedicated integrated circuit, logic implemented in an embedded controller, or logic implemented in the processor 502, etc. In some embodiments, the functionality of the receiver module 524, the training module 526, and the controller module 528 may be implemented using logic, where the logic referred to herein may include any suitable hardware (e.g., a processor, etc.), software (e.g., an application, etc.), firmware, or any suitable combination of hardware, software, and firmware.
[0083] Figure 5B is a block diagram of an example computing device that can be controlled using a DL learning model via a predictive model. Figure 5B The computing device 500B has Figure 5A Like numbered elements. Computing device 500B may be, for example, an edge device such as a laptop, tablet, or smartphone.
[0084] Storage device 522 of computing device 500B also includes receiver module 536. In various examples, receiver module 536 receives input from external computing device 520. For example, the input may be a modified input from computing device 500A of Figure 5A.
[0085] The storage device 522 of the computing device 500B also includes a prediction model module 538. In some examples, the prediction model module 538 can receive a prediction model that is trained to predict the output of the deployed model. For example, the computing device 500A can use the method 300 to train the prediction model. The prediction model module 538 can receive inputs and predict the output of the deployed model for the received inputs. For example, the prediction model module 538 can predict objects detected in an image or predict the classification of an image into one or more classifications. In various examples, the prediction model module 538 can send the prediction output to an external computing device 520, such as the computing device 500A.
[0086] The storage device 522 of the computing device 500B includes a deployment model module 540. The deployment model module 540 can receive a trained deployment model. The deployment model may have been trained to detect objects in an image or classify an image. The deployment model module 540 can receive inputs such as images and detect objects in the image or classify the image. In various examples, the deployment model module 540 can send the detection or classification to the prediction model module 538.
[0087] It should be understood that Figure 5B The block diagram is not intended to indicate that computing device 500B will include Figure 5B In contrast, computing device 500B may include Figure 5B 502. In some embodiments, the functions of the receiver module 536, the prediction model module 538, and the deployment model module 540 may be implemented in part or in whole in hardware and / or the processor 502. For example, the functions may be implemented using a dedicated integrated circuit, logic implemented in an embedded controller, or logic implemented in the processor 502. In some embodiments, the functions of the receiver module 536, the prediction model module 538, and the deployment model module 540 may be implemented using logic, wherein the logic referred to herein may include any suitable hardware (e.g., a processor, etc.), software (e.g., an application, etc.), firmware, or any suitable combination of hardware, software, and firmware.
[0088] Reference now Figure 6, which shows an exemplary cloud computing environment 600. As shown, the cloud computing environment 600 includes one or more cloud computing nodes 602 with which a local computing device used by a cloud computing consumer can communicate. The local computing device can be, for example, a personal digital assistant (PDA) or a mobile phone 604A, a desktop computer 604B, a laptop computer 604C and / or an automobile computer system 604N. The cloud computing nodes 602 can communicate with each other. The cloud computing nodes 602 can be physically or virtually grouped in one or more networks including but not limited to private clouds, community clouds, public clouds or hybrid clouds, or a combination thereof as described above (not shown in the figure). In this way, cloud consumers can request infrastructure as a service (IaaS), platform as a service (PaaS) and / or software as a service (SaaS) provided by the cloud computing environment 600 without having to maintain resources on local computing devices. It should be understood that Figure 6 The various types of computing devices 604A-N shown are merely illustrative, and cloud computing node 602 and cloud computing environment 600 can communicate with any type of computing device on any type of network and / or network addressable connection (eg, using a web browser).
[0089] Reference now Figure 7 , which shows a cloud computing environment 600 ( Figure 6 ) provides a set of functional abstraction layers. First of all, it should be understood that Figure 7 The components, layers and functions shown are only exemplary, and the embodiments of the present invention are not limited thereto. As shown in the figure, the following layers and corresponding functions are provided:
[0090] The hardware and software layer 700 includes hardware and software components. Examples of hardware components include: a mainframe, in one example, systems; servers based on the RISC (Reduced Instruction Set Computer) architecture, in one example the IBM System; IBM System; IBM systems; storage devices; networks and network components. Examples of software components include: network application server software, in one example IBM Application server software; and database software, in one example IBM Database software. (IBM, zSeries, pSeries, xSeries, BladeCenter, WebSphere and DB2 are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide).
[0091] The virtualization layer 702 provides an abstraction layer that can provide examples of the following virtual entities: virtual servers, virtual storage, virtual networks (including virtual private networks), virtual applications and operating systems, and virtual clients. In one example, the management layer 704 can provide the following functions: Resource provisioning function: providing dynamic acquisition of computing resources and other resources for performing tasks in a cloud computing environment; Metering and pricing function: tracking the cost of resource usage within the cloud computing environment and providing bills and invoices for this purpose. In one example, the resources can include application software licenses. Security function: providing identity authentication for cloud consumers and tasks, and providing protection for data and other resources. User portal function: providing access to the cloud computing environment for consumers and system administrators. Service level management function: providing allocation and management of cloud computing resources to meet the required service levels. Service level agreement (SLA) planning and fulfillment function: providing pre-arrangement and provision for future demand for cloud computing resources predicted according to the SLA.
[0092] The workload layer 706 provides examples of functions that may be implemented in a cloud computing environment. In this layer, examples of workloads or functions that may be provided include: mapping and navigation; software development and lifecycle management; virtual classroom teaching provision; data analysis processing; transaction processing; and
[0093] The present invention may be a system, a method or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.
[0094] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. The computer-readable storage medium used herein is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, a transmission medium transmitted through a waveguide or other transmission medium, or a transmission medium transmitted through a waveguide or other transmission medium.
[0095] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0096] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or codes or object codes written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.
[0097] Various aspects of the present technology are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present technology. It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0098] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0099] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0100] Reference now Fig. 8A , depicts a block diagram of a tangible non-transitory computer-readable medium 800A that may be used to control an example DL model that may be deployed via a predictive model. The tangible non-transitory computer-readable medium 800A may be accessed by a processor 802 via a computer interconnect 804. In addition, the tangible non-transitory computer-readable medium 800A may include instructions to direct the processor 802 to execute Figure 3 Method 300 and Figure 4 The operation code of method 400 is as follows.
[0101] The various software components discussed herein may be stored on a tangible, non-transitory computer-readable medium 800A, such as Fig. 8AAs shown. For example, the receiver module 806 includes code for monitoring input data and the predicted performance of the DL model from the prediction model. The training module 808 includes code for training the prediction model based on the prediction error between the performance estimate and the output of the prediction model. For example, the output of the prediction model can be based on the training input, and the performance estimate is based on the training input and the output of the DL model. The prediction controller module 810 includes code for modifying the control input based on the input data and the predicted performance of the DL model. The prediction controller module 810 also includes code for sending the modified control input to the DL model. For example, the prediction controller module 810 includes code for modifying the control input to optimize the predefined performance criteria within a specified time range. In some examples, the prediction controller module 810 includes code for modifying the control input based on the predicted performance in response to detecting a change in the input data exceeding a threshold. In various examples, the prediction controller module 810 includes code for modifying the control input based on the predicted performance in response to detecting a fault of the DL model. It should be understood that, depending on the specific application, any number of additional software components not shown in FIG. 8 may be included in the tangible non-transitory computer-readable medium 800. For example, the computer-readable medium 800 may also include code for generating an alert in response to detecting a change in the predictive performance of the DL model or a failure of the DL model.
[0102] Figure 8B is an example tangible non-transitory computer-readable medium that can store a DL model to be deployed via predictive model control. Figure 8B The computer readable medium 800B includes a Fig. 8A For example, the tangible, non-transitory computer-readable medium 800B may also be accessed by the processor 802 via the computer interconnect 804 .
[0103] The various software components discussed herein may be stored on a tangible, non-transitory computer-readable medium 800B, such as Figure 8BAs shown. For example, the receiver module 818 includes code for receiving input from an external computing device 520. For example, the input can be a modified input from a computing device executing the computer-readable medium 800A. The prediction model module 820 includes code for receiving a prediction model that is trained to predict the output of a deployment model. For example, the prediction model can be trained by the computing device 500A using the computer-readable medium 800A using the method 300. The prediction model module 820 can receive input and predict the output of the deployment model for the received input. For example, the prediction model module 820 may include code for predicting an object detected in an image or predicting one or more categories obtained by classifying an image. In various examples, the prediction model module 820 may include code for sending the prediction output to an external computing device 520 (such as the computing device 500A).
[0104] The deployment model module 822 includes code for receiving a trained deployment model. In various examples, the deployment model may have been trained to detect objects in an image or to classify an image. The deployment model module 822 may also include code for receiving an input (such as an image) and detecting objects in the image or classifying the image. In various examples, the deployment model module 822 may include code for sending the detection or classification to the prediction model module 820.
[0105] The flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a portion of an instruction, and the module, a program segment or a portion of an instruction contains one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions marked in the boxes may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified functions or actions, or can be implemented with a combination of dedicated hardware and computer instructions. It should be understood that depending on the specific application, Fig. 8A and Figure 8B Any number of additional software components not shown in the drawings may be included within the tangible, non-transitory computer-readable media 800A and 800B.
[0106] Embodiments of the present technology have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the present invention.
Claims
1. A system comprising a hardware processor, wherein the hardware processor is configured to: Receiving data input and predicted performance of a deep learning (DL) model deployed on a resource-constrained edge device from a prediction model trained to predict performance characteristics of the deployed DL model on the resource-constrained edge device, wherein the prediction model is trained based on training data and output from a performance estimator, the performance estimator estimating the performance of the deployed DL model based on the training data and output from the DL model generated for the training data; modifying a control input of a deployed DL model based on the data input and the predicted performance; as well as The modified control input is sent to the deployed DL model to modify the performance of the deployed DL model on the resource-constrained edge device.
2. The system according to claim 1, wherein: The hardware processor trains the prediction model based on a prediction error between a performance estimate and an output of the prediction model, wherein the output of the prediction model is based on a training input and the performance estimate is based on the training input and an output of the deployed DL model.
3. The system according to claim 1, wherein: The predicted performance includes runtime performance, memory usage, battery consumption, accuracy, or any combination thereof.
4. The system according to claim 1, wherein: The hardware processor modifies the control input based on the predicted performance in response to detecting a change in the data input exceeding a threshold.
5. The system according to claim 1, wherein: The hardware processor modifies the control input based on the predicted performance in response to detecting that the predicted performance includes a predicted failure of the deployed DL model.
6. The system according to claim 1, wherein: The control inputs are modified to optimize predefined performance criteria within a specified time frame.
7. The system according to claim 1, wherein: The hardware processor will switch the receipt of the control input between the data buffer, the deployed model, and the predictive model.
8. A computer-implemented method comprising: monitoring, via a processor, input data and predicted performance of a deep learning (DL) model deployed on a resource-constrained edge device from a prediction model trained to predict performance characteristics of the deployed DL model on the resource-constrained edge device, wherein the prediction model is trained based on training data and output from a performance estimator, the performance estimator estimating the performance of the deployed DL model based on the training data and output from the DL model generated for the training data; modifying, via the processor, a control input based on the input data and a predicted performance of the deployed DL model; as well as The modified control input is sent via the processor to the deployed DL model.
9. The computer-implemented method of claim 8, wherein: In response to detecting a change in the input data exceeding a threshold change, the control input is modified.
10. The computer-implemented method of claim 8, wherein: In response to detecting a change in the predictive performance of the deployed DL model, the control input is modified.
11. The computer-implemented method of claim 8, wherein: Modifying the control input includes modifying parameters of the deployed DL model to optimize a predefined performance criterion within a specified time frame.
12. The computer-implemented method of claim 8, comprising generating an alert in response to detecting a change in the predictive performance of the deployed DL model or a predictive failure of the deployed DL model.
13. The computer-implemented method of claim 8, comprising training the predictive model based on a prediction error between a performance estimate and a predicted performance of the predictive model, wherein: The performance estimate is based on training input and output performance characteristics of the deployed DL model, and the predicted performance of the prediction model is based on the training input and the performance estimate.
14. The computer-implemented method of claim 13, wherein: The prediction model is modified based on the prediction error until the prediction error does not exceed a threshold error.
15. A computer program product for controlling a deployed deep learning (DL) model, the computer program product comprising a computer-readable storage medium having program code therein, the program code being executable by a processor to cause the processor to: monitoring input data and predicted performance of a deep learning (DL) model deployed on a resource-constrained edge device from a prediction model trained to predict performance characteristics of the deployed DL model on the resource-constrained edge device, wherein the prediction model is trained based on training data and output from a performance estimator, the performance estimator estimating the performance of the deployed DL model based on the training data and output from the DL model generated for the training data; modifying a control input based on the input data and a predicted performance of the deployed DL model; as well as Send the modified control input to the deployed DL model.
16. The computer program product of claim 15, further comprising program code executable by a processor for modifying the control input based on the predicted performance in response to detecting a change in the input data exceeding a threshold.
17. The computer program product of claim 15, further comprising program code executable by a processor for modifying the control input based on the predicted performance in response to detecting that the predicted performance includes a predicted failure of the deployed DL model.
18. The computer program product of claim 15, further comprising program code executable by a processor for modifying the control input to optimize a predefined performance criterion within a specified time frame.
19. The computer program product of claim 15, further comprising program code executable by the processor for generating an alert in response to detecting a change in the predictive performance of the deployed DL model or a predictive failure of the deployed DL model.
20. The computer program product of claim 15, further comprising program code executable by a processor for training the prediction model based on a prediction error between a performance estimate and an output of the prediction model, wherein: The output of the prediction model is based on training inputs, and the performance estimate is based on the training inputs and the output of the deployed DL.
Citation Information
Patent Citations
Method and device for optimizing performance of GPU server
CN110032449A
Systems and methods for real-time forecasting and predicting of electrical peaks and managing the energy, health, reliability, and performance of electrical power systems based on an artificial adaptive neural network
US20090113049A1
Techniques for Mobility-Aware Dynamic Service Placement in Mobile Clouds
US20150245160A1
Systems and methods for mixed-signal computing
WO2018148293A1
A system for maintenance recommendation based on failure prediction
WO2019022737A1
