Deploy parallelizable deep learning models by adapting them to computing devices

Abstract

In an approach for deploying a parallelizable deep learning model by adapting it to a computing device, the deep learning model is divided into a plurality of slices, each slice capable of exchanging data with an associated slice. A virtual model is created from the plurality of slices, the virtual model based on capabilities of a plurality of devices on which the one or more virtual models are deployed, and further, each virtual model includes each slice of the plurality of slices. The one or more virtual models are stored in a cache. In response to determining that the deep learning model is to be deployed on the one or more devices, a candidate model is selected from the virtual models in the cache, the selection being based on information from a device monitor regarding the device.

Description

[Technical Field]

[0001] The present invention relates generally to the field of machine learning, and more particularly to deploying parallelizable deep learning models by adapting them to computing devices. [Background technology]

[0002] Machine learning (ML) is an application of artificial intelligence (AI) that gives systems the ability to automatically learn and improve from experience without being explicitly programmed. Machine learning is the use and development of computer systems that can learn and adapt without following explicit instructions, by using algorithms and statistical models to analyze patterns in data and draw inferences from them. Machine learning algorithms build models based on sample data, known as "training data," to make predictions or decisions. Machine learning algorithms are used in a wide range of applications, such as email filtering and computer vision, where it is difficult or impractical to develop traditional algorithms to perform the required tasks.

[0003] Deep learning is an artificial intelligence (AI) feature that mimics the human brain's ability to process data and create patterns for decision-making. Deep learning is a subset of machine learning in artificial intelligence, with networks capable of learning from unstructured or unlabeled data. Deep learning is a type of machine learning algorithm that uses multiple layers in a network to progressively extract higher-level features from raw input. For example, in image processing, lower layers can identify edges, while higher layers can use those edges to distinguish entire objects. Most modern deep learning models are based on artificial neural networks, specifically convolutional neural networks (CNNs). Deep learning's explosive growth is due to the intersection of deeply layered neural networks and the use of graphics processing units (GPUs) to accelerate their execution.

[0004] To improve model accuracy, data scientists tend to build deeper neural networks. However, as the complexity of neural networks increases, the model size becomes very large, significantly increasing the time it takes to transfer and deploy the model. Model initialization and execution consumes a large amount of processor and memory capacity, and it is difficult to relocate and balance trained models for high availability (HA).

[0005] There are various methods for compressing models for deployment to edge devices. Model compression can be broadly divided into two categories. The first category is pruning, which involves removing redundant connections present in the architecture. The problem with pruning is that the new model formed will be less accurate than the original model because the model was actually trained on the original connections. The second category is quantization, which clusters or rounds weights to allow the same number of connections to be represented using less memory. However, like pruning, quantization also reduces the accuracy of the model. As a result, edge devices often only have a single, non-robust model. The problem with this technique is that there is no way to deploy large models with high accuracy, high availability, and scalability without compression. Summary of the Invention

[0006] Embodiments of the present invention disclose computer-implemented methods, computer program products, and systems for deploying parallelizable deep learning models by adapting them to a computing device. In one embodiment, a deep learning model is divided into multiple slices, each capable of exchanging data with associated slices. A virtual model is created from the multiple slices, the virtual model being based on the capabilities of multiple devices on which the one or more virtual models are deployed, and each virtual model includes each of the multiple slices. The one or more virtual models are stored in a cache. In response to determining that the deep learning model is to be deployed on one or more devices, a candidate model is selected from the virtual models in the cache, the selection being based on information from a device monitor regarding the device. Embodiments of the present invention divide the model into slices based on the capabilities of the edge computing device, and select candidate models and model slices from the virtual model cache to construct an overall model based on the capabilities of the edge computing device.

[0007] In one embodiment, the deep learning model is divided into multiple slices, each slice being a different network layer among multiple different network layers of the deep learning model. Embodiments of the present invention divide the model into slices and deploy them to multiple edge devices.

[0008] In one embodiment, in response to determining that the deep learning model is not readily parallelizable, the deep learning model is partitioned into a plurality of slices based on a predetermined set of rules, the predetermined set of rules partitioning the deep learning model into a plurality of smallest parallelizable layers. Embodiments of the invention partition the model into slices and deploy them to a plurality of edge devices.

[0009] In one embodiment, a device monitor collects information from a plurality of devices on which a deep learning model can be deployed. Characteristics of the plurality of devices are encoded for each time slice, where the characteristics include, but are not limited to, at least one of central processing unit (CPU) capacity, graphics processing unit (GPU) capacity, and disk capacity, and the time slice is a predetermined time interval. A risk factor is predicted for each of the plurality of devices, where the risk factor is predicted using a long-short-term memory (LSTM) model. In response to determining that a risk factor of any of the plurality of devices exceeds a predetermined threshold, the top n devices are selected, where n is a predetermined number, and the top n devices have the lowest risk factors. In response to selecting the top n devices of the plurality of devices, a new virtual model is created from the slice, where the new virtual model is based on the capabilities of the top n devices of the plurality of devices. A cache is updated with the new virtual model. An embodiment of the present invention monitors the health of devices on which the model is deployed and creates a new virtual model for the new device.

[0010] In one embodiment, the health of each device of a plurality of devices is monitored. In response to determining that the health of any device is below a predetermined threshold, the device is marked as a bad device. The bad device is removed from the plurality of devices. A new virtual model is created, the new virtual model not including the bad device. The new virtual model is deployed to one or more of the plurality of devices. The cache is updated with the new virtual model. Embodiments of the invention monitor the health of devices to which models are deployed and replace high-risk or unhealthy devices with other healthy devices.

[0011] In one embodiment, the one or more virtual models are verified to match the deep learning model, and the verification that the one or more virtual models match the deep learning model is determined by one or more predetermined validation rules. Embodiments of the invention use the validation rules to verify that the model is running properly on the edge device. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a functional block diagram illustrating a distributed data processing environment in accordance with an embodiment of the present invention. [Figure 2] 1 is an example of splitting a deep learning model based on different device and virtual models, according to an embodiment of the present invention. [Figure 3] 1 is an example of a model splitting engine, according to an embodiment of the present invention. [Figure 4] 1 is an example of a virtual model cache according to an embodiment of the present invention. [Figure 5] 1 is an example of a monitor according to an embodiment of the present invention. [Figure 6]1 is a flowchart illustrating operational steps of a model generation procedure performed by a model deployment program 112 on a computing device in the distributed data processing environment of FIG. 1 for deploying a parallelizable deep learning model by adapting it to the computing device, according to an embodiment of the present invention. [Figure 7] FIG. 2 is a block diagram of components of a computing device executing a model deployment program within the distributed data processing environment of FIG. 1 in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] Deep learning models are often easily separable because they consist of different layers with their own parameters. Edge devices have different resource capabilities: some are more powerful than others and have different network connectivity.

[0014] To address the technical challenges of the current technology, which lack a method for deploying large models without compression and with high accuracy, high availability, and scalability, the present invention provides a computer-implemented method, computer program product, and system for partitioning models based on different devices and virtual models. Based on the deployment, the present invention monitors and manages model units during runtime to ensure that the model runtime is stable and flexible. To achieve this, in some embodiments, the present invention includes three components: a model partitioning engine, a virtual model cache, and a device monitor.

[0015] In an embodiment, a model partitioning engine partitions a model based on the capabilities of an edge computing device. The model partitioning engine can create multiple different partitions of the model to accommodate different edge devices with different capabilities. A virtual model cache selects candidate models and model slices from the virtual model cache to compose an overall model based on the capabilities of one or more edge computing devices. A device monitor monitors the health of devices and reports that information to the virtual cache to keep the cache updated.

[0016] In an embodiment, the present invention separates the model into different network layers, allowing a particular network layer to exchange data with related layers. Based on the capabilities of different edge devices, such as central processing unit / graphics processing unit (CPU / GPU) capabilities and memory, the present invention dynamically deploys network layers to the most suitable edge devices. For example, edge devices with higher capabilities perform more calculations, and edge devices with lower capabilities perform less calculations.

[0017] In an embodiment, the model partitioning engine works as follows: First, the slice generator partitions the model into different slices. For parallelizable networks, such as CNNs, the slice generator partitions the slices by layer. For complex networks that are difficult to parallelize, the slice generator uses rules to partition the network into the smallest parallelizable layers. To deploy the model to an edge device, the slice generator performs a resource matching search. Based on the monitor information, the slice generator calculates the size and capabilities of the edge device, determines the best match among the available edge devices, and distributes slices based on the resources of the matched device. Furthermore, the slice generator uses validation rules to verify that the model runs properly on the edge device.

[0018] In an embodiment, a virtual model cache holds virtual models for inference and provides ordering using a cognitive model selector. The virtual model cache includes two main components: a cached model list, which is a file that records the top n best constructed models, and a cognitive model generator, which checks the health of the top n devices running the models. In an embodiment, if the virtual model cache detects any anomalies in the running model, the virtual model cache selects a new model and deploys the new model to the appropriate edge device.

[0019] In an embodiment, the monitor collects device health for scheduling, including device resources such as GPU, disk, memory, etc. In an embodiment, the monitor includes a script that collects information from the device according to a schedule and sends this information to a virtual model cache. In an embodiment, the monitor may be an open source tool.

[0020] FIG. 1 is a functional block diagram illustrating a distributed data processing environment, generally designated 100, suitable for operation of a model deployment program 112 in accordance with at least one embodiment of the present invention. As used herein, the term "distributed" refers to a computer system that includes multiple physically distinct devices operating together as a single computer system. FIG. 1 is merely illustrative of one implementation and is not intended to imply any limitation with regard to the environments in which different embodiments may be implemented. Those skilled in the art may appreciate that many modifications to the depicted environment may be made without departing from the scope of the present invention as defined by the appended claims.

[0021] Distributed data processing environment 100 includes computing device 110 connected to network 120. Network 120 may be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the three, and may include wired, wireless, or fiber optic connections. Network 120 may include one or more wired or wireless networks, or both, capable of receiving and transmitting data, voice, or video signals, or combinations thereof, including multimedia signals including voice, data, and video information. In general, network 120 may be any combination of connections and protocols supporting communication between computing device 110 and other computing devices (not shown) in distributed data processing environment 100.

[0022] Computing device 110 may be a standalone computing device, an administrative server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, transmitting, and processing data. In an embodiment, computing device 110 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smartphone, or any programmable electronic device capable of communicating with other computing devices (not shown) in distributed data processing environment 100 via network 120. In another embodiment, computing device 110 may represent a server computing system, such as a cloud computing environment, that utilizes multiple computers as a server system. In yet another embodiment, computing device 110 represents a computing system that utilizes clustered computers and components (e.g., database server computers, application server computers, etc.) that function as a single pool of seamless resources when accessed within distributed data processing environment 100.

[0023] In an embodiment, computing device 110 includes model deployment program 112. In an embodiment, model deployment program 112 is a program, application, or subprogram of a larger program for deploying parallelizable deep learning models by adapting them to the computing device. In another embodiment, model deployment program 112 may be located on another device accessible by computing device 110 via network 120.

[0024] In an embodiment, computing device 110 includes information repository 114. In an embodiment, information repository 114 may be managed by model deployment program 112. In another embodiment, information repository 114 may be managed by the device's operating system, either alone or in conjunction with model deployment program 112. Information repository 114 is a data repository that can store, collect, compare, or combine information, or any combination thereof. In some embodiments, information repository 114 is located external to computing device 110 and is accessed via a communications network, such as network 120. In some embodiments, information repository 114 is stored on computing device 110. In some embodiments, information repository 114 may reside on another computing device (not shown), provided that information repository 114 is accessible by computing device 110. The information repository 114 includes, but is not limited to, deep learning model data, long-short-term memory (LSTM) data, device health data, model slice data, operating system data, configuration data, and other data received by the model deployment program 112 from one or more sources, and data created by the model deployment program 112.

[0025] Information repository 114 may be implemented using any volatile or non-volatile storage medium for storing information, as known in the art. For example, information repository 114 may be implemented using a tape library, an optical library, one or more independent hard disk drives, multiple hard disk drives in a redundant array of independent disks (RAID), a solid-state drive (SSD), or random access memory (RAM). Similarly, information repository 114 may be implemented with any suitable storage architecture known in the art, such as a relational database, a NoSQL database, an object-oriented database, or one or more tables.

[0026] 2 is an example of partitioning a deep learning model based on different devices and virtual models, according to an embodiment of the present invention. In an embodiment, model deployment program 112 partitions the model based on different devices and virtual models. In an embodiment, based on deployment, model deployment program 112 monitors and manages the model during execution to ensure that the model is stable and flexible during execution. In this example, deep learning model 210 represents a deep learning model that will be partitioned based on the capabilities of the computing devices on which the deep learning model may be deployed.

[0027] In an embodiment, the model deployment program 112 includes three main components. The first component is the model partitioning engine 220. This represents the section of the model deployment program 112 that partitions the model based on the capabilities of the edge computing devices to which the model may be deployed. In an embodiment, the same deep learning model may have multiple different partitions for deployment to different environments of edge devices. In an embodiment, the model deployment program 112 creates one or more virtual models for deployment to different sets of edge devices.

[0028] The second component is the virtual model cache 230. This represents a cache from which the model deployment program 112 selects model slices, or existing virtual models, to compose an overall model and then selects candidate models for deployment. The virtual model cache 230 contains models 235. The final component is the device monitor 240. This represents the section of the model deployment program 112 that monitors the health of devices and updates the virtual cache with device health information.

[0029] 2, edge devices 250 represent edge devices to which model deployment program 112 deploys slices of a model to enable efficient operation of the overall model. In this example, edge devices 250 include device 1 251, device 2 252, device 3 253, and device n 254. Once slices are distributed to edge devices 250, each device has the ability to forward data to its associated slice. In this manner, by distributing model slices among edge devices that can communicate with each other and executing the original model, the overall model is executed without compression.

[0030] 3 illustrates an example of a model partitioning engine according to an embodiment of the present invention. In an embodiment, a neural network is divided into different network layers by the model partitioning engine, and one particular network layer can exchange data with related layers. Based on different capabilities of edge devices, such as CPU / GPU and memory, the model deployment program 112 can dynamically deploy network layers to the most suitable edge devices. For example, edge devices with higher capabilities run slices that require larger computational capacity, and edge devices with lower capabilities run slices that require smaller computational capacity.

[0031] In the example of Figure 3, deep learning model 310 represents deep learning model 210 of Figure 2, slice 311, slice 312, slice 313, and slice 314 represent slices of deep learning model 310 after the model has been partitioned, and slice generator 320 represents a section of model deployment program 112, e.g., model partitioning engine 220 of Figure 2, that partitions the model based on the capabilities of edge computing devices on which the model may be deployed. In the example of Figure 3, slice 311, slice 312, and slice 313 represent three different virtual models of deep learning model 310, with slices 311, 312, and 313 each divided at different boundaries to accommodate different edge devices with different capacities.

[0032] For parallelizable networks such as convolutional neural networks (CNNs), the model deployment program 112 divides the slices by layers. For complex networks that are difficult to parallelize, the model deployment program 112 can add rules to divide the network into the smallest parallelizable layers.

[0033] In embodiments, the rules are specific to the type of deep learning network. For example, CNNs typically have multiple network layers, and the calculation results of the current layer are used as input for the next layer, making it possible to split a layer. However, in recurrent neural networks (RNNs), network layers cannot be directly split because they remember the last state of the input. In embodiments, in the case of RNNs, to remember the last state of the input, the split layer records the spliced ​​layers before and after it, creating a network of doubly linked layers.

[0034] In an embodiment, search matching resources 330 represents a section of the model deployment program 112 that calculates the size and capabilities of edge devices based on information collected by a monitor, such as device monitor 240 of FIG. 2. The model deployment program 112 then allocates slices based on the matched devices' available resources. Add rules to validation 340 represents a section of the model deployment program 112 that adds rules to ensure that a computing device can properly execute a model. For example, one rule may state that the model deployment program 112 needs to allocate some capacity on the device for other tasks and may adjust the slices deployed to that device accordingly.

[0035] In an embodiment, edge device 350 represents edge device 250 of Figure 2. Edge device 350 includes device 1 351, which represents device 1 251 of Figure 2. In this example, slice 314 is deployed to device 1 351 by model deployment program 112 based on search matching resource 330.

[0036] Figure 4 shows an example of a virtual model cache according to an embodiment of the present invention. In an embodiment, the virtual model cache stores virtual models for inference and can also organize models using a cognitive model selector. Therefore, this part contains two important components: a model list and a cognitive model generator.

[0037] The model list 410 is a file in which the model deployment program 112 records the top n best constructed models based on the analysis of deep learning models, where n is a predetermined value. This is essentially a list of all models stored in the model cache. In an embodiment, each line in the model list describes one virtual model and the slices that make up that model. For example, model 1 is divided into four slices: s1, s2, s3, and s4. In the example of FIG. 4, s1 is deployed to device 1 in slot 1, s2 to device 3 in slot 2, s3 to device 4 in slot 1, and s4 to device 5 in slot 1. Slots are used to locate / place slices on specific devices.

[0038] In an embodiment, the model deployment program 112 includes a cognitive model generator 420, which is the section of the model deployment program 112 that checks the health of the top n devices (where n is a predetermined value) for each model. If the model deployment program 112 detects any anomalies in the health of the top n devices, the model deployment program 112 selects a new model to run on a different device. This is the section of the model deployment program 112 that determines how to update the model cache.

[0039] FIG. 5 illustrates an example of a monitor according to an embodiment of the present invention. In the example of FIG. 5, device monitor 510 represents a section of model deployment program 112 that collects device health information for scheduling model deployment, including device resources such as CPU / GPU, disk, and memory. This is an example of device monitor 240 in FIG. 2. In an embodiment, model deployment program 112 uses scripts to collect information from devices on a scheduled basis and send this information to a virtual model cache. In an embodiment, model deployment program 112 may use open-source device monitoring tools for the device monitor.

[0040] Device 1 551, device 2 552, device 3 553, and device n 554 represent edge devices monitored by the device monitor 510 of the model deployment program 112. Heartbeat 520 represents the collection of information from the devices by the model deployment program 112 on a scheduled basis.

[0041] 6 is a flowchart illustrating operational steps of a model generation procedure performed by model deployment program 112 on a computing device in the distributed data processing environment of FIG. 1 for deploying a parallelizable deep learning model by adapting it to the computing device, according to an embodiment of the present invention. In alternative embodiments, the steps of workflow 600 may be performed by other programs in cooperation with model deployment program 112.

[0042] In an embodiment, the model deployment program 112 continuously collects data from a monitor to monitor the health of devices running deployed slices of a deep learning model or devices that may be called into service to run slices of a deep learning model that have not yet been deployed. In an embodiment, the model deployment program 112 encodes features, such as GPU, disk speed and capacity, and available memory, over time to determine the capacity each device has to run a model slice. In an embodiment, the model deployment program 112 uses an LSTM model to predict the risk of each device monitored by the device monitor. In an embodiment, the model deployment program 112 determines whether a device is high-risk for running a model. In an embodiment, the model deployment program 112 selects the top n devices (n is a predetermined number) based on the device health assessment by the monitor function of the model deployment program 112 and replaces the high-risk / bad devices with healthy devices. In an embodiment, the model deployment program 112 then updates the model list file with the new split of models determined in step 610. In an embodiment, the model deployment program 112 then ends the cycle.

[0043] It should be understood that embodiments of the present invention are directed to at least providing for deploying parallelizable deep learning models by adapting them to a computing device. However, Figure 6 is intended to provide an illustration of one implementation only and is not intended to imply any limitations with respect to the environments in which different embodiments may be implemented. One skilled in the art could readily implement many modifications to the depicted environment without departing from the scope of the present invention as defined by the appended claims.

[0044] It should be understood that the process depicted in FIG. 6 illustrates one possible iteration of the operational steps performed by the model deployment program 112 to deploy a parallelizable deep learning model by adapting it to a computing device, which is repeated each time a deep learning model is deployed.

[0045] The model deployment program 112 collects monitor information (step 602). In an embodiment, the model deployment program 112 continuously collects data from monitors, such as device monitor 240 of FIG. 2, to monitor the health of devices running deployed slices of deep learning models or that may be called into service to run slices of deep learning models that have not yet been deployed.

[0046] The model deployment program 112 encodes features (GPU, disk, etc.) by time (step 604). In an embodiment, the model deployment program 112 encodes features, such as CPU capacity, GPU capacity, disk speed and capacity, available memory, etc., by time to determine the capacity each device has to execute the model slice. In an embodiment, the time slice used by the monitor section of the model deployment program 112 is a predetermined time interval. In another embodiment, the time slice used by the monitor section of the model deployment program 112 is received from a user. In yet another embodiment, the time slice used by the monitor section of the model deployment program 112 may be selected by other methods known to those skilled in the art.

[0047] The model deployment program 112 uses the LSTM model to predict the risk of the device (step 606). In an embodiment, the model deployment program 112 uses the LSTM model to predict the risk of each device monitored by the device monitor. In an embodiment, the model deployment program 112 uses the LSTM model to calculate a risk factor for the device.

[0048] In an embodiment, the model deployment program 112 uses historical system monitoring data to train an LSTM model. In an embodiment, the encoded features may include, but are not limited to, GPU, memory, disk consumption, time series data, etc., and the labeled result is a device health state. In an embodiment, the model deployment program 112 uses the LSTM to predict a device health state or risk based on the device resource consumption data.

[0049] The model deployment program 112 determines whether there are any high-risk devices (decision block 608). In an embodiment, the model deployment program 112 determines whether the device is a high-risk for running the model. In an embodiment, this is the risk that the deployed model will not be able to run on this device. In an embodiment, if the device is healthy and has sufficient capacity to run the model, there is no risk. If the device is not at risk, there is no need to partition the model. In an embodiment, if the model deployment program 112 determines that the device is not a high-risk for running the model (decision block 608, “No” branch), the model deployment program 112 deploys the model to the device. The model deployment program 112 then ends the cycle. In an embodiment, if the model deployment program 112 determines that the device is a high-risk for running the model (decision block 608, “Yes” branch), the model deployment program 112 proceeds to step 610 and partitions the model.

[0050] The model deployment program 112 selects and replaces the top n devices based on device health (step 610). In an embodiment, if the model deployment program 112 determines that the risk factor of any device determined in step 606 exceeds a predetermined threshold, the model deployment program 112 marks the device as a bad device. In an embodiment, the model deployment program 112 then selects the top n devices (n is a predetermined number) based on the device health assessment by the model deployment program's 112 monitoring functionality and replaces the high-risk / bad devices with healthy devices. In an embodiment, the model deployment program 112 determines the top n devices by selecting the devices with the lowest risk factors. In an embodiment, the model deployment program 112 removes the high-risk / bad devices from the list of available devices.

[0051] The model deployment program 112 updates the model list file (step 612). In an embodiment, the model deployment program 112 then updates the model list file, e.g., model list 410 in Figure 4, with the new partitioning of the model determined in step 610. In an embodiment, the model deployment program 112 then ends the cycle.

[0052] Figure 7 is a block diagram illustrating components of a computing device 110 suitable for a model deployment program 112 in accordance with at least one embodiment of the present invention. Figure 7 illustrates a computer 700, one or more processors 704 (including one or more computer processors), a communications fabric 702, memory 706 including random access memory (RAM) 716 and cache 718, persistent storage 708, a communications unit 712, an I / O interface 714, a display 722, and external devices 720. It should be understood that Figure 7 is only illustrative of one embodiment and is not intended to imply any limitation with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

[0053] As shown, computer 700 operates on a communications fabric 702 that provides communications between a computer processor 704, memory 706, persistent storage 708, communications unit 712, and I / O interface 714. Communications fabric 702 may be implemented with any architecture suitable for passing data or control information between processor 704 (e.g., a microprocessor, communications processor, and network processor), memory 706, external device 720, and any other hardware components in the system. For example, communications fabric 702 may be implemented with one or more buses.

[0054] Memory 706 and persistent storage 708 are computer-readable storage media. In the illustrated embodiment, memory 706 includes RAM 716 and cache 718. In general, memory 706 may include any suitable volatile or non-volatile computer-readable storage media. Cache 718 is high-speed memory that enhances the performance of processor 704 by retaining most recently accessed and recently accessed data from RAM 716.

[0055] Program instructions for the model deployment program 112 may be stored in persistent storage 708, or more generally, any computer-readable storage medium, for execution by one or more of the respective computer processors 704 via one or more memories of memory 706. Persistent storage 708 may be a magnetic hard disk drive, a solid-state disk drive, a semiconductor storage device, a read-only memory (ROM), an electronically erasable programmable read-only memory (EEPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0056] The media used by persistent storage 708 may be removable. For example, a removable hard drive may be used for persistent storage 708. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer-readable storage medium that is also part of persistent storage 708.

[0057] Communications unit 712, in these examples, provides for communication with other data processing systems or devices. In these examples, communications unit 712 includes one or more network interface cards. Communications unit 712 may provide for communication through the use of either or both physical and wireless communications links. In the context of some embodiments of the present invention, various sources of input data may be physically remote from computer 700, and input data may be received, and outputs may be transmitted, via communications unit 712.

[0058] The I / O interface 714 allows for the input and output of data to and from other devices that may be connected to the computer 700. For example, the I / O interface 714 may provide a connection to external devices 720, such as a keyboard, keypad, touchscreen, microphone, digital camera, or other suitable input device or combination thereof. The external devices 720 may also include portable computer-readable storage media, such as thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to implement embodiments of the present invention, such as the model deployment program 112, may be stored on such portable computer-readable storage media and loaded into persistent storage 708 via the I / O interface 714. The I / O interface 714 also connects to a display 722.

[0059] The display 722 provides a mechanism for displaying data to a user and may be, for example, a computer monitor. The display 722 may also function as a touch screen, such as the display on a tablet computer.

[0060] The programs described herein are identified based on the applications for which they are implemented in particular embodiments of the invention. However, it should be understood that the nomenclature of any particular program herein is used merely for convenience, and that the invention should not, therefore, be limited to use in any particular application expressly or implied, or both, by such nomenclature.

[0061] The present invention may be a system, a method, or a computer program product, or a combination thereof. The computer program product may include one or more computer-readable storage media having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0062] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction-execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanical coding devices such as punch cards or raised groove structures with recorded instructions, and any suitable combination of the above. Computer-readable storage medium, as used herein, should not be construed as a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0063] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium into each computing / processing device, or can be downloaded to an external computer or storage device, for example, via the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper cables, optical fibers, wireless networks, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium within the respective computing / processing device for storage.

[0064] The computer-readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can execute computer-readable program instructions by utilizing state information in the computer-readable program instructions to individualize the electronic circuitry to implement aspects of the present invention.

[0065] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0066] These computer-readable program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, whereby the instructions executed by the processor of the computer or other programmable data processing apparatus create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, whereby the computer-readable medium having instructions stored therein comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0067] The computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0068] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may execute a combination of dedicated hardware and computer instructions.

[0069] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or to be limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the present invention. The terms used herein have been selected to best explain the principles, practical applications, or technical improvements over techniques found in the marketplace of the embodiments, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. dividing, by one or more computer processors, the deep learning model into a plurality of slices, wherein each slice of the plurality of slices is capable of exchanging data with one or more associated slices of the plurality of slices; creating, by the one or more computer processors, one or more virtual models from the plurality of slices, the one or more virtual models based on capabilities of a plurality of devices on which the one or more virtual models are deployed, and further wherein each virtual model of the plurality of virtual models includes a respective slice of the plurality of slices; caching, by the one or more computer processors, the one or more virtual models; calculating, by the one or more computer processors, in response to determining that the deep learning model is to be deployed on one or more devices of the plurality of devices, a size and capabilities of the one or more devices, the calculation being based on information from a device monitor for the one or more devices of the plurality of devices; constructing, by the one or more computer processors, an overall model from the one or more virtual models and the plurality of slices in the cache based on the calculated size and capabilities; deploying, by the one or more computer processors, the overall model to the one or more devices of the plurality of devices; 10. A computer-implemented method comprising:

2. Dividing the deep learning model into the plurality of slices, each slice of the plurality of slices being capable of exchanging data with the one or more associated slices, the dividing step comprising: dividing, by the one or more computer processors, the deep learning model into the plurality of slices, each slice being a different network layer of a plurality of different network layers of the deep learning model. The computer-implemented method of claim 1 , comprising:

3. Dividing the deep learning model into the plurality of slices, each slice of the plurality of slices being capable of exchanging data with the one or more associated slices, the dividing step comprising: and in response to determining that the deep learning model is not readily parallelizable, partitioning, by the one or more computer processors, the deep learning model into the plurality of slices based on a set of predetermined rules, the set of predetermined rules dividing the deep learning model into a plurality of smallest parallelizable layers. The computer-implemented method of claim 1 , comprising:

4. In response to determining that the deep learning model is to be deployed on one or more devices of the plurality of devices, calculating the size and capabilities of the one or more devices includes: collecting, by the one or more computer processors, the information by the device monitor from the plurality of devices on which the deep learning model can be deployed; encoding, by the one or more computer processors, characteristics of the plurality of devices for each time slice, the characteristics including at least one of central processing unit (CPU) capacity, graphics processing unit (GPU) capacity, and disk capacity, and further wherein the time slice is a predetermined time interval; Including, constructing, by the one or more computer processors, an overall model from the one or more virtual models in the cache and the plurality of slices based on the calculated size and capabilities, predicting, by the one or more computer processors, a risk factor for each device of the plurality of devices, wherein the risk factor is predicted using a long short-term memory (LSTM) model; and selecting, by the one or more computer processors, a top n number of devices from the plurality of devices in response to determining that the risk factor of any device of the plurality of devices exceeds a predetermined threshold, where n is a predetermined number, and further wherein the top n devices have the lowest risk factors; generating, by the one or more computer processors, one or more new virtual models from the plurality of slices in response to selecting the top n devices of the plurality of devices, the one or more new virtual models being based on capabilities of the top n devices of the plurality of devices; updating, by the one or more computer processors, the cache with the one or more new virtual models; The computer-implemented method of claim 1 , comprising:

5. The computer-implemented method of claim 4 , wherein the device monitor collects the information from the plurality of devices according to a time schedule.

6. Deploying, by the one or more computer processors, the ensemble model to the one or more devices of the plurality of devices includes: monitoring, by the one or more computer processors, the health of each device of the plurality of devices; marking, by the one or more computing devices, any one of the plurality of devices as a bad device in response to determining that the health of the any one of the plurality of devices is below a predetermined threshold; removing, by the one or more computer processors, the defective device from the plurality of devices; creating, by the one or more computer processors, a new virtual model that does not include the defective device; and deploying, by the one or more computer processors, the new virtual model to the one or more devices of the plurality of devices; updating, by the one or more computer processors, the cache with the new virtual model; and The computer-implemented method of claim 1 , further comprising:

7. creating one or more virtual models from the plurality of slices, the one or more virtual models being based on capabilities of a plurality of devices on which the one or more virtual models are deployed, and further wherein each virtual model of the plurality of virtual models includes a respective slice of the plurality of slices; verifying, by the one or more computer processors, that the one or more virtual models match the deep learning model, wherein verifying that the one or more virtual models match the deep learning model is determined by one or more predetermined validation rules. The computer-implemented method of claim 1 , further comprising:

8. A computer program comprising computer readable program instructions for causing a computer to carry out the method of any one of claims 1 to 7.

9. 9. A computer-readable storage medium having stored thereon the computer program of claim 8.

10. one or more computer processors; one or more computer-readable storage media; program instructions stored on said one or more computer-readable storage media for execution by at least one of said one or more computer processors; wherein the stored program instructions include: Dividing a deep learning model into a plurality of slices, wherein each slice of the plurality of slices is capable of exchanging data with one or more associated slices of the plurality of slices; creating one or more virtual models from the plurality of slices, the one or more virtual models being based on capabilities of a plurality of devices on which the one or more virtual models are deployed, and further wherein each virtual model of the plurality of virtual models includes a respective slice of the plurality of slices; caching the one or more virtual models; calculating, by the one or more computer processors, in response to determining that the deep learning model is to be deployed on one or more devices of the plurality of devices, a size and capabilities of the one or more devices, the calculation being based on information from a device monitor for the one or more devices of the plurality of devices; constructing, by the one or more computer processors, an overall model from the one or more virtual models in the cache and the plurality of slices; deploying the overall model to the one or more devices of the plurality of devices; 22. A computer system comprising instructions for:

11. dividing the deep learning model into the plurality of slices, each slice of the plurality of slices capable of exchanging data with the one or more associated slices, the dividing comprising program instructions stored on the one or more computer-readable storage media, the program instructions comprising: Dividing the deep learning model into the plurality of slices, each slice being a different network layer of a plurality of different network layers of the deep learning model.

11. The computer system of claim 10, comprising one or more of the program instructions for:

12. dividing the deep learning model into the plurality of slices, each slice of the plurality of slices capable of exchanging data with the one or more associated slices, the dividing comprising program instructions stored on the one or more computer-readable storage media, the program instructions comprising: In response to determining that the deep learning model is not readily parallelizable, partitioning the deep learning model into the plurality of slices based on a set of predetermined rules, the set of predetermined rules dividing the deep learning model into a plurality of smallest parallelizable layers.

11. The computer system of claim 10, comprising one or more of the program instructions for:

13. In response to determining that the deep learning model is to be deployed on one or more of the plurality of devices, calculating the size and capabilities of the one or more devices comprises program instructions stored on the one or more computer-readable storage media, the program instructions comprising: collecting the information by the device monitor from the plurality of devices on which the deep learning model can be deployed; encoding characteristics of the plurality of devices for each time slice, the characteristics including at least one of a central processing unit (CPU) capacity, a graphics processing unit (GPU) capacity, and a disk capacity, and further wherein the time slice is a predetermined time interval; and one or more of the program instructions for: constructing, by the one or more computer processors, an overall model from the one or more virtual models in the cache and the plurality of slices based on the calculated size and capabilities, comprises program instructions stored on the one or more computer-readable storage media, the program instructions comprising: predicting a risk factor for each device of the plurality of devices, wherein the risk factor is predicted using a long short-term memory (LSTM) model; and in response to determining that the risk factor of any device of the plurality of devices exceeds a predetermined threshold, selecting a top n number of devices from the plurality of devices, n being a predetermined number, and further wherein the top n devices have the lowest risk factors; creating one or more new virtual models from the plurality of slices in response to selecting the top n devices of the plurality of devices, the one or more new virtual models being based on capabilities of the top n devices of the plurality of devices; updating the cache with the one or more new virtual models; 11. The computer system of claim 10, comprising one or more of the program instructions for:

14. Deploying the overall model to the one or more devices of the plurality of devices comprises program instructions stored on the one or more computer-readable storage media, the program instructions comprising: monitoring the health of each device of the plurality of devices; marking any one of the plurality of devices as a bad device in response to determining that the health of the any one of the plurality of devices is below a predetermined threshold; removing the defective device from the plurality of devices; creating a new virtual model that does not include the defective device; deploying the new virtual model to the one or more devices of the plurality of devices; updating the cache with the new virtual model; and 11. The computer system of claim 10, further comprising one or more of the program instructions for:

15. creating one or more virtual models from the plurality of slices, the one or more virtual models based on capabilities of a plurality of devices on which the one or more virtual models are deployed, and further wherein each virtual model of the plurality of virtual models includes a respective slice of the plurality of slices, the creating comprising program instructions stored on the one or more computer-readable storage media: verifying that the one or more virtual models match the deep learning model, wherein verifying that the one or more virtual models match the deep learning model is determined by one or more predetermined validation rules.

11. The computer system of claim 10, further comprising one or more of the program instructions for:

Images