Deploying parallelizable deep learning models by adapting to computing devices

Abstract

In a method of deploying a parallelizable deep learning model by adapting to a computing device, the deep learning model is divided into a plurality of slices, where each slice can exchange data with related slices. A virtual model is created from the plurality of slices, where the virtual model is based on capabilities of a plurality of devices on which one or more virtual models are to be deployed, and further where each virtual model contains each slice of the plurality of slices. The one or more virtual models are stored in a cache. In response to determining to deploy the deep learning model on one or more devices, a candidate model is selected from the virtual models in the cache, where the selection is based on information of the devices from a device monitor.

Description

Background Technology

[0001] This invention relates generally to the field of machine learning, and more specifically to deploying parallelizable deep learning models by adapting them to computing devices.

[0002] Machine learning (ML) is the application of artificial intelligence (AI), which provides systems with the ability to automatically learn and improve from experience without being explicitly programmed. Machine learning is the use and development of computer systems that can learn and adapt without following explicit instructions by analyzing and drawing inferences from patterns in data using algorithms and statistical models. Machine learning algorithms build models based on sample data (called "training data") to make predictions or decisions. Machine learning algorithms are used in a wide variety of applications, such as email filtering and computer vision, where developing traditional algorithms to perform the desired tasks is difficult or infeasible.

[0003] Deep learning is an artificial intelligence (AI) function that mimics how the human brain processes data and creates patterns for decision-making. Deep learning is a subset of AI machine learning, featuring networks capable of learning from unstructured or unlabeled data. Deep learning is a class of machine learning algorithms that use multiple layers in a network to progressively extract higher-level features from the raw input. For example, in image processing, lower layers might identify edges, while higher layers can use those edges to distinguish the entire object. Most modern deep learning models are based on artificial neural networks, particularly convolutional neural networks (CNNs). Deep learning is experiencing explosive growth due to the combination of deep neural networks and the use of graphics processing units (GPUs) to accelerate their execution.

[0004] To improve model accuracy, data scientists tend to build increasingly deep neural networks. However, as neural network complexity increases, the model size becomes quite large, significantly increasing the time spent transferring and deploying the model. Model initialization and operation consume significant amounts of processor and memory capacity, and trained models are difficult to reposition and balance for high availability (HA).

[0005] To deploy models on edge devices, different methods can be used to compress them. Model compression can be divided into two main categories. The first is pruning, which requires removing redundant connections present in the architecture. The problem with pruning is that the resulting new model will have lower accuracy than the original model because the model was actually trained on the original connections. The second is quantization, which involves bundling weights together by clustering or rounding them, allowing the same number of connections to be represented using less memory. However, like pruning, quantization reduces model accuracy. Therefore, edge devices typically only have a single, non-robust model. The problem with existing techniques is that there is no way to deploy large models with high accuracy, high availability, and scalability without compression. Summary of the Invention

[0006] Various embodiments of the present invention disclose computer-implemented methods, computer program products, and systems for deploying parallelizable deep learning models via adapted computing devices. In one embodiment, the deep learning model is divided into multiple slices, wherein each slice can exchange data with associated slices. Virtual models are created from the multiple slices, wherein the virtual models are based on the capabilities of the multiple means of deploying the one or more virtual models described above, and further wherein each virtual model comprises each of the multiple slices. One or more virtual models are stored in a cache. In response to determining that a deep learning model is to be deployed on one or more devices, candidate models are selected from the virtual models in the cache, wherein the selection is based on information about the device from a device monitor. Embodiments of the present invention segment the model into slices based on the capabilities of edge computing devices, and select candidate models and model slices from the virtual model cache to compose the entire model based on the capabilities of the edge computing devices.

[0007] In one embodiment, the deep learning model is divided into multiple slices, where each slice is a different network layer among multiple different network layers of the deep learning model. Embodiments of the present invention segment the model into slices for deployment on multiple edge devices.

[0008] In one embodiment, in response to determining that the deep learning model is not easily parallelizable, the deep learning model is divided into multiple slices based on a predetermined set of rules, wherein the predetermined set of rules divides the deep learning model into multiple minimally parallelizable layers. Embodiments of the invention divide the model into slices for deployment on multiple edge devices.

[0009] In one embodiment, a device monitor collects information from multiple devices on which deep learning models can be deployed. Each time slice encodes characteristics of the multiple devices, said characteristics including, but not limited to, at least one of central processing unit (CPU) capacity, graphics processing unit (GPU) capacity, and disk capacity, and further, wherein the time slice is a predetermined period of time. Risk factors are predicted for each of the multiple devices, wherein the risk factors are predicted using a Long Short-Term Memory (LSTM) model. In response to determining that the risk factor of any device in the multiple devices exceeds a predetermined threshold, the top n devices are selected, where n is a predetermined number, and further, the top n devices have the lowest risk factors. In response to selecting the top n devices, a new virtual model is created from the slice, said new virtual model being based on the capabilities of the top n devices. The cache is updated with the new virtual model. Embodiments of the invention monitor the health of devices on which models are deployed and create new virtual models for new devices.

[0010] In one embodiment, the health status of each of a plurality of devices is monitored. In response to determining that the health of any device is below a predetermined threshold, that device is marked as a faulty device. The faulty device is removed from the plurality of devices. A new virtual model is created that does not include the faulty device. The new virtual model is deployed to one or more of the plurality of devices. The cache is updated with the new virtual model. Embodiments of the invention monitor the health status of the devices on which models are deployed and replace high-risk or unhealthy devices with other healthy devices.

[0011] In one embodiment, one or more virtual models are confirmed to match a deep learning model, wherein the confirmation of a match between the virtual models and the deep learning model is determined by one or more predetermined verification rules. Embodiments of the invention use verification rules to confirm that the model functions correctly on an edge device. Attached Figure Description

[0012] Figure 1 This is a functional block diagram illustrating a distributed data processing environment according to an embodiment of the present invention.

[0013] Figure 2 This is an example of classifying deep learning models based on different devices and virtual models according to an embodiment of the present invention.

[0014] Figure 3 This is an example of a model segmentation engine according to an embodiment of the present invention.

[0015] Figure 4 This is an example of a virtual model cache according to an embodiment of the present invention.

[0016] Figure 5This is an example of a monitor according to an embodiment of the present invention.

[0017] Figure 6 This describes an embodiment of the invention. Figure 1 A flowchart of the operational steps of the model generation process performed by the model deployment procedure 112 on a computing device within a distributed data processing environment for deploying a parallelizable deep learning model by adapting it to the computing device.

[0018] Figure 7 The invention describes an embodiment of the invention. Figure 1 A block diagram of the components of a computing device that executes model deployment programs within a distributed data processing environment. Detailed Implementation

[0019] Most deep learning models are easily decomposed because they consist of different layers with parameters. Edge devices have different resource capabilities; some are more powerful than others and have different network connectivity.

[0020] To address the problem of existing technologies that lack a method for deploying large models with high accuracy, high availability, and scalability without compression, this invention provides a computer-implemented method, computer program product, and system for partitioning models based on different devices and virtual models. Based on this deployment, the invention monitors and manages model units during runtime to ensure model stability and flexibility. To achieve this, in some embodiments, the invention includes three components: a model partitioner engine, a virtual model cache, and a device monitor.

[0021] In this embodiment, the model segmenter engine segments the model based on the capabilities of the edge computing device. The model segmenter engine can create many different segments of the model to accommodate different edge devices with varying capabilities. A virtual model cache selects candidate models and model slices from the virtual model cache to compose the entire model based on the capabilities of one or more edge computing devices. A device monitor monitors the health of the devices and reports information to the virtual cache to keep the cache updated.

[0022] In one embodiment, the invention segments the model into different network layers, where specific network layers can exchange data with related layers. Based on different edge device capabilities, such as CPU / GPU capabilities and memory, the invention dynamically deploys network layers to the most compatible edge devices. For example, edge devices with higher capabilities will perform more computations, while edge devices with lower capabilities will perform fewer computations.

[0023] In this embodiment, the model segmenter engine operates as follows: First, the model is segmented into different slices using a slice generator. For parallelizable networks (e.g., CNNs), the slice generator segments the network by layer. For some complex networks that are difficult to parallelize, the slice generator uses rules to divide the network into minimum parallelizable layers. To deploy the model to edge devices, the slice generator performs a search to match resources. Based on monitoring information, the slice generator calculates the size and capabilities of the edge devices, determines the best match between available edge devices, and allocates slices based on the resources of the matched devices. Furthermore, the slice generator uses validation rules to verify that the model operates correctly on the edge devices.

[0024] In this embodiment, the virtual model cache retains virtual models for inference and also uses a cognitive model selector to provide commands. The virtual model cache includes two key components: a cached model list, which is a file recording the top n optimal models; and a cognitive model generator, which checks the health of the top n devices running the model. In this embodiment, if the virtual model cache detects any anomalies in the running models, it selects a new model and deploys it to the appropriate edge devices.

[0025] In one embodiment, the monitor collects device health data for scheduling purposes and also includes device resources such as GPU, disk, and storage. In another embodiment, the monitor includes scripts that collect information from the device according to a schedule and send that information to a virtual model cache. In yet another embodiment, the monitor may be an open-source tool.

[0026] Figure 1 This is a functional block diagram illustrating a distributed data processing environment (generally designated 100) suitable for operation of a model deployment procedure 112 according to at least one embodiment of the present invention. As used herein, the term "distributed" describes a computer system comprising multiple physically distinct devices that operate together as a single computer system. Figure 1 This illustration provides only one implementation and does not imply any limitation regarding the environments in which different embodiments may be implemented. Those skilled in the art can make numerous modifications to the described environments without departing from the scope of the invention as set forth in the claims.

[0027] The distributed data processing environment 100 includes computing devices 110 connected to a network 120. The network 120 may be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) (such as the Internet), or a combination of all three, and may include wired, wireless, or fiber optic connections. The network 120 may include one or more wired and / or wireless networks capable of receiving and transmitting data, voice, and / or video signals (including multimedia signals containing voice, data, and video information). Generally, the network 120 may be any combination of connections and protocols supporting communication between the computing device 110 and other computing devices (not shown) within the distributed data processing environment 100.

[0028] The computing device 110 may be a standalone computing device, management server, web server, mobile computing device, or any other electronic device or computing system capable of receiving, sending, and processing data. In one embodiment, the computing device 110 may be a laptop computer, tablet computer, netbook computer, personal computer (PC), desktop computer, personal digital assistant (PDA), smartphone, or any programmable electronic device capable of communicating via network 120 with other computing devices (not shown) within the distributed data processing environment 100. In another embodiment, the computing device 110 may represent a server computing system, such as one utilizing multiple computers as server systems in a cloud computing environment. In yet another embodiment, the computing device 110 represents a computing system utilizing clustered computers and components (e.g., database server computers, application server computers, etc.) that act as a single, seamless pool of resources when accessed within the distributed data processing environment 100.

[0029] In one embodiment, computing device 110 includes a model deployment program 112. In this embodiment, model deployment program 112 is a program, application, or subroutine for deploying a parallelizable deep learning model by adapting it to the computing device. In an alternative embodiment, model deployment program 112 may reside on any other device accessible via network 120 from computing device 110.

[0030] In one embodiment, computing device 110 includes an information repository 114. In this embodiment, the information repository 114 may be managed by a model deployment program 112. In an alternative embodiment, the information repository 114 may be managed by the device's operating system alone or in conjunction with the model deployment program 112. The information repository 114 is a data repository capable of storing, collecting, comparing, and / or combining information. In some embodiments, the information repository 114 is located external to computing device 110 and accessed via a communication network (such as network 120). In some embodiments, the information repository 114 is stored on computing device 110. In some embodiments, the information repository 114 may reside on another computing device (not shown), provided that the information repository 114 is accessible by computing device 110. The information repository 114 includes, but is not limited to, deep learning model data, Long Short-Term Memory (LSTM) data, device health data, model slice data, operating system data, configuration data, and other data received by the model deployment program 112 from one or more sources and data created by the model deployment program 112.

[0031] As is known in the art, information repository 114 can be implemented using any volatile or non-volatile storage medium for storing information. For example, information repository 114 can be implemented using a tape library, an optical library, one or more individual hard disk drives, multiple hard disk drives in a redundant array of independent disks (RAID), a solid-state drive (SSD), or random access memory (RAM). Similarly, information repository 114 can be implemented using any suitable storage architecture known in the art, such as a relational database, a NoSQL database, an object-oriented database, or one or more tables.

[0032] Figure 2 This is an example of partitioning a deep learning model based on different devices and virtual models according to an embodiment of the present invention. In this embodiment, the model deployment procedure 112 partitions the model based on different devices and virtual models. In this embodiment, based on deployment, the model deployment procedure 112 monitors and manages the model during runtime to ensure that the model is stable and flexible during runtime. In this example, deep learning model 210 represents a deep learning model that will be partitioned based on the capabilities of the computing devices on which the deep learning model can be deployed.

[0033] In this embodiment, the model deployment procedure 112 includes three main components. The first component is a model segmenter engine 220. This represents the ability of the model deployment procedure 112 to segment models based on the edge computing devices that can deploy the models. In this embodiment, the same deep learning model can have multiple different slices for deployment to different environments on edge devices. In this embodiment, the model deployment procedure 112 will create one or more virtual models for deployment to different sets of edge devices.

[0034] The second component is the virtual model cache 230. This means that the model deployment program 112 will select model slices from it to form a cache of the entire model or an existing virtual model, and then select candidate models for deployment. The virtual model cache 230 contains model 235. The final component is the device monitor 240. This means that the model deployment program 112 will monitor the health of the device and update portions of the virtual cache with device health information.

[0035] exist Figure 2 In the example, edge device 250 refers to the edge device on which model deployment procedure 112 will deploy slices of the model to allow efficient operation of the entire model. In this example, edge device 250 includes device 1251, device 2252, device 3253, and device n 254. When a slice is assigned to edge device 250, each device has the ability to transfer data to the associated slice. In this way, by distributing model slices among edge devices that communicate with each other to run the original model, the entire model can run without compression.

[0036] Figure 3 This is an example of a model segmentation engine according to an embodiment of the present invention. In the embodiment, the neural network is segmented into different network layers by the model segmentation engine, and a specific network layer can exchange data with related layers. Based on the different capabilities of edge devices (e.g., CPU / GPU and memory), the model deployment program 112 can dynamically deploy network layers to the most compatible edge devices. For example, edge devices with higher capabilities will run slices requiring more computing power, while edge devices with lower capabilities will run slices requiring less computing power.

[0037] exist Figure 3 In the example, deep learning model 310 represents Figure 2 In the deep learning model 210, slices 311, 312, 313, and 314 represent slices of the deep learning model 310 after partitioning the model, and slice generator 320 represents the model deployment procedure 112 based on an edge computing device (e.g., capable of deploying the model) that can deploy the model. Figure 2 The model partitioner engine (220) has the capability to partition the model. Figure 3 In the example, slices 311, 312, and 313 represent three different virtual models of the deep learning model 310, wherein each of slices 311, 312, and 313 is segmented on different boundaries to accommodate different edge devices with different capabilities.

[0038] For parallelizable networks, such as convolutional neural networks (CNNs), model deployment procedure 112 will slice the network into layers. For some complex networks that are difficult to parallelize, model deployment procedure 112 may add rules to divide the network into minimal parallelizable layers.

[0039] In this embodiment, the rules are specific to deep learning network types. For example, CNNs typically have multiple network layers, with the result of the current layer's computation serving as the input to the next layer, allowing for layer splitting. However, in recurrent neural networks (RNNs), the network layers remember the last state of the input and therefore cannot be directly split. In this embodiment, for RNNs, to remember the last state of the input, the splitting layer records its backward or forward concatenation layer to construct a two-connected layer network.

[0040] In this embodiment, search matching resource 330 represents the model deployment procedure 112 based on a monitor (e.g., from...). Figure 2 The device monitor 240 collects information to calculate the size and capabilities of the edge device. The model deployment procedure 112 then allocates slices based on the available resources of the matching device. Adding rules to the validator 340 indicates that the model deployment procedure 112, with the added rules, confirms that the computing device can correctly run parts of the model. For example, the rule might specify that the model deployment procedure 112 needs to allocate some capabilities on the device for other tasks, and thus can adjust the slices deployed to that device.

[0041] In this embodiment, edge device 350 represents Figure 2 Edge device 250 in the middle. Edge device 350 includes representations from Figure 2 Device 1251 and device 1351. In this example, slice 314 is deployed to device 1351 by model deployer 112 based on search matching resource 330.

[0042] Figure 4 This is an example of a virtual model cache according to an embodiment of the present invention. In this embodiment, the virtual model cache stores virtual models for inference and may also use a cognitive model selector to organize the models. Therefore, this section will include two key components: a model list and a cognitive model generator.

[0043] Model list 410 is a list of the top n optimal model files compiled by model deployment program 112 based on analysis of deep learning models, where n is a predetermined value. Essentially, this is a list of all models stored in the model cache. In one embodiment, each row in the model list describes a virtual model and the slices that make up that model. For example, model 1 would be split into four slices: s1, s2, s3, and s4. Figure 4In the example, s1 is deployed on device 1 with time slot 1, s2 is deployed on device 3 with time slot 2, s3 is deployed on device 4 with time slot 1, and s4 is deployed on device 5 with time slot 1. Time slots are used to locate / position slices on dedicated devices.

[0044] In this embodiment, the model deployment procedure 112 includes a cognitive model generator 420, which is the part of the model deployment procedure 112 that checks the health status of the first n devices for each model, where n is a predetermined value. If the model deployment procedure 112 detects any anomalies in the health status of the first n devices, it will select a new model to run on different devices. This is the part of the model deployment procedure 112 that determines how to update the model cache.

[0045] Figure 5 This is an example of a monitor according to an embodiment of the present invention. Figure 5 In the example, device monitor 510 represents the part of model deployment program 112 that collects the health status of devices for scheduling model deployment, and also includes device resources such as CPU / GPU, disk, and storage. This is Figure 2 An example of device monitor 240. In this embodiment, model deployment program 112 uses scripts that collect information from the device on a schedule and send that information to the virtual model cache. In this embodiment, model deployment program 112 may use open-source device monitoring tools for device monitoring.

[0046] Devices 1551, 2552, 3553, and n 554 represent edge devices monitored by device monitor 510 of model deployment program 112. Heartbeat 520 indicates that model deployment program 112 is collecting information from the devices as planned.

[0047] Figure 6 This describes an embodiment of the invention. Figure 1 The flowchart illustrates the operational steps of a model generation process performed by a model deployment program 112 on a computing device within a distributed data processing environment, for deploying a parallelizable deep learning model by adapting it to the computing device. In an alternative embodiment, the steps of workflow 600 may be performed by any other program while working in conjunction with the model deployment program 112.

[0048] In one embodiment, model deployment procedure 112 continuously collects data from a monitor to monitor the health of devices running deployed slices of deep learning models, or the health of devices of services that can be invoked to run slices of deep learning models that have not yet been deployed. In one embodiment, model deployment procedure 112 encodes features such as GPU, disk speed and capacity, and available memory over time to determine the capabilities of each device to run model slices. In one embodiment, model deployment procedure 112 uses an LSTM model to predict the risk of each device being monitored by the device monitor. In one embodiment, model deployment procedure 112 determines whether a device is high-risk for running a model. In one embodiment, model deployment procedure 112 selects the top n devices, where n is a predetermined number of assessments of device health based on the monitor functionality of model deployment procedure 112, and replaces high-risk / failed devices with healthy devices. In one embodiment, model deployment procedure 112 then updates the model list file with the new slices of the models determined in step 610. In one embodiment, model deployment procedure 112 then terminates the loop.

[0049] It should be understood that embodiments of the present invention provide at least one method for deploying parallelizable deep learning models by adapting them to computing devices. However, Figure 6 This is merely an illustration of one implementation and does not imply any limitation regarding the environments in which different embodiments may be implemented. Those skilled in the art can make many modifications to the described environments without departing from the scope of the invention as set forth in the claims.

[0050] It should be understood that Figure 6 The process depicted illustrates a possible iteration of the operational steps performed by model deployment procedure 112 for deploying a parallelizable deep learning model by adapting it to a computing device, which is repeated each time a deep learning model is to be deployed.

[0051] Model deployment procedure 112 collects monitoring information (step 602). In an embodiment, model deployment procedure 112 collects monitoring information from a monitor (e.g., from...). Figure 2 The device monitor 240 continuously collects data to monitor the health of devices running deployed slices of deep learning models, or the health of devices that can be invoked to run slices of deep learning models that have not yet been deployed.

[0052] Model deployment procedure 112 encodes features (GPU, disk, etc.) over time (step 604). In one embodiment, model deployment procedure 112 encodes features (e.g., CPU capacity, GPU capacity, disk speed and capacity, available memory, etc.) over time to determine the capability each device must have to run a model slice. In one embodiment, the time slice used by the monitor portion of model deployment procedure 112 is a predetermined time period. In another embodiment, the time slice used by the monitor portion of model deployment procedure 112 is received from the user. In yet another embodiment, the time slice used by the monitor portion of model deployment procedure 112 can be selected by any other method known to those skilled in the art.

[0053] Model deployment procedure 112 uses an LSTM model to predict the risk of devices (step 606). In an embodiment, model deployment procedure 112 uses an LSTM model to predict the risk of each device being monitored by a device monitor. In an embodiment, model deployment procedure 112 uses an LSTM model to calculate the risk factors of the devices.

[0054] In one embodiment, the model deployment procedure 112 uses historical system monitoring data to train an LSTM model. In another embodiment, the encoded features may include, but are not limited to, GPU, memory, disk consumption, time-series data, etc., and the labeled result represents the device's health status. In yet another embodiment, the model deployment procedure 112 uses LSTM based on device resource consumption data to predict the device's health status or risk.

[0055] Model deployment procedure 112 determines whether a high-risk device exists (decision box 608). In an embodiment, model deployment procedure 112 determines whether a device is high-risk for running the model. In an embodiment, this is the risk that the model to be deployed cannot run on the device. In an embodiment, there is no risk when the device is healthy and has sufficient capacity to run the model. If the device is not risky, then the model does not need to be split. In an embodiment, if model deployment procedure 112 determines that the device is not high-risk for running the model (“No” branch, decision box 608), then model deployment procedure 112 deploys the model to the device. Then model deployment procedure 112 ends the loop. In an embodiment, if model deployment procedure 112 determines that the device is high-risk for running the model (“Yes” branch, decision box 608), then model deployment procedure 112 proceeds to step 610 to split the model.

[0056] Model deployment procedure 112 selects and replaces the top n devices based on device health status (step 610). In an embodiment, if model deployment procedure 112 determines that the risk factor of any device, as determined in step 606, exceeds a predetermined threshold, then model deployment procedure 112 marks that device as a faulty device. In an embodiment, model deployment procedure 112 then selects the top n devices, where n is a predetermined number of assessments of device health status based on the monitoring capabilities of model deployment procedure 112, and replaces high-risk / faulty devices with healthy devices. In an embodiment, model deployment procedure 112 determines the top n devices by selecting devices with the lowest risk factors. In an embodiment, model deployment procedure 112 removes high-risk / faulty devices from the list of available devices.

[0057] Model deployment procedure 112 updates the model list file (step 612). In an embodiment, model deployment procedure 112 then updates the model list file with the new partitioning of the models determined in step 610, for example... Figure 4 The model list 410. In an embodiment, the model deployment procedure 112 then ends the loop.

[0058] Figure 7 This is a block diagram describing the components of a computing device 110 suitable for a model deployment program 112 according to at least one embodiment of the present invention. Figure 7 A computer 700 is shown; one or more processors 704 (including one or more computer processors); a communication structure 702; a memory 706, including random access memory (RAM) 716 and cache 718; permanent memory 708; a communication unit 712; an I / O interface 714; a display 722; and an external device 720. It should be understood that... Figure 7 The illustration is provided only as an example and does not imply any limitation regarding the environment in which different embodiments may be implemented. Many modifications can be made to the depicted environment.

[0059] As described, computer 700 operates on communication architecture 702, which provides communication between computer processor 704, memory 706, persistent memory 708, communication unit 712, and I / O interface 714. Communication architecture 702 can be implemented using any architecture suitable for transferring data or control information between processor 704 (e.g., microprocessor, communication processor, and network processor), memory 706, external devices 720, and any other hardware components within the system. For example, communication architecture 702 can be implemented using one or more buses.

[0060] Memory 706 and persistent memory 708 are computer-readable storage media. In the depicted embodiment, memory 706 includes RAM 716 and cache 718. Typically, memory 706 may include any suitable volatile or non-volatile computer-readable storage medium. Cache 718 is a fast memory that enhances the performance of processor 704 by storing recently accessed data from RAM 716 and recently accessed data from other memory locations.

[0061] Program instructions for the model deployment program 112 may be stored in persistent memory 708, or more generally, in any computer-readable storage medium, for execution by one or more of the respective computer processors 704 via one or more memories of memory 706. Persistent memory 708 may be a magnetic hard disk drive, a solid-state drive, a semiconductor storage device, a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0062] The media used in the persistent storage 708 can also be removable. For example, a removable hard disk drive can be used for the persistent storage 708. Other examples include optical discs and disks, thumb drives and smart cards, which are inserted into the drive to be transferred to another computer-readable storage medium that is also part of the persistent storage 708.

[0063] In these examples, communication unit 712 provides communication with other data processing systems or devices. In these examples, communication unit 712 includes one or more network interface cards. Communication unit 712 can provide communication using physical and / or wireless communication links. In some embodiments of the invention, sources of different input data can be physically located away from computer 700, allowing input data to be received and output similarly transmitted via communication unit 712.

[0064] I / O interface 714 allows for data input and output with other devices that can be connected to computer 700. For example, I / O interface 714 may provide connectivity to multiple external devices 720, such as keyboards, keypads, touchscreens, microphones, digital cameras, and / or other suitable input devices. External devices 720 may also include portable computer-readable storage media, such as thumb drives, portable optical discs or disks, and memory cards. Software and data used to implement embodiments of the invention (e.g., model deployment program 112) may be stored on such portable computer-readable storage media and may be loaded onto persistent memory 708 via I / O interface 714. I / O interface 714 is also connected to display 722.

[0065] The display 722 provides a mechanism for displaying data to a user and can be, for example, a computer monitor. The display 722 can also function as a touchscreen, such as the display of a tablet computer.

[0066] The programs described herein are identified based on applications that implement them in specific embodiments of the invention. However, it should be understood that any particular procedural terminology used herein is for convenience only, and therefore the invention should not be limited to use only in any particular application identified and / or implied by such terminology.

[0067] The present invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the present invention.

[0068] Computer-readable storage media can be any tangible device capable of retaining and storing instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0069] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0070] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may be personalized to execute computer-readable program instructions by utilizing state information from the computer-readable program instructions in order to perform aspects of this invention.

[0071] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0072] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other apparatus to operate in a particular manner, such that the computer-readable storage medium storing the instructions comprises an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0073] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other apparatus to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other apparatus to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other apparatus perform the functions / actions specified in one or more boxes in a flowchart and / or block diagram.

[0074] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0075] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements over those found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer implementation method, comprising: A deep learning model comprising a recurrent neural network (RNN) is divided into multiple slices by one or more computer processors, wherein each slice comprises at least one layer of a multilayer deep learning model of the RNN, each slice requires computational power, each slice records its preceding and following slices to remember the last state of the input and constructs a dual-linked layer network. The device's monitor collects multiple device resources from multiple devices; Based on the collected data on the multiple devices, predict the risk factors of the multiple devices; Select the top n devices from the plurality of devices, wherein the top n devices have the lowest risk factor; Based on the computing power required for each slice and the device resources of the top n devices collected, the slices are distributed to the top n devices by the cache of the virtual model; and The deep learning model is run on the first n devices.

2. The computer-implemented method according to claim 1, wherein, Each slice is a different network layer among multiple different network layers in a deep learning model.

3. The computer-implemented method according to claim 1, wherein, The segmentation includes: In response to determining that the deep learning model is not easily parallelizable, one or more computer processors divide the deep learning model into multiple slices based on a predetermined set of rules, wherein the predetermined set of rules divides the deep learning model into multiple minimally parallelizable layers.

4. The computer-implemented method according to claim 1 further includes: Features of the plurality of devices are encoded by one or more computer processors in time slices, wherein the features include, but are not limited to, at least one of the following: central processing unit (CPU) capacity, graphics processing unit (GPU) capacity; And disk capacity, wherein the time slice is a predetermined time period; A risk factor is predicted for each of the plurality of devices by the one or more computer processors, wherein the risk factor is predicted using a Long Short-Term Memory (LSTM) model; In response to determining that the risk factor of any of the plurality of devices exceeds a predetermined threshold, the one or more computer processors select the n devices with the lowest risk factors from the plurality of devices, where n is a predetermined number; In response to selecting the first n devices from the plurality of devices, the one or more computer processors create one or more new virtual models from the plurality of slices, wherein the one or more new virtual models are based on the resources of the first n devices from the plurality of devices; and The cache is updated by the one or more computer processors using the one or more new virtual models.

5. The computer-implemented method according to claim 4, wherein, The device monitor collects device resource information from the multiple devices according to a time schedule.

6. The computer-implemented method according to claim 1, further comprising: The health status of each of the plurality of devices is monitored by one or more computer processors; In response to determining that the health status of any of the plurality of devices is below a predetermined threshold, the one or more computer devices mark any of the devices as a faulty device; The faulty device is removed from the plurality of devices by one or more computer processors; A new virtual model is created by one or more computer processors, wherein the new virtual model does not include the faulty device; The new virtual model is deployed to one or more of the plurality of devices by one or more computer processors; as well as The cache is updated by one or more computer processors using the new virtual model.

7. The computer-implemented method according to claim 1, further comprising: One or more virtual models are created from multiple slices by the one or more computer processors, wherein the one or more virtual models are based on the capabilities of the multiple devices on which the one or more virtual models will be deployed, and wherein each of the multiple virtual models contains each of the multiple slices; The one or more computer processors confirm that the one or more virtual models match the deep learning model, wherein the confirmation that the one or more virtual models match the deep learning model is determined by one or more predetermined verification rules.

8. A computer program product comprising one or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the program instructions including instructions for performing the following operations: The deep learning model, including recurrent neural networks (RNNs), is divided into multiple slices, where... Each of the plurality of slices includes at least one layer of the deep learning model of the recurrent neural network. Each slice requires computational power, and each slice records its preceding and following slices to remember the last state of the input and constructs a dual-linked layer network. The device's monitor collects multiple device resources from multiple devices; Based on the collected data on the multiple devices, predict the risk factors of the multiple devices; Select the top n devices from the plurality of devices, wherein the top n devices have the lowest risk factor; Based on the computing power required for each slice and the device resources of the top n devices collected, the slices are distributed to the top n devices by the cache of the virtual model; and The deep learning model is run on the first n devices.

9. The computer program product according to claim 8, wherein, Each slice is a different network layer among multiple different network layers in a deep learning model.

10. The computer program product according to claim 8, wherein, The computer program product further includes the following program instructions stored on the one or more computer-readable storage media: In response to determining that the deep learning model is not easily parallelizable, the deep learning model is divided into multiple slices based on a predetermined set of rules, wherein the predetermined set of rules divides the deep learning model into multiple minimum parallelizable layers.

11. The computer program product according to claim 8, wherein, The computer program product further includes one or more of the following program instructions stored on the one or more computer-readable storage media: The features of the multiple devices are encoded according to time slices, and the features include, but are not limited to, the capacity of the central processing unit (CPU) and the capacity of the graphics processing unit (GPU). and at least one of the disk capacity, wherein the time slice is a predetermined time period; For each of the plurality of devices, a risk factor is predicted, wherein the risk factor is predicted using a Long Short-Term Memory (LSTM) model; In response to determining that the risk factor of any of the plurality of devices exceeds a predetermined threshold, the n devices with the lowest risk factors among the plurality of devices are selected, where n is a predetermined number; In response to selecting the first n devices from the plurality of devices, one or more new virtual models are created from the plurality of slices, wherein the one or more new virtual models are based on the resources of the first n devices from the plurality of devices; and Update the cache with the one or more new virtual models.

12. The computer program product according to claim 11, wherein, The device monitor collects device resource information from the multiple devices according to a schedule.

13. The computer program product according to claim 8, wherein, The computer program product further includes one or more of the following program instructions stored on one or more computer-readable storage media: Monitor the health status of each of the multiple devices; In response to determining that the health status of any of the plurality of devices is below a predetermined threshold, any of the devices is marked as a faulty device; Remove the faulty device from the plurality of devices; Create a new virtual model, wherein the new virtual model does not include the faulty device; Deploy the new virtual model to one or more of the multiple devices; as well as The cache is updated using the new virtual model.

14. The computer program product according to claim 8, wherein, The computer program product further includes one or more of the following program instructions stored on the one or more computer-readable storage media: One or more virtual models are created from multiple slices, wherein the one or more virtual models are based on the capabilities of the multiple devices on which the one or more virtual models will be deployed, and wherein each of the multiple virtual models contains each of the multiple slices; The one or more virtual models are confirmed to match the deep learning model, wherein the confirmation of the match between the one or more virtual models and the deep learning model is determined by one or more predetermined verification rules.

15. A computer system comprising: One or more computer processors; One or more computer-readable storage media; as well as Program instructions stored on the one or more computer-readable storage media for execution by at least one of the one or more computer processors, the stored program instructions including instructions for performing the following operations: A deep learning model including a recurrent neural network (RNN) is divided into multiple slices, wherein each slice includes at least one layer of the deep learning model of the recurrent neural network, each slice requires computational power, each slice records its preceding and following slices to remember the last state of the input and constructs a dual-linked layer network. The device's monitor collects multiple device resources from multiple devices; Based on the collected data on the multiple devices, predict the risk factors of the multiple devices; Select the top n devices from the plurality of devices, wherein the top n devices have the lowest risk factor; Based on the computing power required for each slice and the device resources of the top n devices collected, the slices are distributed to the top n devices by the cache of the virtual model; and The deep learning model is run on the first n devices.

16. The computer system according to claim 15, wherein, Each slice is a different network layer among multiple different network layers in a deep learning model.

17. The computer system according to claim 15, wherein, The program instructions for segmenting a deep learning model, including a recurrent neural network (RNN), into multiple slices include instructions for performing the following operations: In response to determining that the deep learning model is not easily parallelizable, the deep learning model is divided into multiple slices based on a predetermined set of rules, wherein the predetermined set of rules divides the deep learning model into multiple minimum parallelizable layers.

18. The computer system of claim 15, further comprising one or more of the following program instructions stored on the one or more computer-readable storage media: The features of the plurality of devices are encoded according to time slices, wherein the features include, but are not limited to, the CPU capacity of the central processing unit and the GPU capacity of the graphics processing unit. and at least one of the disk capacity, and of which, The time slice is a predetermined time period; For each of the plurality of devices, a risk factor is predicted, wherein the risk factor is predicted using a Long Short-Term Memory (LSTM) model; In response to determining that the risk factor of any of the plurality of devices exceeds a predetermined threshold, the n devices with the lowest risk factors among the plurality of devices are selected, where n is a predetermined number; In response to selecting the first n devices from the plurality of devices, one or more new virtual models are created from the plurality of slices, wherein the one or more new virtual models are based on the resources of the first n devices from the plurality of devices; and Update the cache with the one or more new virtual models.

19. The computer system according to claim 15, wherein, The system also includes one or more program instructions stored on the one or more computer-readable storage media for the following operations: Monitor the health status of each of the multiple devices; In response to determining that the health status of any of the plurality of devices is below a predetermined threshold, any of the devices is marked as a faulty device; Remove the faulty device from the plurality of devices; Create a new virtual model, wherein the new virtual model does not include the faulty device; Deploying the new virtual model to one or more of the plurality of devices; and Update the cache with the new virtual model.

20. The computer system according to claim 15, wherein, The computer system further includes one or more program instructions stored on the one or more computer-readable storage media for the following operations: The one or more virtual models are created from the plurality of slices, wherein the one or more virtual models are based on the capability of multiple devices on which the one or more virtual models are deployed, and wherein each of the plurality of virtual models contains each of the plurality of slices; The one or more virtual models are confirmed to match the deep learning model, wherein the confirmation that the one or more virtual models match the deep learning model is determined by one or more predetermined verification rules.

Images