Model deployment method, model deployment device, terminal equipment and readable storage medium

Abstract

The invention discloses a model deployment method, a model deployment device, terminal equipment and a readable storage medium. The method comprises the following steps: obtaining an operator model set of a to-be-deployed deep neural network model; performing operator fusion or operator segmentation processing on operator models meeting a preset condition in the operator model set, and obtaining a processed operator model set; obtaining the operation time of each operator model in the processed operator model set on each device in a device set for deploying the model, and obtaining an operation time set; on the basis of the running time set, combining operator models in the processed operator model set through adoption of a preset search method, and obtaining a sub-model set; and deploying a to-be-deployed deep neural network model on the device set based on the sub-model set. According to the method, equipment with different computing power can be fully compatible, and the operation efficiency and the global throughput rate can be improved.

Description

technical field [0001] The invention belongs to the technical field of artificial intelligence, relates to the field of model deployment, and in particular to a model deployment method, a model deployment device, terminal equipment and a readable storage medium. Background technique [0002] Machine Learning (ML) is one of the fastest-growing fields in computer science today. Typical machine learning techniques can train statistical models for specific applications on a large number of pre-collected data sets to update model parameters (also called as "weights") until convergence; use the trained model for inference, that is, predicting outcomes on new data. Deep learning based on neural networks has become the most widely used ML algorithm due to its excellent results. The neural network model is a multi-layer directed acyclic graph (DAG). The model usually consists of operations such as convolution, matrix multiplication, pooling, batch regularization, etc., which are org...

AI Technical Summary

Problems solved by technology

[0004] However, the above studies failed to well address the deployment of large parameter models

Specifically, most of the current work defaults to the fact that the machine running the deep learning model has enough memory and hard disk, and the model can be run directly on a single machine to obtain the output of the model; on cloud server clusters, distributed storage is generally used. Get better file read and write speed, and then get a better throughput rate, but this will waste more time on transmission, and cannot fully utilize the computing power of all devices

With the development of deep learning, researchers have found that computing resources are always limited; in addition, the equipment responsible for computing usually requires stronger computing power, and the computing power responsible for reading and writing is idle most of the time. A huge waste, on the other hand also makes it difficult to improve the throughput

Images