Throughput-optimized deployment method for machine learning inference tasks based on cloud-edge collaboration

Abstract

The invention discloses a throughput-optimized machine learning inference task deployment method based on cloud-edge collaboration. Transmission delay, and then establish a throughput-optimized inference task deployment optimization problem, and design an efficient deployment strategy based on the idea of ​​dynamic programming to obtain the optimal deployment plan. Based on the cloud-edge collaboration scenario, the present invention divides the inference task into serial stages, and uses pipelines to process the inference task in parallel, so as to optimize the throughput of the inference task.

Description

technical field [0001] The invention relates to the field of distributed computing and task scheduling, in particular to a machine learning inference task deployment method based on throughput optimization of cloud-edge collaboration. Background technique [0002] With the rapid development of smart devices, the demand for smart services has surged, and more and more smart devices are connected to the network to process the massively generated streaming sensor data. However, the current machine learning system running on terminal devices faces the following three problems: First, the connection of smart devices to the cloud and the surge in mobile data traffic. According to Cisco's forecast, more than 12 billion smart devices will be connected to the network by 2022 Serve. Therefore, sending raw sensor data, such as video from a surveillance camera, to a remote cloud may congest the backhaul network, resulting in reduced throughput and long response times, as well as the ri...

AI Technical Summary

Problems solved by technology

However, the current machine learning system running on terminal devices faces the following three problems: First, the connection of smart devices to the cloud and the surge in mobile data traffic. According to Cisco's forecast, more than 12 billion smart devices will be connected to the network by 2022 Serve

However, the naive model division method may lead to two problems: first, the amount of data transmitted between different stages of the model is too large, which may lead to significant data transmission delay in the hierarchical mobile network environment; second, the cloud edge The computing device resources in the collaborative system are heterogeneous. If the stage with a large amount of computation is deployed on a device with a low computing capacity, the long computation delay of this stage will become the throughput bottleneck of the parallel processing of the pipeline.

Images