Task scheduling method for reducing response delay of artificial intelligence model on edge device

Abstract

The invention relates to a task scheduling method for reducing response delay of an artificial intelligence model on edge equipment. The method comprises the following steps: 1) collecting and sorting data; 2) constructing and training a prediction model; 3) performing operation scheduling on the prediction model; and 4) decision execution. The method is used for scheduling the model request in the non-perceptual edge computing normal form of the server and reducing the model response delay, is suitable for edge equipment, and can effectively reduce the artificial intelligence model response delay so as to give full play to the potential of edge computing in the field of artificial intelligence and improve the performance of the edge equipment. And the ever-increasing requirements on low-delay and high-efficiency artificial intelligence service are met.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence model reasoning technology, and specifically to a task scheduling method for reducing artificial intelligence model response delay on an edge device. The present invention relates to a method for scheduling model requests in a server-unaware edge computing paradigm to reduce model response delay. Background Art

[0002] In today's digital age, artificial intelligence models have shown extremely wide application scenarios and powerful capabilities. From intelligent voice assistants to help people's daily communication and information acquisition, to the in-depth application of image recognition technology in security monitoring, medical imaging diagnosis and other fields, to the intelligent recommendation system for e-commerce, content distribution and other industries to bring precise service improvement, artificial intelligence models have become the core driving force to promote innovation and efficiency in various industries. However, there are many disadvantages in traditionally deploying and running artificial intelligence models in cloud data centers. Among them, privacy risk is an issue that cannot be ignored. The centralized processing and storage of a large amount of sensitive data in cloud data centers increases the potential threat of data leakage. In addition, the existence of wide area networks inevitably brings significant delays. Since data needs to be transmitted over long distances between terminals and cloud data centers, factors such as network congestion and bandwidth limitations will cause a significant increase in data transmission time, which will seriously affect the response efficiency of artificial intelligence models and cannot meet the needs of some application scenarios with high real-time requirements. In this context, deploying and running artificial intelligence models on edge devices has become an attractive alternative. Edge devices are close to data sources or user terminals, which can effectively reduce the data transmission distance, reduce latency and improve the timeliness of data processing. However, edge devices are resource-constrained, with relatively limited computing power, storage capacity, and energy supply. They cannot run a model and wait for requests like traditional cloud servers. At this point, the server-unaware edge computing paradigm comes into being. In this paradigm, AI models are deployed and run in containers, i.e., they perform reasoning operations. This approach has many advantages, such as being able to achieve fine-grained resource management, flexibly allocating resources according to actual needs, and avoiding resource waste; at the same time, resources can be used on demand, so that edge devices can still efficiently respond to requests from AI models under limited resource conditions.

[0003] The response delay of the model is one of the key indicators to measure the performance of the edge computing platform. Specifically, it refers to the time interval between the server-aware edge computing platform receiving the request and the model returning the inference result, mainly covering the cold start time and model running time of the model. Cold start delay refers to the time taken from starting the container to starting the model inference when a request for a model is received, if the corresponding container has not been started. The inference delay refers to the time taken for the model to perform inference calculations and generate results after receiving the input data. In a classic server-aware edge computing platform, there are usually multiple edge devices that jointly serve requests for artificial intelligence models. Since the load conditions of different edge devices are in dynamic change, for edge devices with limited resources, the cold start time and inference time of the container corresponding to the same model under different load conditions will be significantly different. For example, when an edge device is in a high load state, its system resources are occupied in large quantities. At this time, starting a new model container may significantly extend the cold start time due to insufficient resource allocation; at the same time, when running the model for inference, the inference speed may also be slowed down due to resource competition, and the inference delay may increase. The uncertainty of cold start time and inference time caused by this load change poses a huge challenge to the scheduling of requests.

[0004] Existing research on scheduling requests to reduce model response latency mostly focuses on scenarios where models are run on powerful cloud servers. In a cloud server environment, since the computing power of the device is sufficient, the impact of the load on the cold start and running latency of the container is relatively small. However, the limited resource characteristics of edge devices make them fundamentally different from cloud servers, and the scheduling strategies in the cloud server scenario cannot be directly applied.

[0005] Therefore, there is an urgent need to design a task scheduling method specifically suitable for edge devices that can effectively reduce the response delay of artificial intelligence models, so as to fully tap the potential of edge computing in the field of artificial intelligence and meet the growing demand for low-latency, high-efficiency artificial intelligence services. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and to provide a task scheduling method for reducing the response delay of artificial intelligence models on edge devices. The present invention is suitable for edge devices and can effectively reduce the response delay of artificial intelligence models, so as to give full play to the potential of edge computing in the field of artificial intelligence and meet the growing demand for low-latency and high-performance artificial intelligence services. When the server-unaware edge computing platform receives a model request, it can accurately schedule it to the appropriate device according to the load condition of the edge device, effectively reducing the model cold start and inference delay, overcoming the problem of limited resources, and improving the response efficiency of edge devices running artificial intelligence models.

[0007] The present invention is implemented by the following technical solution: a task scheduling method for reducing the response delay of an artificial intelligence model on an edge device, the method comprising the following steps: 1) Collect and organize data; 2) Build and train prediction models; 3) Scheduling the operation of the prediction model; 4) Decision execution.

[0008] Preferably, the step 1) is specifically as follows: For a variety of different artificial intelligence models, corresponding containers are built on different edge devices, and different device load levels are set, including low load, medium load, high load, etc. At the same time, the two states of cold start and non-cold start are distinguished. For each combination, a large amount of response delay data is collected, and all collected data is sorted, classified and stored to form a set of original data.

[0009] Preferably, the step 2) is specifically as follows: 1) Based on the collected data, select a suitable machine learning model, such as a neural network model, divide the collected raw data into training set, validation set, and test set according to a certain ratio, use the device load, whether it is a cold start, etc. as input features, and use the corresponding response delay as the output label to train the deep neural network. 2) During the training process, a suitable loss function, such as the mean square error loss function, is used to continuously adjust the weights and biases of the neural network through the back propagation algorithm until better performance indicators, such as accuracy and mean square error, are achieved on the validation set and meet the preset threshold requirements; 3) After training, the model is finally evaluated on the test set to ensure that the model has good generalization ability.

[0010] Preferably, the step 3) is specifically as follows: When the server-aware edge computing platform receives a request for a specific artificial intelligence model, it first obtains the current load information of each edge device, organizes this information and data such as the model type corresponding to the request into a format suitable for prediction model input, and then inputs it into the trained deep neural network prediction model. The prediction model calculates the response delay prediction value of the container corresponding to the model running on each edge device based on the input data.

[0011] Preferably, the step 4) is specifically as follows: 1) The prediction model calculates the response delay value of the container corresponding to the model running on different edge devices at this time, compares the response delay prediction values ​​of each edge device output by the prediction model, and determines the edge device with the lowest response delay; 2) Schedule the request of the artificial intelligence model to the edge device with the lowest response delay to realize the operation and reasoning of the model, thereby reducing the overall artificial intelligence model response delay.

[0012] The beneficial effects of the present invention are as follows: Compared with the prior art, the task scheduling method for reducing the response delay of artificial intelligence models on edge devices provided by the present invention has the following significant advantages and positive effects: (1) A data collection and analysis system was built. By collecting response delay data of different AI model containers under various device loads and cold start conditions, and building a prediction model based on a deep neural network, the response delay of model containers running on different edge devices can be estimated, providing data and model support for efficient scheduling.

[0013] (2) The proposed task scheduling decision method effectively addresses the problem of limited edge device resources and multi-task resource competition. Based on the prediction model results, the model request is scheduled to the edge device with the lowest response delay, achieving optimal resource allocation, reducing response delay, and improving the overall system operation efficiency.

[0014] (3) In an experimental environment containing multiple edge devices, a variety of typical artificial intelligence model loads were used for testing and evaluation. The results showed that the latency of this method was lower than that of the lowest load first scheduling method. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of the architecture for running containers containing artificial intelligence models on a server-agnostic edge computing platform. DETAILED DESCRIPTION

[0016] In order to enable those skilled in the art to more clearly understand the objectives, technical solutions and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0017] In the description of the present invention, it is necessary to understand that the orientation or position relationship indicated by the terms such as "up", "down", "left", "right", "inside", "outside", "horizontal" and "vertical" is based on the orientation or position relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention, rather than indicating or implying that the device or original referred to must have a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0018] The present invention will be described in detail below with reference to the accompanying drawings: Figure 1 As shown, a task scheduling method for reducing the response delay of an artificial intelligence model on an edge device comprises the following steps: 1) Collect and organize data; 2) Build and train prediction models; 3) Scheduling the operation of the prediction model; 4) Decision execution.

[0019] The step 1) is specifically as follows: For a variety of different artificial intelligence models, corresponding containers are built on different edge devices, and different device load levels are set, including low load, medium load, high load, etc. At the same time, the two states of cold start and non-cold start are distinguished. For each combination, a large amount of response delay data is collected, and all collected data is sorted, classified and stored to form a set of original data.

[0020] The step 2) is specifically as follows: 1) Based on the collected data, select a suitable machine learning model, such as a neural network model, divide the collected raw data into training set, validation set, and test set according to a certain ratio, use the device load, whether it is a cold start, etc. as input features, and use the corresponding response delay as the output label to train the deep neural network. 2) During the training process, a suitable loss function, such as the mean square error loss function, is used to continuously adjust the weights and biases of the neural network through the back propagation algorithm until better performance indicators, such as accuracy and mean square error, are achieved on the validation set and meet the preset threshold requirements; 3) After training, the model is finally evaluated on the test set to ensure that the model has good generalization ability.

[0021] The step 3) is specifically as follows: When the server-aware edge computing platform receives a request for a specific artificial intelligence model, it first obtains the current load information of each edge device, organizes this information and data such as the model type corresponding to the request into a format suitable for prediction model input, and then inputs it into the trained deep neural network prediction model. The prediction model calculates the response delay prediction value of the container corresponding to the model running on each edge device based on the input data.

[0022] The step 4) is specifically as follows: 1) The prediction model calculates the response delay value of the container corresponding to the model running on different edge devices at this time, compares the response delay prediction values ​​of each edge device output by the prediction model, and determines the edge device with the lowest response delay; 2) Schedule the request of the artificial intelligence model to the edge device with the lowest response delay to realize the operation and reasoning of the model, thereby reducing the overall artificial intelligence model response delay.

[0023] Example 1 For ease of understanding, the task scheduling method provided by the present invention for reducing the response delay of an artificial intelligence model on an edge device is described in detail below.

[0024] Step 1, data collection preparation phase. For a variety of different artificial intelligence models, corresponding containers are built on different edge devices. Different device load levels are set, including low load, medium load, high load, etc., and cold start and non-cold start states are distinguished. For each combination, a large amount of response delay data is collected. For example, for the image recognition model, when edge device A is under low load and cold start, multiple requests are sent and their response delays are recorded; when edge device B is under high load and non-cold start, corresponding data is also collected for the same image recognition model. All collected data are sorted and classified and stored to form a set of original data.

[0025] Step 2, prediction model training phase. A deep neural network model is selected as the prediction model architecture. The collected raw data is divided into training set, validation set and test set according to a certain ratio. The deep neural network is trained with the equipment load condition, whether it is a cold start, etc. as input features and the corresponding response delay as the output label. During the training process, a suitable loss function, such as the mean square error loss function, is used to continuously adjust the weights and biases of the neural network through the back propagation algorithm until good performance indicators are achieved on the validation set, such as accuracy, mean square error, etc., which meet the preset threshold requirements. After the training is completed, the model is finally evaluated on the test set to ensure that the model has good generalization ability.

[0026] Step 3, run the scheduling phase. When the server-aware edge computing platform receives a request for a specific artificial intelligence model, it first obtains the current load information of each edge device, and organizes this information and data such as the model type corresponding to the request into a format suitable for the input of the prediction model. Then it is input into the trained deep neural network prediction model, and the prediction model calculates the response delay prediction value of the container corresponding to the model running on each edge device based on the input data. For example, for a request for a speech recognition model, the prediction model calculates the response delay prediction values ​​on edge devices C, D, E, etc.

[0027] Step 4, decision execution phase. Compare the response delay prediction values ​​of each edge device output by the prediction model to find the edge device with the lowest response delay. For example, after comparison, it is found that the response delay prediction value of edge device D is the lowest, then the request of the speech recognition model is scheduled to edge device D for processing. After receiving the request, edge device D starts the corresponding container (if it is a cold start, complete the container startup process first), runs the artificial intelligence model for reasoning calculation, and returns the final reasoning result to the server-aware edge computing platform. The platform then feeds back the result to the request initiator, thereby completing the entire task scheduling and model operation process, effectively reducing the response delay of the artificial intelligence model.

[0028] Anyone familiar with the technology may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A task scheduling method for reducing artificial intelligence model response delay on edge devices, characterized in that: The method comprises the following steps: 1) Collect and organize data; 2) Build and train prediction models; 3) Scheduling the operation of the prediction model; 4) Decision execution.

2. The task scheduling method for reducing the response delay of an artificial intelligence model on an edge device according to claim 1, characterized in that: The step 1) is specifically as follows: For a variety of different artificial intelligence models, corresponding containers are built on different edge devices, and different device load levels are set, including low load, medium load, and high load. At the same time, the two states of cold start and non-cold start are distinguished. For each combination, a large amount of response delay data is collected, and all collected data is organized and classified and stored to form a set of original data.

3. The task scheduling method for reducing the response delay of an artificial intelligence model on an edge device according to claim 1, characterized in that: The step 2) is specifically as follows: 1) Based on the collected data, select a suitable machine learning model, such as a neural network model, divide the collected raw data into training set, validation set, and test set according to a certain ratio, use the device load and whether it is a cold start as input features, and use the corresponding response delay as the output label to train the deep neural network. 2) During the training process, a suitable loss function, such as the mean square error loss function, is used to continuously adjust the weights and biases of the neural network through the back propagation algorithm until better performance indicators are achieved on the validation set, such as accuracy and mean square error that meet the preset threshold requirements; 3) After training, the model is finally evaluated on the test set to ensure that the model has good generalization ability.

4. The task scheduling method for reducing the response delay of an artificial intelligence model on an edge device according to claim 1, characterized in that: The step 3) is specifically as follows: When the server-aware edge computing platform receives a request for a specific artificial intelligence model, it first obtains the current load information of each edge device, organizes this information and the model type data corresponding to the request into a format suitable for prediction model input, and then inputs it into the trained deep neural network prediction model. The prediction model calculates the response delay prediction value of the container corresponding to the model running on each edge device based on the input data.

5. The task scheduling method for reducing the response delay of an artificial intelligence model on an edge device according to claim 1, characterized in that: The step 4) is specifically as follows: 1) The prediction model calculates the response delay value of the container corresponding to the model running on different edge devices at this time, compares the response delay prediction values ​​of each edge device output by the prediction model, and determines the edge device with the lowest response delay; 2) Schedule the request of the AI ​​model to the edge device with the lowest response delay to realize the operation and reasoning of the model, thereby reducing the overall AI model response delay.

Images