Function caching method in edge service-free computing of service deep learning

Abstract

The invention relates to a function caching method in edge service-free computing of service deep learning, which comprises the following steps of: (1) selection operation in an edge server: on the basis of a greedy strategy, according to a container state and request information on a current multi-edge server, selecting the edge server; selecting the operation with the minimum delay from the local cold start, forwarding to other servers and bypassing to the cloud server to serve the function request; (2) the multi-edge server function in the step (1) is processed in a function caching mode. According to the method designed by the invention, a function caching strategy method can be optimized, so that the total delay of all requests is minimized, and the edge no-service computing performance oriented to deep learning is improved.

Description

Technical Field

[0001] The present invention belongs to the field of edge intelligence technology, and specifically is a function caching method in edge serverless computing that serves deep learning. More specifically, it is a method for optimizing function caching through relay and bypass mechanisms for requests between multiple edge servers in an edge serverless computing environment that serves deep learning, so as to reduce the total delay of request services. Background Art

[0002] Serverless computing is a new computing paradigm that allows developers to run application code without managing servers. In addition, serverless platforms automatically scale up or down capacity to meet demand without the need for capacity planning or provisioning. This greatly simplifies and accelerates application development and deployment, while also reducing costs because developers only pay for the resources they use. For deep learning services, serverless edge computing provides a flexible and efficient solution for model inference services. Traditional deep learning applications usually require provisioned servers and expensive hardware resources, while serverless computing allocates computing resources on demand, allowing users to pay only for the actual computing time used, thereby effectively reducing operation and maintenance costs. In addition, serverless computing supports elastic computing resource scheduling and can dynamically allocate corresponding computing power according to task requirements, thereby maximizing resource utilization efficiency. Edge computing involves deploying computing resources at the edge of the network close to devices and users, making it particularly suitable for latency-sensitive or bandwidth-intensive applications. Serverless edge computing takes advantage of serverless computing and applies it to the edge of the network, which means that model inference is performed closer to the user, thereby reducing latency and improving performance. However, this model also faces some challenges, the most prominent of which is latency, especially cold start latency. In serverless computing, user services (such as model inference requests, user-defined function requests, etc.) are usually called functions, and function execution is usually completed in containers. When a container is first created, it needs to go through steps such as startup, image download, and dependency installation. These steps will introduce additional delays, which are called cold start delays.

[0003] Caching containers in memory is an effective way to alleviate cold starts, that is, after the container processes a function request, it is cached in memory instead of terminating it immediately, and reused when the same function is requested, and an efficient function caching strategy is designed to make full use of limited memory. However, existing work is mainly oriented towards cloud computing environments, lacking consideration of edge resource-constrained environments. Designing function caching strategies in serverless edge computing will bring some new challenges: 1) Limited resources: Edge servers usually have less CPU, memory, and disk space than cloud servers. This means that they can handle fewer function requests and initializing containers may require longer delays. 2) Uneven cold start delay: Cold start delays can vary greatly depending on programming languages, deep learning frameworks, and package sizes. 3) Multiple edge servers and clouds: In serverless edge computing, there are multiple edge servers and cloud servers, and function requests can be routed to different servers. This means that the function may not always be executed on the same server that requested or cached it before. Since functions need to be deployed or migrated between different servers, cold starts may occur more frequently. In some cases, due to the availability of different containers on different servers, it may be necessary to relay requests to other edge servers or bypass to cloud servers to reduce cold start latency.

[0004] Therefore, a function caching method in edge serverless computing for deep learning is designed to overcome the above problems. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a function caching method in edge serverless computing for serving deep learning. The method can optimize the function caching strategy method, thereby minimizing the total delay of all requests and improving the edge serverless computing performance for deep learning.

[0006] The present invention is implemented by the following technical solution: a function caching method in edge serverless computing for serving deep learning, the method comprising the following steps:

[0007] (1) Selecting operations in edge servers: Based on a greedy strategy, according to the current container status and request information on multiple edge servers, the operation with the shortest latency is selected from local cold start, forwarding to other servers, and bypassing to cloud servers to serve the function request;

[0008] (2) The multi-edge server functions in step 1) are processed by function caching.

[0009] Preferably, the selection operation in the edge server in step 1) is specifically:

[0010] 1) Receive input, including cold start delay, execution time, delay in relaying requests to other edge servers, and queuing delay on the server;

[0011] 2) Check whether there is a container serving the current function in the current edge system. If there are no idle resources, cold start the container locally, i.e. create a container service request and calculate the operation delay;

[0012] 3) Check whether there are idle containers serving the current function in other edge servers. If so, calculate the delay of cold-starting the container and executing the function on the local server, queuing and executing the function on the local server, relaying to the hot-start server and executing the function, and select the operation with the smallest delay;

[0013] 4) If there is no idle container for the current function, check whether there is a running container serving the current function. If so, calculate the delay of cold-starting the container on the local server and executing the function, queuing and executing the function on the local server, relaying to the cold-start server to start the function and execute the function, and select the operation with the smallest delay;

[0014] 5) Return the minimum delay and the corresponding server.

[0015] Preferably, the specific steps of the function caching method in step 2) are:

[0016] 1) System initialization: The memory of the edge server is initially empty. The system uses containers to manage computing resources. Each container contains three pieces of information: edge server ID, function ID, and completion time. A completion time of zero indicates that the container is idle.

[0017] 2) When a request is received, the system first checks whether there is an idle container for the corresponding function on the target edge server. If so, the container is used directly to process the request, with a delay equal to the execution time of the function;

[0018] 3) If there are no free containers on the target server and the system only supports relays between edge servers, the system will run the method to select operations in the edge server and select the operation that minimizes the delay based on the current container status;

[0019] 4) After the server is determined, if a new container needs to be created, the system will first check whether there is enough remaining memory. If there is insufficient memory, the existing container needs to be replaced and then a new container needs to be created on the server;

[0020] 5) When cloud server bypass is supported, the system will first assume that the required container exists on the target server and set the confidence of the function. After running the replacement algorithm, it will decide whether to cold start the container on the edge server or forward the request to the cloud server for processing based on the weight value;

[0021] 6) Latency Calculation,If the request is forwarded to the cloud server, the final latency time will include the sum of the processing time of the cloud server and the execution time of the edge server.

[0022] Preferably, the function cache mode supports bypass and relay.

[0023] The beneficial effects of the present invention are as follows:

[0024] Compared with the existing technology, the method of function caching on multiple edge servers in edge serverless computing proposed in this paper effectively solves the problems of limited resources, inconsistent cold start delay, and inter-server relay through relay and detour, and achieves the following positive effects:

[0025] (1) This method supports a practical online function caching method in edge serverless computing that relays or bypasses to cloud servers on multiple edge servers, minimizing the total latency of serving all function requests.

[0026] (2) This method supports distributed requests in serverless edge computing scenarios with multiple servers and designs a greedy strategy to solve the online function caching problem under the challenge of non-uniform cold start delay.

[0027] (3) This method is extensively simulated in the real-world datasets Azure Functions Trace and The Function ColdStart Traces from Alibaba Cloud Function Compute. Experimental results show that compared with existing methods, this method can reduce latency by 52.2% and 73.27% if bypass is not supported, and by 53.63% and 98% if bypass is allowed, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the edge serverless computing model of the present invention.

[0029] Figure 2 A schematic diagram of selecting an operation in the edge server of the present invention.

[0030] Figure 3 Schematic diagram of a multi-edge server function caching method that supports bypassing and relaying. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to more clearly understand the objectives, technical solutions and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0032] In the description of the present invention, it is necessary to understand that the orientation or position relationship indicated by the terms such as "up", "down", "left", "right", "inside", "outside", "horizontal" and "vertical" is based on the orientation or position relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention, rather than indicating or implying that the device or original referred to must have a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0033] The present invention will be described in detail below with reference to the accompanying drawings: Figure 1-3 As shown, a function caching method in edge serverless computing for deep learning is provided, and the method comprises the following steps:

[0034] (1) Selecting operations in edge servers: Based on a greedy strategy, according to the current container status and request information on multiple edge servers, the operation with the shortest latency is selected from local cold start, forwarding to other servers, and bypassing to cloud servers to serve the function request;

[0035] (2) The multi-edge server functions in step 1) are processed by function caching.

[0036] The selection operation in the edge server in step 1) is specifically:

[0037] 1) Receive input, including cold start delay, execution time, delay in relaying requests to other edge servers, and queuing delay on the server;

[0038] 2) Check whether there is a container serving the current function in the current edge system. If there are no idle resources, cold start the container locally, i.e. create a container service request and calculate the operation delay;

[0039] 3) Check whether there are idle containers serving the current function in other edge servers. If so, calculate the delay of cold-starting the container and executing the function on the local server, queuing and executing the function on the local server, relaying to the hot-start server and executing the function, and select the operation with the smallest delay;

[0040] 4) If there is no idle container for the current function, check whether there is a running container serving the current function. If so, calculate the delay of cold-starting the container on the local server and executing the function, queuing and executing the function on the local server, relaying to the cold-start server to start the function and execute the function, and select the operation with the smallest delay;

[0041] 5) Return the minimum delay and the corresponding server.

[0042] The function cache method in step 2) supports bypass and relay, and its specific steps are:

[0043] 1) System initialization: The memory of the edge server is initially empty. The system uses containers to manage computing resources. Each container contains three pieces of information: edge server ID, function ID, and completion time. A completion time of zero indicates that the container is idle.

[0044] 2) When a request is received, the system first checks whether there is an idle container for the corresponding function on the target edge server. If so, the container is used directly to process the request, with a delay equal to the execution time of the function;

[0045] 3) If there are no free containers on the target server and the system only supports relays between edge servers, the system will run the method to select operations in the edge server and select the operation that minimizes the delay based on the current container status;

[0046] 4) After the server is determined, if a new container needs to be created, the system will first check whether there is enough remaining memory. If there is insufficient memory, the existing container needs to be replaced and then a new container needs to be created on the server;

[0047] 5) When cloud server bypass is supported, the system will first assume that the required container exists on the target server and set the confidence of the function. After running the replacement algorithm, it will decide whether to cold start the container on the edge server or forward the request to the cloud server for processing based on the weight value;

[0048] 6) Latency Calculation,If the request is forwarded to the cloud server, the final latency time will include the sum of the processing time of the cloud server and the execution time of the edge server.

[0049] Example 1

[0050] The specific implementation method of the selection operation in the edge server of the present invention is:

[0051] This method delays the cold start of function f. Execution time Latency in relaying requests to other edge servers Queuing delay on server e As input, returns the actual delay t of executing the request real , the edge server that actually handles the request α The method considers three possible scenarios: (1) there is no container for the function in the system, (2) there is an idle container for the function on other edge servers, and (3) there is only a running container for the function on other edge servers. For each case, the method randomly selects a container on another edge server (if any) and chooses the operation corresponding to the minimum latency among the possible options, appending the selected operation to a list C that records all the operations performed by the system.

[0052] The specific implementation method of the function caching method in the present invention is:

[0053] Initially, the memory of the edge server in the serverless edge computing system is initialized to empty. We use C to represent the container in the serverless edge computing system, where the element is a triple (e, f, t), indicating that the container corresponding to the function f on the edge server e will be completed at time t. If t is 0, that is, (e, f, 0), it means that there is an idle container c in the system.

[0054] For a request r:=(e, f), if there is an idle container (e, f, 0) on server e, the function execution time can be used to calculate the Otherwise, when only relaying to other edge servers is supported but bypassing to cloud servers is not supported, if request e has no idle c f Service r: = (e, f), then Algorithm 1 will be run to select operations, and according to the current state C of the container in the edge serverless computing system, the operation that minimizes the delay is greedily selected to serve request r. More specifically, in Algorithm 2, after determining the server that serves request r, if a new container needs to be created, it first checks whether there is enough remaining memory to start a new container of size z. f If there is not enough memory in server e, replace the container in e and create c in e f On the other hand, when supporting bypass to the cloud server, first assume that e β There is c f , and set the confidence level of f. If after running the replacement algorithm, f e β.weight>0, then c f Server β Otherwise, request r will be bypassed to the cloud server with a delay of

[0055] Anyone familiar with the technology may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A function caching method in edge serverless computing for deep learning, characterized in that: The method comprises the following steps: (1) Operation selection in edge servers: Based on the greedy strategy, according to the current container status and request information on multiple edge servers, the operation with the minimum delay is selected from local cold start, forwarding to other servers, and bypassing to cloud servers to serve the function request; (2) The multi-edge server functions in step 1) are processed by function caching.

2. The function caching method in edge serverless computing for serving deep learning according to claim 1 is characterized in that: The selection operation in the edge server in step 1) is specifically: 1) Receive input, including cold start delay, execution time, delay in relaying requests to other edge servers, and queuing delay on the server; 2) Check whether there is a container serving the current function in the current edge system. If there are no idle resources, cold start the container locally, that is, create a container service request and calculate the operation delay; 3) Check whether there are idle containers serving the current function in other edge servers. If so, calculate the delay between cold-starting the container on the local server and executing the function, queuing and executing the function on the local server, relaying to the hot-start server and executing the function, and select the operation with the smallest delay; 4) If there is no idle container for the current function, check whether there is a running container serving the current function. If so, calculate the delay between cold-starting the container on the local server and executing the function, queuing and executing the function on the local server, relaying to the cold-start server to start the function and executing the function, and select the operation with the smallest delay; 5) Return the minimum delay and the corresponding server.

3. The function caching method in edge serverless computing for serving deep learning according to claim 1, characterized in that: The specific steps of the function caching method in step 2) are: 1) System initialization: The memory of the edge server is initially empty. The system uses containers to manage computing resources. Each container contains three pieces of information: edge server ID, function ID, and completion time. A completion time of zero indicates that the container is idle. 2) When a request is received, the system first checks whether there is an idle container for the corresponding function on the target edge server. If so, the container is used directly to process the request, with a delay equal to the execution time of the function. 3) If there are no idle containers on the target server and the system only supports relays between edge servers, the system will run the method to select operations in the edge server and select the operation that minimizes latency based on the current container status; 4) After the server is determined, if a new container needs to be created, the system will first check whether there is enough remaining memory. If the memory is insufficient, the existing container needs to be replaced and then a new container needs to be created on the server; 5) When cloud server bypass is supported, the system will first assume that the required container exists on the target server and set the confidence of the function. After running the replacement algorithm, it will decide whether to cold start the container on the edge server or forward the request to the cloud server for processing based on the weight value; 6) Latency Calculation,If the request is forwarded to the cloud server, the final latency time will include the sum of the processing time of the cloud server and the execution time of the edge server.

4. The function caching method in edge serverless computing for serving deep learning according to claim 1 or 3, characterized in that: The function cache method supports bypass and relay.

Images