Assembly line cold start optimization system and method based on deep learning

By vertically partitioning the deep learning inference service and using historical mode to predict the state of idle containers, the cold start of the serverless platform is optimized, and the problem of high cold start delay in serverless computing is solved, achieving more efficient resource utilization and cost savings.

CN119003092APending Publication Date: 2024-11-22SHANGHAI JIAOTONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411144634.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The cold start delay of deep learning inference services in serverless computing is high. The existing methods focus mainly on runtime load delay, ignore model load delay, and are not effective when converting between different types of models.

Method used

The vertical partitioning method is used to divide the deep learning inference service into multiple slices, and the state of idle containers is predicted through the segment similarity prediction mechanism based on the historical pattern, a scheduling strategy is formulated to optimize cold start, and the container multiplexing is used for hot but idle containers. Reduce model loading latency and communication overhead.

Benefits of technology

It significantly reduces the cold start delay of deep learning inference services on serverless platforms, improves resource efficiency, and reduces the cold start delay by 49.32% to 28.47% compared with the existing technology, saving costs and overhead.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003092A_ABST
    Figure CN119003092A_ABST
Patent Text Reader

Abstract

The invention provides an assembly line cold start optimization system and method based on deep learning, and the system comprises a model partitioning engine which divides a deep learning reasoning service into a plurality of slices through a vertical partitioning method; a scheduler based on idle container estimation uses a prediction mechanism of segment similarity based on a historical mode to predict a future idle container state, and formulates a reuse strategy for scheduling slices; based on a similarity container matcher, a reuse policy of a scheduler based on idle container estimation is evaluated, and idle containers are identified and scheduled. According to the method, the rule of the RICs is determined, the potential relevance between the RICs is comprehensively analyzed, the prediction model based on the RIC use mode is designed, and the cold start delay of the deep learning inference service in the non-service platform is reduced through segmentation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of distributed systems, and in particular to a pipeline cold start optimization system and method based on deep learning. Background Art

[0002] Serverless computing is a major paradigm shift in cloud computing, emphasizing automatic scaling, event-driven architecture, and pay-per-use billing models. The increasing application of deep learning (DL) in various industries requires more efficient methods to deploy trained models and provide deep learning inference services (DLIs) at a large scale. The concept of serverless inference combines the advantages of serverless computing with the specific needs of deploying DL models, creating an innovative way to deliver DLIs at scale. This emerging approach is expected to make DL applications more accessible, scalable, and cost-effective. For serverless inference, the underlying function of DLIs is triggered by receiving a user request, which then starts a container that encapsulates different pre-trained models. This calling process can start a new container (i.e., cold start) or reuse an existing container (i.e., hot start), such as Figure 1 As shown. Cold starts require thorough preparation measures, as new containers need to be created and the application logic loaded into memory. Therefore, cold starts are usually more time-consuming than hot starts. In addition, given resource constraints, it is impractical to maintain a sufficient number of hot containers for each DLI, which makes cold starts an inevitable aspect of serverless platforms.

[0003] Mitigating cold start overhead is a key challenge in serverless computing, motivating various studies to conduct extensive exploration. Cold start mainly consists of two phases: runtime loading and model loading.

[0004] On the one hand, many methods focus on reducing runtime loading latency through image distribution, lightweight containers, etc. For example, SOCK proposes a simplified optimization method for traditional containers. On the other hand, alternative methods aim to reduce runtime loading latency through function preheating or instance sharing. For example, Pagurus is proposed to alleviate cold starts by sharing a hot but idle container with other functions. However, these methods are less effective when dealing with DLIs. Their main focus is to simplify the runtime loading phase, while ignoring the latency caused by model loading. Optimus designs a comprehensive inter-function model conversion strategy aimed at alleviating the latency caused by model loading. Optimus has proven to be effective in converting models of the same category. However, when converting different types of models (e.g., from CNN to Transformers), its latency is even higher than that of methods that start directly from scratch. Summary of the invention

[0005] In view of the defects in the prior art, the object of the present invention is to provide a pipeline cold start optimization system and method based on deep learning.

[0006] A pipeline cold start optimization system based on deep learning provided by the present invention includes:

[0007] Model partitioning engine, which uses vertical partitioning to divide deep learning inference services into multiple slices;

[0008] The scheduler based on idle container estimation uses a prediction mechanism based on segment similarity of historical patterns to predict the future idle container status and formulate a reuse strategy for scheduling slices;

[0009] The similarity-based container matcher evaluates the reuse strategy of the scheduler based on idle container estimation to identify and schedule idle containers.

[0010] Preferably, the model partitioning engine includes generating a vertical partitioning scheme to divide the deep learning inference service into certain fine-grained slices and preheating slices in a sequential, overlapping pipeline manner; the slices can be loaded and executed in parallel.

[0011] Preferably, the scheduler based on idle container estimation includes paying attention to patterns of requests and idle containers, and predicting idle containers based on the patterns to formulate scheduling strategies; the patterns of idle containers include overall periodic patterns and local variability patterns.

[0012] Preferably, the similarity-based container matcher includes using hot but idle containers to avoid cold starts and reusing idle containers using zygote containers based on resource isolation and changes in the runtime and models loaded in each container are affected by the dependencies and configuration of the deep learning inference service.

[0013] Preferably, the system allocates partition slices of each preset type of deep learning inference service to the same server for execution.

[0014] A pipeline cold start optimization method based on deep learning provided by the present invention includes:

[0015] Step S1: Instruct the model partitioning engine to divide the deep learning inference service into multiple slices using a vertical partitioning method;

[0016] Step S2: Let the scheduler based on idle container estimation use the prediction mechanism based on segment similarity of historical patterns to predict the future idle container status and formulate a reuse strategy for scheduling slices

[0017] Step S3: Let the similarity-based container matcher evaluate the reuse strategy of the scheduler based on idle container estimation, identify and schedule idle containers, and complete the cold start optimization process.

[0018] Preferably, the model partitioning engine includes generating a vertical partitioning scheme to divide the deep learning inference service into certain fine-grained slices and preheating slices in a sequential, overlapping pipeline manner; the slices can be loaded and executed in parallel.

[0019] Preferably, the scheduler based on idle container estimation includes paying attention to patterns of requests and idle containers, and predicting idle containers based on the patterns to formulate scheduling strategies; the patterns of idle containers include overall periodic patterns and local variability patterns.

[0020] Preferably, the similarity-based container matcher includes using hot but idle containers to avoid cold starts and reusing idle containers using zygote containers based on resource isolation and changes in the runtime and models loaded in each container are affected by the dependencies and configuration of the deep learning inference service.

[0021] Preferably, the method allocates partition slices of each preset type of deep learning inference service to the same server for execution.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. The present invention determines the law of RIC, comprehensively analyzes the potential correlation between RICs, and designs a prediction model based on RIC usage patterns, which reduces the cold start delay of deep learning inference services in serverless platforms through segmentation; compared with the prior art, the present invention provides a better container reuse mechanism, which can evaluate the supply and demand relationship between the request volume and the number of idle containers, and further reuse idle containers based on the similarity of the runtime and model in the container to improve resource efficiency and avoid cold starts.

[0024] 2. The present invention further reduces the delay of model loading by reducing the model size. A vertical partitioning method is used to divide DLIS into multiple slices consisting of layers. Each slice is preheated in a sequential and overlapping pipeline manner to offset the cold start delay and reduce the communication overhead between slices. An attention mechanism is used based on the overall periodicity of RIC, and the future RIC state is predicted according to the segment similarity of historical patterns. Reuse recommendations for scheduling slices are formulated.

[0025] 3. The container reuse strategy proposed in the present invention evaluates runtime and structural similarities, solving a key shortcoming of the current reuse method that only focuses on runtime loading; since model loading is optimized, the present invention can be seamlessly integrated into the existing cold start optimization framework and has a wide range of application scenarios.

[0026] 4. This paper proposes a model segmentation framework designed for DLIS on serverless platforms to reduce service cold start latency. Compared with the existing algorithms OpenFaaS, SOCK, Tetris, Pagurus, and Optimus, the framework proposed by this paper reduces the cold start latency by 49.32%, 47.5%, 41.32%, 42.9%, and 28.47%, respectively, saving costs and improving work efficiency.

[0027] Other beneficial effects of the present invention will be explained in the specific implementation manner through the introduction of specific technical features and technical solutions. Through the introduction of these technical features and technical solutions, those skilled in the art should be able to understand the beneficial technical effects brought about by the technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0029] Figure 1 Schematic diagram of the cold start process of DLIS.

[0030] Figure 2 It is a framework structure diagram of the present invention.

[0031] Figure 3 Pseudo code of the segmentation algorithm of the present invention.

[0032] Figure 4 This is the overlapping pipeline mechanism of the present invention.

[0033] Figure 5 This is the pattern-based attention mechanism of the present invention.

[0034] Figure 6 This is the pseudo code of the container reuse algorithm of the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0036] The present invention provides a pipeline cold start optimization system and method based on deep learning. Through a comprehensive evaluation of the trace data obtained from the widely used DLIs and feasible experiments, it is found that due to regular user behavior and the continuous creation and termination of containers, for any specific DLIs, two key patterns are determined in the requests and idle containers (RICs) in the host, namely, overall periodicity and local variability. However, there is no correlation between requests and idle containers. Secondly, although both runtime loading and model loading affect the overall latency, the latter is the main bottleneck of DLIs cold start and is significantly affected by the size of the underlying model. Finally, due to the dynamic nature of the request pattern, preheating (PW) and container reuse (CR) strategies are generally ineffective in DLIs, and sometimes even slower than cold start (CO).

[0037] A potential approach is to apply a divide-and-conquer strategy to partition DLIs into multiple slices to reduce the latency of model loading by reducing the model size. First, a vertical partitioning method is adopted to divide DLIs into multiple slices consisting of layers. Each slice is warmed up in a sequential, overlapping pipeline manner to offset the cold start latency and reduce the communication overhead between slices. Subsequently, considering the overall periodicity of RIC, a prediction method is designed that uses an attention mechanism to predict future RIC states based on the segment similarity of historical patterns and formulates reuse recommendations for scheduling slices. In addition, by considering RIC, a container reuse strategy is designed. This strategy evaluates runtime and structural similarity, addressing a key shortcoming of current reuse methods that only focus on runtime loading. Since model loading is optimized, the present invention can be seamlessly integrated into existing cold start optimization frameworks.

[0038] The above is a basic embodiment of the present invention. The technical solution of the present invention is further described below through a preferred embodiment.

[0039] Example 1

[0040] The present invention partitions and reuses models based on miniaturization and pipelining. First, a vertical partitioning approach is adopted to divide DLIs into multiple slices consisting of layers. Each slice is preheated in a sequential, overlapping pipeline manner to offset cold start latency and reduce communication overhead between slices. Subsequently, considering the overall periodicity of RIC, a prediction method is designed that uses an attention mechanism to predict future RIC states based on segment similarities of historical patterns and formulates reuse recommendations for scheduling slices. In addition, by considering RIC, a container reuse strategy is designed. This strategy evaluates runtime and structural similarity, addressing a key shortcoming of current reuse methods that only focus on runtime loading.

[0041] The specific plan is as follows:

[0042] 1. Framework structure

[0043] The present invention designs corresponding modules to implement the above framework. The architecture mainly includes three key components: Model Partition Engine (MPE), RIC Estimation-based Scheduler (RES) and Similarity-based Container Matcher (SCM).

[0044] First, MPE is the basis for alleviating cold starts of DLIs. It generates a vertical partitioning scheme to divide DLIs into finer-grained slices and warm-up slices in a sequential, overlapping pipeline manner. Second, RES uses a prediction mechanism based on segment similarity of historical patterns to predict the next l RIC elements and formulates a container reuse strategy based on this. In addition, SCM designs an idle container reuse strategy to make full use of the resources occupied by idle containers in the cluster.

[0045] like Figure 2 As shown in the figure, it is a process framework diagram for optimizing the cold start of deep learning inference services. In the preparation phase, for each DLIs request submitted by the user, MPE generates a partitioning scheme based on the execution time of the corresponding model collected by the service analyzer. In the deployment phase, RES recommends a reuse strategy based on the prediction results of RIC and schedules partition slices in the cluster. SCM decides whether to start from scratch or reuse idle containers. In order to efficiently reuse idle containers, it is recommended to allocate partition slices of a specific type of DLIs to the same server for execution.

[0046] When a DLIS call arrives, a container is obtained to respond to the call, mainly including the following situations.

[0047] 1. When the requested DLIS type is a new deployment, the scheduler uses a specific selection strategy to allocate servers and start each shard of the DLIS in an overlapping pipeline manner.

[0048] 2. When the DLIs type already exists and the corresponding container remains alive, the request can be processed quickly by reusing the surviving container.

[0049] 3. When the request time exceeds the container's lifetime, the RES decides on the action. If the RES recommends a reuse strategy, the SCM will find a suitable container from all shards of the candidate server based on the runtime and model similarity. However, if no suitable container is found for reuse, the DLIS will start from scratch.

[0050] 4. If RES recommends a fresh start strategy, SCM will directly start each shard of this DLIS from scratch in an overlapping pipeline manner.

[0051] 2. Model Partition Engine (MPE)

[0052] MPE aims to optimize the loading and execution time of DLIS by dividing the model into smaller, manageable slices that can be loaded and executed in parallel. By incorporating the idea of pipeline-based cold start, MPE adopts a vertical partitioning scheme, dividing DLIS into multiple slices and starting each slice in the form of partially overlapping pipelines, thus offsetting the cold start latency issues typically associated with larger models. However, it is worth noting that too many slices may lead to a significant increase in the communication time between slices. Additionally, there may be gaps when each slice runs in the pipeline, resulting in additional overhead.

[0053] 1. Formalization of the Partitioning Strategy

[0054] The problem of partitioning DLIS can be formalized as follows. First, DLIS can be represented by a graph G = <V, E>, where each node l i ∈ V corresponds to a different layer within DLIS, and each edge <l i , l j > ∈ E corresponds to a tensor that serves as the output of layer l i and the input of layer l j . Ultimately, we hope to obtain a vertical partitioning scheme π S to reduce the E2E latency. Given a DLIS with L layers, the vertical partitioning method is to search for k splitting points {S1, S2, …, S k} among these L layers, where 1 ≤ S i ≤ L and when i < j, S i < S j . These k points divide the L layers into k + 1 slices, Ω = {g1, g2, …, g k+1}. The execution time of DLIS is represented as T = {t1, t2, …, t L}, where t i represents the execution time of layer i. Additionally, t r , t m and t c respectively correspond to the three stages of cold start: runtime loading, model loading, and code running. represents the communication time between slices g i and g j . Therefore, the goal of MPE is to find the optimal partitioning scheme π S that effectively reduces the latency of DLIS.

[0055]

[0056] Among them, Constraint 1 ensures that all metrics are non - negative; Constraint 2 ensures that the overlapping parts of the slices in the pipeline remain consistent, thus preventing any gaps; in Constraint 3 is the execution time of each layer before splitting, which stipulates that the total delay using MPE does not exceed the original delay without MPE.

[0057] 2. Vertical partitioning scheme based on dynamic programming

[0058] Obtaining a model partitioning scheme mainly faces the challenge of local dependencies. Specifically, there may be dependencies between different layers within the model, so the partitioning option for a specific layer may affect loading and computing delays as well as communication overhead. The above challenges were addressed by quantitatively evaluating the latency of different partitioning strategies for DLIS. MPE designed a dynamic programming (DP) algorithm to determine the vertical split points in the DAG. Dynamic programming considers layer dependencies and constructs a state transition table to select split points based on the best choice of previous points. In addition, since layer dependencies lead to overlapping subproblems, dynamic programming greatly improves computational efficiency. By considering local dependencies and calculating the optimal solution in stages, dynamic programming is able to determine the global optimal solution.

[0059] Reference Figure 3 As shown in Figure 1, the vertical partitioning (VPAD) process of DLIS mainly includes two steps: cumulative time calculation and vertical partitioning strategy generation. First, VPAD calculates the cumulative delay of each layer and saves it in the pref array (lines 2-3). Then, VPAD traverses each layer in G and calculates the cumulative delay pref of each layer i. [i] -pref [j] and communication overhead Again, VPAD uses DP to iterate and determine the minimum delay (lines 4-9), and then backtracks to determine the optimal partitioning strategy π S (Lines 10-14). Specifically, the dp array is used to store the minimum delay required to reach each layer, where dp [i] represents the minimum total delay from layer 0 to layer i, taking into account both communication and computational overhead. These values ​​are continuously updated during DP by comparing split points to determine the best strategy for each layer. Finally, the values ​​stored in dp [L] The value in (where L represents the total number of layers in the DLIS) corresponds to the minimum total latency of the entire model. During the backtracking phase, the split point array records the best split point for each layer. The time complexity of determining the best partition is O(n 2 ).

[0060] 3. Overlapping pipeline mechanism

[0061] Inspired by the parallel training strategy of large language models, parallelism between slices can also be built into the pipeline. Moreover, pipelining in DLIs introduces less complexity than applying it to model training, mainly because no backpropagation is required.

[0062] Reference Figure 4 As shown in the figure, a partially overlapping pipeline mechanism is adopted. The determinants affecting the end-to-end latency of DLIs are mainly divided into four stages: runtime loading (S1), model loading (S2), calculation (S3), and inter-slice communication (S4). The overlapping pipeline mechanism allows each slice to be loaded in an interleaved manner to form a pipeline. Runtime loading (S1) and model loading (S2) of the initial slice are performed to start the pipeline. Specifically, after starting the first slice (S1 and S2), calculation (S3) can be started without waiting for all slices to be loaded. After the calculation (S3) is completed, the first slice transfers the output (S4) to the second slice. At the same time, the second slice prepares the runtime and model in advance, obtains the output of the previous slice, and performs calculations at the same time. This process is repeated until the entire DLIs is completed.

[0063] 3. Estimated Request and Idle Container Scheduler (RES)

[0064] RIC has two notable characteristics, namely, overall periodicity and local variability. Overall periodicity helps to identify regularities in RICs and maintain a balance of resource efficiency between RICs, thus enabling accurate scheduling decisions. However, small-scale fluctuations and irregularities caused by local variability are inherently unpredictable and, instead, obscure underlying patterns that are critical to making accurate predictions. In addition, the lack of correlation between requests and idle containers suggests that it is unreasonable to rely on only one of them for scheduling decisions. Comprehensively considering the patterns of RICs can ensure that scheduling decisions not only consider the changes in requests but also effectively manage idle containers. In addition, the patterns can grasp the supply and demand dynamics between requests and containers, preventing insufficient resources during request peaks or excessive resource use by idle containers, thereby achieving resource balance. Therefore, RES pays attention to the patterns of requests and idle containers and predicts RICs based on the patterns to formulate scheduling strategies.

[0065] 1. Formalization of the problem of pattern prediction

[0066] A reasonable concept suggests that if historical patterns occur cyclically or repetitively, then it is logical to predict how they will behave in the future. Therefore, forecasting methods involve comparing segments from different periods, extracting common patterns from the segments, and generating future series based on the extracted patterns. The rationale for identifying patterns within segments rather than individual points is that segments are better able to convey semantic information and understand underlying patterns over a longer period of time. Consider a RIC multivariate time series of length ξ, denoted as (x1, x2, …, x ξ )∈R ξ×c , where each element x i contains c attributes. The main goal is to predict the next l elements in RIC. Therefore, given an input sequence of length ξ and the current time t, the prediction result is correspondingly (x t ,x t+1 ,…,x t+l-1 ).

[0067] Inspired by the RIC pattern, predictions are generated by evaluating the similarity between these fragments. Formally, the input sequence It can be cut into a set of d segments using a sliding window, denoted as The length of each segment is equal to the length of the predicted sequence l. Specifically, segment s is determined i and j The correlation between the fragments is calculated and their similarity is evaluated using the dot product calculation method, which can effectively capture the similarity between sequences. i and j The similarity between is defined as:

[0068]

[0069] in, It normalizes the similarity scores according to the number of attributes.

[0070] 2. Pattern-based attention mechanism

[0071] While assessing similarities between segments can be effective in identifying patterns, there are persistent challenges in accurately predicting upcoming RICs. First, RICs are uncorrelated, and fluctuations in RICs result in both long-term and short-term dependencies that can change abruptly. In addition, historical data does not contribute consistently to predictions. Figure 5 As shown, this embodiment introduces a pattern attention mechanism. The attention mechanism can capture not only short-term dependencies but also the global context of the entire RIC. Therefore, firstly, the consistency of the input features is maintained by inputting the RIC fragments separately. Secondly, Softmax is used to standardize the similarity score and prevent numerical instability. Finally, the value vector is aggregated by weighted summation, which improves the performance of the model when dealing with cycles with strong correlations.

[0072] Specifically, in order to maintain the consistency of input features and capture the similarity between input segments, the segments of RIC are input separately. Given an input segment s i , through s i Perform linear transformation to obtain query, key, and value vectors to enhance the model's ability to express input features:

[0073]

[0074] Among them, W q , W k and W vare the linear transformation matrices for queries, keys, and values, respectively.

[0075] In addition, the two fragments i and j The similarity between ij Assessed by:

[0076]

[0077] where k j It is a fragment j The linear transformation result of the key.

[0078] In order to standardize the similarity score, Softmax is used to convert the similarity score to ij is converted to a probability distribution. This conversion clarifies the fragment s i For fragments j In addition, the maximum value is subtracted from the similarity score before Softmax to prevent numerical instability:

[0079]

[0080] To aggregate the value vectors, a weighted sum is performed. Each value vector v i All by adding up the value vectors v from all segments j To update. The weight of the aggregation comes from the normalized similarity value A ij , ensuring that segments with higher similarity contribute more:

[0081]

[0082] Here, d represents the d fragments into which the input sequence is cut.

[0083] Finally, the pattern of request and idle containers is represented by Linear(v i ) is converted to the final output. Subsequently, considering the results of RIC, RES uses the Sigmoid function to determine whether it is more appropriate to execute the reuse strategy or choose to start from scratch.

[0084] 4. Similarity-based container matcher

[0085] SCM acts as an idle container matcher and utilizes hot but idle containers to avoid cold starts. However, not all idle containers are reused. First, considering resource isolation, the namespaces of individual containers do not interfere with each other. Subsequently, the changes in the runtime and models loaded in each container are affected by the dependencies and configurations of the DLIs. Finally, regarding the scheduling overhead of containers, it is recommended to schedule DLIs of the same type on the same server. Therefore, SCM extensively evaluates the reuse strategies of RES, identifies idle containers suitable for scheduling based on package and structural similarities, and uses zygote containers to reuse idle containers.

[0086] 1. Similarity-based matching strategy

[0087] Even if a free container for a slice is identified, the SCM is not free to reuse the container due to the possible differences in the runtime and model structure of the underlying container associated with each DLIs request. Therefore, the SCM develops a similarity matching strategy that combines comparison of package names and versions with an evaluation of the similarity of model internal layers and connectivity.

[0088] A slice f can be viewed as a DAG of packages and underlying models, namely:

[0089] f={(pkg1,pkg2,…),(V f ,E f )};

[0090] Among them, pkg i Specifically specified in the configuration.

[0091] Represents the set of functions on server q when function f is triggered. Package pkg is in f A and another function f B The inclusion relationship in is expressed as:

[0092]

[0093] Then, we can derive the representation functions f and f i ∈F q The vectors of the inclusion relationship are expressed as and Therefore, the functions f and f i The similarity between m packages can be evaluated as follows:

[0094]

[0095] Among them, ω j Indicates the importance of each package.

[0096] f and f iThe structural similarity can be divided into node similarity and edge similarity. Each node of f is represented by a vector which includes the encoding of its type and related parameters. Additionally, the edge is represented by the adjacency matrix A. Therefore, we use the Euclidean distance to compare the similarity of each node between f and f i .

[0097]

[0098] Similarly, the Frobenius norm is used to compare the edge similarity

[0099]

[0100] Combining the edge and node similarities, the structural similarity can be defined as

[0101]

[0102] where α and β are the weights used to balance the node similarity and edge similarity, respectively, and α + β = 1.

[0103] For n requests R = {r1, r2, …, r n}, they arrive at rates of {λ1, λ2, …, λ n}, respectively. According to the Poisson combination property of queue theory, the overall arrival of requests still follows a Poisson process. The average rate is expressed as

[0104]

[0105] To keep the scheduling queue stable, λ(t) < S(t) is required, where S(t) is the service rate. The waiting time of requests in the queue is expressed as

[0106]

[0107] Given M idle containers C = {c1, c2, …, c M} in the cluster, each container c i is equipped with available memory mem(c i ) and a specific number of CPU cores cpu(c i ). Additionally, for each request r i , mem(r i ) represents the required memory, and cpu(c i ) represents the required number of CPU cores. The purpose of obtaining the optimal match is to optimize the minimization of the average response time within the interval Δt:

[0108]

[0109] where t wand t c They represent the waiting time and execution time of the request respectively, and the average values ​​are recorded as and Constraint 2 requires that the memory and number of CPU cores of the idle container are not less than the requested ones; Constraint 3 ensures that the packet inclusion similarity and structure similarity between the idle container and the requested container are not less than the thresholds θ and η respectively; Constraint 4 ensures that the total processing capacity is sufficient to handle the average arrival rate of the requests.

[0110] 2. Idle container matching algorithm based on particle swarm optimization

[0111] There are several challenges in determining containers that are suitable for reuse within a cluster. First, the dynamic container matching problem must be solved because it involves managing changing requests and changing resource conditions. In addition, container matching is complicated by multiple constraints such as memory, CPU, and runtime. To address these challenges, the present invention designs a lightweight heuristic algorithm based on particle swarm optimization (PSO) to dynamically adjust matching schemes and evaluate different plans through a fitness function. PSO provides the best solution in a short time and at a low cost, making it very suitable for adapting to the dynamic changes and frequent adjustments of requests and idle containers under various constraints.

[0112] Reference Figure 6 As shown in Figure 1, the pseudo code of the matching process of free containers is summarized, which includes three different steps: initializing the particle swarm, updating the positions and particle velocities, and calculating the fitness. In step 1, each discrete particle represents an allocation scheme, and its position x i Plot the relationship between requests and free containers, with a velocity v i Represents the moving direction and step size of the particle (lines 1-2). In step 2, during each iteration, all particles are updated according to the current global optimal position and their respective optimal positions (lines 4-8). Specifically, random variables r1 and r2 are introduced to enhance the randomness and diversity in the search (lines 5-6). Subsequently, the speed of each particle is fine-tuned by using the current speed and the individual and global best positions (line 8), where c1 regulates the influence of the particle moving to its own best position, and c2 determines the movement to the global best position. Finally, the position of the particle is modified according to the fine-tuned speed (line 9). In step 3, the optimization objective is used as a fitness metric to evaluate the efficiency of adjustments for individual particles (lines 9-10). The current fitness score of each particle is tied to the fitness score of its historical optimal position. If the current fitness is higher, the individual optimal position of the particle is modified (lines 12-13). In addition, the fitness scores of the individual optimal positions of all particles are compared to determine the global optimal position (line 14).

[0113] This paper proposes a model segmentation framework designed for DLIS on serverless platforms to reduce service cold start latency. Compared with the existing algorithms OpenFaaS, SOCK, Tetris, Pagurus, and Optimus, the framework proposed in this paper reduces cold start latency by 49.32%, 47.5%, 41.32%, 42.9%, and 28.47%, respectively, saving cost and improving work efficiency.

[0114] The present invention also provides a pipeline cold start optimization system based on deep learning. The pipeline cold start optimization system based on deep learning can be implemented by executing the process steps of the pipeline cold start optimization method based on deep learning, that is, those skilled in the art can understand the pipeline cold start optimization method based on deep learning as a preferred implementation of the pipeline cold start optimization system based on deep learning.

[0115] Specifically, a pipeline cold start optimization system based on deep learning includes:

[0116] Model partitioning engine, which uses vertical partitioning to divide deep learning inference services into multiple slices;

[0117] The scheduler based on idle container estimation uses a prediction mechanism based on segment similarity of historical patterns to predict the future idle container status and formulate a reuse strategy for scheduling slices;

[0118] The similarity-based container matcher evaluates the reuse strategy of the scheduler based on idle container estimation to identify and schedule idle containers.

[0119] The model partitioning engine includes generating a vertical partitioning scheme to divide the deep learning inference service into certain fine-grained slices and preheating slices in a sequential, overlapping pipeline manner; the slices can be loaded and executed in parallel.

[0120] The scheduler based on idle container estimation includes paying attention to the patterns of requests and idle containers, and predicting the idle containers according to the patterns to formulate a scheduling strategy; the patterns of the idle containers include overall periodic patterns and local variability patterns.

[0121] The similarity-based container matcher includes using hot but idle containers to avoid cold starts and using zygote containers to reuse idle containers based on resource isolation and the variations of runtimes and models loaded in each container are affected by the dependencies and configurations of deep learning inference services.

[0122] The system allocates partition slices of each preset type of deep learning inference service to the same server for execution.

[0123] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0124] In the description of the present application, it should be understood that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0125] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A pipeline cold start optimization system based on deep learning, characterized in that: include: Model partitioning engine, which uses vertical partitioning to divide deep learning inference services into multiple slices; The scheduler based on idle container estimation uses a prediction mechanism based on segment similarity of historical patterns to predict the future idle container status and formulate a reuse strategy for scheduling slices; The similarity-based container matcher evaluates the reuse strategy of the scheduler based on idle container estimation to identify and schedule idle containers.

2. The pipeline cold start optimization system based on deep learning according to claim 1, characterized in that: The model partitioning engine includes generating a vertical partitioning scheme to divide the deep learning inference service into certain fine-grained slices and preheating slices in a sequential, overlapping pipeline manner; the slices can be loaded and executed in parallel.

3. The pipeline cold start optimization system based on deep learning according to claim 1, characterized in that: The scheduler based on idle container estimation includes paying attention to the patterns of requests and idle containers, and predicting the idle containers according to the patterns to formulate a scheduling strategy; the patterns of the idle containers include overall periodic patterns and local variability patterns.

4. The pipeline cold start optimization system based on deep learning according to claim 1, characterized in that: The similarity-based container matcher includes using hot but idle containers to avoid cold starts and using zygote containers to reuse idle containers based on resource isolation and the variations of runtimes and models loaded in each container are affected by the dependencies and configurations of deep learning inference services.

5. The pipeline cold start optimization system based on deep learning according to claim 1, characterized in that: The system allocates partition slices of each preset type of deep learning inference service to the same server for execution.

6. A pipeline cold start optimization method based on deep learning, characterized in that: include: Step S1: Instruct the model partitioning engine to divide the deep learning inference service into multiple slices using a vertical partitioning method; Step S2: Instruct the scheduler based on idle container estimation to predict the future idle container state using a prediction mechanism based on segment similarity of historical patterns, and formulate a reuse strategy for scheduling slices; Step S3: Let the similarity-based container matcher evaluate the reuse strategy of the scheduler based on idle container estimation, identify and schedule idle containers, and complete the cold start optimization process.

7. The pipeline cold start optimization method based on deep learning according to claim 6, characterized in that: The model partitioning engine includes generating a vertical partitioning scheme to divide the deep learning inference service into certain fine-grained slices and preheating slices in a sequential, overlapping pipeline manner; the slices can be loaded and executed in parallel.

8. The pipeline cold start optimization method based on deep learning according to claim 6, characterized in that: The scheduler based on idle container estimation includes paying attention to the patterns of requests and idle containers, and predicting the idle containers according to the patterns to formulate a scheduling strategy; the patterns of the idle containers include overall periodic patterns and local variability patterns.

9. The pipeline cold start optimization method based on deep learning according to claim 6, characterized in that: The similarity-based container matcher includes using hot but idle containers to avoid cold starts and using zygote containers to reuse idle containers based on resource isolation and the variations of runtimes and models loaded in each container are affected by the dependencies and configurations of deep learning inference services.

10. The pipeline cold start optimization method based on deep learning according to claim 6, characterized in that: The method allocates partition slices of each preset type of deep learning inference service to the same server for execution.