Memory-cooperative DNN layer-level scheduling method based on edge real-time system

Abstract

The present invention discloses a memory-cooperative DNN hierarchical scheduling method based on an edge real-time system, which belongs to the field of real-time system scheduling. The method deeply studies the performance characteristics of tasks, constructs optimization models and uses deep reinforcement learning to achieve the goals of minimizing task delays and maximizing task satisfaction. It mainly includes the following steps: First, by deeply studying the performance characteristics of each DNN task, including memory allocation size and execution time, to ensure that the system achieves optimal performance when running DNN tasks. Secondly, according to the problem definition, an optimization model is established and converted into a Markov decision process to clarify the state space, action space and state transition. Finally, deep reinforcement learning is used to solve the problem, and by learning the optimal scheduling strategy, the optimal scheduling sequence is obtained to maximize task satisfaction. The method described in the present invention can effectively adapt to complex edge system environments and provide a feasible solution for improving user experience.

Description

Technical Field

[0001] The present invention relates to edge computing real-time system scheduling technology, and specifically to a memory-coordinated DNN hierarchical scheduling method based on edge real-time systems. Background Art

[0002] Deep neural networks (DNNs), combined with powerful GPUs and large-scale datasets, have found widespread application in the next-generation information technology sector. With the advent of DNN models, application developers have begun building multi-model applications, such as video surveillance, traffic monitoring, real-time drone footage analysis, and language translation, to process audio or video streams in real time. These multi-model applications consist of several DNN models organized into directed acyclic graphs (DAGs). Compared to traditional applications, the deep learning inference process in these applications is typically computationally and data-intensive. However, the limited computing power of most terminal devices makes it difficult to support model inference, while traditional mobile cloud computing faces uncontrollable network communication latency. As a result, running deep learning applications on smart terminals often fails to meet their real-time requirements.

[0003] To ensure that inference requests meet each application's millisecond latency service level objective (SLO), developers typically deploy their application models on real-time edge systems that offer efficient computing power, high real-time performance, reliability, and security. However, current DNN cluster schedulers for real-time edge systems typically assume that GPUs are the primary resource for scheduling tasks. That is, users request a fixed number of GPUs for their DNN job, and the job is scheduled to run when all requested GPUs are available. However, due to factors such as the coarse-grained resource allocation model (one resource per task) and the asymmetric nature of DNN execution on CPUs and GPUs, existing deep learning frameworks struggle to leverage heterogeneous computing resources to improve the schedulability of real-time DNN tasks. Therefore, effectively utilizing heterogeneous resources to meet SLOs, while fully considering factors such as the system's computing resource heterogeneity, computing performance, and real-time responsiveness, becomes a challenge. Summary of the Invention

[0004] Purpose of the invention: The present invention aims to provide a memory-cooperative DNN hierarchical scheduling method based on edge real-time systems to comprehensively improve the computing performance of GPUs and CPUs and optimize the calculation of inference tasks.

[0005] Technical solution: A memory-cooperative DNN hierarchical scheduling method based on edge real-time systems, including the following steps:

[0006] (1) Predictive analysis of DNN task resource requirements, including the memory allocation size corresponding to each DNN task and the execution time of the task after the corresponding memory is allocated, to obtain the functional relationship between the inference task execution time and the memory size;

[0007] (2) Based on the problem definition, determine the optimization goal, clarify the problem constraints, and determine the range of decision variables to establish an optimization model, including the following considerations:

[0008] (21) For edge systems, including at least a CPU and a GPU, the edge system periodically performs a set of inference tasks. , for each reasoning task The reasoning task is performed by L i DNN layers, reasoning tasks There are dependencies between reasoning tasks, which are organized by a directed acyclic graph;

[0009] (22) x i，j =0 means task J i The jth layer of x is executed on the CPU, i，j =1 indicates task J i The jth layer of S is executed on the GPU, i，j Represents task J i The start time of the jth layer;

[0010] For each node in the DAG, Represents task J i The execution time of the jth layer on the CPU, Represents task J i The execution time of the jth layer on the GPU;

[0011] Task J i In S i The task response time is T i , the estimated deadline is D i ; Each task At the beginning, the optimal M obtained according to step (1) will be allocated i The memory of the specified size is used until the task is completed and the corresponding memory is released;

[0012] The start time Si is determined by the scheduler, and the task response time where w i，j is the DNN layer τ i，j The waiting time, e i，j is the DNN layer τ i，j The running time on GPU or CPU obtained in step S1 is

[0013] (23) In order to make full use of system resources, the scheduler needs to generate an efficient scheduling plan. Therefore, task satisfaction is used as the evaluation indicator. Task satisfaction is defined as the ratio of the expected deadline to the response time, that is:

[0014]

[0015] Among them D i is the expected deadline, T i is the task response time; the larger the ratio, the higher the task satisfaction, so the optimization goal of the scheduler is to maximize the satisfaction of all tasks, which is expressed as:

[0016]

[0017] Therefore, the constraints are as follows:

[0018]

[0019]

[0020]

[0021] The above constraints mean that layers of different tasks cannot run on the CPU or GPU at the same time;

[0022]

[0023] The above constraints indicate that in different layers of the DNN for the same task, the tasks of the next layer must be completed before the previous layer can start to execute;

[0024]

[0025] The above constraints represent any task J i Execution can only begin after its predecessor node has completed execution;

[0026]

[0027] The above constraints mean that the memory allocated at any time is no larger than the system's maximum memory;

[0028]

[0029] The above constraints mean that all tasks must be completed before their deadlines;

[0030] (3) The optimization model is converted into a Markov decision process, the state space, action space and state transition are defined, and deep reinforcement learning is used to solve the optimization problem. The optimal scheduling strategy is learned to obtain the optimal scheduling sequence, thereby maximizing task satisfaction.

[0031] Furthermore, the demand for DNN task resources in step S1 is predicted as follows:

[0032] (11) Simulate reasoning tasks based on the built-in simulation environment of the DNN model, and limit memory usage through system settings to simulate different memory allocation situations;

[0033] (12) Measure the execution time of the DNN task through the library function, and calculate the time difference between the start and end of the task as the execution time of the inference task. This process includes periodically calling the library function to record the memory usage of the current process;

[0034] (13) Based on the data obtained in step (11) and step (12), polynomial regression fitting is performed according to the least squares method to process the relationship between the DNN execution speed and the allocated memory size;

[0035] Assume the fitting polynomial is:

[0036]

[0037] where y is the DNN execution speed, x is the allocated memory size, θ0, θ1, ..., θ k is the parameter of the model, k is an arbitrary constant;

[0038] (14) Calculate the sum of squared errors between the sample and the fitting function. The expression is as follows:

[0039]

[0040] According to the least squares principle, the sum of squared errors SSE about the coefficients θ0, θ1, ..., θ n Find the partial derivative and set it equal to zero to get the estimated value of the coefficient;

[0041] (15) Establish a functional relationship model between the inference task execution time y and the memory size x, expressed as follows:

[0042] y=θ0+θ1x+θ2x 2 +...+θ k x k

[0043] For the above formula, an iterative method is used to select a smaller initial value x0, set the derivative threshold ∈, iteratively increase x, and calculate the derivative. When the derivative of a certain point is found to be less than the threshold ∈, the iteration is stopped to find the optimal task memory allocation x and its corresponding execution time y.

[0044] Furthermore, step (3) includes modeling the task scheduling process as a Markov decision process in solving the optimization problem using deep reinforcement learning, and designing the state space, action space and state transition to solve the task scheduling problem in edge computing;

[0045] (31) State space

[0046] The state s∈S describes the state information of the system, which consists of three parts: resource matrix, waiting matrix and backlog queue length. Therefore, the state of the system is expressed as:

[0047] S={s|s=(V,Q,|b|)}

[0048] Where V represents the resource matrix, Q represents the waiting matrix, and b represents the backlog queue;

[0049] The resource matrix represents the status of GPU, CPU and memory, i.e. the current GPU / CPU ready time and the remaining memory;

[0050] The waiting matrix can be observed by the scheduler, and at most one DNN layer is scheduled per scheduling time step;

[0051] The DNN layers in the backlog queue cannot be scheduled at the current time step;

[0052] The DNN layers to be scheduled are divided into two parts: one in the waiting matrix and the other in the backlog queue. The DNN layers in the waiting matrix can be scheduled at each scheduling time step. At most O DNN layers can be scheduled at each time step, and the excess DNN layers are stored in the backlog queue. The scheduler can observe the complete status information of the waiting matrix and only sees the number of DNN layers in the backlog queue.

[0053] The state of the waiting matrix Q is represented by an O-column matrix, as follows:

[0054] Q=[q1,q2,...,q o ]

[0055] where q i is a DNN layer waiting to be executed;

[0056] By calculating the upward and downward rankings of the reasoning tasks, this method can characterize the dependencies and impacts between tasks, thereby determining the execution order of the tasks.

[0057] For a task node v in a DAG, its upward ranking is determined by recursively calculating the upward ranking of its successor nodes. Assuming u is a successor task of v, its upward ranking is calculated as follows:

[0058]

[0059] The upward ranking represents the critical path length from the current task to the exit task;

[0060] The downstream ranking is determined by recursively calculating the downstream ranking of its predecessor node; assuming u is a predecessor task of v, its downstream ranking is calculated as follows:

[0061]

[0062] The downward ranking represents the longest distance from the entry task to the current task, excluding the execution time of the inference task itself;

[0063] (32) Action Space

[0064] The action space for inference task scheduling includes determining the execution order between O DNN layers and allocating GPU / CPU and memory to each task. To reduce the size of the action space, the scheduling process is divided into scheduling time steps and real-time time steps:

[0065] The scheduling time step is a decision sequence of the scheduler for the scheduler to perform actions;

[0066] The real time step represents the passage of real time and is used to obtain new DNN layers to be scheduled. If the waiting matrix is ​​not full, the new DNN layers will be placed in the waiting matrix; otherwise, they will be placed in the backlog queue.

[0067] At each scheduling time step, the scheduler selects an action A = {A e |A e =(d m ,q n )|m∈{-1, GPU, CPU}, n∈{-1, 1, 2, ..., O}}, where (-1, -1) is an invalid action A ψ , indicating that no DNN layer accepts scheduling, and the rest are valid actions A φ , represents the DNN layer q i Scheduled to GPU or CPU;

[0068] (33)State Transition

[0069] For each scheduling time step, the scheduler makes decisions by observing the system state, including:

[0070] (a) If the scheduler selects a valid action The planned DNN layer is removed from the waiting matrix, and the first DNN layer in the backlog queue is placed in the waiting matrix that has just been removed. At the same time, the ready time of the GPU / CPU in the resource matrix is ​​added to the execution time E of the assigned DNN layer. i，j,If the DNN layer is the first layer of job J, then the remaining memory will be reduced by the memory M required by the assigned DNN layer. Finally, the scheduling time step is increased by 1, and the real time step remains unchanged because there is no need to obtain a new DNN layer;

[0071] (b) If the scheduler selects an invalid action A ψ , it means that there are currently no allocated DNN layers, then the real-time time step will be increased by 1 to obtain the new DNN layer to be scheduled, and the ready time of the GPU / CPU in the resource matrix will be reduced by 1. If the last DNN layer of job J is executed, the memory it occupies will be released, and the remaining memory will increase. At the same time, the waiting time of the unallocated DNN layers in the waiting matrix will be increased by 1;

[0072] (34) Determine the optimization objective and design the reward function

[0073] The goal of the optimization task is to reasonably allocate the system GPU, CPU and memory resources to maximize the long-term task satisfaction of all tasks, that is, The reward is intended to guide the scheduler to achieve the goal of the optimal policy π = p(a|s);

[0074] For valid actions, when there is sufficient remaining memory, the reward is the ratio of the response time to the expected delay requirement; if the action causes the remaining memory to be negative, negative infinity is used as a penalty to make the scheduler allocate memory resources reasonably; if an invalid action is selected, zero reward will be given; therefore, the reward function is designed as follows:

[0075]

[0076] (35) The Monte Carlo policy gradient algorithm is used to solve the optimization problem in step S34, and deep reinforcement learning is also included to optimize the task scheduling strategy.

[0077] Furthermore, for the Monte Carlo policy gradient algorithm, the process is as follows:

[0078] Based on the Markov decision process formula, at each scheduling time step t, the policy network receives the system state s n =(V, Q, |b|), and generates the probability of selecting the action as output, the probability of the action being selected is π θ (a n |s n ), denoted as A e =(d m ,q n ), if A e=(-1, -1), it means that there is no scheduled task at the current scheduling time step and no reward will be obtained. In this case, the real time step moves forward and generates the next state according to the situation (b) in the state transition. In the case of a valid action, a reward will be generated according to the satisfaction of the scheduled task and the memory situation. The next state is obtained by the situation (a) in the state transition. The state, action and reward constitute a scene trajectory, which is used to calculate the cumulative reward for further training.

[0079] Furthermore, the REINFORCE algorithm in the Monte Carlo policy gradient algorithm generates a trajectory τ = (s 0 , a 0 , r 1 , s 1 , a 1 , r 2 ,...,a T , r T+1 , s T+1 ) to update the policy parameters θ;

[0080] In each episode trajectory, the policy gradient at each time step t is transformed into:

[0081]

[0082] The parameter θ is updated according to the gradient ascent, that is:

[0083]

[0084] Where α is the learning rate. If Gn is positive, it means that we want to increase the logarithmic probability of selecting action an in state sn; otherwise, we want to decrease the logarithmic probability.

[0085] Therefore, the agent is guided to find the optimal strategy. After several iterations, the agent will generate an optimal scheduling strategy.

[0086] Furthermore, step (3) also includes optimizing the task scheduling strategy by using deep reinforcement learning to improve the efficiency and performance of the edge computing system and obtain the optimal scheduling strategy;

[0087] By calculating the upward and downward rankings, the task completion time is reduced, and the performance and efficiency of the system are improved;

[0088] When the number of accepted DNN layers is smaller than the waiting slot size, the empty positions are filled with fixed negative values ​​to reduce the probability of being selected.

[0089] Beneficial effects: Compared with the prior art, the method of the present invention has the following three essential features and significant effects:

[0090] (1) The method described in the present invention reasonably considers the computing performance of GPU and CPU, allocates the optimal memory size according to the memory sensitivity of different DNN tasks, fully utilizes heterogeneous computer resources, improves the schedulability of real-time DNN tasks, minimizes task delays, and maximizes task satisfaction.

[0091] (2) The method described in the present invention decouples the scheduling time step and the real-time time step, significantly reducing the dimension of the action space and being able to schedule multiple tasks at the same time.

[0092] (3) The method described in the present invention applies a policy-based deep reinforcement learning algorithm to solve the optimization problem and has good convergence performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 This is a flow chart of the memory-cooperative DNN layer scheduling method based on edge real-time systems;

[0094] Figure 2 An example of a state space in a Markov decision process;

[0095] Figure 3 An example of selecting effective actions for a scheduler in a Markov decision process;

[0096] Figure 4 Example of ineffective action selection for a scheduler in a Markov decision process. DETAILED DESCRIPTION

[0097] To illustrate the technical solution disclosed by the present invention in detail, the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0098] The present invention provides a memory-cooperative DNN hierarchical scheduling method based on an edge real-time system. The method deeply studies the performance characteristics of tasks, builds optimization models, and uses deep reinforcement learning to achieve the goal of minimizing task delays and maximizing task satisfaction. It mainly includes the following steps: First, by deeply studying the performance characteristics of each DNN task (inference task), including memory allocation size and execution time, to ensure that the system achieves optimal performance when running DNN tasks. Secondly, according to the problem definition, an optimization model is established and converted into a Markov decision process to clarify the state space, action space, and state transition. Finally, deep reinforcement learning is used to solve the problem, and by learning the optimal scheduling strategy, the optimal scheduling sequence is obtained to maximize task satisfaction.

[0099] Combine Figure 1 The specific implementation steps and processes of the method of the present invention are as follows:

[0100] S1. Analyze the memory allocation size and the execution time of the task after allocating the corresponding memory

[0101] This step deeply studies the performance characteristics of each DNN task, including the memory allocation size and the execution time of the task after the corresponding memory is allocated, to ensure that the system can achieve optimal performance when running DNN tasks.

[0102] Specifically, step (1) includes the following process:

[0103] S101. Use deep learning frameworks to build various typical DNN models:

[0104] Use deep learning frameworks (such as TensorFlow and PyTorch) to define the model structure for various problems, such as classification, regression, and object detection, including the number of layers, the number of neurons in each layer, and activation functions. Use training data to train the model and ultimately obtain a variety of DNN models.

[0105] S102. Measure the execution time of DNN tasks under different memory allocations:

[0106] Use the simulation environment built into the deep learning framework to simulate inference tasks, and use system settings to limit memory usage to simulate different memory allocation situations.

[0107] We then use the operating system's library functions to measure the execution time of DNN tasks. We record the current time before the task begins and again after the task ends, then calculate the difference between the two to obtain the execution time. During task execution, we periodically call the library functions to record the current process's memory usage.

[0108] S103. Use the fitted data curve to obtain the optimal memory allocation method:

[0109] According to the data collected in step S102, the fitting polynomial is

[0110]

[0111] where y is the DNN execution speed, x is the allocated memory size, θ0, θ1, ..., θ k are the parameters of the model.

[0112] Find the sum of squared errors (SSE) between the sample and the fitted function, that is:

[0113]

[0114] According to the least squares principle, the sum of squared errors SSE about the coefficients θ0, θ1, ..., θ nCalculate the partial derivative and set it equal to zero to obtain the estimated value of the coefficient, and establish a functional relationship model between the inference task execution time (y) and the memory size (x):

[0115] y=θ0+θ1x+θ2x 2 +...+θ k x k

[0116] For the above equation, an iterative method is used to select a smaller initial value x0, set the derivative threshold ∈, iteratively increase x, and calculate the derivative. When the derivative of a certain point is found to be less than the threshold ∈, the iteration is stopped, thereby finding the optimal task memory allocation x and its corresponding execution time y.

[0117] S2. Optimization model and constraints for edge system task scheduling

[0118] This step defines the problem, determines the optimization goal, clarifies the problem constraints, and determines the range of decision variables to establish an optimization model.

[0119] Specifically, step S2 includes the following process:

[0120] S201. Build an optimization model for task scheduling:

[0121] Consider an edge system with a CPU and a GPU. The edge system periodically receives a set of tasks For each task By L i DNN layers, namely There are dependencies between tasks, which are organized by DAG.

[0122] The scheduler determines the execution order of each DNN layer and which device to assign it to for execution.

[0123] x i，j =0 means task J i The jth layer of x is executed on the CPU, i，j =1 indicates task J i The jth layer of S is executed on the GPU. i，j Represents task J i The starting time of the jth layer.

[0124] For each node in the DAG, Represents task J i The execution time of the jth layer on the CPU, Represents task J i The execution time of the jth layer on the GPU.

[0125] Task J i In Si The task response time is T i , the estimated deadline is D i Each task At the beginning, the optimal M obtained in step S1 will be allocated i The memory of the specified size is used until the task is completed and the corresponding memory is released.

[0126] The start time S i Determined by the scheduler, task response time where w i，j is the DNN layer τ i，j The waiting time, e i，j is the DNN layer τ i，j The running time on GPU or CPU obtained in step S1 is

[0127] In order to make full use of system resources, the scheduler needs to generate an efficient scheduling plan. Therefore, task satisfaction is used as the evaluation indicator. Task satisfaction is defined as the ratio of the expected deadline to the response time, that is,

[0128]

[0129] Among them D i is the expected deadline, T i is the task response time. Obviously, the larger the ratio, the higher the task satisfaction. Therefore, the optimization goal of the scheduler is to maximize the satisfaction of all tasks, that is,

[0130]

[0131] S202. Constraints of the design optimization model:

[0132] According to the conditions, layers of different tasks cannot run on the CPU or GPU at the same time, and the memory allocated at any time is not greater than the system maximum memory, so the constraints are as follows

[0133]

[0134]

[0135]

[0136]

[0137] Since there are priority constraints in different layers of the DNN for the same task, the tasks of the latter layer must be executed after the previous layer is completed, so the constraints are as follows

[0138]

[0139] Since the priority constraints between different tasks are organized by DAG, any task J i Execution can only begin after its predecessor node has completed execution, so the constraints are as follows

[0140]

[0141] At the same time, in order to meet the millisecond latency service level objective (SLO) of each application, all tasks must be completed before their deadlines, so the constraints are as follows

[0142]

[0143] S3. Convert the optimization model into a Markov decision process, define the state space, action space and state transition, use deep reinforcement learning to solve the problem, and obtain the optimal scheduling sequence by learning the optimal scheduling strategy to maximize task satisfaction.

[0144] Specifically, step S3 includes the following process:

[0145] S301. Convert the optimization task into a Markov decision process (MDP):

[0146] The design state space, action space and state transition are as follows:

[0147] The state s∈S describes the state information of the system, which consists of three parts: resource matrix, waiting matrix, and backlog queue length. Therefore, the state of the system can be expressed as:

[0148] S={s|s=(V,Q,|b|)}

[0149] Where V represents the resource matrix, Q represents the waiting matrix, and b represents the backlog queue. The resource matrix represents the status of the GPU, CPU, and memory, i.e., the current GPU / CPU ready time and the remaining memory. The waiting matrix can be observed by the scheduler, and at most one DNN layer can be scheduled per scheduling time step. The DNN layer in the backlog queue cannot be scheduled at the current time step. Figure 2 As shown, each part of the status is described in detail below.

[0150] Resource matrix V∈R 3×2 The first column is the GPU, CPU and memory identifiers. In the second column, r1 and r2 respectively indicate how many seconds it will take for the GPU and CPU to complete the currently assigned tasks, and r3 indicates the remaining memory.

[0151] The DNN layers to be scheduled are divided into two parts: one in the waiting matrix and the other in the backlog queue. The DNN layers in the waiting matrix can be scheduled at each scheduling time step. At most O DNN layers can be scheduled at each time step, and the excess DNN layers are stored in the backlog queue. In this case, the scheduler can observe the complete state information of the waiting matrix and only see the number of DNN layers in the backlog queue. Therefore, the state of the waiting matrix Q can be represented by an O-column matrix,

[0152] Q=[q1,q2,...,q o ]

[0153] where q i is a DNN layer waiting to be executed. The rows of the waiting matrix represent the task characteristics of each DNN layer, including DNN layer identifier (τ), required memory (M), upstream ranking (UR), downstream ranking (DR), and estimated execution time (E G , E c ) and the waiting time before scheduling (W).

[0154] In particular, when the number of accepted DNN layers is smaller than the waiting slot size, the empty positions are filled with fixed negative values ​​to reduce the probability of being selected.

[0155] The action space for task scheduling consists of two actions: determining the execution order among the O DNN layers, and allocating GPU / CPU and memory to each task. To reduce the size of the action space, the scheduling process is divided into two time steps: the scheduling time step and the real-time time step. The scheduling time step is the sequence of decisions made by the scheduler, which is used to execute actions; the real-time time step represents the passage of real time and is used to obtain new DNN layers to be scheduled. If the waiting matrix is ​​not full, new DNN layers are placed in the waiting matrix; otherwise, they are placed in the backlog queue.

[0156] At each scheduling time step, the scheduler selects an action A = {A e |A e =(d m ,q n )|m∈{-1, GPU, CPU}, n∈{-1, 1, 2, ..., O}}. Among them, (-1, -1) is an invalid action A ψ , indicating that no DNN layer accepts scheduling, and the rest are valid actions represents the DNN layer q i Scheduled to GPU or CPU.

[0157] The state transitions are as follows:

[0158] The state transfers to the next state according to the state and action (s, a). Figure 3 、 Figure 4 As shown, for each scheduling time step, the scheduler makes a decision by observing the system state.

[0159] (a) If the scheduler selects a valid action The planned DNN layer is removed from the waiting matrix, and the first DNN layer in the backlog queue is placed in the waiting matrix that has just been removed. At the same time, the ready time of the GPU / CPU in the resource matrix is ​​added to the execution time E of the assigned DNN layer. i，j ,If the DNN layer is the first layer of job J, then the remaining memory will be reduced by the memory M required by the assigned DNN layer. Finally, the scheduling time step is increased by 1, and the real time step remains unchanged because there is no need to obtain a new DNN layer;

[0160] (b) If the scheduler selects an invalid action A ψ , indicating that there are currently no allocated DNN layers. The real-time time step is incremented by one to obtain a new DNN layer to be scheduled. The GPU / CPU ready time in the resource matrix is ​​decremented by one. If the last DNN layer of job J completes execution, the memory occupied by it is released, and the remaining memory is increased. At the same time, the waiting time of the unassigned DNN layers in the waiting matrix is ​​incremented by one.

[0161] The goal of the optimization task is to reasonably allocate the system GPU, CPU and memory resources to maximize the long-term task satisfaction of all tasks, that is, The reward is designed to guide the scheduler to achieve the goal of the optimal policy π = p(a|s). For valid actions, when there is sufficient remaining memory, the reward is the ratio of the response time to the expected delay requirement; if the action causes the remaining memory to be negative, negative infinity is used as a penalty to make the scheduler allocate memory resources reasonably. If an invalid action is selected, we will give zero reward; therefore, the reward function is designed as:

[0162]

[0163] S302. Use the Monte Carlo policy gradient algorithm to solve the problem:

[0164] The Monte Carlo policy gradient algorithm is used to solve the problem. Based on the MDP formula, at each scheduling time step t, the policy network receives the system state s. n =(V, Q, |b|), and generates the probability of selecting the action as output. The probability of the action being selected is π θ (a n |s n ), denoted as A e =(d m ,q n ). If A e= (-1, -1), it means that there is no scheduled task at the current scheduling time step and no reward will be obtained. In this case, the real time step moves forward and generates the next state according to the situation (b) in the state transition. ), a reward is generated based on the satisfaction of the scheduled task and the memory status, and the next state is obtained by the state transition in case (a). The state, action, and reward form an episode trajectory, which is used to calculate the cumulative reward for further training.

[0165] REINFORCE generates the trajectory τ = (s 0 , a 0 , r 1 , s 1 , a 1 , r 2 ,...,a T , r T+1 , s T+1 ) to update the policy parameters θ. In each episode trajectory, the policy gradient at each time step t is converted to:

[0166]

[0167] The parameter θ is updated according to the gradient ascent, that is:

[0168]

[0169] Where α is the learning rate. If G n If it is positive, it means that you want to increase the n Select action a n Otherwise, the logarithmic probability is reduced. Therefore, the agent is guided to find the optimal strategy. After multiple iterations, the agent will generate an optimal scheduling strategy.

Claims

1. A memory-cooperative DNN hierarchical scheduling method based on edge real-time system, characterized in that: The following steps are involved: (1) Predictive analysis of DNN task resource requirements, including the memory allocation size corresponding to each DNN task and the execution time of the DNN task after the corresponding memory is allocated, to obtain the functional relationship between the DNN task execution time and memory size; (2) Based on the problem definition, determine the optimization goal, clarify the problem constraints, and determine the range of decision variables to establish an optimization model, including the following considerations: (21) For edge systems, including at least a CPU and a GPU, the edge system periodically performs a set of inference tasks. For each DNN task The DNN task is performed by L i DNN layers, reasoning tasks Moreover, there are dependencies between DNN tasks, which are organized by directed acyclic graphs; (22) x i，j =0 means task J i The jth layer of x is executed on the CPU, i，j =1 means DNN task J i The jth layer of S is executed on the GPU, i，j Denotes DNN task J i The start time of the jth layer; For each node in the DAG, Denotes DNN task J i The execution time of the jth layer on the CPU, Denotes DNN task J i The execution time of the jth layer on the GPU; DNN Task J i In S i The DNN task response time is T i , the estimated deadline is D i ; Each DNN task At the beginning, the optimal M obtained according to step (1) will be allocated i The memory of the specified size is used until the DNN task is completed and the corresponding memory is released; The start time S i Determined by the scheduler, DNN task response time where w i，j is the DNN layer τ i，j The waiting time, e i，j is the DNN layer τ i，j The running time on GPU or CPU obtained in step S1 is (23) In order to make full use of system resources, the scheduler needs to generate an efficient scheduling plan. Therefore, task satisfaction is used as the evaluation indicator. Task satisfaction is defined as the ratio of the expected deadline to the response time, that is: Among them D i is the expected deadline, T i is the task response time; The larger the ratio, the higher the task satisfaction. Therefore, the optimization goal of the scheduler is to maximize the satisfaction of all tasks, which can be expressed as: Therefore, the constraints are as follows: The above constraints mean that layers of different tasks cannot run on the CPU or GPU at the same time; The above constraints indicate that in different layers of the DNN for the same task, the tasks of the next layer must be completed before the previous layer can start to execute; The above constraints represent any task J i Execution can only begin after its predecessor node has completed execution; The above constraints mean that the memory allocated at any time is no larger than the system's maximum memory; The above constraints mean that all tasks must be completed before their deadlines; (3) The optimization model is converted into a Markov decision process, the state space, action space and state transition are defined, and deep reinforcement learning is used to solve the optimization problem. The optimal scheduling strategy is learned to obtain the optimal scheduling sequence, thereby maximizing the DNN task satisfaction.

2. The memory-cooperative DNN hierarchical scheduling method based on edge real-time system according to claim 1 is characterized in that: The demand for DNN task resources in step S1 is predicted as follows: (11) Simulate reasoning tasks based on the built-in simulation environment of the DNN model, limit memory usage through system settings, and simulate different memory allocation situations; (12) Measure the execution time of the DNN task through the library function, and calculate the time difference between the start and end of the task as the execution time of the inference task. This process includes periodically calling the library function to record the memory usage of the current process; (13) Based on the data obtained in step (11) and step (12), polynomial regression fitting is performed according to the least squares method to process the relationship between the DNN execution speed and the allocated memory size; Assume the fitting polynomial is: where y is the DNN execution speed, x is the allocated memory size, θ0, θ1, ..., θ k are the parameters of the model; (14) Calculate the sum of squared errors between the sample and the fitting function. The expression is as follows: According to the least squares principle, the sum of squared errors SSE about the coefficients θ0, θ1, ..., θ n Find the partial derivative and set it equal to zero to get the estimated value of the coefficient; (15) Establish a functional relationship model between the inference task execution time y and the memory size x, expressed as follows: y=θ0+θ1x+θ2x 2 +...+θ k x k For the above formula, an iterative method is used to select an initial value x0, set the derivative threshold ∈, iteratively increase x, and calculate the derivative. When the derivative of a certain point is found to be less than the threshold ∈, the iteration is stopped to find the optimal task memory allocation x and its corresponding execution time y.

3. The memory-cooperative DNN hierarchical scheduling method based on edge real-time system according to claim 1 is characterized in that: Step (3) in the process of solving the optimization problem using deep reinforcement learning, includes modeling the task scheduling process as a Markov decision process, and designing the state space, action space and state transition to solve the task scheduling problem in edge computing; (31) State space The state s∈S describes the state information of the system, which consists of three parts: resource matrix, waiting matrix and backlog queue length. Therefore, the state of the system is expressed as: S={s|s=(V,Q,|b|)} Where V represents the resource matrix, Q represents the waiting matrix, and b represents the backlog queue; The resource matrix represents the status of GPU, CPU and memory, i.e. the current GPU / CPU ready time and the remaining memory; The waiting matrix can be observed by the scheduler, and at most one DNN layer is scheduled per scheduling time step; The DNN layers in the backlog queue cannot be scheduled at the current time step; The DNN layers to be scheduled are divided into two parts: one in the waiting matrix and the other in the backlog queue. The DNN layers in the waiting matrix can be scheduled at each scheduling time step. At most O DNN layers can be scheduled at each time step, and the excess DNN layers are stored in the backlog queue. The scheduler can observe the complete status information of the waiting matrix and only sees the number of DNN layers in the backlog queue. The state of the waiting matrix Q is represented by an O-column matrix, as follows: Q=[q1,q2,...,q o ] where q i is a DNN layer waiting to be executed; By calculating the upward and downward rankings of the reasoning tasks, this method can characterize the dependencies and impacts between tasks, thereby determining the execution order of the tasks. For a task node v in a DAG, its upward ranking is determined by recursively calculating the upward ranking of its successor nodes. Assuming u is a successor task of v, its upward ranking is calculated as follows: The upward ranking represents the critical path length from the current task to the exit task; The downstream ranking is determined by recursively calculating the downstream ranking of its predecessor node; assuming u is a predecessor task of v, its downstream ranking is calculated as follows: The descending ranking represents the longest distance from the entry task to the current task, excluding the execution time of the task itself; (32) Action Space The action space for inference task scheduling includes determining the execution order between O DNN layers and allocating GPU / CPU and memory to each task. To reduce the size of the action space, the scheduling process is divided into scheduling time steps and real-time time steps: The scheduling time step is a decision sequence of the scheduler for the scheduler to perform actions; The real time step represents the passage of real time and is used to obtain new DNN layers to be scheduled. If the waiting matrix is ​​not full, the new DNN layers will be placed in the waiting matrix; otherwise, they will be placed in the backlog queue. At each scheduling time step, the scheduler selects an action A = {A e |A e =(d m ,q n )|m∈{-1, GPU, CPU}, n∈{-1, 1, 2, ..., O}}, where (-1, -1) is an invalid action A ψ , indicating that no DNN layer accepts scheduling, and the rest are valid actions A φ , represents the DNN layer q i Scheduled to GPU or CPU; (33)State Transition For each scheduling time step, the scheduler makes decisions by observing the system state, including: (a) If the scheduler selects a valid action The planned DNN layer is removed from the waiting matrix, and the first DNN layer in the backlog queue is placed in the waiting matrix that has just been removed. At the same time, the ready time of the GPU / CPU in the resource matrix is ​​added to the execution time E of the assigned DNN layer. i，j ,If the DNN layer is the first layer of job J, then the remaining memory will be reduced by the memory M required by the assigned DNN layer. Finally, the scheduling time step is increased by 1, and the real time step remains unchanged because there is no need to obtain a new DNN layer; (b) If the scheduler selects an invalid action A ψ , it means that there are currently no allocated DNN layers, then the real-time time step will be increased by 1 to obtain the new DNN layer to be scheduled, and the ready time of the GPU / CPU in the resource matrix will be reduced by 1. If the last DNN layer of job J is executed, the memory it occupies will be released, and the remaining memory will increase. At the same time, the waiting time of the unallocated DNN layers in the waiting matrix will be increased by 1; (34) Determine the optimization objective and design the reward function The goal of the optimization task is to reasonably allocate the system GPU, CPU and memory resources to maximize the long-term task satisfaction of all tasks, that is, The reward aims to guide the scheduler to achieve the goal of the optimal policy π = p(a|s); For valid actions, when there is sufficient remaining memory, the reward is the ratio of the response time to the expected delay requirement; if the action causes the remaining memory to be negative, negative infinity is used as a penalty to make the scheduler allocate memory resources reasonably; if an invalid action is selected, zero reward will be given; therefore, the reward function is designed as follows: (35) The Monte Carlo policy gradient algorithm is used to solve the optimization problem in step S34, and deep reinforcement learning is also included to optimize the task scheduling strategy.

4. The memory-cooperative DNN hierarchical scheduling method based on edge real-time system according to claim 3 is characterized in that: For the Monte Carlo policy gradient algorithm, the process is as follows: Based on the Markov decision process formula, at each scheduling time step t, the policy network receives the system state s n =(V, Q, |b|), and generates the probability of selecting the action as output, the probability of the action being selected is π θ (a n |s n ), denoted as A e =(d m ,q n ), if A e =(-1, -1), it means that there is no scheduled task at the current scheduling time step and no reward will be obtained. In this case, the real time step moves forward and generates the next state according to the situation (b) in the state transition. In the case of a valid action, a reward will be generated according to the satisfaction of the scheduled task and the memory situation. The next state is obtained by the situation (a) in the state transition. The state, action and reward constitute a scene trajectory, which is used to calculate the cumulative reward for further training.

5. The memory-cooperative DNN hierarchical scheduling method based on edge real-time system according to claim 3 or 4, characterized in that: The REINFORCE algorithm in the Monte Carlo policy gradient algorithm generates a trajectory τ = (s 0 , a 0 , r 1 , s 1 , a 1 , r 2 ,...,a T , r T+1 , s T+1 ) to update the policy parameters θ; In each episode trajectory, the policy gradient at each time step t is transformed into: The parameter θ is updated according to the gradient ascent, that is: Where α is the learning rate. If G n If it is positive, it means that you want to increase the n Select action a n Otherwise, reduce the log probability; Therefore, the agent is guided to find the optimal strategy. After several iterations, the agent will generate an optimal scheduling strategy.

6. The memory-cooperative DNN hierarchical scheduling method based on edge real-time system according to claim 3 is characterized in that: Step (3) also includes using deep reinforcement learning to optimize the task scheduling strategy to improve the efficiency and performance of the edge computing system and obtain the optimal scheduling strategy; By calculating the upward and downward rankings, the task completion time is reduced, and the performance and efficiency of the system are improved; When the number of accepted DNN layers is smaller than the waiting slot size, the empty positions are filled with fixed negative values ​​to reduce the probability of being selected.

Images