A model caching and request routing method and system based on dynamic deep neural network

Abstract

The present invention discloses a model caching and request routing method and system based on a dynamic deep neural network. First, the deep neural network is divided into multiple different sub-models according to the reasoning accuracy at different branch exits, and the floating-point calculation amount and memory usage of these sub-models are counted; then, with the goal of maximizing the total reasoning accuracy of user requests, a target optimization formula is established under constraints such as delay and resources. According to known user requests, a linear programming method is used to solve the target optimization formula to obtain the optimal model cache and request routing fractional solution; the model cache and request routing fractional solution is rounded to an integer solution using a random rounding method; finally, a heuristic method is used to convert the model cache and request routing solution obtained by random rounding into a feasible solution that ultimately meets all constraints. The present invention can improve the reasoning accuracy of user requests on the basis of ensuring low latency, and meet the requirements of high precision and low latency for edge intelligent applications.

Description

Technical Field

[0001] The present invention relates to the field of mobile edge computing, and in particular to a model caching and request routing method and system based on a dynamic deep neural network. Background Art

[0002] In recent years, machine learning methods, particularly deep neural networks (DNNs), have been transforming various application areas such as computer vision, speech recognition, and autonomous driving. To facilitate efficient task processing, mobile edge computing (MEC) has emerged as a promising model, providing end users with low-latency computing, caching, and transmission capabilities. In MEC, DNN models can be pre-cached on base stations (BSs) via edge servers, enabling rapid response to user requests. However, unlike cloud servers, BSs have limited resources and can only cache a few popular DNN models simultaneously to serve a subset of users. Furthermore, in multi-edge collaboration scenarios, request routing is a key factor affecting user quality of experience (QoE). Therefore, how to route user requests to BSs with cached models to serve more users is a critical issue.

[0003] Obviously, the model caching mechanism is closely related to the request routing scheme. Therefore, jointly optimizing model caching and request routing decisions is crucial to improving the QoE of users in MEC. To address this problem, existing solutions can be divided into two categories: (1) based on traditional approximation methods and using relevant mathematical models to formulate relevant model caching and request routing strategies; (2) based on deep reinforcement learning algorithms, which learn the optimal model caching and request routing strategies through the dynamic interaction between the intelligent agent and the environment.

[0004] The above two methods achieve the joint optimization of model caching and request routing in edge environments, but they still have the following limitations: (1) Existing methods mainly focus on caching complete DNN models on resource-limited BSs, resulting in a limited number of DNN models that can be fully cached, making it impossible to effectively utilize BS resources, which ultimately seriously affects user QoE. (2) Existing methods ignore the impact of model loading time on model caching and request routing decisions. Inference requests for different DNN models arrive in real time, and the caching scheme needs to be adjusted in time. The caching schemes of DNN model sub-models of different sizes will result in different model effective service times and inference accuracy, thus affecting user QoE.

[0005] Therefore, the existing joint optimization solution of model caching and request routing in the edge environment still has great limitations and cannot meet the requirements of end users for high precision and low latency, which is why this case came into being. Summary of the Invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a model caching and request routing method and system based on dynamic deep neural networks, which maximizes the overall accuracy of user request reasoning while ensuring latency, thereby meeting the high-precision and low-latency reasoning needs of end users.

[0007] Technical Solution: To achieve the above objectives, the present invention provides a model caching and request routing method based on a dynamic deep neural network, comprising the following steps:

[0008] Step 1: On an existing dataset, analyze the output feature maps of each layer of the deep neural network model to calculate the inference accuracy of the classification task, quantify the floating-point computation amount and memory usage of each layer's branch output, and divide the deep neural network model into multiple sub-models with different inference accuracy and sizes;

[0009] Step 2: Obtain the user request sequence of the current observation window and the cache status of the previous observation window. Based on the sub-models of different sizes divided in step 1 and the known deep neural network model requested by the user, use a linear programming method to solve the solution of the joint optimization problem of model caching and request routing to maximize the benefit under the condition of satisfying constraints; the benefit is the total inference accuracy of the user request. The constraints include: each base station caches one sub-model of each deep neural network model, each user request is routed to at most one base station, the sub-model cached on the base station must be fully loaded into memory before being used to infer the user request, and the base station's memory capacity and the user's maximum tolerable perceived delay constraints;

[0010] Step 3: For the optimal fractional solution obtained in step 2, round it to an integer solution using a random rounding method;

[0011] Step 4: For the rounded integer solutions of model cache and request routing obtained in step 3, a heuristic method is used to obtain the final feasible solutions of model cache and request routing.

[0012] Preferably, the step 1 specifically includes:

[0013] Step 101: Analyze the internal parameter structure of each layer of the deep neural network model, including the size of the convolution kernel, the number of channels, and the number of neurons in the fully connected layer, and calculate the floating-point calculation amount and memory usage of each layer;

[0014] Step 102: Divide the deep neural network model into three parts: an entry network, a hidden layer network, and an exit network according to its network structure;

[0015] Step 103: further divide the hidden layer network in step 102, divide the deep neural network model into multiple sub-models of different sizes, and train an output network for each sub-model separately;

[0016] In step 104, the inference path calculation from the entry network to the exit network of each sub-model is performed in the real device, and the time of loading the sub-model into the memory, the inference accuracy of each exit, and the memory usage of the path are recorded.

[0017] Preferably, the step 2 specifically includes:

[0018] Step 201: M deep neural network models Divide into Sub-models, in Represents the deep neural network model m i The corresponding empty sub-model, H(m i ) represents the deep neural network model m i The number of sub-models; deep neural network model m i The sub-model has an inference accuracy of Divide a period of time into multiple consecutive time periods, each of which constitutes an observation window. Before each observation window begins, obtain the request sequence of the user in the current observation window. and the state of the sub-model cached by each base station in the previous observation window, where U represents the number of requests;

[0019] Step 202: Model caching and request routing decisions need to satisfy the requirement that each base station caches one sub-model of each deep neural network model, including empty sub-models.

[0020] Step 203: Model caching and request routing decisions must ensure that the memory size occupied by the sub-model cached by each base station does not exceed the total memory capacity of the base station.

[0021] Step 204: Model caching and request routing decisions must ensure that each user request is routed to at most one base station, and user requests that cannot be routed are sent to the cloud server.

[0022] In step 205, the model cache and request routing decision must ensure that the delay from the user request being initiated to the receipt of the inference result of the sub-model cached on the routing base station cannot exceed the maximum perceived delay that the user can tolerate. The specific calculation method for the model inference delay is the floating-point calculation amount of the model divided by the maximum computing power that the base station can provide.

[0023] Step 206: Since the deep neural network model must be cached before executing the inference service, the model cache and request routing decision must meet the requirement that the sub-model cached on the base station must be fully loaded into the memory before being used to infer the user request. That is, the completion time of loading the sub-model into the memory must be earlier than the time when the user request is initiated. The sub-model loading time specifically refers to the time to dynamically switch from the sub-model cached in the previous observation window to another sub-model of the same deep neural network that needs to be cached in the current window.

[0024] In step 207, the optimization goal is to maximize the total inference accuracy requested by the user, which is defined as follows:

[0025]

[0026] Among them, x nh and y nu Binary variables representing whether to cache sub-model h to base station n and whether to route user u’s request to base station n, represents the base station set, represents the sub-model set corresponding to the deep neural network model m requested by user u, p h represents the inference accuracy of sub-model h;

[0027] Step 208: transform the nonlinear integer programming problem in step 207, and let the binary integer variable A nuh =x nh ·y nu , then the target formula is transformed into:

[0028]

[0029] Among them, A nuh =1 means that base station n has cached sub-model h and user u’s request is routed to base station n;

[0030] Step 209, relax the range of variables in the integer linear programming problem in step 208 so that x nh ∈[0, 1], A nuh ∈[0, 1];

[0031] Step 210, under the constraints of steps 202 to 206, use the linear programming method to solve the target optimization formula in step 208 after the variables in step 209 are relaxed to obtain the optimal fractional solution

[0032] Preferably, the step 3 specifically includes:

[0033] Step 301, let vector Equal to the indicator vector The probability of the fractional solution is Right now in, Is the size of The unique hot encoding vector of , where the hth element is 1 and the rest are 0, indicates that the sub-model h of the deep neural network model m is cached on the base station n, that is,

[0034] Step 302: Set the intermediate variable The rounding probability is Indicates whether the request of user u is attempted to be routed to the sub-model h of base station n for inference to determine the integer solution

[0035] Step 303: Obtained using the integer solution after random rounding To restore the integer solution of the request routing variable Right now Here 1(a) is the indicator vector. Specifically, when expression a holds, otherwise

[0036] Preferably, the step 4 specifically includes:

[0037] Step 401: for each base station, determine the integer solution after random rounding. Whether the memory capacity constraint is violated, if yes, go to step 404, otherwise go to step 402; A binary variable indicating whether to cache the sub-model h to the base station n;

[0038] Step 402: for each user request, determine the integer solution after random rounding. and Whether the user's maximum tolerable perceived delay constraint or the model loading constraint is violated, if so, go to step 405, otherwise go to step 403; Indicates whether to route the request of user u to base station n;

[0039] Step 403: for each user request, determine the integer solution after random rounding. Is it violated that a user request can only be routed to one base station at most? If so, go to step 406. Otherwise, obtain a feasible integer solution x that satisfies all constraints. nh and y nu ;

[0040] Step 404: Calculate the benefit of each deep neural network model based on the number of user accesses to the deep neural network models cached by the base station and the inference accuracy that the corresponding sub-model can provide. Remove the sub-model corresponding to the model type with the lowest benefit on the base station, and try to cache a smaller sub-model of the model type. If no suitable sub-model can be cached, the user request previously routed to the base station will be redirected to the cloud, and return to step 401.

[0041] Step 405: redirect the user request to the cloud server and return to step 402;

[0042] Step 406 : Routing the user request to the base station with the highest inference accuracy among the cached sub-models of the deep neural network, and returning to step 403 .

[0043] Based on the same inventive concept, the present invention provides a model caching and request routing system based on a dynamic deep neural network, comprising:

[0044] The model partitioning module is used to analyze the output feature maps of each layer of the deep neural network model on an existing dataset to calculate the inference accuracy of the classification task, quantify the floating-point calculation amount and memory usage of the branch output of each layer, and divide the deep neural network model into multiple sub-models with different inference accuracy and different sizes;

[0045] The problem construction and solution module is used to obtain the user request sequence in the current observation window and the cache status of the previous observation window. Based on the divided sub-models of different sizes and the known deep neural network model requested by the user, a linear programming method is used to solve the solution to the joint optimization problem of model caching and request routing to maximize the benefit under the condition of satisfying constraints. The benefit is the total inference accuracy of the user request. The constraints include: each base station caches one sub-model of each deep neural network model, each user request is routed to at most one base station, the sub-model cached on the base station must be fully loaded into memory before being used to infer the user request, and the base station's memory capacity and the user's maximum tolerable perceived delay constraints.

[0046] The random rounding module is used to round the optimal fractional solution obtained by the problem construction and solution module to an integer solution using the random rounding method;

[0047] The feasible solution determination module is used to obtain the final model cache and request routing feasible solution using a heuristic method for the rounded model cache and request routing integer solution.

[0048] Preferably, in the problem construction and solution module, the model cache and request routing decision need to meet the following requirements: each base station caches a sub-model of each deep neural network model, including an empty sub-model; the memory size occupied by the sub-model cached by each base station cannot exceed the total memory capacity of the base station; each user request is routed to at most one base station, and user requests that cannot be routed are sent to the cloud server; the delay from the initiation of the user request to the receipt of the inference result of the sub-model cached on the routed base station cannot exceed the maximum perceived delay that the user can tolerate; and the sub-model cached on the base station must be fully loaded into the memory before being used to infer the user request; the optimization objective is defined as follows:

[0049]

[0050] As an advantage, in the problem construction and solution module, the nonlinear integer programming problem is transformed into a binary integer variable A nuh =x nh ·y nu , the optimization objective is transformed into:

[0051]

[0052] Relax the value range of the variables in the transformed integer linear programming problem so that x nh ∈[0, 1], A nuh ∈[0, 1]; use the linear programming method to solve the target optimization formula after the variables are relaxed and transformed, and obtain the optimal fractional solution

[0053] As an advantage, in the feasible solution determination module, for each base station, the integer solution after random rounding is determined. Whether the memory capacity constraint is violated, where A binary variable indicating whether the sub-model h is cached to the base station n. If yes, then:

[0054] The benefit of each deep neural network model is calculated based on the number of user accesses to the deep neural network model cached by the base station and the inference accuracy that the corresponding sub-model can provide. The sub-model corresponding to the model type with the lowest benefit on the base station is removed, and an attempt is made to cache a smaller sub-model of the model type. If a suitable sub-model cannot be cached, the user request previously routed to the base station will be redirected to the cloud.

[0055] otherwise:

[0056] For each user request, determine the integer solution after random rounding and Whether the user's maximum tolerable perceived delay constraint or model loading constraint is violated, where Indicates whether the request of user u is routed to base station n; if so, the user request is redirected to the cloud server, otherwise for each user request, the integer solution after random rounding is determined. Is the constraint that a user request can only be routed to one base station violated? If so, the user request is routed to the base station with the highest inference accuracy in the cached deep neural network sub-model. Otherwise, a feasible integer solution x that satisfies all constraints is obtained. nh and y nu .

[0057] The present invention also provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the model caching and request routing method based on a dynamic deep neural network.

[0058] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0059] (1) The present invention comprehensively considers the base station resources and delay constraints, as well as the impact of model loading time caused by switching cache schemes on user experience quality, and provides a new solution for flexibly adjusting the cache scheme to increase the effective service time of the deep neural network model.

[0060] (2) The present invention utilizes the fine-grained resource utilization characteristics of the sub-models obtained by dividing the deep neural network model, which can more effectively utilize base station resources, thereby caching more types of deep neural network models and meeting the needs of more users.

[0061] (3) By dynamically switching between sub-models of the same deep neural network model, the present invention can quickly cache the required deep neural network sub-models, reduce the model loading time, and provide more services to more users, compared to the existing method of reloading the model from the first layer each time. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a schematic diagram of the division of the deep neural network model of the present invention;

[0063] Figure 2 Schematic diagram of the model caching and request routing edge computing system of the present invention;

[0064] Figure 3 This is a flowchart of the overall algorithm of the model caching and request routing method of the present invention. DETAILED DESCRIPTION

[0065] The present invention will be further described below with reference to the following examples, but it should not be understood that the scope of the present invention is limited to the following examples. Without departing from the above technical ideas of the present invention, various substitutions and modifications can be made according to common technical knowledge and customary means in the art, and all should be included in the scope of protection of the present invention.

[0066] Example 1:

[0067] See also Figures 1 to 3 , an embodiment of the present invention discloses a model caching and request routing method based on a dynamic deep neural network, comprising the following steps:

[0068] Step 1) First, appropriate outlets are added to the deep neural network model to form a multi-path DNN structure from input to outlet output; on the existing dataset, the output feature maps of each layer of the deep neural network model are analyzed to calculate the inference accuracy obtained by the classification task, the floating-point calculation amount and memory usage of the branch outlets of each layer are quantified, and the deep neural network model is divided into multiple sub-models with different inference accuracy and different sizes.

[0069] In this embodiment, step 1) can be achieved by the following specific steps:

[0070] Step 101) Analyze the internal parameter structure of each layer of the deep neural network, including the size of the convolution kernel, the number of channels, and the number of neurons in the fully connected layer, and calculate the floating-point calculation amount and memory usage of each layer.

[0071] Step 102) According to the network structure of the deep neural network, it is divided into three parts: the entry network, the hidden layer network and the exit network.

[0072] Step 103) Further divide the hidden layer network in step 102, divide the deep neural network model into multiple sub-models of different sizes, and train the output network for each sub-model separately.

[0073] Step 104) Execute the inference path calculation from the entry network to each sub-model exit network in the real device, and record the time to load the sub-model into the memory, the inference accuracy of each exit, and the memory usage of the path.

[0074] For example, Figure 1As shown in the figure, taking the VisionTransformer (ViT) deep neural network model as an example, we divide the deep neural network model into three parts according to its network structure: entry network (EntNet), hidden network (HidNet) and exit network (ExtNet). In order to ensure a certain inference accuracy, we divide a deep neural network model into multiple sub-models of different sizes by splitting HidNet, and each sub-model has its own pre-trained ExtNet. In addition, the dynamic switching between DNN sub-models of the same deep neural network model is the process of adding or deleting the corresponding HidNet and ExtNet. For example, when switching from sub-model 1 to sub-model 2 of ViT, we only need to remove ExtNet1 of sub-model 1, and then add HidNet2 and ExtNet2 to form sub-model 2.

[0075] Step 2) Obtain the user request sequence for the current observation window and the cache status for the previous observation window. Based on the sub-models of different sizes divided according to the DNN at different exits, and knowing the deep neural network model requested by the user, use a linear programming algorithm to solve the joint optimization problem of model caching and request routing to maximize the benefit, subject to constraints. In this embodiment, the benefit is the total inference accuracy of the user requests. The constraints include: each base station caches one sub-model of each deep neural network model, each user request is routed to at most one base station, the sub-model cached on the base station must be fully loaded into memory before being used to infer the user request, and the base station's memory capacity and the user's maximum tolerable perceived delay constraints.

[0076] In this embodiment, step 2) can be achieved by the following specific steps:

[0077] Step 201) M deep neural network models Divide into Sub-models, in Represents the deep neural network model m i The corresponding empty sub-model is introduced for the convenience of modeling and has no effect on base station resource occupancy or model reasoning accuracy; H(m i ) represents the deep neural network model m i The number of sub-models; deep neural network model m i The sub-model has an inference accuracy of A period of time is divided into multiple consecutive time periods, each of which constitutes an observation window. The set of observation windows is represented as Γ = {τ1, τ2, ...τ T}, before each observation window begins, obtain the request sequence of the user in the current observation window and the state of the sub-model cached by each base station in the previous observation window, where U represents the number of requests;

[0078] Step 202) Model caching and request routing decisions need to satisfy the requirement that each base station caches a sub-model of each deep neural network model, including empty sub-models.

[0079] Step 203) Model caching and request routing decisions must ensure that the memory size occupied by the sub-model cached by each base station cannot exceed the total memory capacity of the base station.

[0080] Step 204) Model caching and request routing decisions must ensure that each user request is routed to at most one base station, and user requests that cannot be routed are sent to the cloud server.

[0081] Step 205) Model caching and request routing decisions must ensure that the delay from the initiation of a user request to the receipt of the inference result of the sub-model cached on the routing base station cannot exceed the maximum perceived delay that the user can tolerate; the specific calculation method for the model's inference delay is the floating-point calculation amount of the model divided by the maximum computing power that the base station can provide.

[0082] Step 206) Since the deep neural network model must be cached before executing the inference service, the model cache and request routing decision need to meet the requirement that the sub-model cached on the base station must be completely loaded into the memory before being used to infer the user request, that is, the completion time of loading the sub-model into the memory must be earlier than the time when the user request is initiated; the sub-model loading time specifically refers to the time of dynamically switching from the sub-model cached in the previous observation window to another sub-model of the same deep neural network that needs to be cached in the current window.

[0083] Step 207) The optimization goal is to maximize the total inference accuracy requested by the user. The optimization formula is defined as follows:

[0084]

[0085] Among them, x nh and y nu Binary variables representing whether to cache sub-model h to base station n and whether to route user u’s request to base station n, represents the base station set, Represents the sub-model set corresponding to the deep neural network model m requested by user u. h Represents the inference accuracy of the sub-model h.

[0086] Step 208) Transform the nonlinear integer programming problem in step 207, and let the binary integer variable A nuh =x nh ·y nu, then the target formula is transformed into:

[0087]

[0088] Among them, A nuh =1 indicates that base station n has cached sub-model h and the request of user u is routed to base station n.

[0089] Step 209) Relax the range of the variables in the integer linear programming problem in step 208 so that x nh ∈[0, 1], A nuh ∈[0,1].

[0090] Step 210) Under the constraints of steps 202 to 206, use the linear programming method to solve the target optimization formula in step 208 after the variables in step 209 are relaxed to obtain the optimal fractional solution.

[0091] Step 3) For the optimal fractional solution obtained by the linear programming method in step 2, the random rounding method is used to round it to an integer solution.

[0092] In this embodiment, step 3 can be implemented through the following specific steps:

[0093] Step 301) Let vector Equal to the indicator vector The probability is the fractional solution described in step 2 Right now in, Is the size of The unique hot encoding vector of , where the hth element is 1 and the rest are 0, indicates that the sub-model h of the deep neural network model m is cached on the base station n, that is,

[0094] Step 302) According to the fractional solution described in step 2, let the intermediate variable The rounding probability is Indicates whether the request of user u is attempted to be routed to the sub-model h of base station n for inference to determine the integer solution

[0095] Step 303) Obtained using the integer solution after random rounding To restore the integer solution of the request routing variable Right now Here 1(a) is the indicator vector. Specifically, when expression a holds, otherwise

[0096] For example, the random rounding process algorithm of step 3 can be described by the following pseudo code:

[0097]

[0098] Among them, d u ,ddl u and s u They represent the data size of user u’s request task, the maximum perceived delay that user u can tolerate, and the initiation time of user u’s request respectively; τ represents the current time window, Indicates the status of the cached results of the previous observation window τ-1.

[0099] Step 4): For the rounded integer solutions of model cache and request routing obtained in step 3, a heuristic algorithm is used to obtain the final feasible solutions of model cache and request routing.

[0100] In this embodiment, step 4 can be implemented through the following specific steps:

[0101] Step 401) For each base station, determine the integer solution after random rounding described in step 3. Whether the memory capacity constraint described in step 2 is violated, if so, go to step 404, otherwise go to step 402.

[0102] Step 402) For each user request, determine the integer solution after random rounding described in step 3. and Whether the user's maximum tolerable perceived delay constraint or the model loading constraint described in step 2 is violated, if so, go to step 405, otherwise go to step 403.

[0103] Step 403) For each user request, determine the integer solution after random rounding described in step 3. Is the constraint that a user request can only be routed to one base station as described in step 2 violated? If so, go to step 406. Otherwise, obtain a feasible integer solution x that ultimately satisfies all constraints. nh and y nu .

[0104] Step 404) Based on the number of user accesses to the deep neural network model cached by the base station and the inference accuracy p that the corresponding sub-model can provide h (in ) to calculate each deep neural network model The base station removes the submodel corresponding to the least efficient model type and attempts to cache a smaller submodel of that model type. If no suitable submodel can be cached, the user request previously routed to the base station is redirected to the cloud. Return to step 401.

[0105] Step 405) redirects the user request to the cloud server and returns to step 402.

[0106] Step 406) Routes the user request to the base station with the highest inference accuracy in the cached sub-model of the deep neural network, and returns to step 403.

[0107] Exemplarily, the benefit calculation method of the deep neural network model m in step 404 is as follows: 1) for base station n, if base station n caches a sub-model of model m, then count the number of users who requested model m and were successfully routed to the base station n, and record it as margin1; 2) calculate the accuracy benefit that the sub-model h of model m cached by base station n can bring in the current observation window, that is, margin2 = margin1*p h ;3) Considering the trade-off between accuracy benefit and base station memory resource usage, the benefit is finally calculated as: margin = margin2 / r h , where r h is the memory size of the sub-model h.

[0108] In some embodiments, the model caching and request routing method based on dynamic deep neural network can be described in the following pseudo code to describe the overall process.

[0109]

[0110] Example 2:

[0111] An embodiment of the present invention discloses a model caching and request routing system based on a dynamic deep neural network, comprising:

[0112] The model partitioning module is used to analyze the output feature maps of each layer of the deep neural network model on an existing dataset to calculate the inference accuracy of the classification task, quantify the floating-point calculation amount and memory usage of the branch output of each layer, and divide the deep neural network model into multiple sub-models with different inference accuracy and different sizes;

[0113] The problem construction and solution module is used to obtain the user request sequence in the current observation window and the cache status of the previous observation window. Based on the divided sub-models of different sizes and the known deep neural network model requested by the user, a linear programming method is used to solve the solution to the joint optimization problem of model caching and request routing to maximize the benefit under the condition of satisfying constraints. The benefit is the total inference accuracy of the user request. The constraints include: each base station caches one sub-model of each deep neural network model, each user request is routed to at most one base station, the sub-model cached on the base station must be fully loaded into memory before being used to infer the user request, and the base station's memory capacity and the user's maximum tolerable perceived delay constraints.

[0114] The random rounding module is used to round the optimal fractional solution obtained by the problem construction and solution module to an integer solution using the random rounding method;

[0115] The feasible solution determination module is used to obtain the final model cache and request routing feasible solution using a heuristic method for the rounded model cache and request routing integer solution.

[0116] In some embodiments, in the problem construction and solution module, model caching and request routing decisions need to meet the following requirements: each base station caches one sub-model of each deep neural network model, including empty sub-models; the memory size occupied by the sub-models cached by each base station cannot exceed the total memory capacity of the base station; each user request is routed to at most one base station, and user requests that cannot be routed are sent to the cloud server; the delay from the initiation of the user request to the receipt of the inference result of the sub-model cached on the routed base station cannot exceed the maximum perceived delay that the user can tolerate; and the sub-model cached on the base station must be fully loaded into memory before being used to infer the user request; the optimization objective is defined as follows:

[0117]

[0118] In some embodiments, in the problem construction and solution module, the nonlinear integer programming problem is transformed, and the binary integer variable A is set to nu =x nh ·y nu , the optimization objective is transformed into:

[0119]

[0120] Relax the value range of the variables in the transformed integer linear programming problem so that x nh ∈[0, 1], A nuh ∈[0, 1]; use the linear programming method to solve the target optimization formula after the variables are relaxed and transformed, and obtain the optimal fractional solution

[0121] In some embodiments, in the feasible solution determination module, for each base station, the integer solution after random rounding is determined. Whether the memory capacity constraint is violated, if so:

[0122] The benefit of each deep neural network model is calculated based on the number of user accesses to the deep neural network model cached by the base station and the inference accuracy that the corresponding sub-model can provide. The sub-model corresponding to the model type with the lowest benefit on the base station is removed, and an attempt is made to cache a smaller sub-model of the model type. If a suitable sub-model cannot be cached, the user request previously routed to the base station will be redirected to the cloud.

[0123] otherwise:

[0124] For each user request, determine the integer solution after random rounding and Whether it violates the user's maximum tolerable perceived delay constraint or model loading constraint, if so, redirect the user request to the cloud server, otherwise for each user request, determine the integer solution after random rounding Is the constraint that a user request can only be routed to one base station violated? If so, the user request is routed to the base station with the highest inference accuracy in the cached deep neural network sub-model. Otherwise, a feasible integer solution x that satisfies all constraints is obtained. nh and y nu .

[0125] Example 3:

[0126] A computer program product disclosed in an embodiment of the present invention includes a computer program / instruction, which, when executed by a processor, implements the steps of the model caching and request routing method based on a dynamic deep neural network described in Example 1.

[0127] The program / instruction code for implementing the inventive method can be written in any combination of one or more programming languages. These programs / instruction codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device so that the program / instruction code, when executed by the processor or controller, causes the steps of the inventive method to be implemented. The program / instruction code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or completely on a remote machine or server. The parts not described in detail in the present invention are all known technologies of those skilled in the art.

Claims

1. A model caching and request routing method based on dynamic deep neural network, characterized in that: The steps include: Step 1: On an existing dataset, analyze the output feature maps of each layer of the deep neural network model to calculate the inference accuracy of the classification task, quantify the floating-point computation amount and memory usage of each layer's branch output, and divide the deep neural network model into multiple sub-models with different inference accuracy and sizes; Step 2: Obtain the user request sequence for the current observation window and the cache status for the previous observation window. Based on the sub-models of different sizes divided in step 1 and the known deep neural network model requested by the user, use a linear programming method to solve the joint optimization problem of model caching and request routing to maximize the benefits while satisfying the constraints. The benefit is the total inference accuracy of user requests. The constraints include: each base station caches one sub-model of each deep neural network model, each user request is routed to at most one base station, the sub-model cached on the base station must be fully loaded into memory before being used to infer the user request, as well as the memory capacity of the base station and the maximum tolerable perceived delay of the user. Step 3: For the optimal fractional solution obtained in step 2, round it to an integer solution using a random rounding method; Step 4: For the rounded integer solutions of model cache and request routing obtained in step 3, a heuristic method is used to obtain the final feasible solutions of model cache and request routing.

2. The model caching and request routing method based on a dynamic deep neural network according to claim 1, characterized in that: The step 1 specifically includes: Step 101: Analyze the internal parameter structure of each layer of the deep neural network model, including the size of the convolution kernel, the number of channels, and the number of neurons in the fully connected layer, and calculate the floating-point calculation amount and memory usage of each layer; Step 102: Divide the deep neural network model into three parts: an entry network, a hidden layer network, and an exit network according to its network structure; Step 103: further divide the hidden layer network in step 102, divide the deep neural network model into multiple sub-models of different sizes, and train an output network for each sub-model separately; In step 104, the inference path calculation from the entry network to the exit network of each sub-model is performed in the real device, and the time of loading the sub-model into the memory, the inference accuracy of each exit, and the memory usage of the path are recorded.

3. The model caching and request routing method based on a dynamic deep neural network according to claim 1, characterized in that: The step 2 specifically includes: Step 201: M deep neural network models Divide into Sub-models, in Represents the deep neural network model m i The corresponding empty sub-model, H(m i ) represents the deep neural network model m i The number of sub-models; deep neural network model m i The sub-model has an inference accuracy of Divide a period of time into multiple consecutive time periods, each of which constitutes an observation window. Before each observation window begins, obtain the request sequence of the user in the current observation window. and the state of the sub-model cached by each base station in the previous observation window, where U represents the number of requests; Step 202: Model caching and request routing decisions need to satisfy the requirement that each base station caches one sub-model of each deep neural network model, including empty sub-models. Step 203: Model caching and request routing decisions must ensure that the memory size occupied by the sub-model cached by each base station does not exceed the total memory capacity of the base station. Step 204: Model caching and request routing decisions must ensure that each user request is routed to at most one base station, and user requests that cannot be routed are sent to the cloud server. In step 205, the model cache and request routing decision must ensure that the delay from the user request being initiated to the receipt of the inference result of the sub-model cached on the routing base station cannot exceed the maximum perceived delay that the user can tolerate. The specific calculation method for the model inference delay is the floating-point calculation amount of the model divided by the maximum computing power that the base station can provide. Step 206: Since the deep neural network model must be cached before executing the inference service, the model cache and request routing decision must meet the requirement that the sub-model cached on the base station must be fully loaded into the memory before being used to infer the user request. That is, the completion time of loading the sub-model into the memory must be earlier than the time when the user request is initiated. The sub-model loading time specifically refers to the time to dynamically switch from the sub-model cached in the previous observation window to another sub-model of the same deep neural network that needs to be cached in the current window. In step 207, the optimization goal is to maximize the total inference accuracy requested by the user, which is defined as follows: Among them, x nh and y nu Binary variables representing whether to cache sub-model h to base station n and whether to route user u’s request to base station n, represents the base station set, represents the sub-model set corresponding to the deep neural network model m requested by user u, p h represents the inference accuracy of sub-model h; Step 208: transform the nonlinear integer programming problem in step 207, and let the binary integer variable A nuh =x nh ·y nu , then the target formula is transformed into: Among them, A nuh =1 means that base station n has cached sub-model h and user u’s request is routed to base station n; Step 209, relax the range of variables in the integer linear programming problem in step 208 so that x nh ∈[0, 1], A nuh ∈[0, 1]; Step 210, under the constraints of steps 202 to 206, use the linear programming method to solve the target optimization formula in step 208 after the variables in step 209 are relaxed to obtain the optimal fractional solution 4. The model caching and request routing method based on a dynamic deep neural network according to claim 3, characterized in that: The step 3 specifically includes: Step 301, let vector Equal to the indicator vector The probability of the fractional solution is Right now in, Is the size of The unique hot encoding vector of , where the hth element is 1 and the rest are 0, indicates that the sub-model h of the deep neural network model m is cached on the base station n, that is, Step 302: Set the intermediate variable The rounding probability is Indicates whether the request of user u is attempted to be routed to the sub-model h of base station n for inference to determine the integer solution Step 303: Obtained using the integer solution after random rounding To restore the integer solution of the request routing variable Right now Here 1(a) is the indicator vector. Specifically, when expression a holds, otherwise 5. The model caching and request routing method based on dynamic deep neural network according to claim 1, characterized in that: The step 4 specifically includes: Step 401: for each base station, determine the integer solution after random rounding. Whether the memory capacity constraint is violated, if yes, go to step 404, otherwise go to step 402; A binary variable indicating whether to cache the sub-model h to the base station n; Step 402: for each user request, determine the integer solution after random rounding. and Whether the user's maximum tolerable perceived delay constraint or the model loading constraint is violated, if so, go to step 405, otherwise go to step 403; Indicates whether to route the request of user u to base station n; Step 403: for each user request, determine the integer solution after random rounding. Is it violated that a user request can only be routed to one base station at most? If so, go to step 406. Otherwise, obtain a feasible integer solution x that satisfies all constraints. nh and y nu ; Step 404: Calculate the benefit of each deep neural network model based on the number of user accesses to the deep neural network models cached by the base station and the inference accuracy that the corresponding sub-model can provide. Remove the sub-model corresponding to the model type with the lowest benefit on the base station, and try to cache a smaller sub-model of the model type. If no suitable sub-model can be cached, the user request previously routed to the base station will be redirected to the cloud, and return to step 401. Step 405: redirect the user request to the cloud server and return to step 402; Step 406 : Routing the user request to the base station with the highest inference accuracy among the cached sub-models of the deep neural network, and returning to step 403 .

6. A model caching and request routing system based on dynamic deep neural network, characterized in that: include: The model partitioning module is used to analyze the output feature maps of each layer of the deep neural network model on an existing dataset to calculate the inference accuracy of the classification task, quantify the floating-point calculation amount and memory usage of the branch output of each layer, and divide the deep neural network model into multiple sub-models with different inference accuracy and different sizes; The problem construction and solution module is used to obtain the user request sequence in the current observation window and the cache status of the previous observation window. Based on the divided sub-models of different sizes and the known deep neural network model requested by the user, a linear programming method is used to solve the solution to the joint optimization problem of model caching and request routing to maximize the benefits under the condition of satisfying constraints. The benefit is the total inference accuracy of user requests. The constraints include: each base station caches one sub-model of each deep neural network model, each user request is routed to at most one base station, the sub-model cached on the base station must be fully loaded into memory before being used to infer the user request, as well as the memory capacity of the base station and the maximum tolerable perceived delay of the user. The random rounding module is used to round the optimal fractional solution obtained by the problem construction and solution module to an integer solution using a random rounding method; The feasible solution determination module is used to obtain the final model cache and request routing feasible solution using a heuristic method for the rounded model cache and request routing integer solution.

7. The model caching and request routing system based on dynamic deep neural network according to claim 6, characterized in that: In the problem construction and solution module, model caching and request routing decisions must meet the following requirements: each base station caches one sub-model of each deep neural network model, including empty sub-models; the memory size occupied by the sub-model cached by each base station cannot exceed the total memory capacity of the base station; Each user request is routed to at most one base station, and user requests that cannot be routed are sent to the cloud server; The latency from the time a user request is initiated to the time it receives the inference result of the sub-model cached on the routing base station cannot exceed the maximum perceived latency that the user can tolerate. Furthermore, the sub-model cached on the base station must be fully loaded into memory before being used to infer the user request. The optimization objective is defined as follows: Among them, x nh and y nu Binary variables representing whether to cache sub-model h to base station n and whether to route user u’s request to base station n, Represents the user's request sequence, represents the base station set, represents the sub-model set corresponding to the deep neural network model m requested by user u, p h Represents the inference accuracy of the sub-model h.

8. The model caching and request routing system based on dynamic deep neural network according to claim 7, characterized in that: In the problem construction and solution module, the nonlinear integer programming problem is transformed into a binary integer variable A. nuh =x nh ·y nu , the optimization objective is transformed into: Among them, A nuh =1 means that base station n has cached sub-model h and user u’s request is routed to base station n; Relax the value range of the variables in the transformed integer linear programming problem so that x nh ∈[0,1],A nuh ∈[0,1]; use the linear programming method to solve the target optimization formula after the variables are relaxed and transformed, and obtain the optimal fractional solution 9. The model caching and request routing system based on dynamic deep neural network according to claim 6, characterized in that: In the feasible solution determination module, for each base station, the integer solution after random rounding is determined. Whether the memory capacity constraint is violated, where A binary variable indicating whether the sub-model h is cached to the base station n. If yes, then: The benefit of each deep neural network model is calculated based on the number of user accesses to the deep neural network model cached by the base station and the inference accuracy provided by the corresponding sub-model. The sub-model corresponding to the model type with the lowest benefit on the base station is removed, and an attempt is made to cache smaller sub-models of that model type. If a suitable sub-model cannot be cached, the user request previously routed to the base station will be redirected to the cloud. otherwise: For each user request, determine the integer solution after random rounding and Whether the user's maximum tolerable perceived delay constraint or model loading constraint is violated, where Indicates whether to route the request of user u to base station n; If yes, redirect the user request to the cloud server, otherwise for each user request, determine the integer solution after random rounding. Is the constraint that a user request can only be routed to one base station violated? If so, the user request is routed to the base station with the highest inference accuracy in the cached deep neural network sub-model. Otherwise, a feasible integer solution x that satisfies all constraints is obtained. nh and y nu .

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the model caching and request routing method based on a dynamic deep neural network are implemented.

Images