Model caching and request routing method and system based on dynamic deep neural network

Abstract

The invention discloses a model caching and request routing method and system based on a dynamic deep neural network, and the method comprises the steps: dividing a deep neural network into a plurality of different sub-models according to the reasoning precision of the deep neural network at different branch outlets, and carrying out the statistics of the floating point calculation amount and memory occupation of the sub-models; secondly, establishing a target optimization formula under the constraints of delay, resources and the like by taking maximization of the total reasoning precision of the user requests as a target, and solving the target optimization formula by utilizing a linear programming method according to the known user requests so as to obtain an optimal model cache and request routing score solution; rounding the model cache and the request routing score solution into an integer solution by using a random rounding method; and finally, utilizing a heuristic method to convert the solution of the model cache and the request route obtained by random rounding into a feasible solution which finally meets all constraints. According to the method, the reasoning precision of the user request can be improved on the basis of ensuring relatively low time delay, and the requirements of high precision and low delay of edge intelligence application are met.

Description

Technical Field

[0001] The present invention relates to the field of mobile edge computing, and in particular to a model caching and request routing method and system based on a dynamic deep neural network. Background Art

[0002] In recent years, machine learning methods, especially deep neural networks (DNNs), are transforming various application areas such as computer vision, speech recognition, and autonomous driving. To promote efficient task processing, mobile edge computing (MEC) has become a promising model that can provide low-latency computing, caching, and transmission capabilities to end users. In MEC, DNN models can be pre-cached on base stations (BSs) through edge servers to quickly respond to user requests. However, unlike cloud servers, due to the limited resources of base stations, only a few popular DNN models can be cached at the same time to provide services to some users. In addition, in multi-edge collaboration scenarios, request routing is also a key factor affecting the quality of experience (QoE) of users. Therefore, how to route user requests to the BS with cached related models for execution to serve more users is an important issue.

[0003] Obviously, the model caching mechanism is closely related to the request routing scheme. Therefore, jointly optimizing the model caching and request routing decisions is crucial to improving the QoE of users in MEC. To address this problem, existing solutions can be divided into two categories: (1) based on traditional approximation methods and using related mathematical models to formulate relevant model caching and request routing strategies; (2) based on deep reinforcement learning algorithms, which learn the optimal model caching and request routing strategies through the dynamic interaction between the agent and the environment.

[0004] The above two methods achieve the joint optimization of model caching and request routing in edge environments, but they still have the following limitations: (1) Existing methods mainly focus on caching complete DNN models on BSs with limited resources, resulting in a limited number of DNN models that can be fully cached, which makes it impossible to effectively utilize BS resources and ultimately seriously affects user QoE. (2) Existing methods ignore the impact of model loading time on model caching and request routing decisions. Inference requests for different DNN models arrive in real time, and the caching scheme needs to be adjusted in time. The caching schemes of DNN model sub-models of different sizes will result in different model effective service times and inference accuracy, thus affecting user QoE.

[0005] Therefore, the existing joint optimization solution of model caching and request routing in the edge environment still has great limitations and cannot meet the requirements of end users for high precision and low latency, which leads to this case. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a model caching and request routing method and system based on a dynamic deep neural network, which maximizes the overall accuracy of user request reasoning while ensuring latency, thereby meeting the high-precision and low-latency reasoning requirements of end users.

[0007] Technical solution: In order to achieve the above-mentioned purpose, the present invention provides a model caching and request routing method based on a dynamic deep neural network, comprising the following steps:

[0008] Step 1: On the existing data set, the output feature maps of each layer of the deep neural network model are analyzed to calculate the reasoning accuracy obtained by the classification task, the floating-point calculation amount and memory usage of the branch export of each layer are quantified, and the deep neural network model is divided into multiple sub-models with different reasoning accuracy and different sizes;

[0009] Step 2, obtaining the user request sequence of the current observation window and the cache status of the previous observation window, based on the sub-models of different sizes divided in step 1, and on the basis of the known deep neural network model requested by the user, using the linear programming method to solve the solution of the joint optimization problem of model cache and request routing with maximum benefit under the condition of satisfying constraints; the benefit is the total inference accuracy of the user request, and the constraints include: each base station caches a sub-model of each deep neural network model, each user request is routed to at most one base station, the sub-model cached on the base station must be fully loaded into the memory before being used to infer the user request, as well as the memory capacity of the base station and the maximum tolerable perceived delay constraint of the user;

[0010] Step 3, for the optimal fractional solution obtained in step 2, round it to an integer solution using a random rounding method;

[0011] Step 4: For the rounded integer solutions of model cache and request routing obtained in step 3, a heuristic method is used to obtain the final feasible solutions of model cache and request routing.

[0012] Preferably, the step 1 specifically comprises:

[0013] Step 101, analyzing the internal parameter structure of each layer of the deep neural network model, the parameters include the size of the convolution kernel, the number of channels and the number of neurons in the fully connected layer, and calculating the floating point calculation amount and memory usage of each layer;

[0014] Step 102, dividing the deep neural network model into three parts: an entry network, a hidden layer network, and an exit network according to the network structure of the deep neural network model;

[0015] Step 103, further dividing the hidden layer network in step 102, dividing the deep neural network model into a plurality of sub-models of different sizes, and training an export network for each sub-model separately;

[0016] Step 104, perform the inference path calculation from the entry network to each sub-model exit network in the real device, and record the time of loading the sub-model into the memory, the inference accuracy of each exit and the memory usage of the path.

[0017] Preferably, the step 2 specifically comprises:

[0018] Step 201: M deep neural network models Divide into Sub-models, in Represents the deep neural network model m i The corresponding empty submodel, H(m i ) represents the deep neural network model m i The number of sub-models; deep neural network model m i The sub-model of has an inference accuracy of Divide a period of time into multiple consecutive time periods, each of which constitutes an observation window. Before each observation window begins, obtain the request sequence of the user in the current observation window. and the state of the sub-model cached by each base station in the previous observation window, where U represents the number of requests;

[0019] Step 202, model caching and request routing decisions need to satisfy that each base station caches a sub-model of each deep neural network model, including an empty sub-model;

[0020] Step 203: Model caching and request routing decisions need to satisfy the requirement that the memory size occupied by the sub-model cached by each base station cannot exceed the total memory capacity of the base station;

[0021] Step 204: Model caching and request routing decisions need to satisfy the requirement that each user request is routed to at most one base station, and user requests that cannot be routed are sent to the cloud server;

[0022] Step 205: Model caching and request routing decisions need to satisfy the requirement that the delay from the initiation of a user request to the receipt of the inference result of the sub-model cached on the routing base station cannot exceed the maximum perceived delay that the user can tolerate; the specific calculation method of the model inference delay is the floating-point calculation amount of the model divided by the maximum computing power that the base station can provide;

[0023] Step 206, since the deep neural network model must be cached before executing the inference service, the model cache and request routing decision need to meet the requirement that the sub-model cached on the base station must be fully loaded into the memory before being used to infer the user request, that is, the completion time of loading the sub-model into the memory must be earlier than the time when the user request is initiated; the sub-model loading time specifically refers to the time of dynamically switching from the sub-model cached in the previous observation window to another sub-model of the same deep neural network that needs to be cached in the current window;

[0024] Step 207, the optimization goal is to maximize the total reasoning accuracy requested by the user, which is defined as follows:

[0025]

[0026] Among them, x nh and nu Binary variables representing whether to cache sub-model h to base station n and whether to route user u’s request to base station n, represents the base station set, represents the sub-model set corresponding to the deep neural network model m requested by user u, p h represents the inference accuracy of sub-model h;

[0027] Step 208, transform the nonlinear integer programming problem in step 207, and let the binary integer variable A nuh =x nh ·y nu , then the target formula is transformed into:

[0028]

[0029] Among them, A nuh =1 means that base station n has cached submodel h and the request of user u is routed to base station n;

[0030] Step 209, relax the value range of the variables in the integer linear programming problem in step 208 so that x nh ∈[0, 1], A nuh ∈[0, 1];

[0031] Step 210, under the constraints of steps 202 to 206, use the linear programming method to solve the target optimization formula in step 208 after the variables in step 209 are relaxed to obtain the optimal fractional solution

[0032] Preferably, the step 3 specifically comprises:

[0033] Step 301, let vector Equal to indicator vector The probability of the fractional solution is Right now in, Is the size of The one-hot encoding vector of , where the hth element is 1 and the rest are 0, indicates that the sub-model h of the deep neural network model m is cached on the base station n, that is,

[0034] Step 302, let the intermediate variable The rounding probability is Indicates whether the request of user u is attempted to be routed to the sub-model h of base station n for inference to determine the integer solution

[0035] Step 303, let Using the integer solution after random rounding To restore the integer solution of the request routing variable Right now Here 1(a) is the indicator vector. Specifically, when expression a holds, otherwise

[0036] Preferably, the step 4 specifically comprises:

[0037] Step 401: for each base station, determine the integer solution after random rounding. Whether the memory capacity constraint is violated, if yes, go to step 404, otherwise go to step 402; A binary variable indicating whether to cache sub-model h to base station n;

[0038] Step 402: for each user request, determine the integer solution after random rounding. and Whether the user's maximum tolerable perceived delay constraint or the model loading constraint is violated, if yes, go to step 405, otherwise go to step 403; wherein Indicates whether to route the request of user u to base station n;

[0039] Step 403: for each user request, determine the integer solution after random rounding. Is it a violation of the constraint that a user request can only be routed to one base station at most? If so, go to step 406. Otherwise, obtain a feasible integer solution x that ultimately satisfies all constraints. nh and nu ;

[0040] Step 404: Calculate the benefit of each deep neural network model based on the number of user accesses to the deep neural network model cached by the base station and the inference accuracy that the corresponding sub-model can provide, remove the sub-model corresponding to the model type with the lowest benefit on the base station, and try to cache a smaller sub-model of the model type; if a suitable sub-model cannot be cached, the user request previously routed to the base station will be redirected to the cloud, and return to step 401;

[0041] Step 405, redirect the user request to the cloud server, and return to step 402;

[0042] Step 406 , routing the user request to the base station with the highest reasoning accuracy in the cached sub-model of the deep neural network, and returning to step 403 .

[0043] Based on the same inventive concept, the present invention provides a model caching and request routing system based on a dynamic deep neural network, comprising:

[0044] The model partitioning module is used to analyze the output feature maps of each layer of the deep neural network model on the existing data set to calculate the inference accuracy obtained by the classification task, quantify the floating-point calculation amount and memory usage of the branch export of each layer, and divide the deep neural network model into multiple sub-models with different inference accuracy and different sizes;

[0045] A problem construction and solution module is used to obtain the user request sequence of the current observation window and the cache status of the previous observation window, and based on the divided sub-models of different sizes and the deep neural network model requested by the known user, use a linear programming method to solve the solution of the joint optimization problem of model cache and request routing with maximum benefit under the condition of satisfying constraints; the benefit is the total inference accuracy of the user request, and the constraints include: each base station caches a sub-model of each deep neural network model, each user request is routed to at most one base station, the sub-model cached on the base station must be fully loaded into the memory before being used to infer the user request, as well as the memory capacity of the base station and the maximum tolerable perceived delay constraints of the user;

[0046] A random rounding module is used to round the optimal fractional solution obtained by the problem construction and solution module to an integer solution using a random rounding method;

[0047] The feasible solution determination module is used to obtain the final feasible solution of the model cache and request routing by using a heuristic method for the rounded integer solutions of the model cache and request routing.

[0048] Preferably, in the problem construction and solution module, the model cache and request routing decision need to meet the following requirements: each base station caches a sub-model of each deep neural network model, including an empty sub-model; the memory size occupied by the sub-model cached by each base station cannot exceed the total memory capacity of the base station; each user request is routed to at most one base station, and user requests that cannot be routed are sent to the cloud server; the delay from the initiation of the user request to the receipt of the inference result of the sub-model cached on the routed base station cannot exceed the maximum perceived delay that the user can tolerate; and the sub-model cached on the base station must be fully loaded into the memory before being used to infer the user request; the optimization objective is defined as follows:

[0049]

[0050] Preferably, in the problem construction and solution module, the nonlinear integer programming problem is transformed, and the binary integer variable A is nuh =x nh ·y nu , the optimization objective is transformed into:

[0051]

[0052] Relax the range of variables in the transformed integer linear programming problem so that x nh ∈[0, 1], A nu ∈[0, 1]; use the linear programming method to solve the transformed target optimization formula after variable relaxation and obtain the optimal fractional solution

[0053] Preferably, in the feasible solution determination module, for each base station, the integer solution after random rounding is determined. Whether the memory capacity constraint is violated, where A binary variable indicating whether to cache submodel h to base station n. If yes, then:

[0054] The benefit of each deep neural network model is calculated based on the number of user accesses to the deep neural network model cached by the base station and the inference accuracy that the corresponding sub-model can provide. The sub-model corresponding to the model type with the lowest benefit on the base station is removed, and an attempt is made to cache a smaller sub-model of the model type. If a suitable sub-model cannot be cached, the user request previously routed to the base station will be redirected to the cloud.

[0055] otherwise:

[0056] For each user request, determine the integer solution after random rounding and Whether the user's maximum tolerable perceived delay constraint or model loading constraint is violated, where Indicates whether to route the request of user u to base station n; if yes, redirect the user request to the cloud server, otherwise for each user request, determine the integer solution after random rounding Does it violate the constraint that a user request can only be routed to one base station at most? If so, route the user request to the base station with the highest inference accuracy in the cached deep neural network sub-model. Otherwise, obtain a feasible integer solution x that finally satisfies all constraints. nh and nu .

[0057] The present invention also provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the model caching and request routing method based on a dynamic deep neural network.

[0058] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0059] (1) The present invention comprehensively considers the base station resources and delay constraints, as well as the impact of the model loading time caused by switching cache schemes on the user experience quality, and provides a new solution for flexibly adjusting the cache scheme to increase the effective service time of the deep neural network model.

[0060] (2) The present invention utilizes the fine-grained resource utilization characteristics of the sub-models obtained by dividing the deep neural network model, which can more effectively utilize base station resources, thereby caching more types of deep neural network models to meet the needs of more users.

[0061] (3) By dynamically switching between sub-models of the same deep neural network model, the present invention can quickly cache the required deep neural network sub-models, reduce the model loading time, and provide more services to more users, compared with the existing method of reloading the model from the first layer each time. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of the division of the deep neural network model of the present invention;

[0063] Figure 2 A schematic diagram of the model caching and request routing edge computing system of the present invention;

[0064] Figure 3 The figure is a flow chart of the overall algorithm of the model caching and request routing method of the present invention. DETAILED DESCRIPTION

[0065] The present invention is further described below in conjunction with the embodiments, but it should not be understood that the above subject matter of the present invention is limited to the following embodiments. Without departing from the above technical ideas of the present invention, various substitutions and changes are made according to the common technical knowledge and customary means in the art, which should all be included in the protection scope of the present invention.

[0066] Embodiment 1:

[0067] See also Figures 1 to 3 , a model caching and request routing method based on a dynamic deep neural network disclosed in an embodiment of the present invention comprises the following steps:

[0068] Step 1) First, add appropriate exports to the deep neural network model to form a multi-path DNN structure from input to export output; on the existing data set, analyze the output feature maps of each layer of the deep neural network model to calculate the inference accuracy obtained by the classification task, quantify the floating-point calculation amount and memory usage of the branch exports of each layer, and divide the deep neural network model into multiple sub-models with different inference accuracies and different sizes.

[0069] In this embodiment, step 1) can be achieved by the following specific steps:

[0070] Step 101) Analyze the internal parameter structure of each layer of the deep neural network, including the size of the convolution kernel, the number of channels, and the number of neurons in the fully connected layer, and calculate the floating-point calculation amount and memory usage of each layer.

[0071] Step 102) According to the network structure of the deep neural network, it is divided into three parts: an entry network, a hidden layer network and an exit network.

[0072] Step 103) further divides the hidden layer network in step 102, divides the deep neural network model into multiple sub-models of different sizes, and trains an export network for each sub-model separately.

[0073] Step 104) Execute the inference path calculation from the entry network to each sub-model exit network in the real device, and record the time of loading the sub-model into the memory, the inference accuracy of each exit, and the memory usage of the path.

[0074] For example, Figure 1As shown in the figure, taking the VisionTransformer (ViT) deep neural network model as an example, we divide the deep neural network model into three parts according to its network structure: entry network (EntNet), hidden network (HidNet) and exit network (ExtNet). In order to ensure a certain inference accuracy, we divide a deep neural network model into multiple sub-models of different sizes by splitting HidNet, and each sub-model has its own pre-trained ExtNet. In addition, the dynamic switching between DNN sub-models of the same deep neural network model is the process of adding or deleting the corresponding HidNet and ExtNet. For example, when switching from sub-model 1 to sub-model 2 of ViT, we only need to remove the ExtNet of sub-model 1. 1 , then add HidNet 2 and ExtNet 2 , forming sub-model 2.

[0075] Step 2) Obtain the user request sequence of the current observation window and the cache status of the previous observation window, and based on the above-mentioned sub-models of different sizes divided according to the DNN of different exits, and on the basis of the known deep neural network model requested by the user, use a linear programming algorithm to solve the solution of the joint optimization problem of model cache and request routing with maximum benefit under the condition of satisfying constraints. In this embodiment, the benefit is the total reasoning accuracy of the user request, and the constraints include: each base station caches a sub-model of each deep neural network model, each user request is routed to at most one base station, the sub-model cached on the base station must be fully loaded into the memory before being used to reason about the user request, as well as the memory capacity of the base station and the maximum tolerable perceived delay constraints of the user.

[0076] In this embodiment, step 2) can be achieved by the following specific steps:

[0077] Step 201) M deep neural network models Divide into Sub-models, in Represents the deep neural network model m i The corresponding empty sub-model is introduced for the convenience of modeling and has no effect on base station resource occupancy or model reasoning accuracy; H(m i ) represents the deep neural network model m i The number of sub-models; deep neural network model m i The sub-model of has an inference accuracy of A period of time is divided into multiple continuous time periods, each of which constitutes an observation window. The observation window set is represented by Γ = {τ 1 , τ 2 , ...τT}, before each observation window starts, obtain the request sequence of the user in the current observation window and the state of the sub-model cached by each base station in the previous observation window, where U represents the number of requests;

[0078] Step 202) Model caching and request routing decisions need to satisfy that each base station caches one sub-model of each deep neural network model, including empty sub-models.

[0079] Step 203) Model caching and request routing decisions need to satisfy the requirement that the memory size occupied by the sub-model cached by each base station cannot exceed the total memory capacity of the base station.

[0080] Step 204) Model caching and request routing decisions need to satisfy that each user request is routed to at most one base station, and user requests that cannot be routed are sent to the cloud server.

[0081] Step 205) Model caching and request routing decisions need to satisfy the requirement that the delay from the initiation of a user request to the receipt of the inference result of the sub-model cached on the routing base station cannot exceed the maximum perceived delay that the user can tolerate; the specific calculation method of the model's inference delay is the floating-point calculation amount of the model divided by the maximum computing power that the base station can provide.

[0082] Step 206) Since the deep neural network model must be cached before executing the inference service, the model cache and request routing decision need to satisfy that the sub-model cached on the base station must be completely loaded into the memory before being used to infer the user request, that is, the completion time of loading the sub-model into the memory must be earlier than the time when the user request is initiated; the sub-model loading time specifically refers to the time of dynamically switching from the sub-model cached in the previous observation window to another sub-model of the same deep neural network that needs to be cached in the current window.

[0083] Step 207) The optimization goal is to maximize the total reasoning accuracy requested by the user. The optimization formula is defined as follows:

[0084]

[0085] Among them, x nh and nu Binary variables representing whether to cache sub-model h to base station n and whether to route user u’s request to base station n, represents the base station set, Represents the sub-model set corresponding to the deep neural network model m requested by user u. h Represents the inference accuracy of sub-model h.

[0086] Step 208) The nonlinear integer programming problem in step 207 is transformed, and the binary integer variable A is nu =x nh ·y nu , then the target formula is transformed into:

[0087]

[0088] Among them, A nu =1 indicates that base station n has cached sub-model h and the request of user u is routed to base station n.

[0089] Step 209) Relax the value range of the variables in the integer linear programming problem in step 208 so that x nh ∈[0, 1], A nuh ∈[0,1].

[0090] Step 210) Under the constraints of steps 202 to 206, the linear programming method is used to solve the target optimization formula in step 208 after the variables in step 209 are relaxed to obtain the optimal fractional solution.

[0091] Step 3) For the optimal fractional solution obtained by the linear programming method in step 2, it is rounded to an integer solution using a random rounding method.

[0092] In this embodiment, step 3 can be implemented by the following specific steps:

[0093] Step 301) Let vector Equal to indicator vector The probability is the fractional solution described in step 2 Right now in, Is the size of The one-hot encoding vector of , where the hth element is 1 and the rest are 0, indicates that the sub-model h of the deep neural network model m is cached on the base station n, that is,

[0094] Step 302) According to the fractional solution described in step 2, let the intermediate variable The rounding probability is Indicates whether the request of user u is attempted to be routed to the sub-model h of base station n for inference to determine the integer solution

[0095] Step 303) Using the integer solution after random rounding To restore the integer solution of the request routing variable Right now Here 1(a) is the indicator vector. Specifically, when expression a holds, otherwise

[0096] Exemplarily, the random rounding process algorithm of this step 3 can be described by the following pseudo code:

[0097]

[0098] Among them, d u 、ddl u and u They represent the data size of the request task of user u, the maximum perceived delay that user u can tolerate, and the initiation time of user u's request; τ represents the current time window, Indicates the status of the cached results of the previous observation window τ-1.

[0099] Step 4), for the rounded integer solutions of model cache and request routing obtained in step 3, a heuristic algorithm is used to obtain the final feasible solutions of model cache and request routing.

[0100] In this embodiment, step 4 can be implemented by the following specific steps:

[0101] Step 401) For each base station, determine the integer solution after random rounding described in step 3. Whether the memory capacity constraint described in step 2 is violated, if so, go to step 404, otherwise go to step 402.

[0102] Step 402) For each user request, determine the integer solution after random rounding described in step 3. and Whether the user's maximum tolerable perceived delay constraint or the model loading constraint described in step 2 is violated, if so, go to step 405, otherwise go to step 403.

[0103] Step 403) For each user request, determine the integer solution after random rounding described in step 3. Is it a violation of the constraint that a user request can be routed to at most one base station as described in step 2? If so, go to step 406. Otherwise, obtain a feasible integer solution x that ultimately satisfies all constraints. nh and nu .

[0104] Step 404) Based on the number of user accesses to the deep neural network model cached by the base station and the inference accuracy p that the corresponding sub-model can provide h (in ) to calculate each deep neural network model The sub-model corresponding to the model type with the lowest benefit on the base station is removed, and a smaller sub-model of the model type is attempted to be cached. If a suitable sub-model cannot be cached, the user request previously routed to the base station is redirected to the cloud. Return to step 401.

[0105] Step 405) redirects the user request to the cloud server and returns to step 402.

[0106] Step 406) Routes the user request to the base station with the highest reasoning accuracy in the cached sub-model of the deep neural network, and returns to step 403.

[0107] Exemplarily, the benefit calculation method of the deep neural network model m in step 404 is as follows: 1) For base station n, if base station n caches a sub-model of model m, then the number of users who requested model m and were successfully routed to the base station n is counted and recorded as margin1; 2) Calculate the accuracy benefit that the sub-model h of model m cached by base station n can bring in the current observation window, that is, margin2 = margin1*p h ; 3) Considering the trade-off between accuracy benefit and base station memory resource usage, the benefit is finally calculated as: margin = margin2 / r h , where r h is the memory size of the sub-model h.

[0108] In some embodiments, the model caching and request routing method based on dynamic deep neural network can describe the overall process through the following pseudo code.

[0109]

[0110] Embodiment 2:

[0111] The embodiment of the present invention discloses a model caching and request routing system based on a dynamic deep neural network, comprising:

[0112] The model partitioning module is used to analyze the output feature maps of each layer of the deep neural network model on the existing data set to calculate the inference accuracy obtained by the classification task, quantify the floating-point calculation amount and memory usage of the branch export of each layer, and divide the deep neural network model into multiple sub-models with different inference accuracy and different sizes;

[0113] A problem construction and solution module is used to obtain the user request sequence of the current observation window and the cache status of the previous observation window, and based on the divided sub-models of different sizes and the deep neural network model requested by the known user, use a linear programming method to solve the solution of the joint optimization problem of model cache and request routing with maximum benefit under the condition of satisfying constraints; the benefit is the total inference accuracy of the user request, and the constraints include: each base station caches a sub-model of each deep neural network model, each user request is routed to at most one base station, the sub-model cached on the base station must be fully loaded into the memory before being used to infer the user request, as well as the memory capacity of the base station and the maximum tolerable perceived delay constraints of the user;

[0114] A random rounding module is used to round the optimal fractional solution obtained by the problem construction and solution module to an integer solution using a random rounding method;

[0115] The feasible solution determination module is used to obtain the final feasible solution of the model cache and request routing by using a heuristic method for the rounded integer solutions of the model cache and request routing.

[0116] In some embodiments, in the problem construction and solution module, the model cache and request routing decisions need to meet the following requirements: each base station caches a sub-model of each deep neural network model, including an empty sub-model; the memory size occupied by the sub-model cached by each base station cannot exceed the total memory capacity of the base station; each user request is routed to at most one base station, and user requests that cannot be routed are sent to the cloud server; the delay from the initiation of the user request to the receipt of the inference result of the sub-model cached on the routed base station cannot exceed the maximum perceived delay that the user can tolerate; and the sub-model cached on the base station must be fully loaded into the memory before being used to infer the user request; the optimization objectives are defined as follows:

[0117]

[0118] In some embodiments, in the problem construction and solution module, the nonlinear integer programming problem is transformed, and the binary integer variable A is nu =x nh ·y nu , the optimization objective is transformed into:

[0119]

[0120] Relax the range of variables in the transformed integer linear programming problem so that x nh ∈[0, 1], A nuh ∈[0, 1]; use the linear programming method to solve the transformed target optimization formula after variable relaxation and obtain the optimal fractional solution

[0121] In some embodiments, in the feasible solution determination module, for each base station, the integer solution after random rounding is determined. Whether the memory capacity constraint is violated, if so:

[0122] The benefit of each deep neural network model is calculated based on the number of user accesses to the deep neural network model cached by the base station and the inference accuracy that the corresponding sub-model can provide. The sub-model corresponding to the model type with the lowest benefit on the base station is removed, and an attempt is made to cache a smaller sub-model of the model type. If a suitable sub-model cannot be cached, the user request previously routed to the base station will be redirected to the cloud.

[0123] otherwise:

[0124] For each user request, determine the integer solution after random rounding and Whether it violates the user's maximum tolerable perceived delay constraint or model loading constraint, if so, redirect the user request to the cloud server, otherwise for each user request, determine the integer solution after random rounding Does it violate the constraint that a user request can only be routed to one base station at most? If so, route the user request to the base station with the highest inference accuracy in the cached deep neural network sub-model. Otherwise, obtain a feasible integer solution x that finally satisfies all constraints. nh and nu .

[0125] Embodiment 3:

[0126] A computer program product disclosed in an embodiment of the present invention includes a computer program / instruction, which, when executed by a processor, implements the steps of the model caching and request routing method based on a dynamic deep neural network described in Example 1.

[0127] The program / instruction code for implementing the method of the present invention can be written in any combination of one or more programming languages. These programs / instruction codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program / instruction code, when executed by the processor or controller, enables the steps of the method of the present invention to be implemented. The program / instruction code can be executed completely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine or completely on a remote machine or server. The parts not described in detail in the present invention are all known technologies of those skilled in the art.

Claims

1. A model caching and request routing method based on dynamic deep neural network, characterized in that: The steps include: Step 1: On the existing data set, the output feature maps of each layer of the deep neural network model are analyzed to calculate the reasoning accuracy obtained by the classification task, the floating-point calculation amount and memory usage of the branch export of each layer are quantified, and the deep neural network model is divided into multiple sub-models with different reasoning accuracy and different sizes; Step 2: Obtain the user request sequence of the current observation window and the cache status of the previous observation window. Based on the sub-models of different sizes divided in step 1 and the known deep neural network model requested by the user, use the linear programming method to solve the joint optimization problem of model cache and request routing with maximum benefit under the condition of satisfying constraints. The benefit is the total inference accuracy of user requests, and the constraints include: each base station caches one sub-model of each deep neural network model, each user request is routed to at most one base station, the sub-model cached on the base station must be fully loaded into the memory before being used to infer the user request, as well as the memory capacity of the base station and the maximum tolerable perceived delay constraint of the user; Step 3, for the optimal fractional solution obtained in step 2, round it to an integer solution using a random rounding method; Step 4: For the rounded integer solutions of model cache and request routing obtained in step 3, a heuristic method is used to obtain the final feasible solutions of model cache and request routing.

2. The model caching and request routing method based on dynamic deep neural network according to claim 1, characterized in that: The step 1 specifically includes: Step 101, analyzing the internal parameter structure of each layer of the deep neural network model, the parameters include the size of the convolution kernel, the number of channels and the number of neurons in the fully connected layer, and calculating the floating point calculation amount and memory usage of each layer; Step 102, dividing the deep neural network model into three parts: an entry network, a hidden layer network, and an exit network according to the network structure of the deep neural network model; Step 103, further dividing the hidden layer network in step 102, dividing the deep neural network model into a plurality of sub-models of different sizes, and training an export network for each sub-model separately; Step 104, perform the inference path calculation from the entry network to each sub-model exit network in the real device, and record the time of loading the sub-model into the memory, the inference accuracy of each exit and the memory usage of the path.

3. The model caching and request routing method based on dynamic deep neural network according to claim 1, characterized in that: The step 2 specifically includes: Step 201: M deep neural network models Divide into Sub-models, in Represents the deep neural network model m i The corresponding empty submodel, H(m i ) represents the deep neural network model m i The number of sub-models; deep neural network model m i The sub-model of has an inference accuracy of Divide a period of time into multiple consecutive time periods, each of which constitutes an observation window. Before each observation window begins, obtain the request sequence of the user in the current observation window. and the state of the sub-model cached by each base station in the previous observation window, where U represents the number of requests; Step 202, model caching and request routing decisions need to satisfy that each base station caches a sub-model of each deep neural network model, including an empty sub-model; Step 203: Model caching and request routing decisions need to satisfy the requirement that the memory size occupied by the sub-model cached by each base station cannot exceed the total memory capacity of the base station; Step 204: Model caching and request routing decisions need to satisfy the requirement that each user request is routed to at most one base station, and user requests that cannot be routed are sent to the cloud server; Step 205: Model caching and request routing decisions need to satisfy the requirement that the delay from the initiation of a user request to the receipt of the inference result of the sub-model cached on the routing base station cannot exceed the maximum perceived delay that the user can tolerate; the specific calculation method of the model inference delay is the floating-point calculation amount of the model divided by the maximum computing power that the base station can provide; Step 206, since the deep neural network model must be cached before executing the inference service, the model cache and request routing decision need to meet the requirement that the sub-model cached on the base station must be fully loaded into the memory before being used to infer the user request, that is, the completion time of loading the sub-model into the memory must be earlier than the time when the user request is initiated; the sub-model loading time specifically refers to the time of dynamically switching from the sub-model cached in the previous observation window to another sub-model of the same deep neural network that needs to be cached in the current window; Step 207, the optimization goal is to maximize the total reasoning accuracy requested by the user, which is defined as follows: Among them, x nh and nu Binary variables representing whether to cache sub-model h to base station n and whether to route user u’s request to base station n, represents the base station set, represents the sub-model set corresponding to the deep neural network model m requested by user u, p h represents the inference accuracy of sub-model h; Step 208, transform the nonlinear integer programming problem in step 207, and let the binary integer variable A nuh =x nh ·y nu , then the target formula is transformed into: Among them, A nuh =1 means that base station n has cached submodel h and the request of user u is routed to base station n; Step 209, relax the value range of the variables in the integer linear programming problem in step 208 so that x nh ∈[0, 1], A nuh ∈[0, 1]; Step 210, under the constraints of steps 202 to 206, use the linear programming method to solve the target optimization formula in step 208 after the variables in step 209 are relaxed to obtain the optimal fractional solution 4. The model caching and request routing method based on dynamic deep neural network according to claim 3, characterized in that: The step 3 specifically includes: Step 301, let vector Equal to indicator vector The probability of the fractional solution is Right now in, Is the size of The one-hot encoding vector of , where the hth element is 1 and the rest are 0, indicates that the sub-model h of the deep neural network model m is cached on the base station n, that is, Step 302, let the intermediate variable The rounding probability is Indicates whether the request of user u is attempted to be routed to the sub-model h of base station n for inference to determine the integer solution Step 303, let The integer solution obtained by random rounding To restore the integer solution of the request routing variable Right now Here 1(a) is the indicator vector. Specifically, when expression a holds, otherwise 5. The model caching and request routing method based on dynamic deep neural network according to claim 1, characterized in that: The step 4 specifically includes: Step 401: for each base station, determine the integer solution after random rounding. Whether the memory capacity constraint is violated, if yes, go to step 404, otherwise go to step 402; A binary variable indicating whether to cache the sub-model h to the base station n; Step 402: for each user request, determine the integer solution after random rounding. and Whether the user's maximum tolerable perceived delay constraint or the model loading constraint is violated, if yes, go to step 405, otherwise go to step 403; wherein Indicates whether to route the request of user u to base station n; Step 403: for each user request, determine the integer solution after random rounding. Is it a violation of the constraint that a user request can be routed to at most one base station? If so, go to step 406. Otherwise, obtain a feasible integer solution x that ultimately satisfies all constraints. nh and nu ; Step 404: Calculate the benefit of each deep neural network model based on the number of user accesses to the deep neural network model cached by the base station and the inference accuracy that the corresponding sub-model can provide, remove the sub-model corresponding to the model type with the lowest benefit on the base station, and try to cache a smaller sub-model of the model type; if a suitable sub-model cannot be cached, the user request previously routed to the base station will be redirected to the cloud, and return to step 401; Step 405, redirect the user request to the cloud server, and return to step 402; Step 406 , routing the user request to the base station with the highest reasoning accuracy in the cached sub-model of the deep neural network, and returning to step 403 .

6. A model caching and request routing system based on dynamic deep neural network, characterized in that: include: The model partitioning module is used to analyze the output feature maps of each layer of the deep neural network model on the existing data set to calculate the inference accuracy obtained by the classification task, quantify the floating-point calculation amount and memory usage of the branch export of each layer, and divide the deep neural network model into multiple sub-models with different inference accuracy and different sizes; The problem construction and solution module is used to obtain the user request sequence of the current observation window and the cache status of the previous observation window. Based on the divided sub-models of different sizes and the known deep neural network model requested by the user, the linear programming method is used to solve the solution of the joint optimization problem of model cache and request routing with maximum benefit under the condition of satisfying constraints; The benefit is the total inference accuracy of user requests, and the constraints include: each base station caches one sub-model of each deep neural network model, each user request is routed to at most one base station, the sub-model cached on the base station must be fully loaded into the memory before being used to infer the user request, as well as the memory capacity of the base station and the maximum tolerable perceived delay constraint of the user; A random rounding module is used to round the optimal fractional solution obtained by the problem construction and solution module to an integer solution using a random rounding method; The feasible solution determination module is used to obtain the final feasible solution of the model cache and request routing by using a heuristic method for the rounded integer solutions of the model cache and request routing.

7. The model caching and request routing system based on dynamic deep neural network according to claim 6, characterized in that: In the problem building and solving module, the model cache and request routing decision need to meet the following requirements: each base station caches a sub-model of each deep neural network model, including an empty sub-model; the memory size occupied by the sub-model cached by each base station cannot exceed the total memory capacity of the base station; Each user request is routed to at most one base station, and user requests that cannot be routed are sent to the cloud server; The delay from the user request being initiated to the inference result of the sub-model cached on the routing base station being received cannot exceed the maximum perceived delay that the user can tolerate; and the sub-model cached on the base station must be fully loaded into the memory before being used to infer the user request; the optimization objectives are defined as follows: Among them, x nh and nu Binary variables representing whether to cache sub-model h to base station n and whether to route user u’s request to base station n, Represents the user's request sequence, represents the base station set, represents the sub-model set corresponding to the deep neural network model m requested by user u, p h Represents the inference accuracy of sub-model h.

8. The model caching and request routing system based on dynamic deep neural network according to claim 7, characterized in that: In the problem construction and solution module, the nonlinear integer programming problem is transformed into a binary integer variable A. nuh =x nh ·y nu , the optimization objective is transformed into: Among them, A nuh =1 means that base station n has cached submodel h and the request of user u is routed to base station n; Relax the range of variables in the transformed integer linear programming problem so that x nh ∈[0,1],A nuh ∈[0,1]; use the linear programming method to solve the transformed target optimization formula after variable relaxation and obtain the optimal fractional solution 9. The model caching and request routing system based on dynamic deep neural network according to claim 6, characterized in that: In the feasible solution determination module, for each base station, determine the integer solution after random rounding Whether the memory capacity constraint is violated, where A binary variable indicating whether to cache submodel h to base station n. If yes, then: Calculate the benefit of each deep neural network model based on the number of user visits to the deep neural network model cached by the base station and the inference accuracy that the corresponding sub-model can provide, remove the sub-model corresponding to the model type with the lowest benefit on the base station, and try to cache a smaller sub-model of the model type; If a suitable sub-model cannot be cached, the user request previously routed to the base station will be redirected to the cloud. otherwise: For each user request, determine the integer solution after random rounding and Whether the user's maximum tolerable perceived delay constraint or model loading constraint is violated, where Indicates whether to route the request of user u to base station n; if yes, redirect the user request to the cloud server, otherwise for each user request, determine the integer solution after random rounding Does it violate the constraint that a user request can only be routed to one base station at most? If so, route the user request to the base station with the highest inference accuracy in the cached deep neural network sub-model. Otherwise, obtain a feasible integer solution x that finally satisfies all constraints. nh and nu .

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the model caching and request routing method based on a dynamic deep neural network are implemented according to any one of claims 1 to 5.

Images