A task offloading method and device based on double-layer structure model

By adopting a task unloading method of the two-layer structure model in edge computing, the inner structure uses DNN to make real-time decisions and the outer structure adjusts hyperparameters, solving the problems of increased task unloading costs and insufficient environmental adaptability in the existing technology, realizing the problem of Efficient and low-cost task offload decisions.

CN114237650BActive Publication Date: 2024-12-03XINQING (SHANGHAI) INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111582792.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-22
Publication Date
2024-12-03
Estimated Expiration
2041-12-22

AI Technical Summary

Technical Problem

In the process of task unloading in edge computing, the operating cost of the algorithm increases as the amount of data increases, and the model has to be retrained in the face of changes in the MEC environment, resulting in waste of resources and cost increases.

Method used

Using a task unloading method based on the two-layer structure model, the inner structure completes forward propagation and back propagation through DNN, and combines the Memory mechanism to achieve real-time and high-precision decision-making. The outer structure adjusts hyperparameters through the group optimization algorithm, so that the model can adapt quickly. New environment.

Benefits of technology

In the ever-changing MEC environment, the deep reinforcement learning model's real-time high-precision decision-making on task unloading problems is realized, reducing resource waste and cost, and improving the adaptability and efficiency of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114237650B_ABST
    Figure CN114237650B_ABST
Patent Text Reader

Abstract

The present invention discloses a task unloading method and device based on a double-layer structure model, the method comprising: step S1, constructing and initializing an inner structure model based on a DNN, initializing a current unloading situation matrix, and constructing a total task matrix; step S2, using a swarm optimization algorithm, adjusting model hyperparameters for the inner structure model according to environmental changes and task conditions, training the inner structure models with different hyperparameter combinations, so that the inner structure model can quickly adapt to the new environment, thereby obtaining the inner structure model with the best effect; step S3, the inner structure model accepts the hyperparameters provided by step S2, completes the forward propagation process of the model through a DNN model, analyzes the unloading result of the task, and trains the DNN model according to a memory mechanism and back-propagation feedback, thereby realizing real-time and high-precision decision-making on the task unloading problem using a deep reinforcement learning model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computing offloading technology in edge computing, and in particular to a task offloading method and device based on a double-layer structure model. Background Art

[0002] Computational intensive tasks need to be transferred from IoT devices to cloud servers for execution. However, the task offloading process usually involves a large amount of data transmission, which will lead to high latency in IoT applications. Task offloading, as a distributed computing paradigm, migrates complex tasks from IoT devices to edge cloud servers, which can provide computing services for edge caching, edge training, and edge inference. In the process of task offloading, it is usually affected by various factors in different fields. It needs to dynamically decide whether to offload tasks to edge servers or execute them locally. If a large number of tasks are offloaded to edge servers, it will occupy bandwidth and greatly increase transmission latency. Therefore, an optimal decision-making plan is needed to reasonably assign each task to the server while minimizing the overall cost.

[0003] At present, the research on edge computing task offloading mainly uses intelligent optimization algorithms, such as genetic algorithm, particle swarm algorithm, ant colony algorithm, etc. There are also some methods that apply game theory methods to resource allocation problems in the environment.

[0004] However, although the algorithms in the existing technology can ensure the offloading of tasks, as the amount of data increases, the operating cost of the algorithms becomes increasingly higher. At the same time, in the face of changes in the MEC environment, the models have to be retrained, which leads to waste of resources and increased costs. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a task offloading method and device based on a double-layer structure model. By adopting an inner and outer double-layer structure, the inner structure is used to accept the hyperparameters provided by the external structure, and the forward propagation process of the model is completed through DNN. The task offloading result is analyzed, and then the DNN is fed back according to the Memory mechanism and back propagation, so that the deep reinforcement learning model can make real-time and high-precision decisions on the task offloading problem. At the same time, the outer structure model is used to adjust the model hyperparameters for the deep reinforcement learning model in the constantly changing MEC environment, so that the internal structure model can quickly adapt to the new environment.

[0006] To achieve the above and other purposes, the present invention proposes a task offloading method based on a two-layer structure model, comprising the following steps:

[0007] Step S1, construct and initialize the inner structure model based on DNN, initialize the current unloading situation matrix, and construct the total task matrix;

[0008] Step S2, using a swarm optimization algorithm to adjust model hyperparameters for the internal structure model according to environmental changes and task conditions, and training the internal structure model with different hyperparameter combinations so that the internal structure model can quickly adapt to the new environment, thereby obtaining the internal structure model with the best effect;

[0009] In step S3, the internal structure model accepts the hyperparameters provided in step S2, completes the forward propagation process of the model through the DNN model, analyzes the task offloading result, and trains the DNN model according to the Memory mechanism and back-propagation feedback, thereby realizing real-time and high-precision decision-making on the task offloading problem using the deep reinforcement learning model.

[0010] Preferably, the internal structure model includes:

[0011] The DNN model inputs the current offloading status and the tasks to be offloaded, and outputs the Q value corresponding to the tasks offloaded to each machine;

[0012] The ε-greedy strategy module uses a three-stage ε-greedy strategy to select the device to be unloaded based on the input Q value;

[0013] The replay memory module is used to store the input of the DNN model and the ε-greedy strategy to select the device to be unloaded into the experience replay;

[0014] The sampling training module takes out the data in the experience playback after a certain number of cycles to train the DNN model.

[0015] Preferably, step S2 further comprises:

[0016] Step S200, randomly initializing the population matrix X according to the boundary range, and calculating the fitness matrix;

[0017] Step S201, after sorting the initial fitness matrix, record the current optimal position;

[0018] Step S202, iterating the group optimization algorithm, firstly updating the position of the alerter, then communicating between the individuals in the population to update the position, and finally updating the position of the individuals in the population to explore and update the position, so as to obtain the best population information;

[0019] Step S203, after the iteration is completed, the best hyperparameter combination is output and the hyperparameter combination is substituted into the internal structure model.

[0020] Preferably, in step S200, the method for calculating the fitness matrix is ​​as follows: a row of hyperparameter data in the X matrix is ​​substituted into the inner structure model, and after training, the same set of test data is used to test the effect, and the evaluation standard is the total cost value of this set of data.

[0021] Preferably, in step S202, the location update for the alerter is performed as follows:

[0022] The warning value R2 is randomly generated. If the warning value is small, it means that there are no natural enemies and the population will conduct extensive searches. If the warning value is too large, it means that natural enemies have appeared, threatening the safety of the population.

[0023] And after completing a position update, calculate the new fitness, update the current optimal fitness value and the current optimal fitness value position.

[0024] Preferably, in step S202, the individuals in the population communicate with each other to update their positions as follows:

[0025] Individuals in the population randomly exchange information and go to the best known hunting locations to obtain more food;

[0026] After multiple exchanges, the position update is completed, the new fitness is calculated, and the current optimal fitness value is updated.

[0027] Preferably, in step S202, the population individual exploration performs position update as follows:

[0028] Each individual in the population explores randomly in its own way, either linearly or in spirals;

[0029] After updating the position, calculate the new fitness, update the global optimal fitness, and the global optimal fitness value position.

[0030] Preferably, step S3 further comprises:

[0031] Step S301, select forward propagation, traverse the total task matrix, take each row of task volume data combined with the current unloading situation as input, obtain the output matrix, and calculate the Q value corresponding to each task unloaded to each server, and obtain the server to which each task is unloaded according to the three-stage ε-greedy strategy to obtain the result matrix;

[0032] Step S302, storing the input matrix and the result matrix in the input matrix storage unit and the result matrix storage unit respectively, and updating the current unloading status matrix.

[0033] Step S303, training the DNN model according to the input matrix and the result matrix stored in the input matrix storage unit and the result matrix storage unit, calculating the loss function, and performing back propagation.

[0034] Preferably, step S301 further includes:

[0035] Step S101a, taking out one row of the total task matrix and transposing it to obtain a task matrix, combining the obtained task matrix and the current unloading situation matrix to form an input matrix, and using it as an input of the DNN model to obtain an output matrix of the DNN model;

[0036] Step S301b, calculating the Q value corresponding to each task offloaded to each server;

[0037] Step S301c: According to the calculation result of step S301b, a three-stage ε-greedy strategy is used to select the device to be uninstalled, and a result matrix is ​​obtained.

[0038] To achieve the above object, the present invention further provides a task offloading device based on a double-layer structure model, comprising:

[0039] The inner structure model building unit is used to build and initialize the inner structure model based on DNN, initialize the current unloading situation matrix, and build the total task matrix;

[0040] A hyperparameter adjustment unit is used to use a swarm optimization algorithm to adjust the model hyperparameters for the internal structure model according to environmental changes and task conditions, and to train the internal structure model with different hyperparameter combinations so that the internal structure model can quickly adapt to the new environment, thereby obtaining the best internal structure model;

[0041] The internal structure model training unit enables the internal structure model to accept the hyperparameters provided by the external structure, completes the forward propagation process of the model through the DNN model, analyzes the task offloading results, and trains the DNN model according to the Memory mechanism and back-propagation feedback, thereby realizing real-time and high-precision decision-making on task offloading problems using a deep reinforcement learning model.

[0042] Compared with the prior art, the task offloading method and device based on the double-layer structure model of the present invention utilizes the inner structure to accept the hyperparameters provided by the outer structure, completes the forward propagation process of the model through DNN, analyzes the task offloading result, and then feeds back to DNN according to the Memory mechanism and back propagation, thereby realizing the deep reinforcement learning model to make real-time and high-precision decisions on the task offloading problem. At the same time, the outer structure model is utilized to adjust the model hyperparameters for the deep reinforcement learning model in the constantly changing MEC environment, so that the internal structure model can quickly adapt to the new environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of the steps of a task offloading method based on a double-layer structure model of the present invention;

[0044] Figure 2 It is a schematic diagram of the inner layer structure model in a specific embodiment of the present invention;

[0045] Figure 3 This is a system architecture diagram of a task offloading device based on a double-layer structure model of the present invention. DETAILED DESCRIPTION

[0046] The following describes the implementation of the present invention through specific examples and in conjunction with the accompanying drawings. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific examples, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0047] Figure 1 FIG. 1 is a flowchart of a task offloading method based on a double-layer structure model of the present invention. Figure 1 As shown, the present invention provides a task offloading method based on a double-layer structure model, comprising the following steps:

[0048] Step S1, construct and initialize the inner structure model based on DNN, initialize the current unloading situation matrix, and construct the total task matrix.

[0049] In the present invention, first construct Figure 2 The inner structure model shown includes the following parts according to the workflow:

[0050] 1. DNN model: input the current offloading situation and the tasks to be offloaded, and output the Q value corresponding to the task offloaded to each machine;

[0051] 2. ε-greedy policy module: According to the input Q value, a three-stage ε-greedy policy is used to select the device to be unloaded;

[0052] 3. Replay Memory: used to store the DNN input and the devices to be unloaded selected by the ε-greedy strategy into the experience replay pool. In the present invention, two arrays are set to store the DNN input and the selected devices respectively.

[0053] 3. Sample training module: After a certain number of cycles, take out the data in the experience playback and train the DNN. Figure 2 As shown, that is, when the cycle is repeated a certain number of times, such as a multiple of 10, feedback training of the DNN model is performed.

[0054] In a specific embodiment of the present invention, it is assumed that in the edge computing model, the total number of servers is D+E, where the number of devices is D and the number of edge servers is E. The input of the inner structure model is a matrix of size (D+E)×2, the first column is the task volume, and the second column is the current task unloading status (that is, the task cost incurred by each server). The DNN model has two hidden layers, and the nodes in each layer are 120 and 80 respectively. The input is a matrix of size (D+E)×2. After two hidden layers, the output is a matrix of size (D+E)×(D+E).

[0055] After constructing the above-mentioned DNN-based inner structure model, the current unloading situation matrix is ​​initialized, and each time a task unloading decision is made, the current unloading situation matrix is ​​updated. The current unloading situation matrix initialization is to create a matrix of all zeros. If there are two machines in total, it is [0 0]. Its initialization process is part of the initialization process of the inner structure model; the total task matrix is ​​constructed. The total task matrix is ​​the matrix of tasks to be processed, and the matrix is ​​composed of the task queues to be processed.

[0056] In the present invention, the external structure is designed to cope with the ever-changing MEC environment (changes in server computing performance, bandwidth, etc.). Therefore, based on the swarm optimization algorithm, the present invention provides initial hyperparameters for the internal structure according to environmental changes and task conditions, so that the overall method can adapt to the environment more quickly. In a specific embodiment of the present invention, the three hyperparameters batch_size, learning_rate, and traning_interval in the deep reinforcement learning model (i.e., the internal structure model) need to be searched through the outer structure model.

[0057] Step S2, using the outer structure model (i.e., the swarm optimization algorithm), adjusts the model hyperparameters for the internal structure model according to environmental changes and task conditions, and trains the internal structure models with different hyperparameter combinations so that the internal structure model can quickly adapt to the new environment, thereby obtaining the best internal structure model.

[0058] In the present invention, the external structure is designed to cope with the ever-changing MEC environment (changes in server computing performance, bandwidth, etc.). Therefore, based on the swarm optimization algorithm, the present invention adjusts the hyperparameters for the internal structure model according to environmental changes and task conditions, so that the overall method can adapt to the environment more quickly. In a specific embodiment of the present invention, the three hyperparameters batch_size, learning_rate, and traning_interval in the deep reinforcement learning model (i.e., the internal structure model) need to be searched through the external structure model.

[0059] Specifically, step S2 further includes:

[0060] Step S200, randomly initialize the population matrix X according to the boundary range and calculate the fitness matrix Fitness, where the fitness represents the cumulative cost of each hyperparameter combination model after processing the same task. Initialize the population matrix X as follows:

[0061]

[0062] Among them, n is the number of populations, and d is the dimension, that is, the number of hyperparameters.

[0063] The method for calculating the fitness matrix Fitness is as follows: substitute a row of hyperparameter data in the X matrix into the inner structure model. After training, use the same set of test data to test the effect. The evaluation standard is the total cost value of this set of data.

[0064] Step S201: After sorting the initial fitness matrix, record the current optimal position X best , start the following iteration:

[0065] Step 201a, the first stage is to update the position of the sentinels, which account for 20% of the population. The warning value R2 is randomly generated. If the warning value is small, it means that there are no natural enemies and the population will conduct an extensive search; if the warning value is too large, it means that natural enemies have appeared, threatening the safety of the population.

[0066] The Sentinel positions have been updated as follows:

[0067]

[0068] in, represents the population position after update (after t+1), i, j represents a specific position of the population matrix X, t represents the number of iterations, itermax represents the maximum number of iterations, x i,j represents the position of individuals in the population, a∈0,1] is a random number, Q represents a random number that follows a normal distribution, L is a 1*d matrix with all elements being 1, ST represents the safety threshold, (1-0.2*K) is the second-order oscillation factor, and K is a random sampling value of [-1,1) to prevent the algorithm from falling into a local optimal solution during the search process. Of course, the above formula can be considered as both a position update for the sentinel and a position update for the population, because the sentinel is the first 20% of the individuals in the entire population, and because the matrix has been sorted in descending order, that is, individuals from the first row to the 20%*nth row are updated according to this formula.

[0069] After completing a position update, calculate the new fitness, update the current optimal fitness value and the current optimal fitness value position.

[0070] Step S201b, the second stage is the communication between individuals in the population to update the position, that is, the individuals in the population randomly exchange information and go to the best hunting position known to obtain more food. After multiple exchanges, the position update is completed, the new fitness is calculated, and the current optimal fitness value is updated.

[0071] In a specific embodiment of the present invention, the population individual position update formula in this step is as follows:

[0072]

[0073] Among them, X m,j and X n,j are two randomly selected individuals, r1, r2 are rand(0,1), f(X m,j ) and f(X n,j ) represents the fitness value of the individual, and the best hunting position X best It is the current optimal fitness value position obtained in the previous stage, i.e. step S201a.

[0074] Step S201c, the third stage is to explore and update the position of the population individuals to obtain the best population information, that is, each population individual randomly explores in its own way, or performs linear exploration, or spiral exploration; after updating the position, calculate the new fitness, update the global optimal fitness, and the global optimal fitness value position. The position update in this step is as follows:

[0075]

[0076] Among them, β represents the variable step size, zα represents rand(0,1),

[0077] When the individual population position exceeds the hyperparameter limit, it is automatically processed into the boundary value.

[0078] Step S202, after the iteration is completed, the best hyperparameter combination is output and the hyperparameter combination is substituted into the internal structure model.

[0079] That is to say, after the traversal iteration of step S201a-step S201c is completed, the best population information is output, and the corresponding data is the hyperparameter R2 combination with the best operating effect of the internal structure model of the present invention, and the best hyperparameter combination is substituted into the internal structure model.

[0080] In step S3, the internal structure model accepts the hyperparameters provided by the external structure, completes the forward propagation process of the model through the DNN model, analyzes the task offloading result, and trains the DNN model according to the Memory mechanism and back propagation feedback, thereby realizing real-time and high-precision decision-making on the task offloading problem using the deep reinforcement learning model.

[0081] Specifically, step S3 further includes

[0082] Step S301, select forward propagation (i.e., forward propagation), traverse the total task matrix, take each row of task volume data combined with the current unloading situation as input, obtain the output matrix, calculate the Q value corresponding to each task unloaded to each server, and obtain the server to which each task is unloaded according to the three-stage ε-greedy strategy to obtain the result matrix.

[0083] Specifically, step S301 further includes:

[0084] Step S301a, taking out one row of the total task matrix and transposing it to obtain a task matrix, and combining the obtained task matrix and the current unloading situation matrix to form an input matrix, as shown in Table 1 below:

[0085] Table 1 Input matrix of internal structure model

[0086] <![CDATA[Task X1]]> <![CDATA[Server S1]]> … … <![CDATA[Task X n > <![CDATA[Server S n >

[0087] The first column represents the task volume, and the second column represents the current offloading matrix, that is, the task cost generated by each server. The server here refers to all edge servers and local servers. The task offloading in this method not only includes selecting local (device) execution or edge execution, but also includes selecting a specific server in the local or edge end (these servers have different performances). In a specific embodiment of the present invention, the input matrix size is (D+E)×2, X n and S n It should be X D+E ,S D+E

[0088] The above input matrix is ​​used as the input of the DNN model of the inner structure model to obtain the DNN output matrix, as shown in Table 2 below:

[0089] Table 2 DNN output matrix

[0090]

[0091] Among them, p ij represents the Q value of action a(action) of task i offloading to server j, A 1,。。。 a T It is understood from the perspective of reinforcement learning, which means that in the current state, you can choose a 1,。。。 a T An action in (unload to one of the current D+E servers)

[0092] Step S301b, calculate the Q value corresponding to each task offloaded to each server

[0093] In the present invention, it is assumed that the task volume of each task is x, and the server energy consumption cost E = {e1, e2, e3, …, e D+E}, and the server operation cost P = {p1, p2, p3, …, p D+E}, the server CPU operation rate The uplink speed VU, the downlink speed VD, and the calculation formula of the cost value Q are as follows:

[0094]

[0095] Among them, e up , e down , represent the network uplink speed and downlink speed, w1, w2 represent the weights in the cost evaluation, w1, w2 ∈ (0, 1) and w1 + w2 ≤ 1, which can be adjusted by oneself, local mode means offloading to the device side, edge_mode means offloading to the edge side, and the cost calculations of the two are different because there is also a transmission process at the edge side.

[0096] Step 301c, according to the calculation result of step S301b, adopt a three-segment ε-greedy strategy to select the device to be offloaded, and obtain a result matrix.

[0097] Specifically, generate a random number random. If random > ε, select the server with the smallest DNN output value (that is, the output matrix in Table 2, select the smallest output value in one row of the output matrix). If ε / 2 < random < ε, randomly sample a server for offloading. If neither of the above two situations is met, select the server with the smallest Q value for offloading. Finally, obtain a result matrix, as shown in Table 3:

[0098] Table 3 Result matrix of the internal structure model

[0099] <![CDATA[Uninstallation action A1 of task X1]]> …… <![CDATA[Task X n uninstallation action A n >

[0100] It should be noted that in general reinforcement learning, the goal is to pursue the action that can generate the largest Q value. In the model of the present invention, taking the cost as the Q value, the pursued is the action that generates the smallest Q value.

[0101] Step S302, store the input matrix and the result matrix into the input matrix storage unit Memory_input and the result matrix storage unit Memory_result respectively, and update the current offloading situation matrix.

[0102] Specifically, assume that the current offloading situation matrix is [a, b, c, d, …] T,According to the screening, the current task should be offloaded to the server with index 1, the cost generated is q, and the updated current offloading matrix is ​​[a,b+q,c,d,…] T .

[0103] Step S303, training the DNN model according to the input matrix and result matrix stored in Memory_input and Memory_result, calculating the loss function, and performing back propagation.

[0104] Specifically, assuming training_interval = 20, that is, after storing 20 input matrices and result matrices, the DNN model will be trained once. At the same time, assuming batch_size = 16, if the current number of loops is exactly a multiple of 20, 16 groups of data are randomly sampled from the memory for traversal.

[0105] Each set of data is trained with learning_rate: 0.06, weight_decay: 0.0001, BCELoss as the loss function and Adam as the optimizer.

[0106] After the loss function is calculated, the back propagation method is executed, the loop ends, and the current unloading situation is output.

[0107] It can be seen that the present invention first initializes the internal structure model and then initializes the external structure. The external structure model algorithm (i.e., the swarm optimization algorithm) starts to execute, and the internal structure models with different hyperparameter combinations are trained to finally obtain the internal structure model with the best effect. During this training process, the process of calculating the fitness is also the algorithm execution process of the entire internal structure model. Therefore, the process of the internal structure model executing the task can also be regarded as a part of the external structure model algorithm. Finally, the external model outputs the best hyperparameters. At the same time, the internal structure model has also been trained and can be directly used for unloading after being saved.

[0108] Figure 3 FIG. 1 is a system architecture diagram of a task offloading device based on a double-layer structure model of the present invention. Figure 3 As shown, the present invention provides a task offloading device based on a double-layer structure model, comprising:

[0109] The inner structure model building unit 301 is used to build and initialize the inner structure model based on DNN, initialize the current unloading situation matrix, and build the total task matrix.

[0110] In the present invention, the inner layer structure model building unit 301 builds Figure 2 The inner structure model shown in the figure includes the following parts according to the workflow:

[0111] 1. DNN model: input the current offloading situation and the tasks to be offloaded, and output the Q value corresponding to the task offloaded to each machine;

[0112] 2. ε-greedy policy: According to the input Q value, a three-stage ε-greedy policy is used to select the device to be unloaded;

[0113] 3. Replay Memory: used to store DNN input and ε-greedy strategy to select the device to be unloaded into experience replay;

[0114] 3. Sample: After a certain number of cycles, take out the data in the experience playback and train the DNN.

[0115] In a specific embodiment of the present invention, it is assumed that in the edge computing model, the total number of servers is D+E, where the number of devices is D and the number of edge servers is E. The input of the inner structure model is a matrix of size (D+E)×2, the first column is the task volume, and the second column is the current task unloading status (that is, the task cost incurred by each server). The DNN model has two hidden layers, and the nodes in each layer are 120 and 80 respectively. The input is a matrix of size (D+E)×2. After two hidden layers, the output is a matrix of size (D+E)×(D+E).

[0116] After constructing the above-mentioned DNN-based inner structure model, the current unloading situation matrix is ​​initialized, and each time a task unloading decision is made, the current unloading situation matrix is ​​updated. The current unloading situation matrix initialization is to create a matrix of all zeros. If there are two machines in total, it is [0 0]. Its initialization process is part of the initialization process of the inner structure model; the total task matrix is ​​constructed. The total task matrix is ​​the matrix of tasks to be processed, and the matrix is ​​composed of the task queues to be processed.

[0117] In the present invention, the external structure is designed to cope with the ever-changing MEC environment (changes in server computing performance, bandwidth, etc.). Therefore, based on the swarm optimization algorithm, the present invention provides initial hyperparameters for the internal structure according to environmental changes and task conditions, so that the overall method can adapt to the environment more quickly. In a specific embodiment of the present invention, the three hyperparameters batch_size, learning_rate, and traning_interval in the deep reinforcement learning model (i.e., the internal structure model) need to be searched through the outer structure model.

[0118] The hyperparameter adjustment unit 302 is used to use the outer structure model (i.e., the swarm optimization algorithm) to adjust the model hyperparameters for the internal structure model according to environmental changes and task conditions, and to train the internal structure models with different hyperparameter combinations so that the internal structure model can quickly adapt to the new environment, thereby obtaining the best internal structure model.

[0119] In the present invention, the external structure is designed to cope with the ever-changing MEC environment (changes in server computing performance, bandwidth, etc.). Therefore, based on the swarm optimization algorithm, the present invention adjusts the hyperparameters for the internal structure model according to environmental changes and task conditions, so that the overall method can adapt to the environment more quickly. In a specific embodiment of the present invention, the three hyperparameters batch_size, learning_rate, and traning_interval in the deep reinforcement learning model (i.e., the internal structure model) need to be searched through the external structure model.

[0120] Specifically, the hyperparameter adjustment unit 302 further includes:

[0121] The population matrix initialization module is used to randomly initialize the population matrix X (n is the number of populations, d is the dimension, i.e. the number of hyperparameters) according to the boundary range and calculate the fitness matrix Fitness. The fitness represents the cumulative cost of the model of each hyperparameter combination after processing the same task. The population matrix X is initialized as follows:

[0122]

[0123] Among them, n is the number of populations, and d is the dimension, that is, the number of hyperparameters.

[0124] The method for calculating the fitness matrix Fitness is as follows: substitute a row of hyperparameter data in the X matrix into the inner structure model. After training, use the same set of test data to test the effect. The evaluation standard is the total cost value of this set of data.

[0125] Iterative processing module, after sorting the initial fitness matrix, records the current optimal position X best , start the following iteration:

[0126] The first stage is to update the position of the sentinels, which account for 20% of the population. The warning value R2 is randomly generated. If the warning value is small, it means that there are no natural enemies and the population will conduct an extensive search; if the warning value is too large, it means that natural enemies have appeared, threatening the safety of the population.

[0127]

[0128] in, represents the population position after update (after t+1), i, j represents a specific position of the population matrix X, t represents the number of iterations, itermax represents the maximum number of iterations, x i,j represents the position of individuals in the population, a∈(0,1] is a random number, Q represents a random number that follows a normal distribution, L is a 1*d matrix with all elements being 1, ST represents the safety threshold, (1-0.2*K) is a second-order oscillation factor, and K is a random sampling value of [-1,1) to prevent the algorithm from falling into a local optimal solution during the search process. Of course, the above formula can be considered as both a position update for the sentinel and a position update for the population, because the sentinel is the first 20% of the individuals in the entire population, and because the matrix has been sorted in descending order, that is, individuals from the first row to the 20%*nth row are updated according to this formula.

[0129] After completing a position update, calculate the new fitness, update the current optimal fitness value and the current optimal fitness value position.

[0130] The second stage is the communication between individuals in the population to update the position, that is, the individuals in the population randomly exchange information and go to the best known hunting location to obtain more food. After multiple exchanges, the position update is completed, the new fitness is calculated, and the current optimal fitness value is updated.

[0131]

[0132] Among them, X m,j and X n,j are two randomly selected individuals, r1, r2 are rand(0,1), f(X m,j ) and f(X n,j ) represents the fitness value of the individual, and the best hunting position X best It is the current optimal fitness value position obtained in the previous stage.

[0133] The third stage is the exploration of individual populations to update their positions and obtain the best population information. That is, each individual in the population randomly explores in its own way, either linearly or spirally. After updating the position, the new fitness is calculated, and the global optimal fitness and the position of the global optimal fitness value are updated.

[0134]

[0135] Among them, β represents the variable step size, α represents rand(0,1),

[0136] When the individual population position exceeds the hyperparameter limit, it is automatically processed into the boundary value.

[0137] The hyperparameter combination output module is used to output the best hyperparameter combination after completing the iteration and substitute the hyperparameter combination into the internal structure model.

[0138] That is to say, after the traversal iteration is completed, the best population information is output, and the corresponding data is the hyperparameter combination with the best operating effect of the internal structure model of the present invention, and the best hyperparameter combination is substituted into the internal structure model.

[0139] The internal structure model training unit 303 enables the internal structure model to accept the hyperparameters provided by the external structure, completes the forward propagation process of the model through the DNN model, analyzes the task offloading result, and trains the DNN model according to the Memory mechanism and back propagation feedback, thereby realizing real-time and high-precision decision-making on the task offloading problem using the deep reinforcement learning model.

[0140] Specifically, the internal structure model training unit 303 further includes:

[0141] The forward propagation processing module is used to select forward propagation (i.e., forward propagation), traverse the total task matrix, take each row of task volume data combined with the current unloading situation as input, obtain the output matrix, calculate the Q value corresponding to each task unloaded to each server, and obtain the server to which each task is unloaded according to the three-stage ε-greedy strategy to obtain the result matrix.

[0142] Specifically, the forward propagation processing module is specifically used for:

[0143] Take out one row of the total task matrix and transpose it to get the task matrix. The obtained task matrix and the current unloading situation matrix form an input matrix, which is used as the DNN model input to get the DNN output matrix.

[0144] Calculate the Q value corresponding to each task offloaded to each server;

[0145] In the present invention, assuming that the amount of each task is x, the server energy consumption cost E = {e1, e2, e3, ..., e D+E}, server operation cost P = {p1,p2,p3,…,p D+E}, server CPU operating speed The calculation formulas for the upstream speed VU, the downstream speed VD, and the cost value Q are as follows:

[0146]

[0147] Among them, e up ,e down, represents the network upload speed and download speed. w1 and w2 represent the weights in the cost evaluation, where w1, w2 ∈ (0, 1) and w1 + w2 ≤ 1, which can be adjusted by oneself. local mode means offloading to the device side, and edge_mode means offloading to the edge side. The cost calculations for the two are different because there is also a transmission process on the edge side.

[0148] Adopt a three-stage ε-greedy strategy to select the device to be offloaded, and obtain a result matrix.

[0149] Specifically, generate a random number random. If random > ε, select the server with the smallest DNN output value. If ε / 2 < random < ε, randomly sample a server for offloading. If neither of the above two cases is met, select the server with the smallest Q value (i.e., the output matrix in Table 2, select the smallest output value in a row of the output matrix) for offloading. Finally, obtain a result matrix.

[0150] It should be noted that in general reinforcement learning, the goal is to pursue the action that can generate the largest Q value. In the model of the present invention, taking the cost as the Q value, what is pursued is the action that generates the smallest Q value.

[0151] The storage module is used to store the input matrix and the result matrix into the input matrix storage unit Memory_input and the result matrix storage unit Memory_result respectively, and update the current offloading situation matrix.

[0152] Specifically, assume that the current offloading situation matrix is [a, b, c, d,...] T , according to the screening, it is obtained that the current task should be offloaded to the server with the subscript 1, and the generated cost is q. The updated current offloading situation matrix is [a, b + q, c, d,...] T .

[0153] The training module is used to train the DNN model according to the input matrix and the result matrix stored in Memory_input and Memory_result, calculate the loss function, and perform backpropagation.

[0154] Specifically, assume that training_interval = 20, that is, after every 20 input matrices and result matrices are stored, the DNN model will be trained once. At the same time, assume that batch_size = 16. If the current loop count is exactly a multiple of 20, then randomly sample 16 groups of data from Memory for traversal.

[0155] Train each set of data with learning_rate: 0.06, weight_decay: 0.0001. Use BCELoss as the loss function and Adam as the optimizer.

[0156] After calculating the loss function, execute the backpropagation method. When the loop ends, output the current offloading situation.

[0157] Example

[0158] In this embodiment, a task offloading method based on a double-layer structure model is as follows:

[0159] Step 1: Initialize the swarm optimization algorithm and randomly initialize the population according to the boundary range. The swarm optimization algorithm is only executed when running for the first time or detecting a change in the MEC environment.

[0160] Step 2: Substitute the initialized hyperparameter combination into the inner structure model and initialize the model configuration (total number of servers D + E, where the number of devices is D, the number of edge servers is E, the total task matrix, a DNN model is divided into two-layer structure, and the number of neuron nodes is 120 and 80 respectively).

[0161] Step 3: Forward propagation, traverse the total task matrix, and the following steps are executed in a loop (Steps 4 - 9).

[0162] Step 4: Take out one row and transpose it to get the task matrix. Combine the task matrix and the current offloading situation matrix to form the input matrix, and use it as the DNN input to obtain the output matrix.

[0163] Step 5: Assume the task volume is x, the server energy consumption cost E = {e1, e2, e3, …, e D+E}, the server operation cost P = {p1, p2, p3, …, p D+E}, the server CPU operation rate The calculation formulas for the uplink speed VU, downlink speed VD, and cost value Q are as follows:

[0164]

[0165] Step 6: Calculate the Q value corresponding to each server through the formula. Adopt a three-segment ε-greedy strategy to select the device to be offloaded. Generate a random number random. If random > ε, select the server with the smallest DNN output value. If ε / 2 < random < ε, randomly sample a server for offloading. If neither of the above two situations is met, select the server with the smallest Q value for offloading. Finally, obtain a result matrix.

[0166] In general reinforcement learning, the goal is to pursue the action that produces the largest Q value. In this model, we use the cost as the Q value, and we pursue the action that produces the smallest Q value.

[0167] Step 7: Store the input matrix and result matrix into Memory_input and Memory_result respectively, and update the current unloading status matrix.

[0168] Step 8: In this embodiment, the hyperparameter training_interval=20 is set. When the traversal number reaches a multiple of the training interval, the DNN will be trained once. In this embodiment, the hyperparameter batch_size is set to 16. When the training starts, 16 groups of data are randomly sampled from Meomry.

[0169] Step 9: DNN is trained for each set of data, with learning_rate: 0.06, weight_decay: 0.0001, BCELoss as loss function, and Adam as optimizer. After calculating the loss function, back propagation is performed, the loop ends, and the current unloading status is output. Thus, a training process of the model is completed. From the perspective of the outer model, it can be understood as calculating the fitness value once, and the unloading status matrix obtained at the end corresponds to the fitness value.

[0170] In this embodiment, it is necessary to use an external structural model to search for three hyperparameters batch_size, learning_rate, and traning_interval in the deep reinforcement learning model.

[0171] Step 10: After sorting the initial fitness matrix, record the current optimal position X best , start the algorithm iteration.

[0172] Step 11: Start the group optimization algorithm position update. First, the position of the sentinel is updated. Then, the communication between individuals in the population is updated. Finally, the population individuals explore and update their positions to obtain the best population information.

[0173] Step 12: After the traversal is completed, the best population information is output. The corresponding data is the hyperparameter combination with the best running effect of this model.

[0174] Step 13: Modify the hyperparameters in the inner structure model to the data of the optimal population to complete the model's adaptation to the changing environment.

[0175] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Any person skilled in the art may modify and alter the above embodiments without violating the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be as set forth in the claims.

Claims

1. A task offloading method based on a two-layer structure model comprises the following steps: Step S1, construct and initialize the inner structure model based on DNN, initialize the current unloading situation matrix, and construct the total task matrix; Step S2, using a swarm optimization algorithm to adjust model hyperparameters for the internal structure model according to environmental changes and task conditions, and training the internal structure model with different hyperparameter combinations so that the internal structure model can quickly adapt to the new environment, thereby obtaining the internal structure model with the best effect; Step S2 further comprises: Step S200, randomly initializing the population matrix X according to the boundary range, and calculating the fitness matrix; Step S201, after sorting the initial fitness matrix, record the current optimal position; Step S202, iterating the group optimization algorithm, firstly updating the position of the alerter, then communicating between the individuals in the population to update the position, and finally updating the position of the individuals in the population to explore and update the position, so as to obtain the best population information; The location update for the alerter is as follows: The warning value R2 is randomly generated. If the warning value is small, it means that there are no natural enemies and the population will conduct extensive searches. If the warning value is too large, it means that natural enemies have appeared, threatening the safety of the population. And after completing a position update, calculate the new fitness, update the current optimal fitness value and the current optimal fitness value position; The position update of the sentinel is realized by the following formula: in, represents the population position after updating t+1, i, j represents a specific position of the population matrix X, t represents the number of iterations, itermax represents the maximum number of iterations, x i,j represents the position of the individual in the population, a∈(0,1 is a random number, Q represents a random number that obeys the normal distribution, L is a 1*d matrix with all elements being 1, ST represents the safety threshold, (1-0.2*K) is the second-order oscillation factor, and K is a random sampling value in [-1,1); The communication between individuals in the population to update the location is as follows: Individuals in the population randomly exchange information and go to the best known hunting locations to obtain more food; After multiple exchanges, the position update is completed, the new fitness is calculated, and the current optimal fitness value is updated; The communication position update between individuals in the population is achieved through the following formula: Among them, X m,j and X n,j are two randomly selected individuals, r1, r2 are rand(0,1), f(X m,j ) and f(X n,j ) represents the fitness value of the individual, and the best hunting position X best is the current optimal fitness value position obtained in the previous stage; The population individual exploration performs position update as follows: Each individual in the population explores randomly in its own way, either linearly or in spirals; After updating the position, calculate the new fitness, update the global optimal fitness, and the global optimal fitness value position; The position update of population individual exploration is realized by the following formula: Among them, β represents the variable step size, α represents rand(0,1), Indicates the current optimal fitness value position after updating t times; Step S203, after the iteration is completed, the best hyperparameter combination is output and the hyperparameter combination is substituted into the internal structure model; In step S3, the internal structure model accepts the hyperparameters provided in step S2, completes the forward propagation process of the model through the DNN model, analyzes the task offloading result, and trains the DNN model according to the Memory mechanism and back-propagation feedback, thereby realizing real-time and high-precision decision-making on the task offloading problem using the deep reinforcement learning model.

2. A task offloading method based on a double-layer structure model as claimed in claim 1, characterized in that: The internal structure model includes: The DNN model inputs the current offloading status and the tasks to be offloaded, and outputs the Q value corresponding to the tasks offloaded to each machine; The ε-greedy strategy module uses a three-stage ε-greedy strategy to select the device to be unloaded based on the input Q value; The replay memory module is used to store the input of the DNN model and the ε-greedy strategy to select the device to be unloaded into the experience replay; The sampling training module takes out the data in the experience playback after a certain number of cycles to train the DNN model.

3. A task offloading method based on a double-layer structure model as claimed in claim 2, characterized in that: In step S200, the method for calculating the fitness matrix is ​​as follows: a row of hyperparameter data in the X matrix is ​​substituted into the inner structure model, and after training, the same set of test data is used to test the effect, and the evaluation standard is the total cost value of this set of data.

4. A task offloading method based on a double-layer structure model as claimed in claim 3, characterized in that: Step S3 further comprises: Step S301, select forward propagation, traverse the total task matrix, take each row of task volume data combined with the current unloading situation as input, obtain the output matrix, and calculate the Q value corresponding to each task unloaded to each server, and obtain the server to which each task is unloaded according to the three-stage ε-greedy strategy to obtain the result matrix; Step S302, storing the input matrix and the result matrix in the input matrix storage unit and the result matrix storage unit respectively, and updating the current unloading situation matrix; Step S303, training the DNN model according to the input matrix and the result matrix stored in the input matrix storage unit and the result matrix storage unit, calculating the loss function, and performing back propagation.

5. A task offloading method based on a double-layer structure model as claimed in claim 4, characterized in that: Step S301 further includes: Step S101a, taking out one row of the total task matrix and transposing it to obtain a task matrix, combining the obtained task matrix and the current unloading situation matrix to form an input matrix, and using it as an input of the DNN model to obtain an output matrix of the DNN model; Step S301b, calculating the Q value corresponding to each task offloaded to each server; Step S301c: According to the calculation result of step S301b, a three-stage ε-greedy strategy is used to select the device to be uninstalled, and a result matrix is ​​obtained.

6. A task offloading device based on a double-layer structure model, used to implement the task offloading method based on a double-layer structure model as claimed in any one of claims 1 to 5; The task offloading device based on the two-layer structure model includes: The inner structure model building unit is used to build and initialize the inner structure model based on DNN, initialize the current unloading situation matrix, and build the total task matrix; A hyperparameter adjustment unit is used to use a swarm optimization algorithm to adjust the model hyperparameters for the internal structure model according to environmental changes and task conditions, and to train the internal structure model with different hyperparameter combinations so that the internal structure model can quickly adapt to the new environment, thereby obtaining the best internal structure model; The internal structure model training unit enables the internal structure model to accept the hyperparameters provided by the external structure, completes the forward propagation process of the model through the DNN model, analyzes the task offloading results, and trains the DNN model according to the Memory mechanism and back-propagation feedback, thereby realizing real-time and high-precision decision-making on task offloading problems using a deep reinforcement learning model.