Task unloading method and device applied to edge computing, equipment and medium

Abstract

The embodiment of the invention provides a task unloading method and device applied to edge computing, equipment and a medium, and the method comprises the following steps: taking the unloading cost of an unloading task queue to be unloaded as baseline data; initializing a memory bank, an evaluation network and a target network; training rounds are set, in each training round, an evaluation network is used for unloading each task in the unloading task queue, task unloading cost needed by unloading is calculated through baseline data, unloading memory is obtained, and the total unloading cost corresponding to each training round is generated; memories in the memory bank are transmitted to a target network, weight parameters of the evaluation network are updated through error vectors, and training parameters of the evaluation network are copied to the target network; and taking the training round with the lowest total unloading cost in all the training rounds as a final strategy for task unloading. According to the scheme, through the optimized task unloading method, the service delay of the user and the overall energy consumption of the edge system are reduced.

Description

Technical Field

[0001] The present invention relates to the field of resource allocation technology, and in particular to a task offloading method, device, equipment and medium applied to edge computing. Background Art

[0002] Mobile devices usually have limited computing resources and energy, and some computing tasks may be heavy and have certain latency requirements. In the traditional cloud computing model, devices offload heavy computing tasks to cloud servers for processing, which will cause problems such as high data transmission delay and large network bandwidth usage. With the increase in mobile devices, a large number of data packets are transmitted on the network. Once the data exchange capacity of the core network cannot support its data throughput requirements, the transmission delay will increase significantly.

[0003] like Figure 2 As shown in the figure, mobile edge computing adds an edge server between the user and the cloud server. The task request of the mobile user can be sent to the edge server for processing to reduce the pressure on the core network, greatly reduce the service delay and improve the service quality.

[0004] In such application scenarios, a method is needed to determine where to offload tasks. This method takes service latency and energy consumption as the goal, and optimizes the algorithm to minimize service latency and reduce the overall energy consumption of the system. If computing tasks are offloaded to edge servers, the network latency and energy consumption are small, but the computing performance of edge servers may not be as good as that of cloud servers; offloading to cloud servers can achieve stronger computing performance, but a large number of tasks offloaded to cloud servers at the same time are likely to cause task queuing, resulting in additional latency, and cause network congestion. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a task offloading method applied to edge computing to solve the technical problems of user service delay and high overall energy consumption of edge systems in the prior art. The method includes:

[0006] Pre-unload each task in the unloading task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data B cost ;

[0007] Initialize the memory bank, evaluation network and target network, and initialize the weight parameters and training parameters of the evaluation network;

[0008] Set training rounds. In each training round, use the evaluation network to offload each task in the offload task queue. cost Calculate the task offloading cost O required for offloading each task cost, and obtain the unloaded memory, save the memory to the memory bank, and unload the task cost O corresponding to all tasks cost Sum and generate the total offloading cost O corresponding to each training round cost ;

[0009] The memory in the memory bank is transferred to the target network, the maximum value of the value in the output layer vector of the target network is obtained, the error vector is constructed through the maximum value, the weight parameters of the evaluation network are updated through the error vector, and the training parameters of the evaluation network are copied to the target network;

[0010] In all training rounds, the total offloading cost O cost The lowest training round is taken as the final task offloading solution.

[0011] The embodiment of the present invention also provides a task offloading device applied to edge computing to solve the technical problems of user service delay and high overall energy consumption of edge system in the prior art. The device includes:

[0012] The baseline data acquisition module is used to pre-unload each task in the unloading task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data B cost ;

[0013] The initialization module is used to initialize the memory bank, the evaluation network and the target network, and to initialize the weight parameters and training parameters of the evaluation network;

[0014] The module for calculating the offloading cost is used to set the training rounds. In each training round, the evaluation network is used to offload each task in the offloading task queue. cost Calculate the task offloading cost O required for offloading each task cost , and obtain the unloaded memory, save the memory to the memory bank, and unload the task cost O corresponding to all tasks cost Sum and generate the total offloading cost O corresponding to each training round cost ;

[0015] The network update module is used to transfer the memory in the memory bank to the target network, obtain the maximum value of the value in the output layer vector of the target network, construct the error vector through the maximum value, update the weight parameters of the evaluation network through the error vector, and copy the training parameters of the evaluation network to the target network;

[0016] The final strategy selection module is used to unload the total cost O in all training rounds. cost The lowest training round is taken as the final task offloading solution.

[0017] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned task offloading methods applied to edge computing to solve the technical problems of user service delay and high overall energy consumption of the edge system in the prior art.

[0018] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned task offloading methods applied to edge computing, so as to solve the technical problems of user service delay and high overall energy consumption of the edge system in the prior art.

[0019] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above technical solutions adopted in the embodiments of this specification include at least:

[0020] In the mobile edge computing scenario, the task offloading method according to the embodiment of the present invention can implement a better task offloading strategy, optimize service delay and energy consumption, and improve the stability of the edge system. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 is a flowchart of a task offloading method applied to edge computing provided by an embodiment of the present invention;

[0023] Figure 2 is a schematic diagram of a mobile edge computing scenario model provided by an embodiment of the present invention;

[0024] Figure 3 It is a schematic diagram of the convergence of the task offloading method applied to edge computing provided by an embodiment of the present invention;

[0025] Figure 4 is a comparison chart of experimental effects of a task offloading method applied to edge computing provided by an embodiment of the present invention and an offloading method based on a greedy strategy;

[0026] Figure 5 is a structural block diagram of a computer device provided by an embodiment of the present invention;

[0027] Figure 6 It is a structural block diagram of a task offloading device applied to edge computing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0029] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0030] In an embodiment of the present invention, a task offloading method for edge computing is provided, such as Figure 1 As shown, the method includes:

[0031] Step S101: pre-unload each task in the unload task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data B cost ;

[0032] Step S102: Initializing the memory bank, the evaluation network and the target network, and initializing and setting the weight parameters of the evaluation network and the training parameters of the evaluation network;

[0033] Step S103: Set a training round. In each training round, use the evaluation network to unload each task in the unload task queue. cost Calculate the task offloading cost O required for offloading each task cost , and obtain the unloaded memory, save the memory to the memory bank, and unload the task cost O corresponding to all tasks cost Sum and generate the total offloading cost O corresponding to each training round cost ;

[0034] Step S104: transferring the memory in the memory bank to the target network, obtaining the maximum value of the value in the output layer vector of the target network, constructing an error vector through the maximum value, updating the weight parameters of the evaluation network through the error vector, and copying the training parameters of the evaluation network to the target network;

[0035] Step S105: In all training rounds, the total unloading cost O cost The lowest training round is taken as the final task offloading solution.

[0036] In specific implementation, in order to use the unloading cost obtained by the greedy algorithm as the baseline of the task unloading algorithm of the embodiment of the present invention, each task in the unloading task queue is pre-unloaded through the following steps to obtain the unloading cost of each task, and use the unloading cost as the baseline data:

[0037] Get the CPU frequency of all locations where the task is allowed to be unloaded and the current length of the unload location task accumulation queue, and add all locations to the location set P a ; Traverse the position set P a , place the unloading location with empty unloading location task accumulation queue in the first set P 1 In the unloading position task accumulation queue, the unloading position with non-empty unloading position is placed in the second set P 2 If the first set P 1 is not empty, then in the first set P 1 Find the first unloading position with the highest CPU frequency in the set, and take the first unloading position as the result of greedy unloading; if the first set P 1 is empty, then in the first set P 2 Find the second unloading position with the shortest task accumulation queue in the unloading position, and use the second unloading position as the result of greedy unloading; perform the unloading operation of the task according to the result of greedy unloading, record the unloading cost of each task during greedy unloading, and use the unloading cost as the baseline data B of the deep reinforcement learning algorithm. cost .

[0038] Specifically, refer to the baseline results of greedy unloading to set the reward function to guide the exploration and learning of the reinforcement learning agent. First, traverse and inspect the unloading position task accumulation queue and computing power of each unloading position; second, traverse and inspect the unloading position task accumulation queue status and computing power of each unloading position; finally, if there is no empty unloading position task accumulation queue in the unloading position, select the position with the shortest remaining execution time of the unloading position task accumulation queue among all unloading positions to unload.

[0039] Specifically, initialization also includes determining the capacity of the memory bank and initializing the weight parameters of the neural network (including the evaluation network and the evaluation network). In the embodiment of the present invention, the training parameters are set: ε (parameter used in ε-Greedy) ← 0.9, γ (decay value for future rewards) ← 0.7, α (learning rate) ← 0.0005, and the rule is set (update the target network every 100 rounds).

[0040] In the specific implementation, in order to enhance the interaction between the learning agent and the environment and obtain experience (or memory), the following steps are used to use the evaluation network to unload each task in the unload task queue. cost Calculate the task offloading cost O required for offloading each task cost, generate the uninstalled memory and save the memory to the memory library:

[0041] Initialize the unloading location task accumulation queue Q of each edge node i , define the state s = {O cost ,N r}, where i is the sequence number of the unloading location task accumulation queue, O cost is the task offloading cost, N r is the total number of tasks waiting for the planned unloading location; select action a for executing the unloading operation of the task through the ε-Greedy strategy, execute action a, and calculate the task unloading cost O required for the unloading solution corresponding to the execution of action a cost ; Construct the next state s′ according to state s; Calculate the reward r obtained by action a in state s through reward function R, where reward B cost is the baseline data; generate memory (s, a, r, s′) according to state s, action a, next state s′ and reward r, and save the memory (s, a, r, s′) to the memory bank.

[0042] Specifically, the ε-Greedy strategy is one of the commonly used exploration and exploitation strategies in reinforcement learning. In this strategy, the agent will explore with a certain probability ε (epsilon) when making decisions, that is, randomly select an action to discover new states or update the understanding of the current state; and exploit with a probability of 1-ε, that is, select the currently known best action. The Q value in the Q network is the expected reward of the agent taking a specific action in a given state.

[0043] Specifically, the "ε-Greedy" strategy is used to randomly select actions with a probability of ε, explore unknown states in the environment, and discover potential high-reward paths; the action with the highest Q value in the current state is selected through the Q table / Q network with a probability of 1-ε, and the learned knowledge is used to obtain more rewards. Selecting action a through the ε-Greedy strategy includes:

[0044] Generate a random number rand, where the random number rand is between 0 and 1; if rand is less than the training parameter ε, input the state s into the evaluation network and run it to generate the output layer vector Get the output layer vector The subscript corresponding to the maximum value in is used as the action a selected according to the deep Q network; if rand is greater than the training parameter ε, an action is randomly selected as the action a selected according to the deep Q network.

[0045] In specific implementation, in order to reduce the unloading cost O cost As the optimization target (which includes the total service delay t total and total energy consumption E total ), and adjust the tendency of the target through the coefficient, and calculate the task offloading cost O required for the offloading scheme corresponding to the execution action a through the following steps cost :

[0046] The transmission delay t of the task T , queuing delay t Q and execution delay t e Calculate the total delay t total , where t total =max(t T ,t Q 0+t e ; Through the required process signal T 1 Transmit power Transmission delay and optional process T 2 Signal transmission power Transmission delay Calculate the total energy consumption E total ,in, According to the total delay t total and total energy consumption E total Calculate the minimum task offloading cost O cost ,in, N is the total number of all task offloading requests sent by all users in the time period, i is the i-th task offloading request, α is the delay coefficient, and 1-α is the energy consumption coefficient.

[0047] Specifically, in the edge computing scenario of the embodiment of the present invention, the offloading cost O cost As the optimization target, it includes the total service delay t total and total energy consumption E tital , and adjust the tendency of the target through the coefficient.

[0048] Total delay t total Including the transmission delay of the task t T , queuing delay t Q and execution delay t e , since the task transmission process and queuing process can be carried out simultaneously, the total delay can be expressed as t total =max(t T ,t Q )+t e .

[0049] Transmission delay t TIt refers to the time it takes for task data to be transmitted from the user device to the task execution location. When the task is not unloaded, t T = 0. When offloading to the edge or cloud, the transmission delay consists of two parts: the task data is first transmitted from the user device to the base station near the user, and then the task data is transmitted between base stations according to the offloading location. Therefore, the process of offloading to the edge server includes the necessary process T 1 and optional process T 2 (Process T is not required when offloading to a local edge server 2 ). Process T 1 The transmission delay It is the ratio of task data volume to transmission rate, where d i is the data size of the task, The channel rate at which the user transmits data to the corresponding base station (local base station) serving the user; process T 2 The transmission delay When m 1 =m 2 When in, The channel rate at which the corresponding base station (local base station) serving the user transmits data to the base station performing the task.

[0050] Queuing delay t Q It can be expressed as the sum of the execution time of the previous tasks in the unloading location task accumulation queue, from which the running time of the currently being processed task needs to be subtracted, so the queuing delay Execution delay Among them, t e is the ratio of the task workload (in terms of the number of CPU cycles required) to the CPU frequency, ω i is the workload of the task, f i The CPU frequency of the user device.

[0051] Total energy consumption E total Including transmission energy consumption E T and execution energy consumption E e ,When the task is not unloaded, there is no transmission process,E total =E e =cω i f i 2 The energy consumption of task offloading includes transmission energy consumption and execution energy consumption. When the task is not offloaded, the task data does not need to be transmitted to the edge node or cloud server, so there is no transmission process. When offloading to the edge or to the cloud, there is transmission energy consumption. According to the experiment, E T >>E e, so the main calculation is transmission energy consumption); when offloading to the edge or cloud, the energy consumed by transmitting data dominates, in is the signal transmission power of each transmission process.

[0052] In the specific implementation, in order to obtain knowledge from the memory bank, the parameters of the training evaluation network are used to learn previous experience. The following steps are used to transfer the memory in the memory bank to the target network, obtain the maximum value of the output layer vector of the target network, and construct the error vector through the maximum value:

[0053] Extract a set of memories M from the memory bank, where memory M contains multiple independent memories m i ={s,a,r,s′}, where s is the state, s′ is the next state of state s, r is the reward obtained by unloading the task, and m i Unload the corresponding memory for the i-th task; i The state s in is passed to the evaluation network, and according to the memory m i The index subscript corresponding to action a in the evaluation network is taken out from the output layer vector of the evaluation network, where Q(s,a) is the long-term expected return of taking action a in state s; the memory m is i The next state s′ in the target network is passed to the target network, and the maximum value of the output layer vector of the target network is taken as the maximum value of the output layer vector maxQ(s′); the Q estimate Q is calculated by the maximum value of the output layer vector maxQ(s′) est and Q actual value Q real , through the Q estimate Q est and Q actual value Q real Calculate the error e a ; Through error e a Constructing the error vector in

[0054] Specifically, Q(s,a) is the concept of Q-Learning, which indicates the quality of executing action a in state s.

[0055] In the specific implementation, the following steps are used to calculate the Q estimate Q through the maximum value of the output layer vector maxQ(s′): est and Q actual value Q real , through the Q estimate Q est and Q actual value Q real Calculate the error e a :

[0056] Calculate the Q estimate Q est , where the Q estimate Q est =Q(s,a); calculate the actual value of Qreal , where Q is the actual value Q real = r + γ × maxQ (s ′), r is the reward obtained by action a in state s, γ is the attenuation value of future rewards in the training parameters; through the Q estimate Q est and Q Reality Q real The error e is calculated a , where the error e a =α×(Q real -Q est ), α is the learning rate in the training parameters.

[0057] In the specific implementation, in order to back propagate the error and update the weights of the neural network, the following steps are performed to update the weight parameters of the evaluation network through the error vector and copy the training parameters of the evaluation network to the target network:

[0058] The error vector e is passed into the evaluation network for back propagation, and the weight parameters of the evaluation network are updated; the round number interval is set, and when the training of the evaluation network reaches the round number interval, the weight parameters of the evaluation network are copied to the target network.

[0059] Specifically, the error vector error e is passed to the evaluation network for back propagation to update the weights of the evaluation network. Whenever the training reaches a certain number of rounds, the evaluation network parameters are copied to the target network. Because the role of the target network is to calculate the maximum value of the output layer vector maxQ(s′), the target network is only copied from the evaluation network after a certain number of rounds.

[0060] like Figure 3 As shown, the episode on the horizontal axis represents the training round (the training round is set to 200 in the embodiment of the present invention). After 50 rounds of training, the total unloading cost gradually converges.

[0061] like Figure 4 As shown, a comparison of the total offloading cost after using the task offloading method according to an embodiment of the present invention and using the greedy algorithm to offload tasks is shown.

[0062] In this embodiment, a computer device is provided, such as Figure 5 As shown, it includes a memory 501, a processor 502, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned task offloading methods applied to edge computing is implemented.

[0063] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0064] In this embodiment, a computer-readable storage medium is provided, which stores a computer program for executing any of the above-mentioned task offloading methods applied to edge computing.

[0065] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0066] Based on the same inventive concept, an embodiment of the present invention also provides a task offloading device for edge computing, as described in the following embodiments. Since the principle of solving the problem by the task offloading device for edge computing is similar to that of the task offloading method for edge computing, the implementation of the task offloading device for edge computing can refer to the implementation of the task offloading method for edge computing, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0067] Figure 6 is a structural block diagram of a task offloading device applied to edge computing according to an embodiment of the present invention. Figure 6 As shown, it includes: a baseline data acquisition module 601, an initialization module 602, an unloading cost calculation module 603, a network update module 604 and a final strategy selection module 605. The structure is described below.

[0068] The baseline data acquisition module 601 is used to pre-unload each task in the unloading task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data B. cost ;

[0069] Initialization module 602, used to initialize the memory bank, the evaluation network and the target network, and to initialize and set the weight parameters of the evaluation network and the training parameters of the evaluation network;

[0070] The module 603 for calculating the offloading cost is used to set a training round. In each training round, the evaluation network is used to offload each task in the offloading task queue. cost Calculate the task offloading cost O required for offloading each task cost , and obtain the unloaded memory, save the memory to the memory bank, and unload the task cost O corresponding to all tasks cost Sum and generate the total offloading cost O corresponding to each training round cost ;

[0071] The network update module 604 is used to transfer the memory in the memory bank to the target network, obtain the maximum value of the value in the output layer vector of the target network, construct an error vector through the maximum value, update the weight parameters of the evaluation network through the error vector, and copy the training parameters of the evaluation network to the target network;

[0072] The final strategy selection module 605 is used to select the total unloading cost O in all training rounds. cost The lowest training round is taken as the final task offloading solution.

[0073] In one embodiment, the baseline data acquisition module includes:

[0074] Set P a The acquisition unit is used to obtain the CPU frequency of all locations where the task is allowed to be unloaded and the current length of the unloading location task accumulation queue, and add all locations to the location set P a ;

[0075] Collection classification unit, used to traverse the location set P a , place the unloading location with empty unloading location task accumulation queue in the first set P 1 In the unloading position task accumulation queue, the unloading position with non-empty unloading position is placed in the second set P 2 middle;

[0076] The first greedy result acquisition unit is used for if the first set P 1 is not empty, then in the first set P 1 Find the first unloading position with the highest CPU frequency, and take the first unloading position as the result of greedy unloading;

[0077] The second greedy result acquisition unit is used for if the first set P 1 is empty, then in the first set P 2Find the second unloading position with the shortest unloading position task accumulation queue, and take the second unloading position as the result of greedy unloading;

[0078] The baseline data acquisition unit is used to perform the task unloading operation according to the result of greedy unloading, record the unloading cost of each task during greedy unloading, and use the unloading cost as the baseline data B of the deep reinforcement learning algorithm. cost .

[0079] In one embodiment, the module for calculating the unloading cost includes:

[0080] Initialization queue unit, used to initialize the unloading location task accumulation queue Q of each edge node i , define the state s = {O cost ,N r}, where i is the sequence number of the unloading location task accumulation queue, O cost is the task offloading cost, N r The total number of tasks waiting to be planned for unloading locations;

[0081] The cost calculation unit is used to select an action a for executing the task offloading operation through the ε-Greedy strategy, execute action a, and calculate the task offloading cost O required for executing the offloading solution corresponding to action a. cost ;

[0082] A state construction unit, used to construct a next state s′ according to the state s;

[0083] The reward calculation unit is used to calculate the reward r obtained by action a in state s through the reward function R, where the reward B cost is the baseline data;

[0084] The memory bank generation unit is used to generate memory (s, a, r, s′) according to the state s, action a, next state s′ and reward r, and save the memory (s, a, r, s′) into the memory bank.

[0085] In one embodiment, the cost calculation unit is used to calculate the transmission delay t of the task. T , queuing delay t Q and execution delay t e Calculate the total delay t total , where t total =max(t T ,t Q )+t e ; Through the required process signal T 1 Transmit power Transmission delay and optional process T 2 Signal transmission power Transmission delay Calculate the total energy consumption E total ,in, According to the total delay t total and total energy consumption E total Calculate the minimum task offloading cost O cost ,in, N is the total number of all task offloading requests sent by all users in the time period, i is the i-th task offloading request, α is the delay coefficient, and 1-α is the energy consumption coefficient.

[0086] In one embodiment, the baseline data acquisition module includes:

[0087] The memory extraction unit is used to extract a set of memories M from the memory bank. The memory M contains multiple independent memories m i ={s,a,r,s′}, where s is the state, s′ is the next state of state s, r is the reward obtained by unloading the task, and m i Unload the corresponding memory for the i-th task;

[0088] The reward acquisition unit is used to store the memory m i The state s in is passed to the evaluation network, and according to the memory m i The index subscript corresponding to action a in , takes out Q(s,a) corresponding to the index subscript in the output layer vector of the evaluation network, where Q(s,a) is the long-term expected return of taking action a in state s;

[0089] The output layer vector maximum acquisition unit is used to store the memory m i The next state s′ in is passed to the target network, and the maximum value of the output layer vector of the target network is taken as the maximum value of the output layer vector maxQ(s′);

[0090] Error calculation unit, used to calculate the Q estimate Q through the maximum value of the output layer vector maxQ(s′) est and Q actual value Q real , through the Q estimate Q est and Q actual value Q real Calculate the error e a ;

[0091] The error vector building unit is used to pass the error e a Constructing the error vector in

[0092] In one embodiment, the error calculation unit is used to calculate the Q estimate Q est , where the Q estimate Q est =Q(s,a); calculate the actual value of Qreal , where Q is the actual value Q real = r + γ × maxQ (s ′), r is the reward obtained by action a in state s, γ is the attenuation value of future rewards in the training parameters; through the Q estimate Q est and Q Reality Q real The error e is calculated a , where the error e a =α×(Q real -Q est ), α is the learning rate in the training parameters.

[0093] In one embodiment, the network update module includes:

[0094] An evaluation network parameter updating unit, used to pass the error vector e into the evaluation network for back propagation and update the weight parameters of the evaluation network;

[0095] The target network parameter update unit is used to set the round number interval. When the training of the evaluation network reaches the round number interval, the weight parameters of the evaluation network are copied to the target network.

[0096] The embodiments of the present invention achieve the following technical effects:

[0097] In the mobile edge computing scenario, the task offloading method of the embodiment of the present invention can implement a better task offloading strategy, optimize service delay and energy consumption, and improve the stability of the edge system; with the goal of minimizing the task offloading cost, the service delay and energy consumption are jointly optimized, and the computing offloading requirements of user tasks in mobile edge computing are realized under the constraints of edge node computing power, network environment and other restrictions; the embodiment of the present invention proposes a resource allocation method in the edge computing scenario based on deep reinforcement learning, which reduces the service delay and the energy consumption of the edge system by planning the task offloading location; the test results show that the resource allocation method proposed in the present invention can converge quickly within a limited training round and control the task offloading cost within a relatively low range.

[0098] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A task offloading method applied to edge computing, characterized in that: include: Pre-unload each task in the unload task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data B cost ; Initializing a memory bank, an evaluation network, and a target network, and initializing and setting weight parameters of the evaluation network and training parameters of the evaluation network; Set a training round, in each training round, use the evaluation network to unload each task in the unload task queue, and use the baseline data B cost Calculate the task offloading cost O required for offloading each task cost , and obtain the unloaded memory, save the memory to the memory bank, and calculate the task unloading cost O corresponding to all the tasks cost Sum and generate the total unloading cost O corresponding to each training round cost ; The memory in the memory bank is transferred to the target network, the maximum value of the value in the output layer vector of the target network is obtained, an error vector is constructed by the maximum value, the weight parameter of the evaluation network is updated by the error vector, and the training parameters of the evaluation network are copied to the target network; In all the training rounds, the total offloading cost O cost The lowest training round is taken as the final task offloading solution.

2. The task offloading method for edge computing according to claim 1, characterized in that: Pre-unload each task in the unload task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data B cost ,include: Get the CPU frequency of all locations where the task is allowed to be unloaded and the current length of the unloading location task accumulation queue, and add all the locations to the location set P a ; Traverse the position set P a , placing the unloading positions whose task accumulation queues of the unloading positions are empty into the first set P1, and placing the unloading positions whose task accumulation queues of the unloading positions are not empty into the second set P2; If the first set P1 is not empty, searching for a first unloading position with the highest CPU frequency in the first set P1, and taking the first unloading position as a result of greedy unloading; If the first set P1 is empty, then searching the first set P2 for the second unloading position with the shortest task accumulation queue of the unloading position, and taking the second unloading position as the result of greedy unloading; According to the result of the greedy offloading, the offloading operation of the task is performed, and the offloading cost of each task during the greedy offloading is recorded, and the offloading cost is used as the baseline data B of the deep reinforcement learning algorithm. cost .

3. The task offloading method for edge computing according to claim 1, characterized in that: Use the evaluation network to unload each task in the unload task queue, and use the baseline data B cost Calculate the task offloading cost O required for offloading each task cost , generating an uninstalled memory, and saving the memory to the memory bank, comprising: Initialize the unloading location task accumulation queue Q of each edge node i , define the state s = {O cost ,N r }, where i is the sequence number of the unloading location task accumulation queue, O cost is the task offloading cost, N r The total number of tasks waiting to be planned for unloading locations; Select an action a for executing the offloading operation of the task through the ε-Greedy strategy, execute the action a, and calculate the task offloading cost O required to execute the offloading solution corresponding to the action a. cost ; Construct the next state s according to the state s ′ ; The reward function R is used to calculate the reward r obtained by the action a in state s, where the reward is B cost is the baseline data; According to the state s, the action a, the next state s ′ and reward r to generate memory (s,a,r,s ′ ), and store the memory (s,a,r,s ′ ) is saved in the memory bank.

4. The task offloading method for edge computing according to claim 3, characterized in that: Calculate the task offloading cost O required to execute the offloading solution corresponding to the action a cost ,include: The transmission delay t of the task T , queuing delay t Q and execution delay t e Calculate the total delay t total , where t total =max(t T ,t Q )+t e ; The transmission power of the necessary process signal T1 Transmission delay and optional process T2 signal transmission power Transmission delay Calculate the total energy consumption E total ,in, According to the total delay t total and the total energy consumption E total Calculate the minimum task offloading cost O cost ,in, N is the total number of all task offloading requests sent by all users in the time period, i is the i-th task offloading request, α is the delay coefficient, and 1-α is the energy consumption coefficient.

5. The task offloading method for edge computing according to claim 1, characterized in that: The memory in the memory bank is transferred to the target network, the maximum value of the value in the output layer vector of the target network is obtained, and the error vector is constructed by the maximum value, including: Extract a set of memories M from the memory bank, wherein the memory M includes multiple independent memories m i ={s,a,r,s ′ }, where s is the state, s ′ is the next state of state s, r is the reward obtained by unloading the task, m i Unload the corresponding memory for the i-th task; Memory m i The state s in is passed to the evaluation network, and according to the memory m i The index subscript corresponding to the action a in , takes out Q(s,a) corresponding to the index subscript in the output layer vector of the evaluation network, where Q(s,a) is the long-term expected return of taking action a in the state s; Memory m i The next state s in ′ The target network is passed in, and the maximum value of the output layer vector of the target network is taken as the maximum value of the output layer vector maxQ(s ′ ); The maximum value of the output layer vector maxQ(s ′ ) Calculate the Q estimate Q est and Q actual value Q real , through the Q estimate Q est and the actual value Q real Calculate the error e a ; By the error e a Constructing the error vector in 6. The task offloading method for edge computing according to claim 5, characterized in that: The maximum value of the output layer vector maxQ(s ′ ) Calculate the Q estimate Q est and Q actual value Q real , through the Q estimate Q est and the actual value Q real Calculate the error e a ,include: Calculate the Q estimate Q est , where the Q estimate Q est =Q(s,a); Calculate the actual value Q real , where Q is the actual value Q real = r + γ × maxQ (s ′ ), r is the reward obtained by action a in state s, and γ is the attenuation value of future rewards in the training parameter; The Q estimate Q est And the Q reality Q rea1 The error e is calculated a , where the error e a =α×(Q real -Q est ), α is the learning rate in the training parameters.

7. The task offloading method for edge computing according to claim 1, characterized in that: Updating the weight parameters of the evaluation network by using the error vector and copying the training parameters of the evaluation network to the target network includes: The error vector e is passed into the evaluation network for back propagation, and the weight parameters of the evaluation network are updated; A round number interval is set, and when the training of the evaluation network reaches the round number interval, the weight parameters of the evaluation network are copied to the target network.

8. A task offloading device for edge computing, characterized in that: include: The baseline data acquisition module is used to pre-unload each task in the unloading task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data B cost ; An initialization module, used to initialize the memory bank, the evaluation network and the target network, and to initialize and set the weight parameters of the evaluation network and the training parameters of the evaluation network; The module for calculating the offloading cost is used to set a training round. In each training round, the evaluation network is used to offload each task in the offloading task queue. cost Calculate the task offloading cost O required for offloading each task cost , and obtain the unloaded memory, save the memory to the memory bank, and calculate the task unloading cost O corresponding to all the tasks cost Sum and generate the total unloading cost O corresponding to each training round cost ; A network update module, used to transfer the memory in the memory bank to the target network, obtain the maximum value of the numerical value in the output layer vector of the target network, construct an error vector through the maximum value, update the weight parameters of the evaluation network through the error vector, and copy the training parameters of the evaluation network to the target network; The final strategy selection module is used to select the total unloading cost O in all the training rounds. cost The lowest training round is taken as the final task offloading solution.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the task offloading method applied to edge computing described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the task offloading method for edge computing according to any one of claims 1 to 7.

Images