Task offloading method, device, equipment and medium for edge computing

Abstract

Embodiments of the present invention provide a task offloading method, apparatus, device, and medium for edge computing, wherein the method includes the following steps: using the offloading cost of a pre-offloaded task queue as baseline data; initializing a memory bank, an evaluation network, and a target network; setting training rounds, and in each training round, using the evaluation network to offload each task in the offload task queue, calculating the task offloading cost required for offloading using the baseline data, and obtaining the offloaded memory to generate the total offloading cost corresponding to each training round; transferring the memory in the memory bank to the target network, updating the weight parameters of the evaluation network using an error vector, and copying the training parameters of the evaluation network to the target network; and using the training round with the lowest total offloading cost among all training rounds as the final strategy for task offloading. This solution reduces user service latency and the overall energy consumption of the edge system through an optimized task offloading method.

Description

Technical Field

[0001] The present invention relates to the field of resource allocation technology, and in particular to a task offloading method, device, equipment and medium for edge computing. Background Art

[0002] Mobile devices typically have limited computing resources and energy, while some computing tasks can be demanding and have latency requirements. In traditional cloud computing, devices offload heavy computing tasks to cloud servers for processing, which can lead to high data transmission latency and significant network bandwidth usage. With the increasing number of mobile devices, a large number of data packets are transmitted across the network. If the core network's data exchange capacity cannot support these data throughput requirements, transmission latency will increase significantly.

[0003] like Figure 2 As shown in the figure, mobile edge computing adds an edge server between the user and the cloud server. The task request of the mobile user can be sent to the edge server for processing, which can reduce the pressure on the core network, greatly reduce the service delay and improve the service quality.

[0004] In such application scenarios, a method is needed to determine where to offload tasks. This method targets service latency and energy consumption, using an optimization algorithm to minimize service latency while simultaneously reducing overall system energy consumption. Offloading computing tasks to edge servers reduces network latency and energy consumption, but the computing performance of edge servers may not be as good as that of cloud servers. Offloading computing tasks to cloud servers can achieve higher computing performance, but simultaneously offloading a large number of tasks to cloud servers can easily cause task queuing, resulting in additional latency and network congestion. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a task offloading method for edge computing to solve the technical problems of user service delay and high overall energy consumption of edge systems in the prior art. The method includes:

[0006] Pre-unload each task in the unloading task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data B cost ;

[0007] Initialize the memory bank, evaluation network, and target network, and initialize the weight parameters and training parameters of the evaluation network;

[0008] Set up training rounds. In each training round, use the evaluation network to offload each task in the offload task queue. cost Calculate the task offloading cost O required for each task to be offloaded cost, and obtain the unloaded memory, save the memory to the memory bank, and unload the task cost O corresponding to all tasks cost Sum and generate the total offloading cost O corresponding to each training round cost ;

[0009] The memory in the memory bank is transferred to the target network, the maximum value of the output layer vector of the target network is obtained, the error vector is constructed based on the maximum value, the weight parameters of the evaluation network are updated based on the error vector, and the training parameters of the evaluation network are copied to the target network;

[0010] In all training rounds, the total offloading cost O cost The lowest training round is taken as the final task offloading solution.

[0011] The present invention also provides a task offloading device for edge computing to solve the technical problems of user service delay and high overall energy consumption of edge systems in the prior art. The device includes:

[0012] The baseline data acquisition module is used to pre-unload each task in the unloading task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data B cost ;

[0013] The initialization module is used to initialize the memory bank, evaluation network and target network, and to initialize the weight parameters and training parameters of the evaluation network;

[0014] The module for calculating the offloading cost is used to set the training round. In each training round, the evaluation network is used to offload each task in the offloading task queue. cost Calculate the task offloading cost O required for each task to be offloaded cost , and obtain the unloaded memory, save the memory to the memory bank, and unload the task cost O corresponding to all tasks cost Sum and generate the total offloading cost O corresponding to each training round cost ;

[0015] The network update module is used to transfer the memory in the memory bank to the target network, obtain the maximum value of the value in the output layer vector of the target network, construct the error vector based on the maximum value, update the weight parameters of the evaluation network based on the error vector, and copy the training parameters of the evaluation network to the target network;

[0016] The final strategy selection module is used to unload the total cost O in all training rounds. cost The lowest training round is taken as the final task offloading solution.

[0017] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements any of the above-mentioned task offloading methods applied to edge computing to solve the technical problems of user service delay and high overall energy consumption of the edge system in the prior art.

[0018] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned task offloading methods applied to edge computing, so as to solve the technical problems of user service delay and high overall energy consumption of the edge system in the prior art.

[0019] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0020] In the mobile edge computing scenario, the task offloading method of the embodiment of the present invention can implement a better task offloading strategy, optimize service delay and energy consumption, and improve the stability of the edge system. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 This is a flowchart of a task offloading method for edge computing provided by an embodiment of the present invention;

[0023] Figure 2 is a schematic diagram of a mobile edge computing scenario model provided by an embodiment of the present invention;

[0024] Figure 3 Schematic diagram of the convergence of the task offloading method applied to edge computing provided by an embodiment of the present invention;

[0025] Figure 4 This is a comparison chart of the experimental results of the task offloading method applied to edge computing provided by an embodiment of the present invention and the offloading method based on the greedy strategy;

[0026] Figure 5 This is a structural block diagram of a computer device provided by an embodiment of the present invention;

[0027] Figure 6 This is a structural block diagram of a task offloading device for edge computing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0029] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the features in the following embodiments and embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of this application.

[0030] In an embodiment of the present invention, a task offloading method for edge computing is provided, such as Figure 1 As shown, the method includes:

[0031] Step S101: Pre-unload each task in the unloading task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data B cost ;

[0032] Step S102: Initializing the memory bank, the evaluation network, and the target network, and initializing and setting the weight parameters and training parameters of the evaluation network;

[0033] Step S103: Set up training rounds. In each training round, use the evaluation network to unload each task in the unload task queue. cost Calculate the task offloading cost O required for each task to be offloaded cost , and obtain the unloaded memory, save the memory to the memory bank, and unload the task cost O corresponding to all tasks cost Sum and generate the total offloading cost O corresponding to each training round cost ;

[0034] Step S104: transferring the memory in the memory bank to the target network, obtaining the maximum value of the output layer vector of the target network, constructing an error vector based on the maximum value, updating the weight parameters of the evaluation network based on the error vector, and copying the training parameters of the evaluation network to the target network;

[0035] Step S105: In all training rounds, the total offloading cost O cost The lowest training round is taken as the final task offloading solution.

[0036] In specific implementation, in order to use the offloading cost obtained by the greedy algorithm as the baseline of the task offloading algorithm of the embodiment of the present invention, the following steps are performed to pre-offload each task in the offloading task queue, obtain the offloading cost of each task, and use the offloading cost as the baseline data:

[0037] Get the CPU frequency of all locations where the task is allowed to be unloaded and the current length of the unloading location task accumulation queue, and add all locations to the location set P a ; Traverse the position set P a , place the unloading positions whose task accumulation queue is empty in the first set P1, and place the unloading positions whose task accumulation queue is not empty in the second set P2; if the first set P1 is not empty, find the first unloading position with the highest CPU frequency in the first set P1, and use the first unloading position as the result of greedy unloading; if the first set P1 is empty, find the second unloading position with the shortest task accumulation queue in the first set P2, and use the second unloading position as the result of greedy unloading; perform the unloading operation of the task according to the result of greedy unloading, record the unloading cost of each task during greedy unloading, and use the unloading cost as the baseline data B of the deep reinforcement learning algorithm cost .

[0038] Specifically, a reward function is set based on the baseline results of greedy offloading to guide the reinforcement learning agent's exploration and learning. First, the task accumulation queue and computing power of each offloading location are traversed and inspected. Second, the status and computing power of the task accumulation queue of each offloading location are traversed and inspected. Finally, if there is no empty task accumulation queue at an offloading location, the location with the shortest remaining execution time in the task accumulation queue is selected for offloading.

[0039] Specifically, initialization also includes determining the memory bank capacity and initializing the weight parameters of the neural network (including the evaluation network and the evaluation network). In this embodiment of the present invention, the training parameters are set as follows: ε (parameter used in ε-Greedy) ← 0.9, γ (decay value for future rewards) ← 0.7, α (learning rate) ← 0.0005, and a rule is set (updating the target network every 100 rounds).

[0040] In the specific implementation, in order to strengthen the interaction between the learning agent and the environment and obtain experience (or memory), the following steps are used to use the evaluation network to unload each task in the unloading task queue, and the baseline data B is used to calculate the unloading task. cost Calculate the task offloading cost O required for each task to be offloaded cost , generate the uninstalled memory and save the memory to the memory library:

[0041] Initialize the offloading location task accumulation queue Q of each edge nodei , define state s={O cost ,N r}, where i is the sequence number of the unloading location task accumulation queue, O cost is the task offloading cost, N r is the total number of tasks waiting for the planned unloading location; select action a for executing the unloading operation of the task through the ε-Greedy strategy, execute action a, and calculate the task unloading cost O required for the unloading solution corresponding to action a. cost ; Construct the next state s' according to state s; calculate the reward r obtained by action a in state s through the reward function R, where the reward B cost is the baseline data; generate memory (s, a, r, s′) based on state s, action a, next state s′ and reward r, and save the memory (s, a, r, s′) to the memory bank.

[0042] Specifically, the ε-Greedy strategy is one of the commonly used exploration and exploitation strategies in reinforcement learning. In this strategy, when making decisions, the agent explores with a probability of ε (epsilon), randomly selecting an action to discover a new state or update its understanding of the current state; and exploits with a probability of 1-ε, selecting the currently best known action. The Q-value in a Q-network is the expected reward of the agent taking a specific action in a given state.

[0043] Specifically, the "ε-Greedy" strategy is used to randomly select actions with a probability of ε, explore unknown states in the environment, and discover potential high-reward paths; the action with the highest Q value in the current state is selected through the Q table / Q network with a probability of 1-ε, and the learned knowledge is used to obtain more rewards. The ε-Greedy strategy is used to select action a, including:

[0044] Generate a random number rand, where the random number rand is between 0 and 1; if rand is less than the training parameter ε, input the state s into the evaluation network and run it to generate the output layer vector Get the output layer vector The subscript corresponding to the maximum value in the , takes the action corresponding to the subscript as the action a selected according to the deep Q network; if rand is greater than the training parameter ε, randomly select an action as the action a selected according to the deep Q network.

[0045] In specific implementation, in order to reduce the unloading cost O cost As the optimization target (which includes the total service delay t total and total energy consumption Etotal ), and adjust the tendency of the target through the coefficient, and calculate the task offloading cost O required for the offloading solution corresponding to action a through the following steps cost :

[0046] The transmission delay t of the task T , queuing delay t Q and execution delay t e Calculate the total delay t total , where t total =max(t T ,t Q 0+t e ; The transmission power of the necessary process signal T1 Transmission delay and the signal transmission power of the optional process T2 Transmission delay Calculate the total energy consumption E total ,in, According to the total delay t total and total energy consumption E total Calculate the minimum task offloading cost O cost ,in, N is the total number of all task offloading requests sent by all users in the time period, i is the i-th task offloading request, α is the delay coefficient, and 1-α is the energy consumption coefficient.

[0047] Specifically, in the edge computing scenario of the embodiment of the present invention, the offloading cost O cost As the optimization target, it includes the total service delay t total and total energy consumption E tital , and adjust the tendency of the target through the coefficient.

[0048] Total delay t total Including the task transmission delay t T , queuing delay t Q and execution delay t e , since the task transmission process and queuing process can be carried out simultaneously, the total delay can be expressed as t total =max(t T ,t Q )+t e .

[0049] Transmission delay t T It refers to the time it takes for task data to be transmitted from the user device to the task execution location. When the task is not unloaded, t T= 0. When offloading to the edge or cloud, the transmission delay consists of two parts: the task data is first transmitted from the user device to the base station near the user, and then the task data is transmitted between base stations based on the offloading location. Therefore, the process of offloading to the edge server includes the required process T1 and the optional process T2 (process T2 is not required when offloading to the local edge server). The transmission delay of process T1 It is the ratio of task data volume to transmission rate, where d i is the data size of the task, The channel rate at which the user transmits data to the corresponding base station (local base station) serving the user; the transmission delay of process T2 When m1=m2, it is offloaded to the local edge server. in, The channel rate at which the corresponding base station serving the user (local base station) transmits data to the base station performing the task.

[0050] Queuing delay t Q It can be expressed as the sum of the execution time of the previous tasks in the unloading position task accumulation queue, from which the running time of the currently being processed task needs to be subtracted, so the queuing delay Execution delay Among them, t e is the ratio of the task workload (in terms of the number of CPU cycles required) to the CPU frequency, ω i is the workload of the task, f i The CPU frequency of the user device.

[0051] Total energy consumption E total Including transmission energy consumption E T and execution energy consumption E e ,When the task is not offloaded, there is no transmission process,E total =E e =cω i f i 2 The energy consumption of task offloading includes transmission energy consumption and execution energy consumption. When the task is not offloaded, the task data does not need to be transmitted to the edge node or cloud server, so there is no transmission process. When offloading to the edge or to the cloud, there is transmission energy consumption. According to the experiment, E T >>E e , so the main calculation is transmission energy consumption); when offloading to the edge or cloud, the energy consumed by transmitting data dominates, in is the signal transmission power of each transmission process.

[0052] In practice, in order to acquire knowledge from the memory bank and train the parameters of the evaluation network to learn from previous experience, the following steps are performed to transfer the memory in the memory bank to the target network, obtain the maximum value of the output layer vector of the target network, and construct the error vector based on the maximum value:

[0053] Extract a set of memories M from the memory bank, which contains multiple independent memories m i ={s,a,r,s′}, where s is the state, s′ is the next state of state s, r is the reward obtained by unloading the task, and m i Unload the corresponding memory for the i-th task; i The state s in the evaluation network is passed to the evaluation network, and the memory m i The index subscript corresponding to the action a in the evaluation network is taken out from the output layer vector of the evaluation network, where Q(s,a) is the long-term expected return of taking action a in state s; the memory m i The next state s′ in the target network is passed to the target network, and the maximum value of the output layer vector of the target network is used as the maximum value of the output layer vector maxQ(s′); the Q estimate Q is calculated by the maximum value of the output layer vector maxQ(s′) est and Q actual value Q real , through the Q estimate Q est and Q actual value Q real Calculate the error e a ; Through error e a Construct error vector in

[0054] Specifically, Q(s,a) is the concept of Q-Learning, which indicates the quality of executing action a in state s.

[0055] In the specific implementation, the following steps are performed to calculate the Q estimate Q by the maximum value of the output layer vector maxQ(s′): est and Q actual value Q real , through the Q estimate Q est and Q actual value Q real Calculate the error e a :

[0056] Calculate the Q estimate Q est , where the Q estimate Q est =Q(s,a); calculate the actual value of Q real , where Q is the actual value Q real = r + γ × maxQ (s ′), r is the reward obtained by action a in state s, γ is the attenuation value of future rewards in the training parameter; through the Q estimate Q estand Q Reality Q real The error e is calculated a , where the error e a =α×(Q real -Q est ), α is the learning rate in the training parameters.

[0057] In specific implementation, in order to backpropagate the error and update the weights of the neural network, the following steps are performed to update the weight parameters of the evaluation network through the error vector and copy the training parameters of the evaluation network to the target network:

[0058] The error vector e is passed into the evaluation network for back propagation, and the weight parameters of the evaluation network are updated; the round interval is set, and when the training of the evaluation network reaches the round interval, the weight parameters of the evaluation network are copied to the target network.

[0059] Specifically, the error vector e is passed to the evaluation network for backpropagation, updating the evaluation network's weights. Every time training reaches a certain number of rounds, the evaluation network parameters are copied to the target network. Because the target network's role is to calculate the maximum value of the output layer vector, maxQ(s′), the target network is only copied from the evaluation network after a certain number of rounds.

[0060] like Figure 3 As shown, the episode on the horizontal axis represents the training round (the number of training rounds is set to 200 in the embodiment of the present invention). After 50 rounds of training, the total unloading cost gradually converges.

[0061] like Figure 4 FIG. 4 shows a comparison of the total offloading cost after using the task offloading method according to an embodiment of the present invention and using the greedy algorithm to offload tasks.

[0062] In this embodiment, a computer device is provided, such as Figure 5 As shown, it includes a memory 501, a processor 502 and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned task offloading methods applied to edge computing is implemented.

[0063] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0064] In this embodiment, a computer-readable storage medium is provided, which stores a computer program for executing any of the above-mentioned task offloading methods applied to edge computing.

[0065] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include transitory media such as modulated data signals and carrier waves.

[0066] Based on the same inventive concept, an embodiment of the present invention further provides a task offloading device for edge computing, as described in the following embodiments. Since the principle of solving the problem by the task offloading device for edge computing is similar to that of the task offloading method for edge computing, the implementation of the task offloading device for edge computing can refer to the implementation of the task offloading method for edge computing, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements the predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0067] Figure 6 This is a structural block diagram of a task offloading device applied to edge computing according to an embodiment of the present invention. Figure 6 As shown, it includes: a baseline data acquisition module 601, an initialization module 602, an offloading cost calculation module 603, a network update module 604 and a final strategy selection module 605. The structure is described below.

[0068] The baseline data acquisition module 601 is used to pre-unload each task in the unloading task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data B cost ;

[0069] Initialization module 602, used to initialize the memory bank, the evaluation network and the target network, and to initialize and set the weight parameters and training parameters of the evaluation network;

[0070] The calculation module 603 is used to set a training round. In each training round, the evaluation network is used to unload each task in the unloading task queue. cost Calculate the task offloading cost O required for each task to be offloaded cost , and obtain the unloaded memory, save the memory to the memory bank, and unload the task cost O corresponding to all tasks cost Sum and generate the total offloading cost O corresponding to each training round cost ;

[0071] The network update module 604 is used to transfer the memory in the memory bank to the target network, obtain the maximum value of the value in the output layer vector of the target network, construct an error vector based on the maximum value, update the weight parameters of the evaluation network based on the error vector, and copy the training parameters of the evaluation network to the target network;

[0072] The final strategy selection module 605 is used to select the total unloading cost O in all training rounds. cost The lowest training round is taken as the final task offloading solution.

[0073] In one embodiment, the baseline data acquisition module includes:

[0074] Set P a The acquisition unit is used to obtain the CPU frequency of all locations where the task is allowed to be unloaded and the current length of the unloading location task accumulation queue, and add all locations to the location set P a ;

[0075] Collection classification unit, used to traverse the location set P a , placing the unloading locations whose unloading location task accumulation queue is empty into the first set P1, and placing the unloading locations whose unloading location task accumulation queue is not empty into the second set P2;

[0076] a first greedy result obtaining unit, configured to, if the first set P1 is not empty, search for a first unloading position with the highest CPU frequency in the first set P1, and use the first unloading position as a result of greedy unloading;

[0077] A second greedy result obtaining unit is configured to, if the first set P1 is empty, search for a second unloading position with the shortest unloading position task accumulation queue in the first set P2, and use the second unloading position as the result of greedy unloading;

[0078] The baseline data acquisition unit is used to perform the task offloading operation according to the result of greedy offloading, record the offloading cost of each task during greedy offloading, and use the offloading cost as the baseline data B of the deep reinforcement learning algorithm. cost .

[0079] In one embodiment, the module for calculating the uninstallation cost includes:

[0080] Initialization queue unit, used to initialize the unloading location task accumulation queue Q of each edge node i , define state s={O cost ,N r}, where i is the sequence number of the unloading location task accumulation queue, O cost is the task offloading cost, N r The total number of tasks waiting to be planned for unloading locations;

[0081] The cost calculation unit is used to select the action a for executing the offloading operation of the task through the ε-Greedy strategy, execute the action a, and calculate the task offloading cost O required for executing the offloading solution corresponding to the action a. cost ;

[0082] A state construction unit, used to construct the next state s′ according to the state s;

[0083] The reward calculation unit is used to calculate the reward r obtained by action a in state s through the reward function R, where the reward B cost is the baseline data;

[0084] The memory bank generation unit is used to generate memory (s, a, r, s′) according to state s, action a, next state s′ and reward r, and save the memory (s, a, r, s′) into the memory bank.

[0085] In one embodiment, the cost calculation unit is used to calculate the transmission delay t of the task. T , queuing delay t Q and execution delay t e Calculate the total delay t total , where t total =max(t T ,t Q )+t e ; The transmission power of the necessary process signal T1 Transmission delay and the signal transmission power of the optional process T2 Transmission delay Calculate the total energy consumption E total ,in, According to the total delay t total and total energy consumption E total Calculate the minimum task offloading cost O cost ,in, N is the total number of all task offloading requests sent by all users in the time period, i is the i-th task offloading request, α is the delay coefficient, and 1-α is the energy consumption coefficient.

[0086] In one embodiment, the baseline data acquisition module includes:

[0087] The memory extraction unit is used to extract a set of memories M from the memory bank. The memory M contains multiple independent memories m i ={s,a,r,s′}, where s is the state, s′ is the next state of state s, r is the reward obtained by unloading the task, and m i Unload the corresponding memory for the i-th task;

[0088] The reward acquisition unit is used to store the memory m i The state s in the evaluation network is passed to the evaluation network, and the memory m i The index subscript corresponding to action a in the evaluation network is taken out from the output layer vector of the evaluation network, where Q(s,a) is the long-term expected return of taking action a in state s.

[0089] The output layer vector maximum acquisition unit is used to store the memory m i The next state s′ in the target network is passed to the target network, and the maximum value of the output layer vector of the target network is used as the maximum value of the output layer vector maxQ(s′);

[0090] Error calculation unit, used to calculate the Q estimate Q by the maximum value of the output layer vector maxQ(s′) est and Q actual value Q real , through the Q estimate Q est and Q actual value Q real Calculate the error e a ;

[0091] Error vector building unit, used to pass the error e a Construct error vector in

[0092] In one embodiment, the error calculation unit is used to calculate the Q estimate Q est , where the Q estimate Q est =Q(s,a); calculate the actual value of Q real , where Q is the actual value Q real = r + γ × maxQ (s ′), r is the reward obtained by action a in state s, γ is the attenuation value of future rewards in the training parameter; through the Q estimate Q est and Q Reality Q real The error e is calculated a , where the error ea =α×(Q real -Q est ), α is the learning rate in the training parameters.

[0093] In one embodiment, the network update module includes:

[0094] The evaluation network parameter update unit is used to pass the error vector e into the evaluation network for back propagation and update the weight parameters of the evaluation network;

[0095] The target network parameter update unit is used to set the round interval. When the training of the evaluation network reaches the round interval, the weight parameters of the evaluation network are copied to the target network.

[0096] The embodiments of the present invention achieve the following technical effects:

[0097] In the mobile edge computing scenario, the task offloading method of the embodiment of the present invention can achieve a better task offloading strategy, optimize service delay and energy consumption, and improve the stability of the edge system; with the goal of minimizing the task offloading cost, the service delay and energy consumption are jointly optimized, and the computing offloading requirements of user tasks in mobile edge computing are realized under the constraints of edge node computing power, network environment and other restrictions; the embodiment of the present invention proposes a resource allocation method in the edge computing scenario based on deep reinforcement learning, which reduces the service delay and the energy consumption of the edge system by planning the task offloading location; the test results show that the resource allocation method proposed in the present invention can converge quickly within a limited training round and control the task offloading cost within a low range.

[0098] Obviously, those skilled in the art should understand that the various modules or steps of the above-mentioned embodiments of the present invention can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices. Alternatively, they can be implemented using program code executable by the computing device, so that they can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into separate integrated circuit modules, or multiple modules or steps can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0099] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A task offloading method applied to edge computing, characterized in that: include: Pre-unload each task in the unload task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data ; Initializing the memory bank, the evaluation network, and the target network, and initializing and setting the weight parameters of the evaluation network and the training parameters of the evaluation network; Set a training round, in each training round, use the evaluation network to unload each task in the unloading task queue, and use the baseline data Calculate the task offloading cost required for each task to be offloaded , and obtain the memory of the uninstallation, save the memory to the memory bank, and calculate the task uninstallation cost corresponding to all the tasks Sum up and generate the total offloading cost corresponding to each training round ; Use the evaluation network to unload each task in the unloading task queue, and use the baseline data Calculate the task offloading cost required for each task to be offloaded , generating an uninstalled memory, and saving the memory to the memory bank, comprising: Initialize the unloading location task accumulation queue of each edge node , define the state ,in, The sequence number of the accumulation queue for the unloading location task, Offloading cost for tasks, is the total number of tasks waiting for the planned unloading location; the action for performing the unloading operation of the task is selected by the ε-Greedy strategy , perform the action , calculate and perform the action The task offloading cost required by the corresponding offloading solution ; According to the state Construct the next state ; Through the reward function , calculate the action in state s Rewards received , among which, rewards , is the baseline data; according to the status 、The action , the next state and rewards Generate Memory , the memory saving to the memory bank; Calculate the task offloading cost required to execute the offloading solution corresponding to action a ,include: Transmission delay through tasks , queuing delay and execution delay Calculate the total delay ,in, ; Through the necessary process signal Transmit power , transmission delay and optional process Signal transmission power , transmission delay Calculate the total energy consumption ,in, According to the total delay and the total energy consumption Calculate the minimum task offloading cost ,in, , N is the total number of all task offloading requests sent by all users in the time period, i is the i-th task offloading request, is the delay coefficient, is the energy consumption coefficient; Transferring the memory in the memory bank to the target network, obtaining the maximum value of the numerical value in the output layer vector of the target network, constructing an error vector based on the maximum value, updating the weight parameters of the evaluation network based on the error vector, and copying the training parameters of the evaluation network to the target network; The total offloading cost of all the training rounds is The lowest training round is taken as the final task offloading solution.

2. The task offloading method for edge computing according to claim 1, wherein: Pre-unload each task in the unload task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data ,include: Get the CPU frequency of all locations where the task is allowed to be offloaded and the current length of the offload location task accumulation queue, and add all the locations to the location set ; Traverse the location set , place the unloading location with an empty unloading location task accumulation queue in the first set In the process, the unloading location whose task accumulation queue is not empty is placed in the second set middle; If the first set is not empty, then in the first set Find the first unloading position with the highest CPU frequency, and use the first unloading position as the result of greedy unloading; If the first set is empty, then in the first set Searching for the second unloading position with the shortest task accumulation queue in the unloading position, and taking the second unloading position as the result of greedy unloading; Execute the task unloading operation according to the result of the greedy unloading, record the unloading cost of each task during the greedy unloading, and use the unloading cost as the baseline data of the deep reinforcement learning algorithm .

3. The task offloading method for edge computing according to claim 1, wherein: The memory in the memory bank is transferred to the target network, the maximum value of the value in the output layer vector of the target network is obtained, and the error vector is constructed by the maximum value, including: Extract a set of memories from the memory bank , the memory Contains multiple independent memories ,in, For status, Status The next state, The reward for uninstalling the task. Unload the corresponding memory for the i-th task; The memory Status in Enter the evaluation network, according to the memory Actions in The corresponding index subscript is taken out from the output layer vector of the evaluation network. ,in, For the state The long-term expected reward of taking action a under The memory The next state in The target network is passed in, and the maximum value of the output layer vector of the target network is used as the maximum value of the output layer vector ; The maximum value of the vector through the output layer Calculate the Q estimate Q est and Q actual value Q real , through the Q estimate Q est and the Q reality value Q real Calculated error ; Through the error Construct error vector ,in .

4. The task offloading method for edge computing according to claim 3, wherein: The maximum value of the vector through the output layer Calculate the Q estimate Q est and Q actual value Q real , through the Q estimate Q est and the Q reality value Q real Calculated error ,include: Calculating Q estimates , where the Q estimate ; Calculate the Q actual value , where Q is the actual value , To act in state s The rewards received, is the decay value of future rewards in the training parameters; The Q estimate and the Q reality The error calculated , where the error , is the learning rate in the training parameters.

5. The task offloading method for edge computing according to claim 1, wherein: Updating the weight parameters of the evaluation network using the error vector and copying the training parameters of the evaluation network to the target network include: The error vector Passing the evaluation network into the evaluation network for back propagation and updating the weight parameters of the evaluation network; A round interval is set, and when the training of the evaluation network reaches the round interval, the weight parameters of the evaluation network are copied to the target network.

6. A task offloading device for edge computing, characterized in that: include: The baseline data acquisition module is used to pre-unload each task in the unloading task queue, obtain the unloading cost of each task, and use the unloading cost as the baseline data. ; An initialization module is used to initialize the memory bank, the evaluation network and the target network, and to initialize and set the weight parameters of the evaluation network and the training parameters of the evaluation network; The module for calculating the unloading cost is used to set a training round. In each training round, the evaluation network is used to unload each task in the unloading task queue. Calculate the task offloading cost required for each task to be offloaded , and obtain the memory of the uninstallation, save the memory to the memory bank, and calculate the task uninstallation cost corresponding to all the tasks Sum up and generate the total offloading cost corresponding to each training round ; Module for calculating unloading costs, including: Initialization queue unit, used to initialize the unloading location task accumulation queue of each edge node , define the state ,in, The sequence number of the queue for the unloading location task accumulation, Offloading cost for tasks, The total number of tasks waiting to be planned for unloading locations; Cost calculation unit, used to select the action for performing the offloading operation of the task through the ε-Greedy strategy , perform the action , calculate execution action The task offloading cost required by the corresponding offloading solution ; State construction unit, used to Construct the next state ; Reward calculation unit, used to calculate the reward function , calculate the action in state s Rewards received , among which, rewards , is the baseline data; Memory bank generation unit, used to generate memory according to the state ,action , next state and rewards Generate Memory , the memory Save to memory; Cost calculation unit for transmission delay through tasks , queuing delay and execution delay Calculate the total delay ,in, ; Through the necessary process signal Transmit power , transmission delay and optional process Signal transmission power , transmission delay Calculate the total energy consumption ,in, ; According to the total delay and total energy consumption Calculate the minimum task offloading cost ,in, , N is the total number of all task offloading requests sent by all users in the time period, i is the i-th task offloading request, is the delay coefficient, is the energy consumption coefficient; a network update module, configured to transfer the memory in the memory bank to the target network, obtain the maximum value of the numerical value in the output layer vector of the target network, construct an error vector based on the maximum value, update the weight parameters of the evaluation network based on the error vector, and copy the training parameters of the evaluation network to the target network; The final strategy selection module is used to select the total offloading cost in all the training rounds The lowest training round is taken as the final task offloading solution.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the task offloading method for edge computing according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program for executing the task offloading method for edge computing according to any one of claims 1 to 5.

Images