A method for independent task offloading based on local and edge collaborative caching

Abstract

This paper discloses an independent task offloading method based on local and edge collaborative caching, introduces the concept of service caching, and proposes a task offloading and resource allocation method based on service caching. First, taking the weighted sum of energy consumption and time delay as the optimization objective, the task offloading, service caching, and resource allocation problems are formulated as mixed integer nonlinear programming problems, and corresponding communication models, computational models, and cache models are established. Second, the above problems are approximated as a Markov decision process, and a deep deterministic policy gradient algorithm is proposed to solve the task offloading, service caching, and resource allocation problems, ultimately reducing the energy consumption and time delay caused by the task offloading process.

Description

Technical Field

[0001] The present invention relates to big data and edge computing technologies, and in particular to an independent task offloading method based on local and edge collaborative caching. Background Art

[0002] In recent years, the Internet of Things (IoT) has been a major revolution in information and communications technology. The IoT extends internet connectivity from traditional smart devices to transportation and industrial manufacturing, inevitably requiring extensive computing and generating massive amounts of data. According to a global data traffic forecast released by the International Data Corporation (IDC), global data traffic is projected to reach 163 zettabytes by 2025. To reduce the cost of processing this massive amount of data, one effective approach is cloud computing. This approach uses the cloud to distribute the massive data processing process into numerous smaller programs. These programs are then processed across a network of servers, ultimately returning the results to users. This massive amount of data can be processed in seconds, enabling powerful network services. However, with the rapid adoption of emerging applications, centralized cloud-based IoT solutions are facing challenges, such as high latency due to the long distance between local devices and the cloud, as well as energy consumption issues with backhaul links between base stations and the core network. Furthermore, a growing number of business scenarios are demanding the real-time, security, and privacy of massive data analysis and processing. Therefore, single-cloud computing is no longer sufficient to meet future development needs. This is why the concept of edge computing has been proposed.

[0003] Edge computing is the provision of intelligent services at the edge of the network, close to local end-devices, through a distributed, open platform that integrates storage, computing, and other capabilities. Edge computing analyzes data collected from end-devices directly on local devices or networks near where the data is generated, eliminating the need to transmit the data to cloud-based data processing centers. Compared to local servers, edge computing offers more abundant computing resources and greater computing and storage capabilities. Furthermore, compared to cloud computing centers, edge computing is closer to users and end-devices, allowing for more flexible geographical distribution. Uploading massive amounts of data to edge computing modules for processing reduces data transmission latency from local to cloud computing centers, thereby better meeting the requirements of low-latency tasks. Therefore, edge computing is a promising approach to addressing current challenges in the development of the Industrial Internet of Things (IIoT). However, it also presents numerous challenges, including the implementation of appropriate strategies for scheduling tasks to appropriate locations for computation. Furthermore, task scheduling in different scenarios requires different energy consumption and latency metrics. In recent years, research has focused not only on scheduling independent and dependent tasks with the goal of low latency and energy consumption, but also on caching strategies for edge servers. Therefore, how to coordinate task offloading and caching to achieve the goal of reducing latency and energy consumption is the focus and difficulty of current edge computing. Summary of the Invention

[0004] The purpose of the present invention is to provide an independent task offloading method based on local and edge collaborative caching to solve the problem of repeated calculation of the same task wasting time, energy and other resources without considering the caching strategy.

[0005] The technical solution to achieve the purpose of the present invention is: a method for independent task offloading based on local and edge collaborative caching, comprising the following steps:

[0006] Step S1: Determine the edge computing (MEC) architecture, including local devices, wireless access points (APs), and edge servers (ESs), where local devices are connected to the APs and ESs.

[0007] Step S2: Establish the communication model, computing model, and cache model of the MEC architecture, determine the latency and energy consumption of a single local device, establish an equation with the weighted sum of energy consumption and latency as the optimization target, and determine the constraints;

[0008] Step S3: Approximate the optimization problem proposed in S2 as a Markov decision process. The remaining computing resources of the ES, the remaining bandwidth resources, the remaining cache capacity of the local device and the ES, the weighted sum of energy consumption and latency are designed as the state vector. The task offloading strategy, cache strategy, computing resource and bandwidth resource allocation strategy are designed as the action vector. The equation with the weighted sum of energy consumption and latency as the optimization objective is used as the reward. This is transformed into solving the optimal task offloading strategy problem and the cache strategy problem.

[0009] Step S4: Based on the DDPG (Deep Deterministic Policy Gradient) algorithm, the policy value network (Actor), action value network (Critic), and experience replay buffer of the deep reinforcement learning model are initialized. The Actor randomly performs action A based on the current state S, and obtains reward R and a new state S'. At the same time, the Critic updates the Actor based on the action A and the current state S performed by the Actor. The historical state S, next state S', action A, reward R, etc. after the interaction are stored as sample transmission tuples in the experience replay buffer. Tuples are selected from the experience replay buffer and the above process is repeated to train the Actor and Critic.

[0010] Step S5: Obtain the system state of the current time slot, input the system state into the trained deep reinforcement learning model, and use the trained deep reinforcement learning model to obtain the optimal caching strategy and optimal task offloading strategy for each time slot.

[0011] Furthermore, in step S1, the MEC architecture is determined, including local devices, wireless access points APs, and edge servers ESs. The specific method is as follows:

[0012] The MEC architecture includes: a set of n local devices UE = {UE1, UE2, UE3, ... UE n}, a set of K wireless access points AP = {AP1, AP2, ... AP K Each local device is connected to the ES through an AP. The local device and the ES have both computing and caching capabilities.

[0013] Assume that the system adopts a discrete time model t∈{0,1,2,...,T}, and for any local device UE in the same time slot t i Only one independent and inseparable task will be generated That It is represented by a four-tuple in Indicates the data size of the task, Indicates the cache size of the required service, Indicates the number of CPU clock cycles required to complete the task. Indicates the maximum tolerable delay of the task;

[0014] ES is represented by a two-tuple {F, V}, where F represents the computing resources of ES and V represents the total cache capacity of ES.

[0015] Furthermore, in step S2, the communication model, computing model, and cache model of the architecture are established. The specific method is as follows:

[0016] (1) Communication model

[0017] The local device accesses the ES through the AP and then offloads the task to the ES. When multiple local devices choose to send tasks to the ES, the frequency division multiple access technology is used to enable each local device to be allocated a certain channel resource. Then the local device UE i The transmission rate between ES and ES in a certain time slot t for:

[0018]

[0019] Among them B i Indicates that ES is the local device UE i The allocated bandwidth, the total bandwidth is B total , assuming that the bandwidth allocated to each local device is uniform, P i Indicates the local device UE i The transmission power, h i Indicates the local device UE i The channel gain, σ 2 represents Gaussian white noise, then the transmission delay from the local device to the ES in time slot t is for:

[0020]

[0021] Because the data size of the task after ES processing is smaller than the data before processing, and the downlink rate from ES to the local device is higher than the uplink rate from the local device to ES, the energy consumption of the task upload in time slot t is not considered. for:

[0022]

[0023] (2) Computational model

[0024] Each independent task generated by a local device is either processed locally or completely offloaded to ES for calculation, defining a binary indicator variable To represent the offloading decision of the local device, Indicates that the local execution task is selected in time slot t, Indicates the local device UE i The number of CPU clock cycles required to complete the task in time slot t, f i local Indicates the local device UE i The computing power of UE i The generated task is executed locally at the time Expressed as:

[0025]

[0026] κ local The energy consumed by the CPU during one rotation is:

[0027]

[0028] Indicates that the local device UE at time slot t i Choose to offload to ES for processing. The total computing capacity of ES is f ES , assigned to the local device UE i The computing resources are f i ES , then the computation delay of ES for:

[0029]

[0030] Therefore, the local device UE is obtained i The total transmission delay to ES is:

[0031]

[0032] Local device UE i Energy consumption generated by transmitting tasks to ES Including the energy consumption during transmission Energy consumption generated by local device standby when calculating ES

[0033]

[0034] (3) Cache Model

[0035] To further reduce latency and energy consumption and conserve limited computing resources on the ES, some hot content results are stored on the ES and local devices. Before executing a computing task, the local device will first communicate with the ES and other local devices in the area to query their cache information, and then determine whether to execute the task locally or offload the calculation to the ES. If the calculation is performed on the local device, the local device will directly return the result if it already has the cached result, or cache it on another local device and transmit it to the local device through the ES. Otherwise, the local device will perform the calculation itself. If the calculation is offloaded to the ES, the calculation result of the task is already stored in the ES cache and the result is returned directly. Otherwise, the calculation is performed on the ES.

[0036] In addition, considering the cache strategy based on content popularity, the data required for the computing task is stored according to its highest popularity until the storage capacity reaches the upper limit. The probability of the local device requesting popular data follows the Zipf distribution in the statistics, then the local device UE i The probability of requesting ES to compute task v is as follows:

[0037]

[0038] e>0 is the Zipf distribution impact factor, and V represents all content;

[0039] definition is a binary cache variable, ES stands for edge server, when Indicates that the ES caches the local device UE in time slot t i The calculation result of a task K, It means that the calculation result of task K is not cached, δ i Indicates the local device UE i The capacity of the cache in ES, assuming the capacity of ES is M ES , the usage of ES cache capacity by n local devices cannot exceed the maximum cache capacity of ES:

[0040]

[0041] definition It is a binary cache variable, UD represents the local device, Indicates the local device UE i The calculation results of task K are cached, and Indicates the local device UE i The calculation results of task K are not cached, ε i Indicates that it is cached in the local device UE i Assume that the total cache capacity of all local devices is M UD , cache capacity usage of n local devices The maximum cache capacity cannot be exceeded:

[0042]

[0043] Furthermore, in step S2, the latency and energy consumption of a single local device are determined, and an equation is established with the weighted sum of energy consumption and latency as the optimization objective, while also determining the constraints. The specific method is as follows:

[0044] Single local device UE i The delay in time slot t for:

[0045]

[0046] Similarly, a single local device UE i Energy consumption in time slot t for:

[0047]

[0048] The optimization goal is to minimize the weighted sum of energy consumption and latency, with the following constraints: tasks must be executed on either the local device or the ES; the task execution time cannot exceed the maximum tolerable latency; the channel bandwidth resources and the computing resources allocated by the ES cannot exceed the specified maximum values; and the cache capacity of the local device and ES cannot exceed the maximum capacity. The specific model is as follows:

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] Furthermore, in step S3, the optimization problem proposed in step S2 is approximated as a Markov decision process, which is transformed into solving the optimal task offloading strategy and caching strategy problem. The specific method is as follows:

[0059] The state in the Markov decision process reflects the environment of the network space. The state S at time t includes the following: the first considers the remaining computing resources of the ES, the second considers the remaining bandwidth resources, the third and fourth consider the remaining cache capacity of the ES and the local device, and the last considers the weighted sum of energy consumption and delay ψ t ,as follows:

[0060]

[0061] The action space is used to represent the computing offloading strategy, service caching strategy, computing resource and bandwidth resource allocation strategy, so the action vector A is designed as:

[0062]

[0063] After performing an action in a certain state, the agent will obtain feedback from the environment to obtain rewards, choose behaviors based on the rewards obtained, and continuously update strategies to maximize rewards and select optimal behaviors. Therefore, the reward function r t Set to the negative value of the optimization goal:

[0064]

[0065] Furthermore, in step S4, based on the DDPG algorithm, the policy value network Actor, the action value network Critic, and the experience replay buffer of the deep reinforcement learning model are initialized. The Actor performs random actions based on the current state to obtain rewards and new states. At the same time, the Critic updates the Actor based on the actions and current state of the Actor, and stores the historical state, future state, action, and reward after the interaction as sample transmission tuples in the experience replay buffer to train the Actor and Critic, where:

[0066] The deep reinforcement learning model is divided into two main parts: the policy value network Actor and the action value network Critic. Both the policy value network Actor and the action value network Critic contain a current network Eval and a target network Target. The specific training process is as follows:

[0067] Step S51: Initialize the experience playback buffer R and set its size to D;

[0068] Step S52: Randomly initialize the parameters θ and ω of the Eval network of the Actor and Critic, and assign the parameters θ' and ω' of the Target network of the Actor and Critic to θ and ω respectively;

[0069] Step S53: Actor based on state S tAnd the current strategy π and random noise N get action A t =π(S t ,θ)+N;

[0070] Step S54: Actor performs action A t , get reward R t and the next state S', the tuple (S t ,A t ,R t ,S') is stored in the experience replay pool R;

[0071] Step S55: Randomly select a transmission tuple from the experience replay buffer to calculate the Q value of the Critic's Eval network;

[0072] Step S56: Use the loss function to update the Eval network parameters of the Critic and Actor, and then update the Target network parameters of the Actor and Critic, and finally obtain the trained deep reinforcement learning model.

[0073] A system for offloading independent tasks based on local and edge collaborative caching realizes offloading independent tasks based on local and edge collaborative caching based on the independent task offloading method.

[0074] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, independent task offloading based on local and edge collaborative caching is implemented based on the independent task offloading method.

[0075] A computer-readable storage medium stores a computer program thereon. When the computer program is executed by a processor, the independent task offloading method described in the item is used to implement independent task offloading based on local and edge collaborative caching.

[0076] Compared with existing technologies, this invention offers the following significant advantages: 1) The caching technology considered can significantly reduce the cost of task offloading. 2) The invention weights latency and energy consumption as two optimization objectives and considers them comprehensively rather than separately. 3) The DDPG (Deep Deterministic Policy Gradient) algorithm employed can significantly reduce system overhead and significantly outperform other benchmark algorithms in terms of long-term energy consumption across various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 This is a diagram of the MEC architecture.

[0078] Figure 2 This is the DDPG structure diagram.

[0079] Figure 3 Schematic diagram of the performance comparison between A2C and DDPG.

[0080] Figure 4 A diagram showing the performance comparison between the two systems with and without caching capabilities. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0082] The present invention provides an independent task offloading method based on local and edge collaborative caching, comprising the following steps:

[0083] Step S1: Determine the MEC (Mobile Edge Computing) architecture;

[0084] like Figure 1 As shown, a set of n local devices UE = {UE1, UE2, UE3, ...UE n}、The set of K wireless access points (AP) is AP={AP1,AP2,...AP K} and a separate edge server (ES). Each local device is connected to the ES through an AP. The local device and the ES have not only computing capabilities but also caching capabilities.

[0085] Assume that the system adopts a discrete time model t∈{0,1,2,...,T}, and for any local device UE in the same time slot t i (UE i ∈UE) will only generate an independent and inseparable task That It can be represented by a four-tuple in Indicates the data size of the task, Indicates the cache size of the required service, Indicates the number of CPU clock cycles required to complete the task. Indicates the maximum tolerable delay of the task. ES is represented by a tuple {F, V}, where F represents the computing resources of ES and V represents the total cache capacity of ES.

[0086] Step S2: Establishing the communication model, computing model, and cache model of the architecture;

[0087] (1) Communication model

[0088] The local device and ES are connected by a wireless link. The local device can access the ES through the wireless link and then offload the task to the ES. When multiple local devices choose to send tasks to the ES, the frequency division multiple access technology is used to enable each local device to be allocated a certain channel resource. Therefore, for the local device UE i The transmission rate between ES and ES in a certain time slot t for:

[0089]

[0090] Among them B i Indicates that ES is the local device UE i The allocated bandwidth, the total bandwidth is B total , assuming that the bandwidth allocated to each local device is uniform, P i Indicates the local device UE i The transmission power, h i Indicates the local device UE i The channel gain, σ 2 represents Gaussian white noise, then the transmission delay from the local device to the ES in time slot t is for:

[0091]

[0092] Because the data size of the task after ES processing is generally smaller than the data before processing, and the downlink rate from ES to the local device is much higher than the uplink rate from the local device to ES, the delay of downlink transmission is not considered. for:

[0093]

[0094] (2) Computational model

[0095] Each independent task generated by a local device is either processed locally or completely offloaded to ES for computation. Define a binary indicator variable To represent the offloading decision of the local device, Indicates that the local execution task is selected in time slot t, Indicates the local device UE i The number of CPU clock cycles required to complete the task in time slot t, f i local Indicates the local device UE i Therefore, for UE i The generated task is executed locally at the time Expressed as:

[0096]

[0097] κ local The energy consumed by the CPU during one rotation is:

[0098]

[0099] Indicates that the local device UE at time slot t i Choose to offload to ES for processing. The total computing capacity of ES is f ES , assigned to the local device UE i The computing resources are f i ES , we can get the computation delay in ES for:

[0100]

[0101] Therefore, the local device UE can be obtained i The total transmission delay to ES is:

[0102]

[0103] Local device UE i Energy consumption generated by transmitting tasks to ES Including the energy consumption during transmission Energy consumption generated by local device standby when calculating ES

[0104]

[0105] (3) Cache Model

[0106] To further reduce latency and energy consumption and conserve limited computing resources on the ES, some hot content results are stored on the ES and locally. Before executing a computing task, the local machine will first communicate with the ES and other local devices in the area to query their cache information, and then determine whether to execute the task locally or offload the calculation to the ES. If the calculation is performed locally, the result will be returned directly if the local cache already exists, or it will be cached on another local device and transferred to the local machine through the ES. Otherwise, the local machine will perform the calculation itself. If the calculation is offloaded to the ES, the result of the task can be returned directly if it is already stored in the ES cache. Otherwise, the calculation will be performed on the ES.

[0107] The present invention considers a cache strategy based on content popularity, which will store the data required by the computing task according to its highest popularity until the storage capacity reaches the upper limit. The probability of local devices requesting popular data follows the Zipf distribution in statistics. Usually, several of the most popular computing tasks will be requested by a large number of devices at different times. i The probability of requesting ES to compute task v is as follows:

[0108]

[0109] e>0 is the Zipf distribution impact factor, and V represents all content.

[0110] definition is a binary cache variable, ES stands for edge server, when Indicates that the ES caches the local device UE in time slot t i The calculation result of a task K, It means that the calculation result of task K is not cached, δ i Indicates the local device UE i The capacity of the cache in ES, assuming the capacity of ES is M ES ,The usage of ES cache capacity by n local devices cannot exceed the maximum cache capacity of ES:

[0111]

[0112] definition It is a binary cache variable, UD represents the local device, Indicates the local device UE i The calculation results of task K are cached, and Indicates the local device UE i The calculation results of task K are not cached, ε i Indicates that it is cached in the local device UE i Assume that the total cache capacity of all local devices is M UD , cache capacity usage of n local devices The maximum cache capacity cannot be exceeded:

[0113]

[0114] Step S3: Determine the delay and energy consumption of a single user, and establish an equation with the weighted sum of energy consumption and delay as the optimization target;

[0115] set up To represent the local device UE i The unloading decision at time slot t, where Indicates that the local device is selected to execute the task. Indicates that offloading to ES processing is selected, a single local device UE i The delay T in time slot t i t for:

[0116]

[0117] Similarly, a single local device UE i Energy consumption in time slot t for:

[0118]

[0119] The optimization goal is to minimize the weighted sum of energy consumption and delay, with the following constraints: tasks must be executed on either the local device or the ES; the execution time of the task cannot exceed the maximum tolerable delay; the channel bandwidth resources and the computing resources allocated by the ES cannot exceed the specified maximum; and the cache capacity of the local device and the ES cannot exceed the maximum capacity. Finally, from equations (12) and (13), we can obtain the equation for minimizing the weighted sum of energy consumption and delay of n local devices as follows:

[0120]

[0121]

[0122]

[0123]

[0124]

[0125]

[0126]

[0127]

[0128]

[0129] Step S4: approximate the optimization problem of step S3 to a Markov decision process, and transform it into solving the optimal task offloading strategy problem and the caching strategy problem, which specifically includes the following steps.

[0130] The state in the Markov decision process reflects the environment of the network space. In the system and environment considered in this method, the state S at time t includes the following: the first considers the remaining computing resources of the ES, the second considers the remaining bandwidth resources, the third and fourth consider the remaining cache capacity of the ES and the local device, and the last considers the weighted sum of energy consumption and delay ψt ,as follows:

[0131]

[0132] The action space is used to represent the computing offloading strategy, service caching strategy, computing resource and bandwidth resource allocation strategy, so the action vector is designed as:

[0133]

[0134] After performing an action in a certain state, the agent will receive feedback from the environment to obtain rewards, select behaviors based on the rewards, and continuously update the strategy to maximize the rewards and select the optimal behavior. Therefore, the reward function is set to the negative value of the optimization target:

[0135]

[0136] Step S5: Initialize the policy value network (Actor) and action value network (Critic) of the deep reinforcement learning model, as well as the experience replay buffer, and randomly select tuples from the experience replay pool to train the current (Eval) network and target (Target) network of the Actor and Critic.

[0137] Depend on Figure 2 The network structure diagram of the DDPG algorithm is shown. It consists of two main components: the actor and the critic. The actor is responsible for interacting with the environment to generate actions, while the critic is responsible for scoring the actor's actions to guide the actor's future action generation strategy. Both components consist of an Eval network and a Target network, for a total of four neural networks. Each neural network is configured as a fully connected network with an input layer, two hidden layers, and an output layer. The input and output layers have 64 neurons, and each hidden layer has 128 neurons. The Relu activation function is used, and the output layer uses the Sigmoid activation function. The environment begins with an initial state S, and before the start of the game, the experience replay buffer is initialized to 5000.

[0138] First, the actor's Eval network selects action A based on the current state S and a certain strategy π. In order to add some randomness to the learning process and increase the coverage of learning, DDPG adds a certain amount of noise N to the selected A, that is, the expression of the final action A that interacts with the environment:

[0139] A=π θ (S)+N (18)

[0140] Then the Actor interacts with the environment to generate the next state S' and reward R. The system will store the historical state S, future state S', action A, reward R, etc. after the interaction as sample transmission tuples in the experience replay buffer. The tuple size is set to 64. These tuples may be randomly selected and input into the Actor's Eval network for repeated training in the next iteration. At the same time, the Critic's Eval network will also calculate the Q value based on the initial state S in the environment and the action A made by the Actor, or obtain the tuple from the experience replay buffer to calculate the Q value. The Q value obtained by the Critic will promote the update of the Actor's Eval network parameters, and use the gradient loss function to update the Actor's Eval network. The m Q values ​​are accumulated and then averaged, expressed as J(θ):

[0141]

[0142] Q(S j ,A j ,θ) is the Q value calculated by the Eval network of the actor. DDPG uses a sliding average update to copy the parameters from the Eval network to the Target network. This is to mix the newly added elements and the old average value with a certain ratio τ to form a new average value. Therefore, the target network parameter θ' of the actor is updated as follows:

[0143] θ'=τθ+(1-τ)θ' (20)

[0144] The Critic's Target network parameters ω' are regularly copied and updated from the Critic's Eval network. Similarly,

[0145] ω'=τω+(1-τ)ω' (21)

[0146] The Actor's Target network selects action A' based on state S', and the Critic's Target network calculates the actual action value based on action A' and state S':

[0147] y j =R+γQ'(S',A',ω') (22)

[0148] Where Q(S', A', ω') represents the estimated Q value obtained by the Critic's Target network, S' is the next state the system enters after the Actor performs the action, A' represents the action selected by the Actor's Target network in the next state S', ω' is the Critic's Target network parameter, and γ represents the attenuation factor, which is set to 0.9; then the gradient loss function is used to update the Critic's Eval network, and the m differences are squared and accumulated and then averaged, expressed as L(ω):

[0149]

[0150] y j represents the actual action value, Q(S j ,A j ,ω) represents the Q value calculated by the Critic’s Eval network.

[0151] During neural network training, the learning rate is set to 0.002, and the Eval networks of Actor and Critic are continuously trained and their network parameters are updated until the number of iterations reaches the maximum iteration value of 500.

[0152] Step S6: Obtain the system status of the current time slot and input the system status into the trained deep reinforcement learning model. Using the trained deep reinforcement learning model, the optimal caching strategy and task offloading strategy can be obtained.

[0153] In order to verify the effectiveness and superiority of the DDPG algorithm for task offloading, it is compared with the A2C (AdvantageActorCritic) algorithm. The final results are as follows: Figure 3 As shown; In addition, the impact of cache on the performance of the system model is verified by comparing the performance of the DDPG algorithm without using the cache function and the performance of the DDPG algorithm with the cache function. The results are shown in Figure 4 shown.

[0154] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for independent task offloading based on local and edge collaborative caching, comprising the following steps: Step S1: Determine an edge computing architecture, including a local device, a wireless access point AP, and an edge server ES, wherein the local device is connected to the ES via the AP; Step S2: Establish the communication model, computing model, and cache model of the MEC architecture, determine the latency and energy consumption of a single local device, establish an equation with the weighted sum of energy consumption and latency as the optimization target, and determine the constraints; Step S3: Approximate the optimization problem proposed in S2 as a Markov decision process. The remaining computing resources of the ES, the remaining bandwidth resources, the remaining cache capacity of the local device and the ES, the weighted sum of energy consumption and latency are designed as the state vector. The task offloading strategy, cache strategy, computing resource and bandwidth resource allocation strategy are designed as the action vector. The equation with the weighted sum of energy consumption and latency as the optimization objective is used as the reward. This is transformed into solving the optimal task offloading strategy problem and the cache strategy problem. Step S4: Based on the DDPG algorithm, the policy value network Actor, action value network Critic, and experience replay buffer of the deep reinforcement learning model are initialized. The Actor randomly performs action A based on the current state S and obtains reward R and a new state S'. At the same time, the Critic updates the Actor based on the action A and the current state S performed by the Actor, and stores the historical state S, next state S', action A, and reward R after the interaction as sample transmission tuples in the experience replay buffer. Tuples are selected from the experience replay buffer and the above process is repeated to train the Actor and Critic. Step S5: Obtain the system state of the current time slot, input the system state into the trained deep reinforcement learning model, and use the trained deep reinforcement learning model to obtain the optimal caching strategy and optimal task offloading strategy for each time slot; The cache model is as follows: To further reduce latency and energy consumption and conserve limited computing resources on the ES, some hot content results are stored on the ES and local devices. Before executing a computing task, the local device will first communicate with the ES and other local devices in the area to query their cache information, and then determine whether to execute the task locally or offload the calculation to the ES. If the calculation is performed on the local device, the local device will directly return the result if it already has the cached result, or cache it on another local device and transmit it to the local device through the ES. Otherwise, the local device will perform the calculation itself. If the calculation is offloaded to the ES, the calculation result of the task is already stored in the ES cache and the result is returned directly. Otherwise, the calculation is performed on the ES. In addition, considering the cache strategy based on content popularity, the data required for the computing task is stored according to its highest popularity until the storage capacity reaches the upper limit. The probability of the local device requesting popular data follows the Zipf distribution in the statistics, then the local device UE i The probability of requesting ES to compute task v is as follows: e>0 is the Zipf distribution impact factor, and V represents all content; definition is a binary cache variable, ES stands for edge server, when Indicates that the ES caches the local device UE in time slot t i The calculation result of a task K, It means that the calculation result of task K is not cached, δ i Indicates the local device UE i The capacity of the cache in ES, assuming the capacity of ES is M ES , the usage of ES cache capacity by n local devices cannot exceed the maximum cache capacity of ES: definition It is a binary cache variable, UD represents the local device, Indicates the local device UE i The calculation results of task K are cached, and Indicates the local device UE i The calculation results of task K are not cached, ε i Indicates that it is cached in the local device UE i Assume that the total cache capacity of all local devices is M UD , cache capacity usage of n local devices The maximum cache capacity cannot be exceeded:

2. The independent task offloading method based on local and edge collaborative caching according to claim 1 is characterized in that: Step S1: Determine the MEC architecture, including local devices, wireless access point APs, and edge servers ES. The specific method is as follows: The MEC architecture includes: a set of n local devices UE = {UE1, UE2, UE3, ... UE n }, a set of K wireless access points AP = {AP1, AP2, ... AP K Each local device is connected to the ES through an AP. The local device and the ES have both computing and caching capabilities. Assume that the system adopts a discrete time model t∈{0,1,2,...,T}, and for any local device UE in the same time slot t i Only one independent and inseparable task will be generated That It is represented by a four-tuple in Indicates the data size of the task, Indicates the cache size of the required service, Indicates the number of CPU clock cycles required to complete the task. Indicates the maximum tolerable delay of the task; ES is represented by a two-tuple {F, V}, where F represents the computing resources of ES and V represents the total cache capacity of ES.

3. The independent task offloading method based on local and edge collaborative caching according to claim 1 is characterized in that: Step S2: Establish the communication model, calculation model, and cache model of the architecture. The specific method is as follows: (1) Communication model The local device accesses the ES through the AP and then offloads the task to the ES. When multiple local devices choose to send tasks to the ES, the frequency division multiple access technology is used to enable each local device to be allocated a certain channel resource. Then the local device UE i The transmission rate between ES and ES in a certain time slot t for: Among them B i Indicates that ES is the local device UE i The allocated bandwidth, the total bandwidth is B total , assuming that the bandwidth allocated to each local device is uniform, P i Indicates the local device UE i The transmission power, h i Indicates the local device UE i The channel gain, σ 2 represents Gaussian white noise, then the transmission delay from the local device to the ES in time slot t is for: Because the data size of the task after ES processing is smaller than the data before processing, and the downlink rate from ES to the local device is higher than the uplink rate from the local device to ES, the energy consumption of the task upload in time slot t is not considered. for: (2) Computational model Each independent task generated by a local device is either processed locally or completely offloaded to ES for calculation, defining a binary indicator variable To represent the offloading decision of the local device, Indicates that the local execution task is selected in time slot t, Indicates the local device UE i The number of CPU clock cycles required to complete the task in time slot t, f i local Indicates the local device UE i The computing power of UE i The generated task is executed locally at the time Expressed as: κ local The energy consumed by the CPU during one rotation is: Indicates that the local device UE at time slot t i Choose to offload to ES for processing. The total computing capacity of ES is f ES , assigned to the local device UE i The computing resources are f i ES , then the computation delay of ES for: Therefore, the local device UE is obtained i The total transmission delay to ES is: Local device UE i Energy consumption generated by transmitting tasks to ES Including the energy consumption during transmission Energy consumption generated by local device standby when calculating ES 4. The independent task offloading method based on local and edge collaborative caching according to claim 3 is characterized in that: Step S2: Determine the latency and energy consumption of a single local device, establish an equation with the weighted sum of energy consumption and latency as the optimization objective, and determine the constraints. The specific method is as follows: Single local device UE i The delay T in time slot t i t for: Similarly, a single local device UE i Energy consumption in time slot t for: The optimization goal is to minimize the weighted sum of energy consumption and latency, with the following constraints: tasks must be executed on either the local device or the ES; the task execution time cannot exceed the maximum tolerable latency; the channel bandwidth resources and the computing resources allocated by the ES cannot exceed the specified maximum values; and the cache capacity of the local device and ES cannot exceed the maximum capacity. The specific model is as follows:

5. The independent task offloading method based on local and edge collaborative caching according to claim 4 is characterized in that: In step S3, the optimization problem proposed in step S2 is approximated as a Markov decision process, which is transformed into solving the optimal task offloading strategy and caching strategy problem. The specific method is as follows: The state in the Markov decision process reflects the environment of the network space. The state S at time t includes the following: the first considers the remaining computing resources of the ES, the second considers the remaining bandwidth resources, the third and fourth consider the remaining cache capacity of the ES and the local device, and the last considers the weighted sum of energy consumption and delay ψ t ,as follows: The action space is used to represent the computing offloading strategy, service caching strategy, computing resource and bandwidth resource allocation strategy, so the action vector A is designed as: After performing an action in a certain state, the agent will obtain feedback from the environment to obtain rewards, choose behaviors based on the rewards obtained, and continuously update strategies to maximize rewards and select optimal behaviors. Therefore, the reward function r t Set to the negative value of the optimization goal:

6. The independent task offloading method based on local and edge collaborative caching according to claim 4 is characterized in that: Step S4: Based on the DDPG algorithm, the policy value network Actor, action value network Critic, and experience replay buffer of the deep reinforcement learning model are initialized. The Actor randomly performs an action A based on the current state S to obtain a reward R and a new state S'. At the same time, the Critic updates the Actor based on the action A and the current state S performed by the Actor, and stores the historical state S, next state S', action A, and reward R after the interaction as sample transmission tuples in the experience replay buffer. Tuples are selected from the experience replay buffer and the above process is repeated to train the Actor and Critic. The deep reinforcement learning model is divided into two main parts, namely the policy value network Actor and the action value network Critic. Both the policy value network Actor and the action value network Critic contain a current network Eval and a target network Target. The specific training process is as follows: Step S51: Initialize the experience playback buffer R and set its size to D; Step S52: Randomly initialize the parameters θ and ω of the Eval network of the Actor and Critic, and assign the parameters θ' and ω' of the Target network of the Actor and Critic to θ and ω respectively; Step S53: Actor based on state S t And the current strategy π and random noise N get action A t =π(S t ,θ)+N; Step S54: Actor performs action A t , get reward R t and the next state S', the tuple (S t ,A t ,R t ,S') is stored in the experience replay pool R; Step S55: Randomly select a transmission tuple from the experience replay buffer to calculate the Q value of the Critic's Eval network; Step S56: Use the loss function to update the Eval network parameters of the Critic and Actor, and then update the Target network parameters of the Actor and Critic, and finally obtain the trained deep reinforcement learning model.

7. An independent task offloading system based on local and edge collaborative caching, characterized in that: Based on the independent task offloading method described in any one of claims 1 to 6, independent task offloading based on local and edge collaborative caching is implemented.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, independent task offloading based on local and edge collaborative caching is implemented based on the independent task offloading method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the independent task offloading method according to any one of claims 1 to 6 is used to implement independent task offloading based on local and edge collaborative caching.

Images