A decentralized adaptive computing offloading method based on DDPG algorithm
By adopting a decentralized adaptive computing and unloading method based on DDPG algorithm on the highway, using V2V communication and edge computing resources of the truck fleet system, the problems of insufficient computing resources and high latency on the highway are solved, and efficient Compute the unloading strategy to reduce energy consumption and task latency.
Patent Information
- Application Number
- CN202311371217.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-23
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2043-10-23
AI Technical Summary
In highway scenarios, traditional vehicle edge computing mode faces the problems of high computing latency and insufficient resources, especially the sparse deployment of RSU/BS and high cost. The dynamic topology of the vehicle network causes communication links to be disconnected, affecting the computing and offloading efficiency.
Using a decentralized adaptive computing and unloading method based on DDPG algorithm, the V2V communication between the head truck and member truck in the truck fleet system is used to optimize channel state using NOMA technology and AR model, and the intelligent driver model IDM optimizes fleet spacing. Combining the computing resources of DVFS and EC servers, dynamically adjust the power allocation of local computing and offload tasks to reduce energy consumption and latency.
The average energy consumption minimized and cache queue length minimized under the maximum power constraint, and an efficient unloading strategy that is independent and adaptable is reduced, which reduces the load and task calculation delay of the truck fleet system and improves the efficiency of computing resource utilization.
Smart Images

Figure CN117641288B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to a decentralized adaptive computing offloading method based on a DDPG algorithm. Background Art
[0002] With the development of smart cars and the Internet of Vehicles, various delay-sensitive and computationally intensive applications have emerged, such as autonomous driving, augmented reality, automatic navigation, and online games, which pose a challenge to vehicle resources. Fortunately, thanks to 5G communication technology, vehicle edge computing (VEC) can offload part or all of the computing tasks to physically proximal RSU / BS or other vehicles through vehicle-to-RSU (V2R) or vehicle-to-vehicle (V2V) communications based on IEEE 802.11p, providing vehicle users with more efficient computing and communication services, thereby solving the problem of insufficient vehicle resources. This computing paradigm has been widely used in urban traffic scenarios. However, in highway scenarios, the traditional VEC model faces some challenges:
[0003] (1) Deploying RSU / BS along highways is a non-negligible expense. The coverage area of RSU / BS on highways is wider than that of urban roads. If the offloaded tasks are only processed on the edge computing (EC) server of RSU / BS, the total computing delay will exceed the tolerable threshold.
[0004] (2) Most highways are built in remote areas, and the deployment of RSU / BS is relatively sparse, which cannot meet the computing offloading needs of vehicles on the highway.
[0005] (3) Vehicles travel quickly on highways, and the vehicle network has a highly dynamic topology. Frequent position switching between vehicles and RSU / BS usually causes the wireless communication link to be disconnected, thereby reducing the effective communication time and quality of experience (QoE) of the vehicle.
[0006] To address the above challenges, vehicle platooning is an effective solution. As one of the most important application scenarios of platooning technology, truck platooning can reduce energy consumption, improve logistics efficiency and driving safety. At present, the research on truck platooning mainly focuses on platoon control and coordination, and there is less research on the computational unloading scheme in truck platooning systems. Summary of the invention
[0007] In order to reduce the average energy consumption and task processing delay of a fleet system, the present invention provides a decentralized adaptive computing offloading method based on the DDPG algorithm.
[0008] A decentralized adaptive computing offloading method based on the DDPG algorithm of the present invention is used for a truck fleet system. The truck fleet system model and the communication model are:
[0009] The truck convoy consists of a lead truck PL and M member trucks PM, where PM is represented by: {1,2,...,m,...,M}; PL is equipped with a small BS with an edge computing server; each PM communicates with PL via V2V.
[0010] Applying the classic vehicle following model to the PM in the platoon: the intelligent driver model IDM, and assuming that the distance between any two adjacent trucks in the platoon system is equal and represented as d, the equilibrium spacing within the platoon is given by:
[0011]
[0012] Where d0 is the minimum distance within the convoy, t h is the ideal headway, v is the driving speed, v max is the maximum travel speed.
[0013] Assume that the length of each truck is d t , then the communication distance between member m and the lead vehicle is written as:
[0014] d m =m(d+d t )(2)
[0015] In the truck platoon system, NOMA technology is used for communication between PM and PL. Since multiple PMs upload task data synchronously on the same channel, there will be signal interference when unloading data; therefore, in the uplink at time t, the signal received by PL from PM m can be calculated as follows:
[0016]
[0017] is the transmit power of PM m, is the maximum transmit power, is the channel vector of PM m, x m (t) is a complex data signal with unit variance; y m The first part is the effective signal from the target PM, the second part is the interference signal from other PMs sharing the channel, and the third part is n(t)~CN(0,σ 2 ) is a variable with variance σ 2 The additive white Gaussian noise vector of is: For PM m, the AR model is used to characterize the channel state transition between time t and t+1:
[0018]
[0019] Among them, ρ mis the normalized correlation coefficient between time t and t+1 and is close to 1. e(t+1) is an error vector and also a m (t+1) uncorrelated complex Gaussian distribution, the channel vector of each PM is initialized to h m (0)~CN(0,h0(d ref / d m ) α ), h0 is a path loss constant, d ref is the reference distance, α is the path loss exponent.
[0020] Consider M PMs sharing the same channel in time t in descending order of their channels: h1(t)≥h2(t)≥…≥h m (t)≥…≥h M (t), the interference signal of PM m can be written as:
[0021]
[0022] Then the corresponding signal interference plus noise (SINR) of PM m at time t is:
[0023]
[0024] σ 2 is the Gaussian white noise power.
[0025] Therefore, the task data transmission rate between PM m and PL can be obtained according to the bandwidth B:
[0026] r m,o (t) = Blog2(1+ζ m (t)) (7)
[0027] Fleet system calculation model:
[0028] a m (t) quantifies the number of arrival task data of PM m during time t, assuming that at all times a m (t) are all independent and identically distributed, a m (t) is temporarily stored in the cache queue and then processed at the next time t+1; for each PM, it is assumed that the arrival rate of the computing task follows the Poisson distribution λ and the data size of the task follows the uniform distribution u~U(u min ,u max ), then a m (t) satisfies: E[a m (t)] = λ·u.
[0029] (1) Local computing model
[0030] For each PM, assuming that the task data is temporarily stored in the cache queue system in time order and then popped out of the cache queue and either computed locally or offloaded to the PL, the size of the task data computed locally for PM m can be written as:
[0031]
[0032] The CPU cycle frequency of PM m is scheduled by adjusting the chip voltage using DVFS technology. is the local computing power allocated to the CPU, is the maximum local computing power, κ m is the effective conversion capacitance parameter of the PM m processor, L m The number of CPU cycles required to execute the task data can be obtained through offline measurement.
[0033] The energy consumed by the local computation of PM m in time t can be written as:
[0034] E m,l (t) = τ0p m,l (t) (9)
[0035] (2) Calculation of unloading model
[0036] Due to insufficient computing resources of PM, part of the task data will be offloaded to PL and executed by the EC server. According to formula (8), the time required for the EC server to process the offloaded task data of PM m is expressed as:
[0037]
[0038] is the power allocated to PM m by the EC server, is the maximum computing power allocated to PM m, κ H is the effective conversion capacitance parameter of the EC server, L H It is the number of CPU cycles required by the PL to process the task data.
[0039] According to formula (7), the size of the task data to be unloaded from PM m at time t is calculated as follows:
[0040]
[0041] The energy consumed by PM m to unload task data in time t can be written as:
[0042] E m,o (t) = τ0p m,o (t) (12)
[0043] The energy consumed by PL to calculate the unloading task data of PM m in time t can be written as:
[0044] E m,H (t) = τ0p m,H (t) (13)
[0045] Modeling of Fleet System Problems:
[0046] At time t, PM m locally computes and offloads task data to PL respectively. m,l (t) and q m,o (t), therefore, the amount of data for all the calculated tasks is q m,l (t)+q m,o (t); Q m (t) represents the current length of the cache queue, given the arriving task data a m (t), the cache queue length at the next time t+1 is expressed as:
[0047] Q m (t+1)=min([max((Q m (t)-q m,l (t)-q m,o (t)),0)+a m (t)],Q m max ) (14)
[0048] Q m max The maximum cache queue length.
[0049] According to the M / M / 1 model and Little formula in queuing theory, the average length of the queue is equal to the average task arrival rate multiplied by the average time the task stays in the queue system, that is, the average cache queue length is proportional to the task data delay. Therefore, the average cache queue length is used to represent the average task calculation delay. The average cache queue length of the fleet system can be written as:
[0050]
[0051] Assume that the EC server has a high-frequency multi-core CPU and sufficient computing resources, and can process different tasks of PM unloading in parallel; therefore, the computational delay and energy consumption of PL are ignored; according to formulas (9) and (12), the average energy consumed by the truck platoon system in time t is calculated as:
[0052]
[0053] Considering the limited battery resources of the fleet system and the task calculation delay, the problem to be solved can be expressed as: minimizing the average energy consumption cost under the maximum power constraint and minimizing the average cache queue length under the maximum cache queue length, written as:
[0054]
[0055]
[0056] ω is a non-negative weighting coefficient; (17-b) and (17-c) indicate that the power must not exceed its corresponding maximum value, and (17-d) ensures the limit of the cache queue length.
[0057] Decentralized adaptive computing offloading based on DDPG:
[0058] DDPG consists of two parts: the actor network and the critic network. Each part contains an online main network μ(s; θ), Q(s, a; w) and a target network μ(s; θ - )、Q(s,a;w - ); To ensure more efficient random exploration of the environment state space to obtain the optimal strategy, OU random noise is added to the action space as follows:
[0059] a t =μ(s t ;θ)+σ OU (t) (18)
[0060] Initialize the parameters of all networks, make DDPG interact with the environment, and then generate the trajectory tuple (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool as a training data set. During the training phase, a small batch of samples containing N trajectory tuples is randomly selected from the experience replay pool and sent to DDPG to update the network parameters.
[0061] The actor's main network transforms the state s according to the strategy π t Mapped to a deterministic continuous action a t , and pass it to the critic's main network; the critic's main network outputs the corresponding Q value, which can evaluate action a t According to the Bellman equation, the Q value is calculated as follows:
[0062] y t =r t +γQ(s t+1 ,μ(s t+1 θ -);w - ) (19)
[0063] The main network of critic uses the stochastic gradient descent SGD method to update its parameters:
[0064]
[0065]
[0066] The actor's main network uses the deterministic policy gradient DPG method, according to the gradient provided by the critic main network To update the parameters of its network:
[0067]
[0068] Using the soft update method, the parameters of the two target networks are partially updated through the two main online networks:
[0069] θ-←τ·θ+(1-τ)·θ - (twenty four)
[0070] w-←τ·w+(1-τ)·w - (25)
[0071] (1) State space
[0072] In the truck platoon system, the state space of each PM includes the buffer queue length, channel status and SINR in the uplink. Based on the optimized computation offloading strategy learned by dynamic information, the agent adaptively allocates power, which can reduce the energy consumption of task execution and the length of the buffer queue. Therefore, the state space of the queuing system is defined as:
[0073]
[0074] (2) Action Space
[0075] Based on the state of the convoy system, the agent adaptively and continuously selects the local computation power and transmission power at time t; therefore, the action space of the convoy system in the continuous domain is defined as:
[0076] A(t)={p 1,l (t),p 1,o (t);,...,p m,l (t),p m,o (t),...,p M,l (t),p M,o (t)} (27)
[0077] (3) Reward Function
[0078] In the fleet system, the cost function of the PM includes the length of the cache queue, the energy consumption of local computing, and the energy consumption of task offloading, so the designed reward function weighs the length of the cache queue and the total energy consumption; therefore, the reward that PM m obtains from the environment after performing an action is defined as:
[0079]
[0080] Therefore, the average reward for the truck fleet is defined as:
[0081]
[0082] Since the goal of DRL is to maximize the sum of all rewards, the energy consumption and cache queue length are negative; ω is a weighting coefficient that weighs the cache queue length and total energy consumption; is the penalty function that limits the length of the cache queue, and P is the penalty constant; It is a binary function, which is equal to 1 when the cache queue length exceeds the maximum value, and equal to 0 otherwise; the goal of the fleet system reward function is expressed as a long-term reward. When the long-term reward converges to a stable value, the optimal strategy can be obtained, which can be expressed as:
[0083]
[0084] The beneficial technical effects of the present invention are:
[0085] 1. The decentralized adaptive computing offloading method based on the DDPG algorithm proposed in the present invention can independently and adaptively perform continuous and accurate power control according to time-varying information to achieve an efficient offloading strategy, which helps to reduce the average energy consumption of the truck fleet system and the average computing delay of the task.
[0086] 2. The decentralized adaptive computing offloading method based on the DDPG algorithm proposed in the present invention enables fleet members to independently make offloading decisions based on local environmental information, thereby reducing the load on the fleet system; this method is based on the DDPG algorithm, and vehicles can make continuous power allocation decisions to obtain the optimal computing offloading decision. BRIEF DESCRIPTION OF THE DRAWINGS
[0087] Figure 1 A truck platoon system model.
[0088] Figure 2 It is a computation offloading scheme for PM.
[0089] Figure 3 This is the framework of the decentralized adaptive computing offloading method based on the DDPG algorithm of the present invention.
[0090] Figure 4The training results of PM under different algorithms. (a) Truck 1, (b) Truck 2, (c) Truck 3.
[0091] Figure 5 Algorithm training results for Truck 1 with different weighting coefficients.
[0092] Figure 6 Comparison of the test performance of the three methods under different task arrival rates. (a), (b) and (c) show the average reward, energy consumption and cache queue length of the truck queue respectively.
[0093] Figure 7 Comparison of the test performance of the three methods at different driving speeds. (a), (b), and (c) show the average reward, energy consumption, and cache queue length of the truck queue, respectively.
[0094] Figure 8 Comparison of the test performance of the three methods under different numbers of PMs. (a), (b) and (c) show the average reward, energy consumption and cache queue length of the truck queue respectively. DETAILED DESCRIPTION
[0095] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0096] A decentralized adaptive computing offloading method based on the DDPG algorithm of the present invention is used for a truck fleet system. The truck fleet system model and the communication model are:
[0097] like Figure 1 As shown in the figure, a truck convoy is traveling on a highway. The truck convoy consists of a lead truck PL and M member trucks PM, where PM is represented by {1,2,...,m,...,M}. The PL is equipped with a small BS with an edge computing (EC) server, which can support sufficient computing resources to calculate the on-board tasks. Each PM communicates with the PL via V2V.
[0098] When a truck convoy is driving on a highway, all trucks can sense the traffic environment through sensors such as cameras and lidar. The collected data can be converted into application tasks that must be processed to maintain the safety and stability of the convoy system. In the truck convoy system, continuous time is divided into discrete time, the execution cycle is represented by τ0 with equal length, and the time index is represented by T = {0, 1, 2, 3, ...}. Figure 2 As shown, during task processing, the battery can power the processor and the transmitter, and the computing task can be performed locally by the CPU or partially or completely offloaded by the transmitter to the PL.
[0099] Applying the classic vehicle following model to the PM in the platoon: the intelligent driver model IDM, and assuming that the distance between any two adjacent trucks in the platoon system is equal and represented as d, the equilibrium spacing within the platoon is given by:
[0100]
[0101] Where d0 is the minimum distance within the convoy, t h is the ideal headway, v is the driving speed, v max is the maximum travel speed.
[0102] Assume that the length of each truck is d t , then the communication distance between member m and the lead vehicle is written as:
[0103] d m =m(d+d t )(2)
[0104] In the truck platoon system, NOMA technology is used for communication between PM and PL. Since multiple PMs upload task data synchronously on the same channel, there will be signal interference when unloading data; therefore, in the uplink at time t, the signal received by PL from PM m can be calculated as follows:
[0105]
[0106] is the transmit power of PM m, is the maximum transmit power, is the channel vector of PM m, x m (t) is a complex data symbol with unit variance; y m The first part is the effective signal from the target PM, the second part is the interference signal from other PMs sharing the channel, and the third part is n(t)~CN(0,σ 2 ) is a variable with variance σ 2 For PM m, the AR (time correlation autoregressive) model is used to characterize the channel state transition between time t and t+1:
[0107]
[0108] Among them, ρ m is the normalized correlation coefficient between time t and t+1 and is close to 1. e(t+1) is an error vector and also a m (t+1) uncorrelated complex Gaussian distribution, the channel vector of each PM is initialized to hm (0)~CN(0,h0(d ref / d m ) α ), h0 is a path loss constant, d ref is the reference distance, α is the path loss exponent.
[0109] To handle the interference caused by channel sharing between multiple PMs, PL (i.e., small BS) uses the successive interference cancellation (SIC) technique. After receiving the signal, PL performs the SIC decoding scheme to reduce the interference from other PMs according to the descending order of channel gain. Consider M PMs sharing the same channel in time t in descending order of their channels: h1(t)≥h2(t)≥…≥h m (t)≥…≥h M (t), the interference signal of PM m can be written as:
[0110]
[0111] Then the corresponding signal interference plus noise (SINR) of PM m at time t is:
[0112]
[0113] σ 2 is the Gaussian white noise power.
[0114] Therefore, the task data transmission rate between PM m and PL can be obtained according to the bandwidth B:
[0115] r m,o (t) = Blog2(1+ζ m (t)) (7)
[0116] Fleet system calculation model:
[0117] a m (t) quantifies the number of arrival task data of PM m during time t, assuming that at all times a m (t) are all independent and identically distributed. Figure 2 As shown, a m (t) is temporarily stored in the cache queue and then processed at the next time t+1; for each PM, it is assumed that the arrival rate of the computing task follows the Poisson distribution λ and the data size of the task follows the uniform distribution u~U(u min ,u max ), then a m (t) satisfies: E[a m (t)] = λ·u.
[0118] (1) Local computing model
[0119] For each PM, assuming that the task data is temporarily stored in the cache queue system in time order and then popped out of the cache queue and either computed locally or offloaded to the PL, the size of the task data computed locally for PM m can be written as:
[0120]
[0121] The CPU cycle frequency of PM m is scheduled by adjusting the chip voltage using DVFS technology. is the local computing power allocated to the CPU, is the maximum local computing power, κ m is the effective conversion capacitance parameter of the PM m processor, L m The number of CPU cycles required to execute the task data can be obtained through offline measurement.
[0122] The energy consumed by the local computation of PM m in time t can be written as:
[0123] E m,l (t) = τ0p m,l (t) (9)
[0124] (2) Calculation of unloading model
[0125] Due to insufficient computing resources of PM, part of the task data will be offloaded to PL and executed by the EC server. According to formula (8), the time required for the EC server to process the offloaded task data of PM m is expressed as:
[0126]
[0127] is the power allocated to PM m by the EC server, is the maximum computing power allocated to PM m, κ H is the effective conversion capacitance parameter of the EC server, L H It is the number of CPU cycles required by the PL to process the task data.
[0128] Since the output data size of the EC server is usually much smaller than the input data size, the transmission delay of the backhaul link (i.e., downlink) can be ignored. Therefore, according to formula (7), the size of the task data offloaded from PM m in time t is calculated as follows:
[0129]
[0130] The energy consumed by PM m to unload task data in time t can be written as:
[0131] E m,o (t) = τ0pm,o (t) (12)
[0132] The energy consumed by PL to calculate the unloading task data of PM m in time t can be written as:
[0133] E m,H (t) = τ0p m,H (t) (13)
[0134] Modeling of Fleet System Problems:
[0135] Assuming that PL has sufficient cache resources, the EC server will immediately calculate the offloaded task data, so the PL data cache delay can be ignored. At time t, PM m locally calculates and offloads task data to PL for q m,l (t) and q m,o (t), therefore, the amount of data for all the calculated tasks is q m,l (t)+q m,o (t); Q m (t) represents the current length of the cache queue, given the arriving task data a m (t), the cache queue length at the next time t+1 is expressed as:
[0136] Q m (t+1)=min([max((Q m (t)-q m,l (t)-q m,o (t)),0)+a m (t)],Q m max ) (14)
[0137] Q m max The maximum cache queue length.
[0138] According to the M / M / 1 model and Little formula (L = λW) in queuing theory, the average length of the queue is equal to the average task arrival rate multiplied by the average time the task stays in the queue system, that is, the average cache queue length is proportional to the task data delay. Therefore, the average cache queue length is used to represent the average task calculation delay. The average cache queue length of the fleet system can be written as:
[0139]
[0140] Assume that the EC server has a high-frequency multi-core CPU and sufficient computing resources, and can process different tasks of PM unloading in parallel; therefore, the computational delay and energy consumption of PL are ignored; according to formulas (9) and (12), the average energy consumed by the truck platoon system in time t is calculated as:
[0141]
[0142] Considering the limited battery resources of the fleet system and the task calculation delay, the problem to be solved can be expressed as: minimizing the average energy consumption cost under the maximum power constraint and minimizing the average cache queue length under the maximum cache queue length, written as:
[0143]
[0144] ω is a non-negative weighting coefficient; (17-b) and (17-c) indicate that the power must not exceed its corresponding maximum value, and (17-d) ensures the limit of the cache queue length.
[0145] It is worth noting that low latency is very important for time-sensitive tasks. If the PL fully observes the necessary global information and then distributes the collected information to each PM, it will lead to high system load and considerable latency. To solve this problem, it is assumed that each PM can only make decisions based on its local environment information. That is: at each time t, PM m receives the arriving task data a m (t) to update its cache queue and estimate the current channel vector h m (t), receiving SINR, and then PM m makes independent power allocation decision: p m,l (t), p m,o (t).
[0146] Decentralized adaptive computing offloading based on DDPG:
[0147] like Figure 3 As shown in the figure, DDPG consists of two parts: the actor network and the critic network. Each part contains an online main network μ(s; θ), Q(s, a; w) and a target network μ(s; θ-), Q(s, a; w - ); To ensure more efficient random exploration of the environment state space to obtain the optimal strategy, OU random noise is added to the action space as follows:
[0148] a t =μ(s t ;θ)+σ OU (t) (18)
[0149] Initialize the parameters of all networks, make DDPG interact with the environment, and then generate the trajectory tuple (s t ,a t ,r t ,s t+1) is stored in the experience replay pool as a training data set. During the training phase, a small batch of samples containing N trajectory tuples is randomly selected from the experience replay pool and sent to DDPG to update the network parameters.
[0150] The actor's main network transforms the state s according to the strategy π t Mapped to a deterministic continuous action a t , and pass it to the critic's main network; the critic's main network outputs the corresponding Q value, which can evaluate action a t According to the Bellman equation, the Q value is calculated as follows:
[0151] y t =r t +γQ(s t+1 ,μ(s t+1 θ - );w - ) (19)
[0152] The main network of critic uses the stochastic gradient descent SGD method to update its parameters:
[0153]
[0154] The actor's main network uses the deterministic policy gradient DPG method, according to the gradient provided by the critic main network To update the parameters of its network:
[0155]
[0156] Using the soft update method, the parameters of the two target networks are partially updated through the two main online networks:
[0157] θ-←τ·θ+(1-τ)·θ - (twenty four)
[0158] w-←τ·w+(1-τ)·w - (25)
[0159] The decentralized adaptive computation offloading method based on the DDPG algorithm can make an efficient computation offloading strategy and reduce the energy consumption and task computation delay of the truck platoon system. In other words, PM does not need to obtain global information about the entire fleet system and performs effective power allocation in the continuous domain based on only partial information. The definitions of the state space, action space, and reward function are described in detail below.
[0160] (1) State space
[0161] In the truck platoon system, the state space of each PM includes the buffer queue length, channel status and SINR in the uplink. Based on the optimized computation offloading strategy learned by dynamic information, the agent adaptively allocates power, which can reduce the energy consumption of task execution and the length of the buffer queue. Therefore, the state space of the queuing system is defined as:
[0162]
[0163] (2) Action Space
[0164] Based on the state of the convoy system, the agent adaptively and continuously selects the local computation power and transmission power at time t; therefore, the action space of the convoy system in the continuous domain is defined as:
[0165] A(t)={p 1,l (t),p 1,o (t);,...,p m,l (t),p m,o (t),...,p M,l (t),p M,o (t)} (27)
[0166] (3) Reward Function
[0167] In the fleet system, the cost function of the PM includes the length of the cache queue, the energy consumption of local computing, and the energy consumption of task offloading, so the designed reward function weighs the length of the cache queue and the total energy consumption; therefore, the reward that PM m obtains from the environment after performing an action is defined as:
[0168]
[0169] Therefore, the average reward for the truck fleet is defined as:
[0170]
[0171] Since the goal of DRL is to maximize the sum of all rewards, the energy consumption and cache queue length are negative; ω is a weighting coefficient that weighs the cache queue length and total energy consumption; is the penalty function that limits the length of the cache queue, and P is the penalty constant; It is a binary function, which is equal to 1 when the cache queue length exceeds the maximum value, and equal to 0 otherwise; the goal of the fleet system reward function is expressed as a long-term reward. When the long-term reward converges to a stable value, the optimal strategy can be obtained, which can be expressed as:
[0172]
[0173] The detailed information of the algorithm of the present invention is shown in Algorithm 1.
[0174]
[0175]
[0176] Example:
[0177] 1. Numerical simulation experiment and result analysis
[0178] Two hidden layers with 300 and 200 neurons were set in the neural network respectively, and the Adam method was used to train the parameters of the network.
[0179] (1) Algorithm convergence results
[0180] Consider a truck platoon system with M=3, where each PM makes a computational unloading decision based on the proposed unloading method (DACODRL). Extensive numerical simulation experiments were conducted to compare the performance of DACODRL. In the training phase, both DACODRL and DDQN algorithms were trained for 20 rounds using different random seeds, and the average of the results of 5 rounds with good training was selected to compare the algorithm convergence results. In the algorithm testing phase, the best performing DACODRL and DDQN were selected for algorithm testing experiments.
[0181] In the training phase, we first set the weighting coefficient ω = 0.5, the task arrival rate λ = 2 Mbps, and the driving speed v = 30 m / s. Figure 4 The convergence results of different algorithms for PM 1, PM 2, and PM 3 are illustrated; (1) the exact number of convergence steps of DACODRL and DDQN is shown, and (2) a detailed comparison of the rewards after convergence of the three methods is shown. It can be seen that DACODRL of the three PMs can converge before 40 episodes, and the rewards of DACODRL convergence are larger than those of DDQN and the local computation greedy policy (LCGP). Since the action space of DACODRL is continuous, the agent of this method can make more precise power control than DDQN, so the reward of DACODRL is the largest. This shows that in continuous control problems, the DDPG-based agent can explore the action space more effectively than the DDQN-based agent. In addition, LCGP prioritizes tasks with the maximum power without considering the optimal solution, and the reward of this method is lower than that of DACODRL and DDQN.
[0182] (2) Analysis of weighted coefficients of three methods
[0183] In order to analyze the different effects of energy consumption and cache queue length on the proposed method, different weighting coefficients in the reward function are adjusted: Consider a truck fleet system with M = 3, fixed task arrival rate λ = 2Mbps, driving speed v = 30m / s, and set weighting coefficients ω = 0.2, ω = 0.5, and ω = 0.8 respectively. This experiment focuses on PM1. Figure 5 As shown in the figure, when ω=0.2, ω=0.5 and ω=0.8, DACODRL converges at about 20 epsodes, 30 epsodes and 40 epsodes respectively, and the reward after convergence tends to be stable. It is worth noting that under different weighting coefficients, the convergence reward of DACODRL is higher than that of DDQN, which shows that the strategy learned by DACODRL always has the best performance in terms of energy consumption and task calculation delay, and also proves the superiority of the continuous power control strategy based on the DDPG algorithm. As the weighting coefficient increases, the rewards of DACODRL and DDQN also increase. This is because as the weighting coefficient decreases, the penalty effect of energy consumption in the reward function becomes smaller, and the penalty weight of the cache queue length increases. If the cache queue length exceeds the threshold, the penalty of the cache queue length will be greater. Therefore, the reward of the weighting coefficient ω=0.2 is the lowest.
[0184] (3) Analysis of task arrival rate of three methods
[0185] Consider a truck platoon system with M = 3, fixed weight coefficient ω = 0.5, and driving speed v = 30 m / s. Based on the training algorithm, the impact of task arrival rate on the calculation of unloading of the platoon system is tested, and the performance of three methods is compared. Figure 6 (a) It can be seen that as the arrival rate of tasks increases, the computational cost of PM increases, the energy consumed by processing tasks and the length of the cache queue increase, so the average reward of the fleet gradually decreases. It can be seen that DACODRL makes more effective decisions than DDQN, and its average reward is higher than DDQN. Since the trained DACODRL and DDQN consider the optimal unloading plan, their average rewards are much greater than LCGP. Figure 6 (b) and Figure 6 As shown in (c), the average energy consumption of the three methods increases with the increase of the task arrival rate, the consumption of the fleet system and the average buffer queue length, while DACODRL has the lowest energy consumption and the best performance. The buffer queue length of DACODRL is similar to that of DDQN. Since LCGP always allocates the maximum computing power to calculate tasks, the energy consumption and buffer queue length of this method are the largest and smallest, respectively.
[0186] (4) Analysis of driving speed using three methods
[0187] Consider a truck platoon system with M = 3, and fix the weight coefficient ω = 0.5 and the task arrival rate λ = 2Mbps. The effect of driving speed on the computational unloading of the platoon system is tested, and the performance of the three methods is compared through the training algorithm. Figure 7 (a) It is proved that the average reward of DACODRL is higher than that of DDQN and LCGP at different driving speeds. It is worth noting that with the increase of driving speed, when the driving speed is less than 25m / s, the average reward of DACODRL and DDQN changes little, and when the driving speed is greater than 25m / s, the average reward gradually decreases. This is because when the driving speed increases, in order to ensure the safety and stability of the convoy, the distance between PM and PL increases, which affects their communication performance. Figure 7 (b) and Figure 7 (c) shows that under time-varying communication conditions, the transmission power required for communication of DACODRL and DDQN gradually increases, and the average energy consumption of the two methods increases. In addition, the buffer queue lengths of DACODRL and DDQN are close and grow with the increase of driving speed. For LCGP, since the task arrival rate is fixed and not affected by communication conditions, the arriving task data can be processed immediately in the next moment. Therefore, the average energy consumption of this method changes slightly, and the average buffer queue length is minimal and remains unchanged.
[0188] (5) Analysis of PM quantity in three methods
[0189] With fixed weighting coefficient ω = 0.5, task arrival rate λ = 2Mbps, and driving speed v = 30m / s, the effects of different numbers of PMs on the calculation unloading of the platoon system are tested, and the performance of the three algorithms is compared. Figure 8 As shown in (a), DACODRL has the largest average reward and the best performance under different numbers of PM. As M increases, the average rewards of DACODRL and DDQN gradually decrease, and the average reward of LCGP changes slightly. Figure 8 (b) and Figure 8(c) It can be seen that the average energy consumption and average cache queue length of DACODRL and DDQN both increase with the growth of M, while the average energy consumption and cache queue length of LCGP change slightly and remain unchanged, respectively. This is because all PMs share the same channel when communicating with PL. As the number of PMs increases, the spectral efficiency of data transmission decreases, and more transmission power is required to transmit the same size of data, resulting in more energy consumption. In addition, the number of PMs is also one of the factors that determine the length of the fleet, affecting the communication distance between PM and PL. Therefore, the average energy consumption and cache queue length of DACODRL and DDQN related to computational offloading gradually increase, and the average reward gradually decreases. Since LCGP always processes tasks with maximum computing power and is not affected by communication conditions, its average energy consumption does not change much, and the average cache queue length is the smallest and remains unchanged, so the average reward of this method changes less.
[0190] The unloading method of the present invention can reduce the energy consumption cost, task processing delay and system load of the fleet system. The unloading method can independently and adaptively perform continuous and accurate power control based on time-varying information, and learn the optimal decision by balancing energy consumption and cache queue length, thereby realizing an efficient computational unloading strategy. Experimental results show that the proposed DACODRL method has excellent performance. Under different task arrival rates, driving speeds and PM numbers, the test performance of DACODRL is better than that of the local computation greedy strategy (LCGP) and the DDQN based on discrete action space.
Claims
1. A decentralized adaptive computation offloading method based on DDPG algorithm for truck platoon system, It is characterized in that The truck convoy system model and communication model are: The truck convoy consists of a lead truck PL and M member trucks PM, where PM is represented by: {1,2,...,m,...,M}; PL is equipped with an edge computing EC server; each PM communicates with PL via V2V; Applying the classic vehicle following model to the PM in the platoon: the intelligent driver model IDM, and assuming that the distance between any two adjacent trucks in the platoon system is equal and represented as d, the equilibrium spacing within the platoon is given by: Where, d 0 is the minimum intra-platoon spacing, t h is the ideal headway, v is the driving speed, v max is the maximum travel speed; Assume that the length of each truck is d t , then the communication distance between member m and the lead vehicle is written as: d m =m(d+d t )(2) In the truck platoon system, NOMA technology is used for communication between PM and PL. Since multiple PMs upload task data synchronously on the same channel, there will be signal interference when unloading data; therefore, in the uplink at time t, the signal received by PL from PM m can be calculated as follows: is the transmit power of PM m, is the maximum transmit power, is the channel vector of PM m, x m (t) is a complex data signal with unit variance; y m The first part is the effective signal from the target PM, the second part is the interference signal from other PMs sharing the channel, and the third part is n(t)~CN(0,σ 2 ) is a variable with variance σ 2 The additive white Gaussian noise vector of is: For PM m, the AR model is used to characterize the channel state transition between time t and t+1: Among them, ρ m is the normalized correlation coefficient between time t and t+1 and is close to 1. e(t+1) is an error vector and also a m (t+1) uncorrelated complex Gaussian distribution, the channel vector of each PM is initialized to h m (0)~CN(0,h 0 (d ref / d m ) α ), h 0 is a path loss constant, d ref is the reference distance, α is the path loss exponent; Consider M PMs sharing the same channel in time t in descending order of their channels: 1 (t)≥h 2 (t)≥…≥h m (t)≥…≥h M (t), the interference signal of PM m can be written as: Then the corresponding SINR of PM m at time t is: σ 2 is the Gaussian white noise power; Therefore, the task data transmission rate between PM m and PL can be obtained according to the bandwidth B: r m,o (t)=Blog 2 (1+ζ m (t)) (7) Fleet system calculation model: a m (t) quantifies the number of arrival task data of PM m during time t, assuming that at all times a m (t) are all independent and identically distributed, a m (t) is temporarily stored in the cache queue and then processed at the next time t+1; for each PM, it is assumed that the arrival rate of the computing task follows the Poisson distribution λ and the data size of the task follows the uniform distribution u~U(u min ,u max ), then a m (t) satisfies: E[a m (t)] = λ·u; (1) Local computing model For each PM, assuming that the task data is temporarily stored in the cache queue system in time order and then popped out of the cache queue and either computed locally or offloaded to the PL, the size of the task data computed locally for PM m can be written as: The CPU cycle frequency of PM m is scheduled by adjusting the chip voltage using DVFS technology. is the local computing power allocated to the CPU, is the maximum local computing power, κ m is the effective conversion capacitance parameter of the PM m processor, τ 0 is the time slot time; L m The number of CPU cycles required to execute the task data that can be obtained through offline measurement; The energy consumed by the local computation of PM m in time t can be written as: E m,l (t)=τ 0 p m,l (t) (9) (2) Calculation of unloading model Due to insufficient computing resources of PM, part of the task data will be offloaded to PL and executed by the EC server. According to formula (8), the time required for the EC server to process the offloaded task data of PM m is expressed as: is the power allocated to PM m by the EC server, is the maximum computing power allocated to PM m, κ H is the effective conversion capacitance parameter of the EC server, L H is the number of CPU cycles required by the PL to process task data; According to formula (7), the size of the task data to be unloaded from PM m at time t is calculated as follows: The energy consumed by PM m to unload task data in time t can be written as: E m,o (t)=τ 0 p m,o (t) (12) The energy consumed by PL to calculate the unloading task data of PM m in time t can be written as: E m,H (t)=τ 0 p m,H (t) (13) Modeling of Fleet System Problems: At time t, PM m locally computes and offloads task data to PL respectively. m,l (t) and q m,o (t), therefore, the amount of data for all the calculated tasks is q m,l (t)+q m,o (t); Q m (t) represents the current length of the cache queue, given the arriving task data a m (t), the cache queue length at the next time t+1 is expressed as: Q m (t+1)=min([max((Q m (t)-q m,l (t)-q m,o (t)),0)+a m (t)],Q m max ) (14) Q m max is the maximum cache queue length; According to the M / M / 1 model and Little formula in queuing theory, the average length of the queue is equal to the average task arrival rate multiplied by the average time the task stays in the queue system, that is, the average cache queue length is proportional to the task data delay. Therefore, the average cache queue length is used to represent the average task calculation delay. The average cache queue length of the fleet system can be written as: Assume that the EC server has a high-frequency multi-core CPU and sufficient computing resources, and can process different tasks of PM unloading in parallel; therefore, the computational delay and energy consumption of PL are ignored; according to formulas (9) and (12), the average energy consumed by the truck platoon system in time t is calculated as: Considering the limited battery resources of the fleet system and the task calculation delay, the problem to be solved can be expressed as: minimizing the average energy consumption cost under the maximum power constraint and minimizing the average cache queue length under the maximum cache queue length, written as: ω is a non-negative weighting coefficient used to weigh the cache queue length and total energy consumption; (17-b) and (17-c) indicate that the power must not exceed its corresponding maximum value, and (17-d) ensures the limit of the cache queue length; Decentralized adaptive computing offloading based on DDPG: DDPG consists of two parts: the actor network and the critic network. Each part contains an online main network μ(s; θ), Q(s, a; w) and a target network μ(s; θ - )、Q(s,a;w - ); To ensure more efficient random exploration of the environment state space to obtain the optimal strategy, OU random noise is added to the action space as follows: a t =μ(s t ;θ)+σ OU (t) (18) Initialize the parameters of all networks, make DDPG interact with the environment, and then generate the trajectory tuple (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool as a training data set. During the training phase, a small batch of samples containing N trajectory tuples is randomly selected from the experience replay pool and sent to DDPG to update the network parameters; The actor's main network transforms the state s according to the strategy π t Mapped to a deterministic continuous action a t , and pass it to the critic's main network; the critic's main network outputs the corresponding Q value, which can evaluate action a t According to the Bellman equation, the target Q value is calculated as follows: y t =r t +γQ(s t+1 ,μ(s t+1 ;θ - );w - ) (19) The main network of critic uses the stochastic gradient descent SGD method to update its parameters: The actor's main network uses the deterministic policy gradient DPG method, according to the gradient provided by the critic main network To update the parameters of its network: Using the soft update method, the parameters of the two target networks are partially updated through the two main online networks: i - ←τ·θ+(1-τ)·θ - (24) w - ←τ·w+(1-τ)·w - (25) (1) State space In the truck platoon system, the state space of each PM includes the buffer queue length, channel status and SINR in the uplink. Based on the optimized computation offloading strategy learned by dynamic information, the agent adaptively allocates power, which can reduce the energy consumption of task execution and the length of the buffer queue. Therefore, the state space of the queuing system is defined as: (2) Action Space Based on the state of the convoy system, the agent adaptively and continuously selects the local computation power and transmission power at time t; therefore, the action space of the convoy system in the continuous domain is defined as: A(t)={p 1,l (t),p 1,o (t);,...,p m,l (t),p m,o (t),...,p M,l (t),p M,o (t)} (27) (3) Reward Function In the fleet system, the cost function of the PM includes the length of the cache queue, the energy consumption of local computing, and the energy consumption of task offloading, so the designed reward function weighs the length of the cache queue and the total energy consumption; therefore, the reward that PM m obtains from the environment after performing an action is defined as: Therefore, the average reward for the truck fleet is defined as: Since the goal of DRL is to maximize the sum of all rewards, the energy consumption and cache queue length are negative; is the penalty function that limits the length of the cache queue, and P is the penalty constant; It is a binary function, which is equal to 1 when the cache queue length exceeds the maximum value, and equal to 0 otherwise; the goal of the fleet system reward function is expressed as a long-term reward. When the long-term reward converges to a stable value, the optimal strategy can be obtained, which can be expressed as:
