Deep reinforcement learning based in-vehicle network computation offloading and power allocation method

Abstract

The application discloses a kind of based on deep reinforcement learning's computing unloading and power distribution method in vehicle fleet network, specifically is: with the minimum vehicle fleet system average energy consumption and task data processing delay as target, the computing model of vehicle fleet system is established;According to the method based on multi-agent DRL, vehicle fleet network is regarded as an environment, and each PM is regarded as the agent that can interact with dynamic environment;The goal that each agent pursues is to maximize long-term discount reward;In the learning process, the decision made by agent based on state is driven by reward, and computing unloading and power distribution joint optimization are carried out.The computing unloading and power distribution optimization model proposed in the application comprehensively considers the mobility of vehicle, communication interference and vehicle cooperation;It can adapt to fast-moving vehicle fleet network environment;It can capture the time sequence characteristics of environment state, and can make more accurate power distribution actions.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, and in particular relates to a computational offloading and power allocation method for fleet networks based on deep reinforcement learning. Background Technology

[0002] With the development of vehicle-to-everything (V2X) technology and the application of artificial intelligence algorithms, connected and autonomous vehicles have become an important solution for improving traffic safety and reducing environmental pollution, and are also a crucial component of intelligent transportation systems. Due to their high degree of autonomy and connectivity, merging connected and autonomous vehicles into a convoy is an efficient and eco-friendly driving solution. A convoy consists of a lead vehicle (PL) and a group of member vehicles (PM) traveling in the same direction. In the convoy network, member vehicles periodically share information with the lead vehicle, allowing them to travel closely together with minimal spatial spacing. This driving method reduces vehicle energy consumption and improves driving safety and comfort.

[0003] However, the successful implementation of platoon cooperative control and the integration of upper-level control applications largely depend on the high-performance communication and computing capabilities of onboard equipment. In particular, autonomous driving systems must rely on powerful computing capabilities to perceive surrounding traffic conditions, fuse heterogeneous data from multiple sources, and make AI-based driving decisions. Furthermore, the implementation of some onboard application tasks also requires significant onboard resources. Therefore, advanced communication technologies and assisted computing play a crucial role in achieving connectivity, autonomy, and driver comfort for platooned vehicles. To overcome these challenges, vehicle edge computing is an effective solution. However, this solution also presents some challenges: 1) In some remote areas, roadside infrastructure coverage is sparse, making it impractical to lay infrastructure along the road to achieve full coverage. Therefore, real-time V2I-based offloading services are difficult to achieve. 2) The high mobility of vehicles leads to frequent changes in the V2V network topology, resulting in unstable V2V offloading services and difficulty in meeting user experience quality requirements. 3) The highly dynamic traffic environment requires vehicle users to make real-time decisions based on the current environmental conditions. Summary of the Invention

[0004] This invention provides a modeling approach for computation offloading and resource allocation schemes in fleet networks, and offers a solution to the joint optimization problem in dynamic fleet networks. Specifically, it presents a computation offloading and power allocation method for fleet networks based on deep reinforcement learning.

[0005] The present invention provides a method for computational offloading and power allocation within a fleet network based on deep reinforcement learning, specifically comprising:

[0006] The fleet system is as follows:

[0007] The convoy consists of a lead vehicle (PL) and a group of member vehicles (PMs) traveling in the same direction. The PL is equipped with an edge computing (EC) server to provide computational offloading services for the resource-constrained PMs. The timeline is discretized into T time slots, each with a duration of τ0. The index set of the time slots is represented as: T = {1, 2, 3, ..., t, ..., T}. In each time slot t, the PM can generate task data. The convoy vehicles are represented as a set M = {0, 1, 2, ..., m, ..., M}, where m = 0 represents the PL and M represents the total number of PMs. Each PM can execute task data locally or offload it to the PL via V2V communication. NOMA technology is used to implement the link between the PMs and PLs.

[0008] Creating a dynamic motion model for the vehicle fleet:

[0009] The Intelligent Driving Model (IDM) is used to characterize the following behavior of platooned vehicles. The initial position of the last PM is defined as the origin of the coordinate system, and only the longitudinal velocity and position of the vehicles in the platoon are considered. At time slot t, the velocity and acceleration of PM m are v and v, respectively. m (t) and a m (t); The head coordinates of PM m at time slot t are x m Let m ∈ {1,2,...,M}, then the relative velocity is Δv. m (t)=v m (t)-v m-1 (t); The inter-vehicle distance between PM m and the vehicle in front, i.e., the distance between the front of the vehicle and the rear of the vehicle in front, is Δd. m (t)=x m-1 (t)-x m (t)-d m-1,len , where d m-1,len Let PMm represent the vehicle length; the motion state of PMm depends on the kinematic characteristics of PMm-1. According to IDM, the acceleration of PMm can be written as:

[0010]

[0011] Among them, a max It is the maximum acceleration, v max The ideal speed under steady traffic flow, δ∈[1,5] represents the driver sensitivity characteristics, and f(·) is a function related to speed and relative speed, defined as:

[0012]

[0013] Where d0 represents the ideal frontage distance, t0 represents the ideal headway distance, and b max This indicates maximum deceleration; therefore, the velocity and position of PMm at time slot t are:

[0014] v m (t)=v m (t-1)+a m (t-1)·τ0 (3)

[0015]

[0016] Therefore, the communication distance between PM m and PL is:

[0017] d m，0 (t)=x0(t)-x m (t) (5)

[0018] Team communication model

[0019] At time t, the signal received by PL from PM m in the uplink can be calculated as follows:

[0020]

[0021] in: It is the transmission power of PM m. It is the maximum transmit power of PM m, h m (t) is the channel gain between PM m and PL, c m (t) is a complex data signal; y 0,m The first part of n(t) is the desired signal from the target PM, the second part is the interference from other PMs, and the third part is additive white Gaussian noise.

[0022] When the PL receives a signal, it uses the SIC decoding scheme to reduce interference from other PMs by following the descending order of the signal:

[0023]

[0024] Where, p i,o (t)(h i (t)) 2 ≤p m,o (t)(h m (t)) 2 For i∈{1,2,...,M}, i≠m, the channel gain between PMm and PL at time slot t is:

[0025]

[0026] Where g0 is the channel power gain at a reference distance of one meter, α is the path loss exponent, and n0(t) is a complex Gaussian variable, then the signal-to-interference-plus-noise ratio (SINR) corresponding to PMm is:

[0027]

[0028] Where σ 2 It is white noise power.

[0029] According to Shannon's formula, given a bandwidth B, the data transmission rate of PM m in the uplink of time slot t is:

[0030] r m，o (t)=B log2(1+ζ m (t)) (10)

[0031] Establish a computational model:

[0032] Assume that the task data for PM m is temporarily stored in a buffer queue; for PM m, assume that the generated task data follows an average value of λ. m The Poisson distribution.

[0033] (1) Local computing model

[0034] The amount of task data that PM m can process during time slot t is:

[0035]

[0036] in It is the number of CPU cycles per second allocated by PM m. It is the allocated local execution power. It is the maximum computing power, κ m L0 is the effective switching capacitor of the PM m processor, and L0 is the number of CPU cycles required to compute each bit of task data.

[0037] (2) Calculate the unloading model

[0038] The computational unloading process of PM m consists of three steps: 1) PM m transmits the task data to PL; 2) PL's EC server computes the unloaded task data; 3) PL returns the execution result to PM m. Since the data size after execution at PL is much smaller than the input data size, the transmission time is ignored in the third step. Assume that the first and second steps can be completed within one time slot τ0, and the transmission time of the task data is t. m,off (t), then the data processing time in the EC server is τ0-t m,off (t), according to (10), we can obtain the system of equations:

[0039]

[0040] Where q m,o (t) represents the task data that PM m uninstalls. It is the number of CPU cycles allocated to PM m by the EC server. It is the calculated power that PL allocates to PM m. κ0 is the maximum computing power, and κ0 is the effective switched capacitor of the EC server processor; by solving (12), the transmission time t can be obtained. m,off (t); The size of the task data that PM m can unload in time slot t is:

[0041]

[0042] Problem modeling:

[0043] For PM m in time slot t, the sizes of the task data computed locally and unloaded to PL are q, respectively. m,l (t) and q m,o (t), therefore, the size of all executed task data is q. m,l (t)+q m,o (t), using Q m (t) represents the current length of the buffer queue, o m (t) represents the number of task data generated in time slot t, and the length of the buffer queue in the next time slot t+1 is:

[0044]

[0045] Based on queuing theory and Little's formula, the average buffer queue length is used to represent the average task data computation latency. Assuming the EC server installed in the PL has a high-frequency multi-core CPU, it can compute the unloading task data in parallel. To ensure that the PM's unloading task data can be processed immediately, Q0(t) = 0 is set. The average buffer queue length of the fleet system can be calculated as follows:

[0046]

[0047] The average energy consumption of a fleet system can be written as:

[0048]

[0049] The model aims to minimize the weighted sum of the average energy consumption and task data processing latency of the fleet system, and jointly optimize computational offloading and power allocation under dynamic conditions; let P = {p m,l p m,o p 0,m If} is the decision variable, then the power decision problem is expressed as:

[0050]

[0051] DDPG-based computational offloading and power allocation:

[0052] Based on the multi-agent DRL approach, the fleet network is viewed as an environment, and each PM is considered an agent capable of interacting with the dynamic environment. Since the PL has a high-performance multi-core CPU, it processes the offloading task data immediately, ignoring the PL's computational latency, i.e., t... m,off (t) = τ0; the goal of each agent is to maximize the long-term discounted reward; during the learning process, the agent's decisions based on the state are reward-driven, therefore, how the reward function is defined will directly affect the quality of the agent's learning strategy; the agent's environment state space, action space, and reward function are as follows:

[0053] 1) State Space

[0054] PM m's state s m (t)∈S m It consists of local state and global state; the local state includes the generated task data. m (t) Length of the buffer queue Q m (t) and channel gain h m (t), the global state includes SINRζ m (t); Therefore, the state space of the m-th agent is:

[0055] s m (t)={o m (t), Q m (t), h m (t), ζ m (t)} (18)

[0056] 2) Action Space

[0057] Based on state s m (t), the agent performs actions including local computation of power p m,l (t) and transmission power p m,o The joint action of (t); therefore, the action space of the m-th agent is:

[0058] μ m (t)={p m，l (t), p m，o (t)} (19)

[0059] In each learning step, to ensure more efficient stochastic exploration of the environment state space to obtain the optimal policy, OU random noise is added to the operation space:

[0060] u m (t)=μ m (t)+σ OU (t) (20)

[0061] 3) Reward function

[0062] Based on the reward function defined in (17), the agent can learn the optimal policy step by step in the iteration. The design of the reward function is as follows:

[0063] Energy Consumption Reward: During the learning process, to enable the agent to learn the optimal power allocation strategy, the energy consumption at each moment is added to the reward function; the energy consumption calculated from the task data is:

[0064] E m (t)=p m，l (t)·τ0+p m，o (t)·τ0 (21)

[0065] Calculating the latency reward: The buffer queue length can measure the computation latency of task data; to reduce the execution latency of task data, Q is introduced into the reward function. m (t).

[0066] Punishment and reward: The agent's behavior affects the length of the buffer queue; in (14), if Q m If (t) is less than 0, it indicates that the agent is over-allocating computational power; if Q m If (t) exceeds the threshold, it means that the power allocated to the agent is insufficient to fully compute the arriving task data, resulting in task data accumulation. To prevent this problem, two penalty terms are added to the reward function to ensure that the agent performs more accurate operations:

[0067]

[0068] Where p m,over (t) represents the over-allocated local computing power, and P1 and P2 are the over-allocation power and the penalty factor for exceeding the threshold, respectively. It is a binary function that equals 1 when the length of the buffer queue exceeds its maximum value, and equals 0 otherwise.

[0069] Therefore, based on the above definition, by summing the reward values ​​using weighted coefficients ω1 and ω2, the reward function for agent m is:

[0070] r m (t)=-ω1θ1E m (t)-ω2θ2Q m (t)-P m (t) (23)

[0071] The learning algorithm consists of a learning process and a testing process; for each learning step, the m-th agent obtains its local state s from the current environment. m (t) and output the deterministic action μm (t); In order for the agent to fully explore the environment to obtain the optimal strategy, the OU noise σ OU (t) added to μ m In (t), the agent will receive r after completing the action output. m (t) Reward, evaluating how good the action is; after all agents complete the process, the environment enters the next state, and each agent will receive a transition tuple (s). m (t), u m (t), r m (t), s m+1 (t)), this tuple will be stored in the experience replay pool B. m The network is used for model training; therefore, for each agent, a mini-batch of samples containing N transformation tuples is randomly selected from the experience replay pool and fed into the DDPG to update the network parameters; the target Q-value of the target network can be calculated as follows:

[0072]

[0073] Where γ is the discount factor; the online critic network updates its parameters using stochastic gradient descent, as shown below:

[0074]

[0075]

[0076] Where α w It is the learning rate of the online critic network; based on the gradient provided by the online critic. The Actor online network updates network parameters using a deterministic policy gradient method.

[0077]

[0078]

[0079] Where α θ It is the learning rate of the actor's online network; the soft update method is used to update two target networks through two online networks:

[0080] θ - ←τ·θ+(1-τ)·θ - (29)

[0081] w - ←τ·w+(1-τ)·w - (30)

[0082] Where τ is the update coefficient, and τ << 1.

[0083] The beneficial technical effects of this invention are as follows:

[0084] The proposed optimization model for computational unloading and power allocation in fleet networks comprehensively considers vehicle mobility, communication interference, and vehicle cooperation. It can adapt to fast-moving fleet network environments and capture the temporal characteristics of environmental states, enabling more accurate power allocation actions. Attached Figure Description

[0085] Figure 1 For fleet network models.

[0086] Figure 2 The computational offloading and power allocation scheme for PM.

[0087] Figure 3 This is an LSTM model used to process PM state features.

[0088] Figure 4 This provides the framework for the DDPG algorithm.

[0089] Figure 5 The training results of the COPA scheme based on DDPG are shown under different PMs. (a) M=5, (b) M=7, (c) M=9, (d) M=11.

[0090] Figure 6 The training results are for the COPA scheme based on DDPG with different weighting coefficients.

[0091] Figure 7 The test performance of four schemes is compared under different task data arrival rates. (a), (b), and (c) show the average reward, average allocation power, and average buffer queue length of the test results, respectively.

[0092] Figure 8 The test performance is compared for four different weighting schemes. (a), (b), and (c) show the average reward, energy consumption, and buffer queue length of the test results, respectively.

[0093] Figure 9 This paper compares the test performance of four schemes at different driving speeds for the PL system. (a) shows the balanced inter-vehicle distance of the fleet system, and (b) shows the average reward of the test results.

[0094] Figure 10 Performance comparison tests of four schemes under different PM numbers. Detailed Implementation

[0095] The present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0096] The present invention provides a method for computational offloading and power allocation within a fleet network based on deep reinforcement learning, specifically comprising:

[0097] The fleet system is as follows:

[0098] like Figure 1 As shown, the convoy consists of a lead vehicle (PL) and a group of member vehicles (PMs) traveling in the same direction. The PL is equipped with an edge computing (EC) server (or vehicles with larger resource capacity are considered PLs), providing computational offloading services to the resource-constrained PMs. The timeline is discretized into T time slots, each with a duration of τ0. The index set of the time slots is represented as: T = {1, 2, 3, ..., t, ..., T}. In each time slot t, the PM can generate task data (e.g., vehicle driving information, location information, application data, and surrounding environment information). The convoy vehicles are represented as a set M = {0, 1, 2, ..., m, ..., M}, where m = 0 represents the PL and M represents the total number of PMs. Each PM can execute task data locally or offload it to the PL via V2V communication. NOMA technology is used to implement the link between the PMs and PLs. In this case, PMs can simultaneously upload data to the PL on the same channel, thereby improving spectral efficiency.

[0099] Consider the mobility and following behavior of vehicles in the platoon to create a dynamic motion model of the platoon.

[0100] The PM (Production Manager) adjusts its speed based on the speed changes of the preceding vehicle in the platoon to ensure a safe following distance. An Intelligent Driving Model (IDM) is used to characterize the following behavior of the platooned vehicles. The initial position of the last PM is defined as the origin of the coordinate system, considering only the longitudinal velocity and position of the vehicles in the platoon. At time slot t, the velocity and acceleration of PM m are vm and vm, respectively. m (t) and a m (t); The head coordinates of PM m at time slot t are x m Let m ∈ {1,2,...,M}, then the relative velocity is Δv. m (t)=v m (t)-v m-1 (t); The inter-vehicle distance between PM m and the vehicle in front, i.e., the distance between the front of the vehicle and the rear of the vehicle in front, is Δd. m (t)=x m-1 (t)-x m (t)-d m-1,len , where d m-1,len Let PMm represent the vehicle length; the motion state of PMm depends on the kinematic characteristics of PMm-1. According to IDM, the acceleration of PMm can be written as:

[0101]

[0102] Among them, a maxIt is the maximum acceleration, v max The ideal speed under steady traffic flow, δ∈[1,5] represents the driver sensitivity characteristics, and f(·) is a function related to speed and relative speed, defined as:

[0103]

[0104] Where d0 represents the ideal frontage distance, t0 represents the ideal headway distance, and b max This indicates maximum deceleration; therefore, the velocity and position of PMm at time slot t are:

[0105] v m (t)=v m (t-1)+a m (t-1)·τ0 (3)

[0106]

[0107] Therefore, the communication distance between PM m and PL is:

[0108] d m，0 (t)=x0(t)-x m (t) (5)

[0109] Team communication model

[0110] Using NOMA communication technology can improve the utilization of the fleet's limited communication resources. Since PMs can transmit data simultaneously on the same channel, they will interfere with each other when unloading task data. At time t, the signal received by PL from PM m in the uplink can be calculated as follows:

[0111]

[0112] in: It is the transmission power of PM m. It is the maximum transmit power of PM m, h m (t) is the channel gain between PM m and PL, c m (t) is a complex data signal; y 0,m The first part of n(t) is the desired signal from the target PM, the second part is the interference from other PMs, and the third part is additive white Gaussian noise.

[0113] Continuous Interference Cancellation (SIC) is used to recover the desired signal. That is, when the PL receives a signal, it employs a SIC decoding scheme to reduce interference from other PMs by reducing the signal's descending order.

[0114]

[0115] Where, p i,o (t)(h i (t)) 2 ≤p m,o (t)(h m (t)) 2 For i∈{1,2,...,M}, i≠m, the channel gain between PMm and PL at time slot t is:

[0116]

[0117] Where g0 is the channel power gain at a reference distance of one meter, α is the path loss exponent, and n0(t) is a complex Gaussian variable, then the signal-to-interference-plus-noise (SINR) corresponding to PMm is:

[0118]

[0119] Where σ 2 It is white noise power.

[0120] According to Shannon's formula, given a bandwidth B, the data transmission rate of PM m in the uplink of time slot t is:

[0121] r m，o (t)=B log2(1+ζ m (t)) (10)

[0122] Establish a computational model:

[0123] like Figure 2 As shown, it is assumed that the task data of PM m is temporarily stored in a buffer queue; for PM m, it is assumed that the generated task data follows an average value of λ. m The Poisson distribution.

[0124] (1) Local computing model

[0125] The amount of task data (in bits) that PM m can process during time slot t is:

[0126]

[0127] in It is the number of CPU cycles per second allocated by PM m. It is the allocated local execution power. It is the maximum computing power, κ m L0 is the effective switching capacitor of the PM m processor, and L0 is the number of CPU cycles required to compute each bit of task data.

[0128] (2) Calculate the unloading model

[0129] The computational unloading process of PM m consists of three steps: 1) PM m transmits the task data to PL; 2) PL's EC server computes the unloaded task data; 3) PL returns the execution result to PM m. Since the data size after execution at PL is much smaller than the input data size, the transmission time is ignored in the third step. Assume that the first and second steps can be completed within one time slot τ0, and the transmission time of the task data is t. m,off (t), then the data processing time in the EC server is τ0-t m,off (t), according to (10), we can obtain the system of equations:

[0130]

[0131] Where q m,o (t) represents the task data that PM m uninstalls. It is the number of CPU cycles allocated to PM m by the EC server. It is the calculated power that PL allocates to PM m. κ0 is the maximum computing power, and κ0 is the effective switched capacitor of the EC server processor; by solving (12), the transmission time t can be obtained. m,off (t); The size of the task data that PM m can unload in time slot t is:

[0132]

[0133] Problem modeling:

[0134] For PM m in time slot t, the sizes of the task data computed locally and unloaded to PL are q, respectively. m,l (t) and q m,o (t), therefore, the size of all executed task data is q. m,l (t)+q m,o (t), using Q m (t) represents the current length of the buffer queue, o m (t) represents the number of task data generated in time slot t, and the length of the buffer queue in the next time slot t+1 is:

[0135]

[0136] Based on queuing theory and Little's formula, the average buffer queue length is used to represent the average task data computation latency. Assuming the EC server installed in the PL has a high-frequency multi-core CPU, it can compute the unloading task data in parallel. To ensure that the PM's unloading task data can be processed immediately, Q0(t) = 0 is set. The average buffer queue length of the fleet system can be calculated as follows:

[0137]

[0138] The average energy consumption of a fleet system can be written as:

[0139]

[0140] The model aims to minimize the weighted sum of the average energy consumption and task data processing latency of the fleet system, and jointly optimize computational offloading and power allocation under dynamic conditions; let P = {p m,l p m,o p 0,m If} is the decision variable, then the power decision problem is expressed as:

[0141]

[0142] DDPG-based computational offloading and power allocation:

[0143] In PDMP, dynamic indicators of vehicles traveling in platoons, such as vehicle speed v, are used. m (t), acceleration a m (t) and position x m (t) is time-dependent, resulting in the vehicle channel gain h m (t) The time-series characteristics of the changing data. Furthermore, the arrival process of PM's task data follows a Poisson distribution, and the task data o m The magnitude of (t) is random. Therefore, the environmental states observed by the agent at different times in time period T can be regarded as temporally sequential data. In order to enable the agent to better capture the temporal correlation of state changes and make the power allocation operation more efficient, a Long Short-Term Memory (LSTM) neural network is selected as the training network.

[0144] LSTM is a variant of Recurrent Neural Networks (RNNs) that can address the problems of long-term information dependence and lack of short-term information input. For example... Figure 3 As shown, the core of LSTM is the LSTM unit, which consists of a candidate memory and three gates: a forget gate, an input gate, and an output gate. The LSTM unit controls the flow of environmental states in the candidate memory through three gating mechanisms. Therefore, by training the neural network, the agent can learn the temporal correlation of environmental states to make better power allocation decisions.

[0145] To improve the accuracy of power allocation decisions made by the agent, DDPG is used as the learning algorithm. It is an extension of the traditional DPG method, introducing a deep neural network (DNN) to address the high-dimensional state and action space problems of traditional RL. Compared to the traditional Deep Q-Network (DQN) algorithm, DDPG, by combining DNN and DPG algorithms, can effectively solve continuous control problems. Figure 4As shown, DDPG is an Actor-Critic architecture. The participant network is a deterministic policy network that learns action policies. The critic network is a Q-value function network used to evaluate the action values ​​output by the participant network. To stabilize the learning process and improve algorithm performance, both the Actor network and the critic network have a target network.

[0146] Traditional DRL-based methods primarily rely on single agents. In these cases, a single agent does not need to consider the behaviors of other agents, thus the relationships between agents are relatively independent. However, in multi-agent scenarios, the decisions of agents influence each other, and the joint actions of all agents determine the transition of the environment state. Therefore, convergence of single-agent algorithms cannot be guaranteed in multi-agent scenarios. In this problem, the power allocation behavior of the PM affects the SINR of all PMs through the shared network environment, thereby influencing the change of the environment state. Specifically, PM decisions need to consider the actions of other PMs simultaneously.

[0147] Based on the multi-agent DRL approach, the fleet network is viewed as an environment, and each PM is considered an agent capable of interacting with the dynamic environment. Since the PL has a high-performance multi-core CPU, it processes the offloading task data immediately, ignoring the PL's computational latency, i.e., t... m,off (t) = τ0; the goal of each agent is to maximize the long-term discounted reward; during the learning process, the agent's state-based decisions are reward-driven, therefore, how the reward function is defined directly affects the quality of the agent's learning strategy. The agent's environment state space, action space, and reward function are as follows:

[0148] 1) State Space

[0149] PM m's state s m (t)∈S m It consists of local state and global state; the local state includes the generated task data. m (t) Length of the buffer queue Q m (t) and channel gain h m (t), the global state includes SINRζ m (t); Therefore, the state space of the m-th agent is:

[0150] s m (t)={o m (t), Q m (t), h m (t), ζ m (t)} (18)

[0151] 2) Action Space

[0152] Based on state sm (t), the agent performs actions including local computation of power p m,l (t) and transmission power p m,o The joint action of (t); therefore, the action space of the m-th agent is:

[0153] μ m (t)={p m，l (t), p m，o (t)} (19)

[0154] In each learning step, to ensure more efficient stochastic exploration of the environment state space to obtain the optimal policy, OU random noise is added to the operation space:

[0155] u m (t)=μ m (t)+σ OU (t) (20)

[0156] 3) Reward function

[0157] Based on the reward function defined in (17), the agent can learn the optimal policy step by step in the iteration. The design of the reward function is as follows:

[0158] Energy Consumption Reward: During the learning process, to enable the agent to learn the optimal power allocation strategy, the energy consumption at each moment is added to the reward function; the energy consumption calculated from the task data is:

[0159] E m (t)=p m，l (t)·τ0+p m，o (t)·τ0 (21)

[0160] Calculating the latency reward: The buffer queue length can measure the computation latency of task data; to reduce the execution latency of task data, Q is introduced into the reward function. m (t).

[0161] Punishment and reward: The agent's behavior affects the length of the buffer queue; in (14), if Q m If (t) is less than 0, it indicates that the agent is over-allocating computational power; if Q m If (t) exceeds the threshold, it means that the power allocated to the agent is insufficient to fully compute the arriving task data, resulting in task data accumulation. To prevent this problem, two penalty terms are added to the reward function to ensure that the agent performs more accurate operations:

[0162]

[0163] Where p m,over(t) represents the over-allocated local computing power, and P1 and P2 are the over-allocation power and the penalty factor for exceeding the threshold, respectively. It is a binary function that equals 1 when the length of the buffer queue exceeds its maximum value, and equals 0 otherwise.

[0164] Therefore, based on the above definition, by summing the reward values ​​using weighted coefficients ω1 and ω2, the reward function for agent m is:

[0165] r m (t)=-ω1θ1E m (t)-ω2θ2Q m (t)-P m (t) (23)

[0166] The learning objective of the agent is to maximize the discounted reward over a period of time. Since the goal of PDMP is to minimize the weighted sum of the average energy consumption and processing latency of the fleet system, the agent's reward function is set to negative. Different combinations of ω1 and ω2 represent a balance between different performance metrics. For example, setting ω1 = 1 and ω2 = 0 means that the agent m learns the optimal policy only by minimizing the average energy consumption.

[0167] The learning algorithm consists of a learning process and a testing process; for each learning step, the m-th agent obtains its local state s from the current environment. m (t) and output the deterministic action μ m (t); In order for the agent to fully explore the environment to obtain the optimal strategy, the OU noise σ OU (t) added to μ m In (t), the agent will receive r after completing the action output. m (t) Reward, evaluating how good the action is; after all agents complete the process, the environment enters the next state, and each agent will receive a transition tuple (s). m (t), u m (t), r m (t), s m+1 (t)), this tuple will be stored in the experience replay pool B. m The network is used for model training; therefore, for each agent, a mini-batch of samples containing N transformation tuples is randomly selected from the experience replay pool and fed into the DDPG to update the network parameters; the target Q-value of the target network can be calculated as follows:

[0168]

[0169] Where γ is the discount factor; the online critic network updates its parameters using stochastic gradient descent, as shown below:

[0170]

[0171]

[0172] Where α w It is the learning rate of the online critic network; based on the gradient provided by the online critic. The Actor online network updates network parameters using a deterministic policy gradient method.

[0173]

[0174]

[0175] Where α θ It is the learning rate of the actor's online network; the soft update method is used to update two target networks through two online networks:

[0176] θ - ←τ·θ+(1-τ)·θ - (29)

[0177] w - ←τ·w+(1-τ)·w - (30)

[0178] Where τ is the update coefficient, and τ << 1.

[0179] Detailed information on the computational offloading and power allocation scheme for DDPG is shown in Algorithm 1.

[0180]

[0181]

[0182] Example

[0183] The simulation was run on a PC equipped with an Intel Core i7-12700 CPU at 2.10 GHz. The graphics card was an NVIDIA GeForce RTX 3060 with 12 GB of RAM. TensorFlow 2.30 and Python 3.8 were used to implement a DDPG-based computational offloading and power allocation (COPA) scheme in the simulation. In the simulation setup, the PM was driven by an IDM. To simulate interference from other vehicles, the initial speed of the PL was randomly generated between 15 m / s and 20 m / s in each training session, and the initial vehicle distances were randomly generated between 10 m and 15 m. Each vehicle was 5 m long. Two LSTM units and a fully connected neural network were used as the training network.

[0184] 1. Algorithm convergence analysis

[0185] During training, the weighting coefficients ω1=ω2=0.5 and the task data arrival rate λ=4Mbps were set. Figure 5 The correlation between the learning stage j and the average reward obtained on the corresponding training set is shown for different numbers of PMs. It can be seen that the DDPG-based COPA scheme converges after sufficient training and performs well for different numbers of PMs. In all four subgraphs, the average reward fluctuates greatly at the beginning of training because the agent is in the environment exploration phase. As the number of iterations increases, the agent continuously learns from the environment, and the average reward gradually increases. After approximately 30, 20, 40, and 30 training sessions, respectively, the corresponding values ​​are... Figure 5 In (a), (b), (c), and (d), the average reward of the system becomes stable. This indicates that the agent has learned the optimal COPA scheme in the dynamic environment. The average reward fluctuates less after convergence. This is because the environment is dynamically changing, and the agent should still make different power allocation decisions to better adapt to the dynamic environment.

[0186] To analyze the different impacts of energy consumption and buffer queue length on the COPA scheme, we adjusted different weight coefficients in the training reward function. We fixed the number of PMs M = 5, the task data arrival rate λ = 4 Mbps, and set the weight coefficients ω1 = 0.1, ω1 = 0.5, and ω1 = 0.9. Figure 6 The training results of the DDPG-based COPA scheme with different weight coefficients are shown. It can be seen that the scheme converges after 50 episodes with different weight coefficient settings. Notably, in the comparison after algorithm convergence, the average reward of the system increases with the increase of the weight coefficient. The average reward is lowest when ω1 = 0.1, highest when ω1 = 0.9, and falls between the two when ω1 = 0.5. This is because when ω1 is small, the agent is less sensitive to energy consumption and allocates more power to reduce buffer queue backlog, thus increasing the average reward.

[0187] 2. Performance Comparison

[0188] We compare the traditional DDPG algorithm (DDPG-FC), DDQN, and the Local Computation Greedy Method (LCGP) as benchmark algorithms.

[0189] 1) Impact of task data arrival rate: First, compare the test results of the four schemes under different task arrival rates. During the test, the number of PMs was fixed at M=5, the weighting coefficient ω1=0.5, and the PL driving speed v0=15m / s. Figure 7 The test results for the average reward, average allocation power, and average buffer queue length of four schemes under different task arrival rates are presented. Figure 7(a) It can be seen that the average reward of all four schemes decreases as the task completion rate increases. Furthermore, the DDPG-COPA scheme has a higher average reward than all three baseline schemes. All three learning schemes (i.e., DDPG-COPA, DDPG-FC, and DDQN) achieve higher average rewards than the LCGP scheme. Figure 7 The test results in (b) show that the average allocation power of all three learning schemes is lower than that of the LCGP scheme, and the DDPG-COPA scheme has the lowest average allocation power. Furthermore, as the task arrival rate increases, the average allocation power of the three learning schemes increases much more slowly compared to LCGP. Figure 7 (c) shows that all four schemes produce roughly the same average queue length, which increases with task arrival rate. Because the DDPG-COPA scheme learns the temporal relevance of environmental states, its power allocation actions based on these states are more accurate. Furthermore, as task data increases, all three learning schemes focus on allocating more power to increase rewards. Therefore, the DDPG-COPA scheme has a lower average power allocation than the other two learning schemes, and the average buffer queue lengths of the three learning schemes are similar.

[0190] 2) Impact of weighting coefficients: The performance of the four schemes under different weighting coefficients was compared. During the test, the number of PMs was fixed at M=5, the arrival rate of task data λ=4Mbps, and the PL driving speed v0=15m / s. Figure 8 The test results for four schemes are shown in terms of average reward, average energy consumption, and average buffer queue length. From Figure 8 (a) It can be seen that the average reward of the DDPG-COPA scheme is greater than that of the other three benchmark schemes with different ω1 values. As ω1 increases, the average reward of all four schemes increases, and the proposed DDPG-COPA consistently has the highest average reward. Figure 8 The test results in (b) show that the DDPG-COPA scheme has the lowest average energy consumption for all ω1 values. As ω1 increases, the average energy consumption of all three learning schemes decreases, while the average energy consumption of the LCGP scheme remains constant. Figure 8As shown in (c), the average buffer queue length of all three learning schemes increases with increasing ω1, and the LCGP scheme has the smallest average buffer queue length, which remains constant. This is because when ω1 is small, the low penalty for energy consumption allows all three learning schemes to allocate higher power, which helps reduce the buffer queue length. Since the DDPG-COPA scheme can perform continuous and accurate power allocation, its average energy consumption is significantly lower than the other three benchmark schemes. Compared to the other two learning schemes, the average buffer queue length of DDPG-COPA is less affected by the weighting coefficient ω1. Furthermore, the average buffer queue length of the LCGP scheme is not affected by ω1 because it always allocates maximum power.

[0191] 3) Impact of PL driving speed: The performance of the four schemes at different PL driving speeds was evaluated. During the test, the number of PMs was fixed at M=5, the task data arrival rate was λ=4Mbps, and the weighting coefficient ω1=0.5. Figure 9 (a) shows the trend of platoon balance distance at different PL speeds. As can be seen from the figure, the platoon balance distance increases with increasing PL speed to maintain a safe following distance. Figure 9 The test results in (b) show that the average reward remains essentially unchanged across the four schemes, and the training models of the three learning schemes can adapt to different driving speeds of the PL. Furthermore, the average reward of DDPG-COPA is superior to the other three schemes. This is because the environmental state observed by the agent is related to the motion characteristics of the convoy vehicles. The agent in the DDPG-COPA scheme can learn a more reasonable power allocation strategy based on different driving states of the convoy vehicles. Therefore, at different PL driving speeds, the test performance of this scheme is better than other benchmark schemes.

[0192] 4) Impact of PM quantity: The performance of four schemes with different PM quantities was compared. During the test, the data arrival rate of the fixed task was λ = 4 Mbps, the weighting coefficient ω1 = 0.5, and the travel speed of the PL was v0 = 15 m / s. Figure 10The test results for four schemes under different numbers of PMs are presented. The graph shows that the DDPG-COPA scheme achieved the highest reward for different fleet sizes. The average reward of all three learning schemes was higher than that of the LCGP scheme. Furthermore, as the number of PMs increases, the average reward of all four schemes decreases, with the three learning schemes experiencing a greater decrease than the LCGP scheme. This is because as the fleet size increases, the communication distance between the PM at the end of the fleet and the PL (Pulse Controller) is longer. Additionally, PMs consume more energy in data transmission to combat co-channel interference caused by more vehicles in the fleet. Therefore, the performance of the three learning schemes is significantly affected by the number of PMs. In the LCGP scheme, only a small portion of the task data is offloaded to the PL, so changes in communication conditions have a smaller impact on this scheme.

[0193] This invention designs a computation offloading and power allocation scheme for dynamic fleet networks, and addresses the problem of minimizing the computational cost of the fleet system under conditions of random task arrival, time-varying wireless channels, and vehicle cooperation. The goal is to reduce the average energy consumption and task processing latency of the fleet system. It provides a modeling approach for computation offloading and resource allocation schemes in fleet networks and offers a solution to the joint optimization problem in dynamic fleet networks.

Claims

1. A method for computational offloading and power allocation within a fleet network based on deep reinforcement learning, characterized in that, The fleet system is as follows: The convoy consists of a lead vehicle (PL) and a group of member vehicles (PM) traveling in the same direction. The PL is equipped with an edge computing EC server to provide computation offloading services for resource-constrained PMs; the timeline is discretized into T time slots, each with a duration of τ0, and the index set of the time slots is represented as: T={1,2,3,...,t,...,T}. In each time slot t, the PM can generate task data; the queue of vehicles is represented as a set M={0,1,2,...m,...,M}, where m=0 represents the PL and M represents the total number of PMs; each PM can execute task data locally or offload it to the PL via V2V communication; NOMA technology is used to implement the link between PMs and PLs; Creating a dynamic motion model for the vehicle fleet: The Intelligent Driving Model (IDM) is used to characterize the following behavior of platooned vehicles. The initial position of the last PM is defined as the origin of the coordinate system, and only the longitudinal velocity and longitudinal position of the vehicles in the platoon are considered. At time slot t, PM m The velocity and acceleration are respectively v m (t) and a m (t); PM at time slot t m The head coordinate is x m Let m ∈ {1,2,...,M}, then the relative velocity is (t). PM m The inter-vehicle distance between the vehicle and the vehicle in front, that is, the distance between the front of the vehicle and the rear of the vehicle in front. ,in express Vehicle length; PM m The state of motion depends on Kinematic characteristics, according to IDM, PM m The acceleration can be written as: （1） Among them, a max It is the maximum acceleration, v max The ideal speed under steady traffic flow, δ∈[1,5] represents the driver sensitivity characteristics, and f(·) is a function related to speed and relative speed, defined as: （2） Where d0 represents the ideal frontage distance, t0 represents the ideal headway distance, and b max This indicates maximum deceleration; therefore, PM m The velocity and position at time slot t are: （3） （4） Therefore, PM m The communication distance between PL and PL is: （5） Team communication model At time t, PL receives PM from the uplink. m The received signal is calculated in the following way: （6） Where: p m,o (t)∈ It is PM m The transmission power, It is PM m Maximum transmit power, h m (t) is PM m The channel gain between PL and c m (t) is a complex data signal; y 0,m The first part of n(t) is the desired signal from the target PM, the second part is the interference from other PMs, and the third part is additive white Gaussian noise. When the PL receives a signal, it uses the SIC decoding scheme to reduce interference from other PMs by following the descending order of the signal: （7） in, PM at time slot t m The channel gain between PL and PL is: （8） Where: g0 is the channel power gain at a reference distance of one meter, α is the path loss exponent, and n0(t) is a complex Gaussian variable, then PM m The corresponding signal interference plus noise SINR is: （9） Where σ 2 It is white noise power; According to Shannon's formula, given a bandwidth B, the PM in the uplink of time slot t... m The data transmission rate is: （10） Establish a computational model: PM m Task data is temporarily stored in a buffer queue; for PM m The generated task data follows an average value of λ. m The Poisson distribution; (1) Local computing model PM m The amount of task data that can be processed during time slot t is: （11） in It is PM m The allocated CPU cycles per second It is the allocated local execution power. It is the maximum computing power, κ m It is PM m The effective switched capacitor of the processor, L0 is the number of CPU cycles required to compute each bit of task data; (2) Calculate the unloading model PM m The computational unloading process includes three steps: 1) PM m 1) Transfer the task data to PL; 2) PL's EC server calculates the unloaded task data; 3) PL returns the execution results to PM. m Since the data size after execution at PL is much smaller than the input data size, the transmission time is ignored in the third step. The first and second steps can be completed within one time slot τ0, and the transmission time of the task data is t. m,off (t), then the data processing time in the EC server is According to (10), the system of equations can be obtained: （12） Where q m,o (t) is PM m Uninstallation task data, The EC server is assigned to PM m CPU cycle count It is PL that assigns to PM m Calculated power, κ0 is the maximum computing power, and κ0 is the effective switched capacitor of the EC server processor; by solving (12), the transmission time t can be obtained. m,off (t); PM m The size of the task data that can be unloaded in time slot t is: （13） Problem modeling: For PM in time slot t m The sizes of the task data computed locally and unloaded to the PL are q respectively. m,l (t) and q m,o (t), therefore, the size of all executed task data is q. m,l (t)+q m,o (t), using Q m (t) represents the current length of the buffer queue, o m (t) represents the number of task data generated in time slot t, and the length of the buffer queue in the next time slot t+1 is: （14） Based on queuing theory and Little's formula, the average buffer queue length is used to represent the average task data computation latency. The EC server installed in the PL has a high-frequency multi-core CPU, and the EC server can compute the unloading task data in parallel. To ensure that the PM's unloading task data can be processed immediately, Q0(t) = 0 is set. The average buffer queue length of the fleet system is calculated as follows: （15） The average energy consumption of the fleet system is written as: （16） The model aims to minimize the weighted sum of the average energy consumption and task data processing latency of the fleet system, and jointly optimize computational offloading and power allocation under dynamic conditions; let P = {p m,l p m,o p 0,m If} is the decision variable, then the power decision problem is expressed as: （17） DDPG-based computational offloading and power allocation: Based on the multi-agent DRL approach, the fleet network is viewed as an environment, and each PM is considered an agent capable of interacting with the dynamic environment. Since the PL has a high-performance multi-core CPU, it processes the offloading task data immediately, ignoring the PL's computational latency, i.e., t... m,off (t)=τ0; the goal of each agent is to maximize the long-term discounted reward; during the learning process, the agent's state-based decisions are reward-driven, therefore, how the reward function is defined will directly affect the quality of the agent's learning strategy; the agent's environment state space, action space, and reward function are as follows: 1) State Space PM m state s m (t)∈S m It consists of local state and global state; the local state includes the generated task data. m (t) Length of the buffer queue Q m (t) and channel gain h m (t), the global state includes SINR ζ m (t); Therefore, the state space of the m-th agent is: （18） 2) Action Space Based on state s m (t), the agent performs actions including local computation of power p m,l (t) and transmission power p m,o The joint action of (t); therefore, the action space of the m-th agent is: （19） In each learning step, to ensure more efficient stochastic exploration of the environment state space to obtain the optimal policy, OU random noise is added to the operation space: （20） 3) Reward function Based on the reward function defined in (17), the agent can learn the optimal policy step by step in the iteration. The design of the reward function is as follows: Energy Consumption Reward: During the learning process, to enable the agent to learn the optimal power allocation strategy, the energy consumption at each moment is added to the reward function; the energy consumption calculated from the task data is: （21） Calculating the latency reward: The buffer queue length can measure the computation latency of task data; to reduce the execution latency of task data, Q is introduced into the reward function. m (t); Punishment / Reward: The agent's behavior affects the length of the buffer queue; in (14), if Q m If (t) is less than 0, it indicates that the agent is over-allocating computational power; if Q m If (t) exceeds the threshold, it means that the power allocated to the agent is insufficient to fully compute the arriving task data, resulting in task data accumulation. To prevent this problem, two penalty terms are added to the reward function to ensure that the agent performs more accurate operations: （22） Where p m,over (t) represents the over-allocated local computing power, and P1 and P2 are the over-allocation power and the penalty factor for exceeding the threshold, respectively; φ m (t) is a binary function that equals 1 when the length of the buffer queue exceeds its maximum value, and equals 0 otherwise; Therefore, based on the above definition, by summing the reward values ​​using weighted coefficients ω1 and ω2, the reward function for agent m is: （23） The learning algorithm consists of a learning process and a testing process; for each learning step, the m-th agent obtains its local state s from the current environment. m (t) and output the deterministic action µ m (t); In order for the agent to fully explore the environment to obtain the optimal strategy, the OU noise σ OU (t) added to µ m In (t), the agent will receive r after completing the action output. m (t) Reward, evaluating how good the action is; after all agents complete the process, the environment enters the next state, and each agent will receive a transition tuple (s). m (t), u m (t), r m (t), s m+1 (t)), this tuple will be stored in the experience replay pool B. m The network is used for model training; therefore, for each agent, a mini-batch of samples containing N transformation tuples is randomly selected from the experience replay pool and fed into the DDPG to update the network parameters; the target Q-value of the target network is calculated as follows: （24） Where γ is the discount factor; the online critic network updates its parameters using stochastic gradient descent, as shown below: （25） （26） Where α w It is the learning rate of the online critic network; based on the gradient provided by the online critic. The Actor online network uses a deterministic policy gradient method to update network parameters: （27） （28） Where α θ It is the learning rate of the actor's online network; the soft update method is used to update two target networks through two online networks: （29） （30） Where τ is the update coefficient, and τ << 1.

Images