RSU-assisted Internet of Vehicles multi-hop task unloading optimization method based on deep reinforcement learning

Abstract

The invention discloses an RSU-assisted Internet of Vehicles multi-hop task unloading optimization method based on deep reinforcement learning, and belongs to the technical field of edge computing and task unloading. The method comprises the following steps of: 1, actively sensing road information by a sensor of the RSU according to a road environment, and generating a sensing task; 2, establishing an unloading time delay model and an unloading cost model according to different task unloading nodes, and then constructing a target function for unloading decision according to the established unloading time delay model and unloading cost model; 3, according to the objective function, converting a task decision problem into a Markov decision process problem, and selecting a calculation unloading node which enables task processing time delay and task processing cost to be minimum; and 4, realizing common optimization of task processing time delay and task processing cost by using an A3C algorithm to obtain an optimal task decision. According to the invention, multi-hop task unloading under vehicle movement is realized, computing resources outside the RSU communication range are reasonably utilized, and the communication link quality is ensured.

Description

Technical Field

[0001] The present invention relates to the field of edge computing technology, and in particular to an RSU-assisted vehicle network multi-hop task offloading optimization method based on deep reinforcement learning. Background Art

[0002] In recent years, the rapid development of autonomous driving technology has fundamentally changed the transportation system. The integration of mobile edge computing (MEC) and the Internet of Vehicles (IoV) provides users with safer and more convenient transportation services and experiences. In the IoV sector, the use of roadside units (RSUs) helps vehicles perceive their road environment, which is crucial for autonomous navigation.

[0003] Due to limited computing power, RSU needs to offload the tasks it perceives to cloud servers, other RSUs, or vehicles for processing. However, various task offloading schemes in the prior art usually only consider single-hop communication, and RSU can only offload tasks to vehicles within its coverage area for processing. Although vehicles outside the RSU coverage area also have idle computing resources, and from the perspective of utilizing the computing resources of the entire system, ideally, RSU can offload tasks to vehicles outside its coverage area through multi-hop communication. However, the high mobility of vehicles poses a challenge to maintaining communication links during multi-hop transmission.

[0004] Researchers have proposed collaborative edge computing frameworks to reduce computing latency and improve the reliability of in-vehicle networks. Others have developed collaborative computing methods based on deep reinforcement learning to determine strategies for task offloading, computation, and result delivery between vehicles. This approach aims to find optimal solutions in complex urban transportation networks to minimize computational service latency and penalties for service failures. However, these existing methods fail to consider the multi-hop offloading of tasks between vehicles, failing to fully utilize the computing resources of the entire system. Summary of the Invention

[0005] 1. Technical problem to be solved by the invention

[0006] In view of the above-mentioned deficiencies in the existing technology, the present invention provides an RSU-assisted vehicle network multi-hop task offloading optimization method based on deep reinforcement learning. The present invention realizes multi-hop task offloading under vehicle movement, rationally utilizes computing resources outside the RSU communication range, and ensures the quality of the communication link.

[0007] 2. Technical solution

[0008] In order to achieve the above object, the technical solution provided by the present invention is:

[0009] The present invention provides an RSU-assisted vehicle network multi-hop task offloading optimization method based on deep reinforcement learning, which comprises the following steps:

[0010] Step 1: Based on the road environment, the sensors on the RSU actively sense the road information and generate perception tasks;

[0011] Step 2: Establish an offloading delay model and an offloading cost model for the corresponding sensing task according to different task offloading nodes, and then construct an objective function for offloading decision based on the established offloading delay model and offloading cost model;

[0012] Step 3: According to the objective function, the task decision problem is transformed into a Markov decision process problem, and the computation offloading node that minimizes the task processing delay and task processing cost is selected;

[0013] Step 4: Use the A3C algorithm to jointly optimize task processing delay and task processing cost to obtain the optimal task decision.

[0014] Furthermore, the offloading delay model is the total processing delay D of the task being calculated on the vehicle, on the RSU, and on the cloud server. r (t); The offloading cost model is the total processing cost C of the task calculated on the vehicle, on the RSU and on the cloud server r (t).

[0015] Furthermore, when the task is offloaded to the vehicle for processing, the communication link quality needs to be evaluated to find a suitable communication link, including V2V link quality evaluation and R2V link quality evaluation. Among them, the probability that the change in the relative distance between vehicle i and vehicle j is less than the remaining communication distance between vehicle i and vehicle j is used as the evaluation of V2V link quality; the probability that the change in the relative distance between RSUr and vehicle i is less than the remaining communication distance between RSUr and vehicle i is used as the evaluation of R2V link quality.

[0016] Furthermore, offloading the task f(t) from RSUr to vehicle v involves three steps: finding candidate vehicles, determining the transfer path, and executing the offloading decision; where:

[0017] In the process of searching for candidate vehicles, the RSUr sends a service request to all vehicles within its communication range. Upon receiving the request, each vehicle further relays the request to other vehicles within the communication range. When a vehicle receives a service request, it records all the forwarding nodes that the request passes through before reaching the vehicle. The vehicle then sends the recorded information back to the RSUr together with the current status.

[0018] After finding the candidate vehicles, we will get the set ζ of all candidate vehicles that can communicate with RSUr r (t), assuming that there is a set of η paths between RSUr and candidate vehicle v Given a path h(r, v) consisting of u single-hop links: RSU r→vehicle h(1)→vehicle h(2)→…→vehicle h(u-2)→vehicle h(u-1)→vehicle v, where The quality of a path h(r, v) is defined as the product of the connection probabilities of each single-hop link, expressed as follows:

[0019]

[0020] The optimal path for sending task f(t) from RSUr to candidate vehicle v is the path with the highest quality among all paths;

[0021] After obtaining candidate vehicles and the optimal transmission path, RSUr makes an offloading decision and offloads the task to a suitable vehicle for computation.

[0022] Furthermore, the total processing delay of task offloading to the vehicle, the total processing delay of task offloading to the RSU, and the total processing delay of task offloading to the cloud server are all composed of transmission delay, waiting delay, calculation delay and result transmission delay, among which the result transmission delay is negligible.

[0023] Furthermore, the total processing delay of the task on the vehicle, on the RSU, and on the cloud server is:

[0024]

[0025] Among them, {1, 2, ..., v, ... V} is the vehicle set, {1, 2, ..., r, ... R} is the RSU set, is the total processing delay of unloading the task to a vehicle in the vehicle set, is the total processing delay of the task offloading to an RSU in the RSU set, is the total processing delay of the task offloading to the cloud server, x r，y (t)∈{0, 1} is a binary variable, and y∈{1,…,V,V+1,…,V+R,V+R+1} represents the unloading node.

[0026] Furthermore, the total processing cost of task offloading to the vehicle and the total processing cost of task offloading to the RSU are both composed of relay cost and computation cost, while the total processing cost of task offloading to the cloud server is composed of computation cost;

[0027] The total processing cost of the task on the vehicle, on the RSU, and on the cloud server is:

[0028]

[0029] Among them, {1, 2, ..., v, ... V} is the vehicle set, {1, 2, ..., r, ... R} is the RSU set, is the total processing cost of offloading the task to a vehicle in the vehicle set, is the total processing cost of offloading the task to an RSU in the RSU set, is the total processing cost of offloading tasks to the cloud server, x r，y (t)∈{0, 1} is a binary variable, and y∈{1,...,V,V+1,...,V+R,V+R+1} represents the unloading node.

[0030] Furthermore, the objective function is constructed as follows:

[0031]

[0032] st

[0033]

[0034]

[0035]

[0036]

[0037] Where α is a weight factor, constraints C1-C2 indicate that the task is calculated on a single offloading node; constraint C3 ensures that the task is calculated on its valid time d r Constraint C4 ensures that the connection probability of the communication link will not be 0, and constraint C5 ensures that the task will not be offloaded to vehicles that cannot communicate with the RSUs that perceive them.

[0038] Furthermore, the specific process of step three is as follows:

[0039] The Markov decision process includes three key elements: state, action, and reward, which are defined as follows:

[0040] state:

[0041] The state space consists of the task queue information Q(t) and computational cost information q(t) of each computing node. In each time slot t, the agent monitors the network environment and dynamically collects the system status:

[0042] s(t)={Q(t),q(t)}

[0043] action:

[0044] After observing the state s(t) from the environment, the agent makes an unloading decision:

[0045] a(t)={x1,x2,...,x r ,...,x R}

[0046] where x r =x r，y (t), y∈{1,...,V,V+1,...,V+R,V+R+1},x r Offloading decisions made for RSUr;

[0047] award:

[0048] The design of the reward function is the key to reinforcement learning. Generally speaking, the reward function is related to the established objective function. The optimization goal is to obtain the maximum system reward. If the constraints C1-C5 are met, the reward function is the negative value of the objective function. If the constraints C1-C5 are not met, the reward function is -Z. The reward function is expressed as:

[0049]

[0050] Furthermore, the specific process of step 4 is as follows:

[0051] The state information obtained by the interaction between RSU and the environment in each time slot is input into the global actor network θ and the global critic network θ v , as well as the local actor network θ′ and the local critic network θ′ in each worker thread v , the initial values ​​of the parameters of the local network are synchronized with the global network;

[0052] In each training cycle, each worker thread uses the local networks θ′ and θ′ v Interact with the environment and collect sample data, including state s(t), action a(t), reward r(t), and next state s(t+1); based on these samples, the training goal is to minimize the loss of the local actor network and the local critic network;

[0053] Each worker thread uses the local network to calculate the gradient, and after a certain number of training steps, sends the gradient information of the local network to the global network θ and θ v , the global network updates its parameters accordingly; subsequently, the local network synchronizes the latest parameters from the global network;

[0054] After training, the optimized global policy networks θ and θ are used v , the optimal task decision process is given based on the state information obtained by RSU.

[0055] 3. Beneficial effects

[0056] Compared with the existing known technologies, the technical solution provided by the present invention has the following significant effects:

[0057] (1) The present invention proposes an RSU-assisted vehicle network multi-hop task offloading optimization method based on deep reinforcement learning. This method combines deep reinforcement learning with edge computing structure and introduces the combination of the two into the RSU-assisted vehicle network multi-hop task offloading optimization problem. The RSU interacts with the road environment, perceives the road environment, generates corresponding tasks to be processed, and communicates with adjacent RSUs, vehicles within its communication range, and cloud servers to perform single-hop task offloading. At the same time, in order to utilize computing resources outside the RSU communication range, the RSU communicates with vehicles outside its coverage area through relay vehicles, realizing multi-hop task offloading.

[0058] (2) The present invention selects a suitable computing offloading node based on the task processing delay and task processing cost after task offloading. When the task is selected to be offloaded to the vehicle for calculation, due to the mobility of the vehicle, there is a risk of communication link disconnection during the task transmission process. The present invention models and evaluates the quality of the communication link to find a suitable communication link. Due to the complexity of road information, the joint optimization of task processing delay and processing cost is a difficult non-convex problem to solve. The present invention uses the exploration-exploitation strategy in deep reinforcement learning, which can effectively explore in the strategy space to find a better solution. At the same time, the present invention converts the problem into a Markov decision process and applies the asynchronous advantage actor-critic (A3C) algorithm in the deep reinforcement learning algorithm to solve the non-convex problem well. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a system scenario diagram of the present invention;

[0060] Figure 2 This is a flowchart of the RSU-assisted vehicle network multi-hop task offloading optimization method based on deep reinforcement learning of the present invention;

[0061] Figure 3 Schematic diagram of the A3C parameterized target strategy of the present invention;

[0062] Figure 4 、 5 ,6 are comparison charts of reward value, task processing delay, and task processing cost under different algorithms. DETAILED DESCRIPTION

[0063] In order to further understand the content of the present invention, the present invention is described in detail with reference to the accompanying drawings and embodiments.

[0064] like Figure 1As shown, this embodiment considers a road environment with a cloud server, multiple RSUs, and multiple vehicles, which are considered three different types of computing nodes. Each computing node has a queue of tasks to be processed. In the road environment, each RSU actively perceives road environment information and generates perception tasks. During the task processing process, it can choose to offload tasks to adjacent RSUs, vehicles within its communication range, the cloud server, or even to vehicles outside its coverage area via relay vehicles.

[0065] See Figure 2 , this embodiment specifically includes the following steps:

[0066] Step 1: Based on the road environment, the sensors on the RSU actively perceive road information and generate perception tasks.

[0067] Step 2: Calculate the corresponding task offloading delay and offloading cost according to different task offloading nodes, and then design the optimization objective function based on the calculated task offloading delay and offloading cost to minimize the task processing delay and processing cost. Specifically:

[0068] When an RSU generates a sensing task, the processing of the sensing task can only be offloaded to a certain computing offload node. According to different computing offload nodes, this embodiment calculates the task processing delay and task processing cost respectively. Among them:

[0069] The offloading delay model is the total processing delay of the task calculation on the vehicle, on the RSU, and on the cloud server. The details are as follows:

[0070] When the task is offloaded to the vehicle for processing, the communication link quality needs to be evaluated to find a suitable communication link, including V2V link quality evaluation and R2V link quality evaluation.

[0071] The speed change of vehicle i in the time interval Δt follows the standard Gaussian distribution, expressed as Δv i ~N(0,σ i 2 Δt), where σ i 2 is the variance of vehicle i’s speed. The speed change of vehicle j in the time interval Δt follows the standard Gaussian distribution, expressed as Δv j ~N(0,σ j 2 Δt). According to the principle of linear combination of Gaussian variables, Δv i -Δv j Also follows the Gaussian distribution, expressed as:

[0072] Δv i -Δv j ~N(0,σ i2 Δt+σ j 2 Δt)

[0073] The relative distance change between adjacent vehicles i and j under Δt is expressed as:

[0074]

[0075] Therefore, the connectivity probability between vehicle i and vehicle j within Δt can be predicted. The connectivity probability is the probability that the change in the relative distance between the two is less than the remaining communication distance between the two. It is used as an evaluation of the V2V link quality and is calculated as follows:

[0076] p i，j (Δt)=p{d i，j (Δt)<D i，j}

[0077] Among them D i，j is the remaining communication distance between vehicle i and vehicle j.

[0078] Similarly, the relative distance change between RSUr and vehicle i under Δt is expressed as:

[0079]

[0080] Therefore, the connectivity probability between RSUr and vehicle i within Δt can be predicted. The connectivity probability is the probability that the change in the relative distance between RSUr and vehicle i is less than the remaining communication distance between them. It is used as an evaluation of the R2V link quality and is calculated as follows:

[0081] p r，i (Δt)=p{d r，i (Δt)<D r，i}

[0082] Among them D r，i is the remaining communication distance between RSUr and vehicle i.

[0083] The evaluation results of V2V link quality and R2V link quality are used to calculate the quality of the path h(r, v) below.

[0084] Offloading a task f(t) from RSUr to vehicle v involves three steps: finding candidate vehicles, determining the transfer path, and executing the offloading decision.

[0085] When searching for candidate vehicles, the RSUr sends a service request to all vehicles within its communication range. Upon receiving a request, each vehicle further relays the request to other vehicles within its communication range. When a vehicle receives a service request, it records all forwarding nodes the request passed through before reaching the vehicle. The vehicle then sends this information back to the RSUr along with its current status.

[0086] After finding the candidate vehicles, we will get the set ζ of all candidate vehicles that can communicate with RSUr r (t). Assume that there is a set of η paths between RSUr and candidate vehicle v Given a path h(r, v) consisting of u single-hop links: RSU r→vehicle h(1)→vehicle h(2)→…→vehicle h(u-2)→vehicle h(u-1)→vehicle v, where The quality of a path h(r, v) is defined as the product of the connection probabilities of each single-hop link, expressed as follows:

[0087]

[0088] The optimal path for sending task f(t) from RSUr to candidate vehicle v is the path with the largest product (i.e., the highest quality) among all paths.

[0089] After obtaining candidate vehicles and the optimal transmission path, RSUr makes an offloading decision and offloads the task to a suitable vehicle for computation.

[0090] According to Shannon's theorem, the data transmission rate between two adjacent vehicles i and j in time slot t is The data transmission rate between RSUr and vehicle i is

[0091] The total processing delay of task offloading to the vehicle includes transmission delay, waiting delay, calculation delay and result transmission delay.

[0092] The task f(t) perceived by RSUr is offloaded to vehicle v through the optimal path P(r, v). The optimal path consists of u single-hop links: RSUr → vehicle P(1) → vehicle P(2) → … → vehicle P(u-2) → vehicle P(u-1) → vehicle v. The transmission delay is calculated as follows:

[0093]

[0094] where t max Indicates the maximum transmission waiting time. If the sender cannot receive a response from the receiver within this period, the task transmission fails. r is the task size. When u=1, it indicates a single-hop transmission. Otherwise, it indicates a multi-hop transmission.

[0095] Before vehicle v can execute the computation of task f(t), it must first process all tasks in the queue. The waiting delay is calculated as follows:

[0096]

[0097] Where L(v, t) is the task queue of vehicle v in time slot t, represents the mth task in the task queue, ξ v represents the handling capability of vehicle v.

[0098] The calculation formula for calculating the delay is as follows:

[0099]

[0100] in is the computing resources required to process the task, ξ v represents the handling capability of vehicle v.

[0101] The processing result is very small compared to the task size, so the result transmission delay is negligible. Therefore, the total processing delay of the task offloaded to vehicle v is The calculation formula is as follows:

[0102]

[0103] If the unloading decision selects a vehicle v that can communicate with RSUr, the total processing delay is calculated according to the formula. Otherwise, the total processing delay is set to 0.

[0104] When the task is offloaded to the RSU for calculation:

[0105] The task f(t) is offloaded from RSUr to RSUk via z single-hop links for computation. The total processing delay includes transmission delay, waiting delay, computation delay, and result transmission delay. The transmission delay is calculated as follows:

[0106]

[0107] in Indicates the transmission rate between RSUs. r is the task size. If the task is processed locally, the transmission delay is zero. Otherwise, the transmission delay is calculated according to the formula.

[0108] The waiting delay calculation formula is as follows:

[0109]

[0110] Where L(k, t) is the task queue of RSUk in time slot t. represents the mth task in the task queue, ξ k is the processing capability of RSUk.

[0111] The calculation formula for calculating the delay is as follows:

[0112]

[0113] in is the computing resources required to process the task, ξ k Indicates the processing capability of RSUk.

[0114] The processing result is very small compared to the task size, so the result transmission delay can be ignored. Therefore, the total processing delay of the task offloaded to RSUk is The calculation formula is as follows:

[0115]

[0116] When tasks are offloaded to cloud servers for computation:

[0117] The task f(t) is offloaded from the RSUr to the cloud server for computation. The total processing delay includes transmission delay, waiting delay, computation delay, and result transmission delay. The transmission delay calculation formula is as follows:

[0118]

[0119] in Indicates the transmission rate between RSUr and cloud server. r Task size

[0120] The waiting delay calculation formula is as follows:

[0121]

[0122] Where L(c, t) is the task queue of the cloud server at time slot t. represents the mth task in the task queue, ξ c The processing power of the cloud server.

[0123] The calculation formula for calculating the delay is as follows:

[0124]

[0125] in is the computing resources required to process the task. c The processing power of the cloud server.

[0126] The processing result is very small compared to the task size, so the result transmission delay is negligible. Therefore, the total processing delay of the task offloaded to the cloud server is The calculation formula is as follows:

[0127]

[0128] Based on the processing delay of different computing offloading nodes, the total processing delay of the task calculation on the vehicle, on the RSU, and on the cloud server is:

[0129]

[0130] Among them, {1, 2, ..., v, ... V} is the vehicle set, {1, 2, ..., r, ... R} is the RSU set, is the total processing delay of unloading the task to a vehicle in the vehicle set, is the total processing delay of the task offloading to an RSU in the RSU set, is the total processing delay of the task offloading to the cloud server, x r，y (t)∈{0, 1} is a binary variable, and y∈{1,...,V,V+1,...,V+R,V+R+1} represents the unloading node.

[0131] The offloading cost model is the total processing cost of the task calculation on the vehicle, on the RSU, and on the cloud server.

[0132] The details are as follows:

[0133] When the task is calculated on the vehicle:

[0134] Task f(t) is unloaded to vehicle v via the optimal path P(r, v): (RSU r → vehicle P(1) → vehicle P(2) → … → vehicle P(u-2) → vehicle P(u-1) → vehicle v), paying a certain relay cost to the relay vehicles on the path. The total processing cost includes relay cost and computation cost. The formula for calculating relay cost is as follows:

[0135]

[0136] in is the relay cost of the relay vehicle, s r is the task size. If it is a single-hop transmission, the relay cost is zero. Otherwise, the relay cost is calculated according to the formula.

[0137] The calculation formula for calculating the cost is as follows:

[0138]

[0139] in is the computational cost of vehicle v, It is the computing resources required to process the task.

[0140] The total processing cost of task offloading to vehicle v The calculation formula is as follows:

[0141]

[0142] If the offloading decision selects a vehicle v that is able to communicate with RSUr, the total processing cost is calculated according to Eq. Otherwise, the total processing cost is set to 0.

[0143] When tasks are calculated on RSUs:

[0144] The task f(t) is offloaded from RSUr to RSUk via z single-hop links for computation. The total processing cost includes relay cost and computation cost. The relay cost is calculated as follows:

[0145]

[0146] in is the relay cost of relay RSU. r is the task size. If the task is processed locally or transmitted in a single hop, the relay cost is zero. Otherwise, the relay cost is calculated according to the formula.

[0147] The calculation formula for calculating the cost is as follows:

[0148]

[0149] in is the computational cost of RSUk, It is the computing resources required to process the task.

[0150] Total processing cost of task offloading to RSUk The calculation formula is as follows:

[0151]

[0152] When the task is calculated on the cloud server:

[0153] The task f(t) is offloaded from RSUr to the cloud server for computation, which only has computational cost. Therefore, the total processing cost is expressed as:

[0154]

[0155] in is the computing cost of the cloud server, It is the computing resources required to process the task.

[0156] According to the processing costs of different computing offloading nodes, the total processing cost of the task calculated on the vehicle, on the RSU, and on the cloud server is:

[0157]

[0158] Among them, {1, 2, ..., v, ... V} is the vehicle set, {1, 2, ..., r, ... R} is the RSU set, is the total processing cost of offloading the task to a vehicle in the vehicle set, is the total processing cost of offloading the task to an RSU in the RSU set, is the total processing cost of offloading tasks to the cloud server, x r，y (t)∈{0, 1} is a binary variable, and y∈{1,...,V,V+1,...,V+R,V+R+1} represents the unloading node.

[0159] The goal of this invention is to continuously optimize the offloading decision to minimize the total processing delay and total processing cost of the task perceived by each RSU. The optimization objective function is as follows:

[0160]

[0161] st

[0162]

[0163]

[0164]

[0165]

[0166] Where α is a weight factor. Constraints C1-C2 indicate that the task is calculated on a single offloading node. Constraint C3 ensures that the task is calculated on its valid time d r Constraint C4 ensures that the connection probability of the communication link will not be 0. Constraint C5 ensures that the task will not be offloaded to vehicles that cannot communicate with the RSU that perceives them. The proposed optimization problem is a typical 0-1 planning problem.

[0167] Step 3: According to the objective function, the task decision problem is transformed into a Markov decision process problem, and the computation offloading node that minimizes the task processing delay and task processing cost is selected.

[0168] In step three, the specific contents are as follows:

[0169] The Markov decision process includes three key elements: state, action, and reward, which are defined as follows:

[0170] state:

[0171] The state space consists of the task queue information Q(t) and computational cost information q(t) of each computing node. In each time slot t, the agent monitors the network environment and dynamically collects the system state:

[0172] s(t)={Q(t),q(t)}

[0173] action:

[0174] After observing the state s(t) from the environment, the agent makes an unloading decision:

[0175] a(t)={x1,x2,...,x r ,...,x R}

[0176] where x r =x r，y (t), y∈{1,…,V,V+1,…,V+R,V+R+1},x r Offloading decisions made for RSUr.

[0177] award:

[0178] The design of the reward function is key to reinforcement learning. Generally speaking, the reward function is related to the established objective function. The optimization goal is to obtain the maximum system reward. If the constraints C1-C5 are met, the reward function is the negative value of the objective function. If the constraints C1-C5 are not met, the reward function is -Z. The reward function is expressed as:

[0179]

[0180] Step 4: Use the A3C algorithm to solve the joint optimization problem of task processing delay and task processing cost to obtain the optimal task decision.

[0181] In step 4, the specific contents are as follows: the state information obtained by the interaction between RSU and the environment in each time slot is input into the global actor network θ and the global critic network θ v , as well as the local actor network θ′ and the local critic network θ′ in each worker thread v The initial values ​​of the parameters of the local network are synchronized with the global network.

[0182] In each training cycle, each worker thread uses the local network (θ′ and θ′ v ) interacts with the environment and collects sample data, including state s(t), action a(t), reward r(t), and next state s(t+1). Based on these samples, the training goal is to minimize the loss of the local actor network and the local critic network:

[0183] The loss function of the local actor network is expressed as:

[0184] L π (θ′)=-logπ(a(t)|s(t);θ′)A(s(t),a(t))

[0185] where π(a(t)|s(t);θ′) is the output of the local actor network, A(s(t), a(t)) is the advantage function, and A(s(t), a(t)) = R t -V(s(t);θ′ v ), V(s(t);θ′ v ) is the output of the local critic network, R t It is the cumulative discount reward under the e-step.

[0186] The loss function of the local critic network is expressed as:

[0187] L V (θ′ v )=(R t -V(s t θ′ v )) 2

[0188] Each worker thread uses the local network to calculate the gradient, and after a certain number of training steps, sends the gradient information of the local network to the global network (θ and θ v ), the global network updates its parameters accordingly. Subsequently, the local networks synchronize the latest parameters from the global network.

[0189] After training, the optimized global policy network (θ and θ v ), and gives the optimal task decision process based on the state information obtained by RSU.

[0190] Figure 4 、 5 6 are comparison diagrams of reward value, task processing delay, and task processing cost under different algorithms. It can be seen from the figure that the present invention achieves a larger reward value, lower task processing delay and task processing cost.

[0191] In summary, the above-described embodiment of a deep reinforcement learning-based RSU-assisted multi-hop task offloading optimization method for connected vehicles combines deep reinforcement learning with edge computing architectures, applying the two approaches to the RSU-assisted multi-hop task offloading optimization problem. The RSU interacts with the road environment, perceives it, and generates corresponding tasks to be processed. The RSU can communicate with neighboring RSUs, vehicles within its communication range, and cloud servers. Furthermore, to utilize computing resources outside the RSU's communication range, the RSU communicates with vehicles outside its coverage area via relay vehicles for multi-hop task offloading. Tasks can be offloaded to the RSU, cloud server, or vehicle for computation. The appropriate offloading node is selected based on the post-task processing latency and cost. When tasks are offloaded to vehicles for computation, due to vehicle mobility, there is a risk of communication link disconnection during task transmission. Therefore, the quality of the communication link is modeled and evaluated to identify a suitable communication link. Due to the complexity of road information, the joint optimization of task processing latency and cost is a difficult non-convex problem. Deep reinforcement learning utilizes an exploration-exploitation strategy, enabling efficient exploration within the policy space to find optimal solutions. The problem is converted into a Markov decision process and the asynchronous advantage actor-critic (A3C) algorithm in deep reinforcement learning algorithm is applied to solve this non-convex problem.

Claims

1. A RSU-assisted multi-hop task offloading optimization method for Internet of Vehicles based on deep reinforcement learning, characterized in that: The steps are: Step 1: According to the road environment, the sensors on the RSU actively sense the road information and generate perception tasks; Step 2: Establish an unloading delay model and an unloading cost model of the corresponding sensing task according to different task unloading nodes, and then construct an objective function for unloading decision-making according to the established unloading delay model and unloading cost model; Step 3: According to the objective function, the task decision problem is transformed into a Markov decision process problem, and the computation offloading node that minimizes the task processing delay and the task processing cost is selected; Step 4: Use the A3C algorithm to jointly optimize the task processing delay and task processing cost to obtain the optimal task decision.

2. According to claim 1, a RSU-assisted vehicle networking multi-hop task offloading optimization method based on deep reinforcement learning is characterized by: The offloading delay model is the total processing delay D of the task calculation on the vehicle, on the RSU and on the cloud server. r (t); The offloading cost model is the total processing cost C of the task calculated on the vehicle, on the RSU and on the cloud server r (t).

3. According to claim 2, a RSU-assisted vehicle networking multi-hop task offloading optimization method based on deep reinforcement learning is characterized in that: When the task is offloaded to the vehicle for processing, it is necessary to evaluate the quality of the communication link to find a suitable communication link, including V2V link quality evaluation and R2V link quality evaluation; the probability that the change in the relative distance between vehicle i and vehicle j is less than the remaining communication distance between vehicle i and vehicle j is used as the evaluation of the V2V link quality; the probability that the change in the relative distance between RSU r and vehicle i is less than the remaining communication distance between RSU r and vehicle i is used as the evaluation of the R2V link quality.

4. According to claim 3, a RSU-assisted vehicle networking multi-hop task offloading optimization method based on deep reinforcement learning is characterized in that: Offloading task f(t) from RSU r to vehicle v consists of three steps: finding candidate vehicles, determining the transfer path, and executing the offloading decision; where: In the process of searching for candidate vehicles, RSU r sends a service request to all vehicles within its communication range. Upon receiving the request, each vehicle further relays the request to other vehicles within the communication range. When a vehicle receives a service request, the vehicle records all forwarding nodes that the request passes through before reaching the vehicle. The vehicle then sends the recorded information back to RSU r together with the current status. After finding the candidate vehicles, we will obtain the set ζ of all candidate vehicles that can communicate with RSU r r (t), assuming that there is a set of η paths between RSUr and candidate vehicle v Given a path h(r, v) consisting of u single-hop links: RSU r→vehicle h(1)→vehicle h(2)→…→vehicle h(u-2)→vehicle j(u-1)→vehicle v, where The quality of a path h(r, v) is defined as the product of the connection probabilities of each single-hop link, expressed as follows: The optimal path for sending task f(t) from RSU r to candidate vehicle v is the path with the highest quality among all paths; After obtaining the candidate vehicles and the optimal transmission path, RSU r makes an offloading decision and offloads the task to a suitable vehicle for calculation.

5. According to claim 4, a RSU-assisted vehicle network multi-hop task offloading optimization method based on deep reinforcement learning is characterized in that: The total processing delay of task offloading to the vehicle, the total processing delay of task offloading to the RSU, and the total processing delay of task offloading to the cloud server are all composed of transmission delay, waiting delay, calculation delay and result transmission delay, among which the result transmission delay is negligible.

6. The RSU-assisted multi-hop task offloading optimization method for Internet of Vehicles based on deep reinforcement learning according to claim 5 is characterized by: The total processing delay of the task calculation on the vehicle, on the RSU, and on the cloud server is: Among them, {1, 2, ..., v, ... V} is the vehicle set, {1, 2, ..., r, ... R} is the RSU set, is the total processing delay of unloading the task to a vehicle in the vehicle set, is the total processing delay of the task offloading to an RSU in the RSU set, is the total processing delay of the task offloading to the cloud server, x r，y (t)∈{0, 1} is a binary variable and y∈{1,...,V,V+1,...,V+R,V+R+1} represents the unloading node.

7. The RSU-assisted multi-hop task offloading optimization method for Internet of Vehicles based on deep reinforcement learning according to claim 6 is characterized by: The total processing cost of task offloading to the vehicle and the total processing cost of task offloading to the RSU are both composed of relay cost and computation cost, and the total processing cost of task offloading to the cloud server is composed of computation cost; The total processing cost of the task on the vehicle, on the RSU, and on the cloud server is: Among them, {1, 2, ..., v, ... V} is the vehicle set, {1, 2, ..., r, ... R} is the RSU set, is the total processing cost of offloading the task to a vehicle in the vehicle set, is the total processing cost of offloading the task to an RSU in the RSU set, is the total processing cost of offloading tasks to the cloud server, x r，y (t)∈{0, 1} is a binary variable and y∈{1,...,V,V+1,...,V+R,V+R+1} represents the unloading node.

8. The RSU-assisted multi-hop task offloading optimization method for Internet of Vehicles based on deep reinforcement learning according to claim 7 is characterized by: The constructed objective function is as follows: s.t. Where α is a weight factor, constraints C1-C2 indicate that the task is calculated on a single offloading node; constraint C3 ensures that the task is calculated within its valid time d r Constraint C4 ensures that the connection probability of the communication link will not be 0, and constraint C5 ensures that the task will not be offloaded to vehicles that cannot communicate with the RSUs that perceive them.

9. A RSU-assisted vehicle networking multi-hop task offloading optimization method based on deep reinforcement learning according to any one of claims 1 to 8, characterized in that: The specific process of step three is as follows: The Markov decision process includes three key elements: state, action, and reward, which are defined as follows: state: The state space consists of the task queue information Q(t) and computational cost information q(t) of each computing node. In each time slot t, the agent monitors the network environment and dynamically collects the system status: s(t)={Q(t),q(t)} action: After observing the state s(t) from the environment, the agent makes an unloading decision: a(t)={x1,x2,...,x r ,...,x R } where x r =x r，y (t), y∈{1,...,V,V+1,...,V+R,V+R+1},x r The offloading decision made for RSU r; award: The design of the reward function is the key to reinforcement learning. Generally speaking, the reward function is related to the established objective function. The optimization goal is to obtain the maximum system reward. If the constraints C1-C5 are met, the reward function is the negative value of the objective function. If the constraints C1-C5 are not met, the reward function is -Z. The reward function is expressed as:

10. The RSU-assisted vehicle networking multi-hop task offloading optimization method based on deep reinforcement learning according to claim 9 is characterized in that: The specific process of step 4 is as follows: The state information obtained by the interaction between RSU and the environment in each time slot is input into the global actor network θ and the global critic network θ v , and the local actor network θ′ and the local critic network θ′ in each worker thread v , the initial values ​​of the parameters of the local network are synchronized with the global network; In each training cycle, each worker thread uses local networks θ′ and θ' v Interact with the environment and collect sample data, including state s(t), action a(t), reward r(t), and next state s(t+1); based on these samples, the training goal is to minimize the loss of the local actor network and the local critic network; Each worker thread uses the local network to calculate the gradient, and after a certain number of training steps, sends the gradient information of the local network to the global network θ and θ v , the global network updates its parameters accordingly; subsequently, the local network synchronizes the latest parameters from the global network; After training, the optimized global policy networks θ and θ are used v , the optimal task decision process is given based on the state information obtained by RSU.

Images