A method for optimizing dependent task offloading in connected vehicles using deep reinforcement learning

Abstract

The present invention discloses a method for optimizing dependent task offloading in the Internet of Vehicles using deep reinforcement learning. The steps of the present invention include: S1, constructing a vehicle system network; S2, constructing a task model for an application; S3, constructing a task load model: according to the vehicle network system and the task model, respectively calculating the time delay, energy consumption and incentive payment of the three unloading methods of local calculation, unloading to nearby vehicles, and unloading to nearby roadside units; S4, determining the task priority: first determine the priority of each subtask based on the allocation of the predecessor nodes of the subtask and in combination with the dynamic network environment, and then schedule according to the multi-queue algorithm; S5, finding the optimal unloading strategy through deep reinforcement learning. The method of the present invention does not require too much prior knowledge, and has good reusability in similar application scenarios, and has high practical value.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle edge computing technology, and specifically relates to a method for optimizing dependent task offloading in a vehicle network using deep reinforcement learning. Background Art

[0002] With the rapid growth in the number of vehicle users worldwide and their increasing demand for services, the Internet of Vehicles (IoV) faces increasing challenges, including high network bandwidth requirements, high transmission reliability, and robust data processing capabilities. The IoV integrates traditional vehicle ad hoc networks with telematics technologies to effectively enhance vehicle service capabilities. In IoV, intelligent vehicles can execute a variety of applications, such as collision warning, autonomous driving, and automatic navigation. However, these applications require extensive computing and storage resources and have strict latency requirements. Executing these applications on resource-limited vehicles within low latency constraints is a challenging task.

[0003] In-vehicle edge computing (VEIC), which integrates mobile edge computing (MEC) into the Internet of Vehicles (IoV), is a promising solution that can effectively address the aforementioned issues. VEIC can improve vehicle service quality by deploying the computing and storage resources of MEC servers near the vehicle. Computationally intensive and latency-sensitive tasks can be offloaded to MEC servers over the wireless network for execution. While VEIC can reduce task execution latency, edge servers have limited computing and storage capabilities, making load balancing unavailable. Therefore, effective offloading strategies can reduce task processing latency and improve user service quality.

[0004] Current in-vehicle edge computing focuses primarily on optimizing the computation of individual tasks, lacking consideration of inter-task dependencies. This limits the scope and effectiveness of in-vehicle edge computing technology. Furthermore, for environmental reasons, most vehicles are electric, and energy consumption is crucial for their use. In practical applications, vehicles or infrastructure providing computing services typically require corresponding incentives, but existing research has rarely focused on energy consumption and the corresponding incentives. To address these issues, the present invention aims to minimize energy consumption and incentives during computing, while meeting the computing service requirements of applications, thereby improving user service quality. Specifically, the present invention proposes a system and method for optimizing dependent task offloading in the Internet of Vehicles (IoV) using deep reinforcement learning. This method considers inter-task dependencies, minimizing energy consumption during computing and the incentives paid to the vehicle or infrastructure providing the computing services while ensuring latency constraints are met. Specifically, the system first mathematically models the computing resources, energy costs, and incentives of the vehicle and infrastructure. Then, deep reinforcement learning techniques are used, combined with inter-task dependencies, to develop appropriate task scheduling and offloading strategies. This method minimizes energy consumption during computing and the incentives paid to the vehicle or infrastructure providing the computing services while ensuring latency constraints are met.

[0005] In summary, the present invention proposes a new vehicle-mounted edge computing technology, which can better solve the computational offloading problems of dependencies between tasks, energy consumption costs, and incentive rewards, and improve the efficiency and quality of computing services. Summary of the Invention

[0006] The purpose of the present invention is to address the deficiencies of the existing technology and provide a method for optimizing dependent task offloading in the Internet of Vehicles using deep reinforcement learning.

[0007] In order to achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0008] S1. Building a vehicle system network: The proposed vehicle system network consists of a MEC server, RSU, and vehicle user end connected in sequence;

[0009] S2. Build the application task model: Tasks with timing and data dependencies are modeled as a DAG task model;

[0010] S3. Build a task load model: Based on the vehicle network system and task model, calculate the latency, energy consumption, and incentive compensation for three offloading methods: local computing, offloading to nearby vehicles, and offloading to nearby roadside units.

[0011] S4. Determine task priority: First, determine the priority of each subtask based on the distribution of its predecessor nodes and the dynamic network environment, and then schedule it based on the multi-queue algorithm;

[0012] S5. Find the optimal offloading strategy through deep reinforcement learning: The optimal offloading strategy is learned through deep reinforcement learning. That is, each subtask selects the corresponding execution device to complete the offloading of the task so as to minimize the energy consumption and the incentive reward paid under the delay constraint, thereby improving the user's service quality.

[0013] Beneficial effects of the present invention:

[0014] The present invention is applicable to V2I and V2V collaborative offloading scenarios in a vehicle network environment. The method decouples the nonlinear integer programming problem into two sub-problems: the associated subtask scheduling problem and the task offloading problem. In combination with the allocation of subtask predecessor tasks and the dynamic network environment, the method prioritizes the prepared subtask set and then uses deep reinforcement learning based on value functions and policy functions to solve the offloading problem. Under the premise of ensuring task dependencies and delay constraints, the method optimizes the task offloading problem, minimizes the energy consumption in the calculation process and the incentive compensation paid to the vehicle or infrastructure for the computing service. The method of the present invention does not require too much prior knowledge, has good reusability in similar application scenarios, and has high practical value.

[0015] This invention addresses the problem of offloading dependent tasks in V2I and V2V collaborative edge computing based on deep reinforcement learning. It effectively minimizes the weighted sum of client vehicle energy consumption and incentive payments under latency constraints, thereby improving the user experience. The entire process is standardized and easy to operate. Furthermore, this invention has strong practical value and reusability, and has broad application prospects in similar scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the V2I and V2V collaborative offloading model provided by an example of the present invention;

[0017] Figure 2 This is a schematic diagram of the deep reinforcement learning algorithm proposed in this example;

[0018] Figure 3 This is a flowchart of the deep reinforcement learning algorithm in the Internet of Vehicles scenario proposed by the example of the present invention. DETAILED DESCRIPTION

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and be able to implement it. However, the embodiments cited are not intended to limit the present invention.

[0020] The agent takes an action in its current state, receives a reward from the environment, and then proceeds to the next state. Through continuous interaction, the agent gradually learns the optimal offloading strategy. During this process, the agent continuously updates its value function to better estimate the value of each action. Simultaneously, the agent continuously adjusts its strategy to better explore the search space and find the optimal offloading decision.

[0021] In order to minimize the weighted sum of energy consumption and incentive rewards in the vehicle network edge computing scenario of V2I and V2V collaborative offloading while meeting the delay constraint, the present invention provides a method for offloading dependent tasks based on deep reinforcement learning.

[0022] The edge computing scenario described in the present invention refers to the vehicle nodes in the vehicle network communication system with V2I and V2V collaborative offloading. These vehicle nodes can perform V2I communication with the edge MEC server and V2V communication with other vehicle nodes. In this scenario, we call the vehicle that generates the application the client vehicle, and the vehicle that provides computing services for the application the server vehicle. The application is composed of a series of inseparable subtasks, which have timing dependencies and data dependencies and can be represented using a directed acyclic graph (DAG). These subtasks can be executed locally, transmitted to the server vehicle for execution, or uploaded to the RSU next to the road for execution. In order to minimize the weighted sum of energy consumption and incentive rewards in this scenario, the present invention proposes a method for offloading dependent tasks based on deep reinforcement learning. This method only requires very little prior knowledge and simple interaction with the environment to obtain learning experience, and obtains a near-optimal solution that minimizes the weighted sum of energy consumption cost and incentive rewards.

[0023] This paper proposes a method to optimize the offloading of dependent tasks in the Internet of Vehicles using deep reinforcement learning. Figure 3 As shown, the following steps are included:

[0024] S1. Build vehicle system network;

[0025] The proposed vehicle system network consists of a sequentially connected MEC server, RSUs, and vehicle user terminals. Multiple vehicle user terminals are connected via V2V transmission links, enabling inter-vehicle communication and collaborative computing. Vehicle user terminals and RSUs are connected via V2I transmission links, enabling vehicle-to-network communication. MEC servers, as edge computing resource providers, provide computing, storage, and network resources, offering edge computing services to vehicle user terminals. RSUs, as network infrastructure, offer network connectivity and road condition information, providing data and collaborative computing support to vehicle user terminals. The entire vehicle system network forms a distributed edge computing platform, enabling collaborative computing and communication between vehicles and between vehicles and the network.

[0026] like Figure 1 As shown in the figure, in this scenario, vehicles can be divided into two types according to their roles in task processing: client vehicles and service vehicles. The data transmission link includes the task data V2V transmission link, the task data upload to RSU link, the RSU and server connection link, and the RSU and RSU connection link. For client vehicles, their tasks can be processed through the following three offloading methods: ① Local computing: The client vehicle processes the task by itself. ② Offloading to nearby vehicles: The client vehicle uploads the task to the adjacent service vehicle. ③ Offloading to nearby roadside units: The client vehicle uploads the task to a nearby roadside unit.

[0027] S2. Build the application task model:

[0028] Model the tasks with timing and data dependencies as a DAG task model. Model the DAG task as G = (V, E), where the vertex set V = {φi|1≤i≤N}, where N represents the number of subtasks in V. Define the triple as φ i =(w i ,d i ,τ i ), where w i ,d i ,τ i Represents tasks φ respectively i The computational workload, the amount of data, and the maximum completion time tolerance delay. E represents the set of edges, each edge is a directed edge indicating the execution priority between them, indicating that subtask φ i Must be in φ j Completed before (equivalent to φ i The output of φ j If the input from φ i to φ j There is an edge, φ i is called φ jThe direct predecessor of φ j is φ i All predecessors must be completed before their successors. A subtask without any predecessor nodes is a start node, also known as a source task, denoted by φ entry ; A subtask without any successor node is an exit node, denoted as φ exit The DAG task model can clearly represent the timing and data dependencies between tasks, providing a basis for task scheduling.

[0029] S3. Build a task load model: Based on the vehicle network system and task model, calculate the latency, energy consumption, and incentive compensation for the three unloading methods: local computing, unloading to nearby vehicles, and unloading to nearby roadside units.

[0030] 3.1) The interdependent subtasks executed on different vehicle nodes need to transfer data. Specifically, the subtask φ executed on vehicle node m j The calculation result of is passed to the subtask φ executed on the vehicle node n i It takes a certain amount of time. The data transfer time is defined as follows:

[0031]

[0032] Among them, the symbols k, m, and n represent the labels of vehicle nodes respectively, φ j is φ i The subtask predecessor node, r mn represents the transmission bandwidth between vehicle node m and vehicle node n, Represents subtask φ i P j The data size of the dependent data, e mn Indicates whether vehicle nodes m and n are within the communication range. Represents roadside unit RSU i The set of edges established by vehicles within the communication range through single-hop V2V communication.

[0033] 3.2) Calculate the time to execute the task locally.

[0034] Specifically, for the case of executing tasks locally, the subtask φ i The completion time is determined by the waiting time and calculation time It consists of two parts.

[0035]

[0036]

[0037]

[0038] Among them, V c Indicates the client vehicle, avail(V c ) represents the time when the client vehicle completes all the scheduled tasks, w i Represents subtask φ i The amount of computation required; f c Indicates the computing power of the client vehicle.

[0039] For RSU j When executing tasks on the i The completion time is determined by the upload time Waiting time and calculation time It consists of three parts.

[0040]

[0041]

[0042]

[0043]

[0044] in, Indicates that the client vehicle arrives at the specified RSU j The transmission rate between i Represents subtask φ i The amount of data, Represents roadside unit RSU j computing power.

[0045] Similarly, the time to execute the task on the service vehicle is calculated by the transmission time Waiting time and calculation time It consists of three parts.

[0046]

[0047]

[0048]

[0049]

[0050] in Represents the client vehicle V c Go to the designated service vehicle V k The transmission rate between kRepresents the service-side vehicle V k computing power.

[0051] 3.3) Calculate the energy consumption of executing tasks locally:

[0052]

[0053] Among them, ξ is a coefficient related to the architecture of the computer chip. Since the execution is done locally, no incentive compensation is paid.

[0054] Calculate the energy consumption of executing tasks on RSU:

[0055]

[0056] Where p represents the upload power, so the roadside unit RSU j Processing subtask φ i The incentive compensation required to be paid is expressed by the following formula:

[0057]

[0058] Among them, p j Represents roadside unit RSU j The fee charged per unit time (s).

[0059] Calculate the energy consumption of executing tasks on the server vehicle:

[0060]

[0061] Among them, the service-side vehicle k processes the subtask φ i The incentive compensation required to be paid is expressed by the following formula:

[0062]

[0063] Among them, ρ k It represents the fee charged by the service vehicle k per unit resource (1G clocks).

[0064] 3.4) Execute a subtask φ i The time required t i Energy consumption i and the fees you need to pay i They are:

[0065]

[0066]

[0067]

[0068] in, Indicates the decision to uninstall the subtask. The values ​​of are all 0 or 1, and only one of the three can be 1.

[0069] The total incentive reward cost, that is, the incentive reward that the client vehicle needs to pay to complete all tasks, is recorded as:

[0070]

[0071] The total energy cost, that is, the energy consumed by the client vehicle to complete all tasks, is recorded as:

[0072]

[0073] Where X represents the offloading decision vector and N represents the number of tasks.

[0074] These formulas describe the relationship between task offloading decisions, execution time, energy consumption, and incentive rewards, and can be used to optimize task offloading schemes to minimize total energy consumption and total incentive rewards.

[0075] 3.5) In order to make the optimal offloading decision, an optimization problem considering the energy consumption and incentive reward of completing the task is established:

[0076]

[0077] Where C(X) represents the total incentive reward cost, and E(X) represents the total computational cost. C1 indicates that each subtask must be offloaded to a certain device. C2 and C3 indicate that if the subtask is offloaded to the service vehicle or RSU, its completion time cannot exceed the maximum allowed time threshold l. c,k or C4 indicates that the sum of the weights of energy consumption and incentive compensation is 1, and both must be positive numbers.

[0078] The optimization problem can be decomposed into two subproblems: task prioritization and offloading decision-making. First, task prioritization determines the execution order of each subtask to satisfy latency and task dependency constraints. Then, the task scheduling order is used for offloading decision-making. Deep reinforcement learning is used to determine whether each subtask should be executed locally, offloaded to the RSU, or offloaded to the service vehicle to minimize the weighted sum of energy consumption and incentive compensation.

[0079] S4. Determine task priority: First, determine the priority of each subtask based on the distribution of its predecessor nodes and the dynamic network environment, and then schedule it based on the multi-queue algorithm.

[0080] 4.1) For the start subtask, assuming Rank D (φ ready)=0. The priority of the subsequent subtask is determined according to the scheduling of the predecessor subtask, and the priority of the existing subtask φ is determined. i ∈φ ready The ranking of , for which the dynamic downward sorting values ​​are calculated as follows:

[0081]

[0082] Among them, pred(φ i ) is the subtask φ i The set of direct predecessor nodes of is the dynamic average computation time, which is affected by φ j ∈pred(φ i ) distribution, C VC (φ i ) indicates that the subtask φ can be executed i The set of candidate service-side vehicles or infrastructures, φ ready Represents a collection of preparation subtasks. Its definition is as follows:

[0083]

[0084] and, represents the dynamic average transmission time, which is also affected by φ j ∈pred(φ i ) is defined as follows:

[0085]

[0086] 4.2) Task scheduling depends not only on priority but also on the dependencies between tasks. A subtask must be executed after all its predecessor subtasks are completed. Therefore, it is necessary to maintain a queue Q of ready subtasks. ready and executing subtask queue Q exe , and there is also a subtask queue Q that has been calculated finished and the set T of all subtasks in the application task .

[0087] Initially, φ entry Subtasks from the set T task Put in Q ready According to the subtask priority confirmation formula, Q ready The tasks in are sorted in ascending order. Take out the head element and put the task into the executing subtask queue Q exe , if the queue Q exe If there are completed tasks in the Q finished This may generate new preparation subtasks if the set T taskA subtask φ i All predecessor node subtasks are in queue Q finished , then the subtask φ i Become a preparation subtask and put it into queue Q ready Then repeat the above steps until the set T task During the entire subtask priority determination process, the preparation subtask is dynamically determined based on the execution results of the subtask predecessor, and the priority of the preparation subtask is then determined.

[0088] S5. Find the optimal offloading strategy through deep reinforcement learning: that is, each subtask selects the corresponding execution device to complete the offloading of the task so as to minimize energy consumption and incentive rewards under the delay constraint, thereby improving the user's service quality. Figure 2 shown.

[0089] In deep reinforcement learning methods, the design of actions, states, and rewards plays a crucial role. To adapt to the system environment and enable the agent to output the optimal action based on the current state of the environment, the state space, action space, and reward function are designed as follows:

[0090] State space S: The state is represented in vector form. It describes the state of the client vehicle, including mission information, location, available computing resources, and connection time. t ={T,N}, where T and N represent the roadside unit RSU j The set of candidate vehicles available within the communication range and the set of preparation subtasks, where

[0091] N=(w i ,τ i ,succ(φ i ),pred(φ i )) (27)

[0092] Action space A: Action is the decision made by the requester through the offloading strategy π: A←S, which has a direct impact on the external environment. The action at time t is recorded as This is a one-hot encoding. Each element can have a value of 0 or 1.

[0093] Reward function R: Intuitively, the reward R←(S,A), the feedback obtained by the agent during the offloading process, directly determines the strategy. The optimization goal is to minimize energy consumption and overhead costs, but in reality, each decision step can only serve one subtask. Therefore, the cost of a single subtask m at time t is expressed as R t (s,a)=-(e i +c i ).

[0094]

[0095] 5.1) Initialize the experience replay pool and define the maximum number of executions and the initial episode for training and evaluation.

[0096] 5.2) After initialization, the parameters μ and θ of the actor and critic networks need to be initialized. This is because before starting training, the network parameters need to be initialized to some random values ​​so as not to fall into local optimal solutions during the learning process.

[0097] 5.3) Next, we need to initialize the target actor network parameters μ'←μ and the target critic network parameters θ'←θ. These networks will be used to calculate the TD error and update the parameters of the current network during the training process of the algorithm.

[0098] 5.4) Get the initial state of the environment s t This is an important step in the algorithm. At the same time, we also need to prepare the initial random noise n t , in order to conduct action exploration. Through such exploration, we can discover more state and action pairs and help the network better converge to the optimal solution.

[0099] 5.5) From time t=1 until t=n (n represents the number of indivisible subtasks in the application). Based on the output of the Actor-C network of the current network and the random noise n t Get a t :

[0100]

[0101] 5.6) Execute action a t , get reward r t , the environment state becomes s t+1

[0102] 5.7) will (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool, so that we can use the historical data in the experience replay pool to perform offline training and improve the stability and robustness of the model.

[0103] 5.8) Sample N tuples {(s t ,a t ,r t ,s t+1 )}, these tuples are used for training the target network and updating the current network. Usually, we choose random sampling to ensure the randomness of the samples.

[0104] 5.9) Use the target network to calculate the target value for each tuple y t , which is a t and action a t The relevant value. t By the current reward r t and the next state s t+1 Specifically, the target value is calculated by the following formula:

[0105]

[0106] where γ is the discount factor, is the target critic network, is the target actor network, and these network parameters are initialized in step 5.3.

[0107] 5.10) Update the parameters θ of the current critic network by minimizing the target loss L(θ). L(θ) is calculated by the mean squared error (MSE) loss function:

[0108]

[0109] Here N is the number of tuples sampled from the experience replay pool, y i is the target value, is the value output by the current critic network. This MSE loss function is a common supervised learning loss function that can reflect the fitting ability of the current critic network and help the network better adapt to the target value y i .

[0110] 5.11) Update the Actor network of the current network by calculating the sampled policy gradient. Where Q(s,a|θ) represents the predicted value of the current critic network for state s and action a, represents the action gradient with respect to the predicted value, represents the action output of the current actor network for state s, Represents the parameter gradient of the action output. The calculation formula of the policy gradient is a Monte Carlo method that estimates the expected value using sampled tuple data:

[0111]

[0112] 5.12) If the current iteration number t is divisible by C, i.e., t%C == 0, update the network parameters. Where θ' and u' represent the parameters of the target critic network and the target actor network, respectively, and v is a hyperparameter less than 1 used to smooth the update:

[0113] θ'=vθ+(1-v)θ' (33)

[0114] u'=vu+(1-v)u' (34)

[0115] 5.13) t=t+1, repeat steps 5.5 to 5.12.

[0116] 5.14) If the current number of iterations t is equal to n, the maximum number of executions has been reached, and episode = episode + 1. Then repeat steps 5.4 to 5.13 until the training process is complete and the optimal offloading strategy is obtained.

[0117] It should be understood that any portion not elaborated in detail in this description belongs to the prior art. Those skilled in the art should understand that the above embodiments are merely intended to help readers understand the principles and implementation methods of the present invention, and the scope of protection of the present invention is not limited to such embodiments. All equivalent substitutions made based on the present invention are within the scope of protection of the present invention.

Claims

1. A method for optimizing dependent task offloading in the Internet of Vehicles using deep reinforcement learning, characterized in that: The steps include: S1. Building a vehicle system network: The proposed vehicle system network consists of a MEC server, RSU, and vehicle user end connected in sequence; S2. Build the application task model: Tasks with timing and data dependencies are modeled as a DAG task model; S3. Build a task load model: Based on the vehicle network system and task model, calculate the latency, energy consumption, and incentive compensation for the three offloading methods: local computing, offloading to nearby vehicles, and offloading to nearby roadside units. The specific implementation is as follows: 3.1) Interdependent subtasks executed on different vehicle nodes require data transfer; Subtask φ to be executed on vehicle node m j The calculation result of is passed to the subtask φ executed on the vehicle node n i It takes a certain amount of time, and the data transmission time is defined as follows: Among them, the symbols k, m, and n represent the labels of vehicle nodes respectively, φ j is φ i The subtask predecessor node, r mn represents the transmission bandwidth between vehicle node m and vehicle node n, Represents subtask φ i P j The data size of the dependent data, e mn Indicates whether vehicle nodes m and n are within the communication range; Represents roadside unit RSU i The set of edges established by vehicles within the communication range through single-hop V2V communication; 3.2) Calculate the time to execute the task locally; 3.3) Calculate the energy consumption of executing tasks locally: 3.4) Execute a subtask φ i The time required t i Energy consumption i and the fees you need to pay i 3.5) In order to make the optimal offloading decision, an optimization problem considering the energy consumption and incentive reward of completing the task is established as follows: C4:α+β＝1,α＞0 β＞0 Where C(X) represents the total incentive reward cost, E(X) represents the total computing cost; C1 means that each subtask must be offloaded to a certain device; Indicates the decision to offload the subtask; C2 and C3 indicate that if the subtask is offloaded to the service vehicle or RSU, its completion time cannot exceed the maximum allowed time threshold l c,k or C4 means that the sum of the weights of energy consumption and incentive compensation is 1, and both must be positive numbers; The optimization problem is decomposed into two sub-problems: task prioritization and offloading decision-making. First, task prioritization determines the execution order of each subtask to meet latency and task dependency constraints. Then, the task scheduling order is used for offloading decision-making, using deep reinforcement learning to determine whether each subtask should be executed locally, offloaded to the RSU, or offloaded to the service vehicle to minimize the weighted sum of energy consumption and incentive rewards. S4. Determine task priority: First, determine the priority of each subtask based on the distribution of its predecessor nodes and the dynamic network environment, and then schedule it based on the multi-queue algorithm; S5. Find the optimal offloading strategy through deep reinforcement learning: The optimal offloading strategy is learned through deep reinforcement learning. That is, each subtask selects the corresponding execution device to complete the offloading of the task so as to minimize the energy consumption and the incentive reward paid under the delay constraint, thereby improving the user's service quality.

2. The method for optimizing dependent task offloading in the Internet of Vehicles using deep reinforcement learning according to claim 1 is characterized in that Step S2 builds the task model of the application, which is specifically implemented as follows: The tasks with timing dependency and data dependency are modeled as a DAG task model; the DAG task is modeled as G = (V, E), where the vertex set V = {φ i |1≤i≤N}, where N represents the number of subtasks in V; define the triple as φ i =(w i ,d i ,τ i ), where w i ,d i ,τ i Represents tasks φ respectively i The computational workload, the amount of data, and the maximum completion time tolerance delay; E represents the set of edges, each edge is a directed edge indicating the execution priority between them, indicating that subtask φ i Must be in φ j Completed before, equivalent to φ i The output of φ j Part of the input; if from φ i to φ j There is an edge, φ i is called φ j direct predecessor; on the contrary, φ j is φ i The direct successor of ; all predecessors must be completed before their successors; A subtask without any predecessor node is a start node, also called a source task, denoted as φ entry ; A subtask without any successor node is an exit node, denoted as φ exit ; Through the DAG task model, the timing dependency and data dependency between tasks can be clearly expressed, providing a basis for task scheduling.

3. The method for optimizing dependent task offloading in the Internet of Vehicles using deep reinforcement learning according to claim 1 is characterized in that Step 3.2) is specifically implemented as follows: For the case of executing tasks locally, the subtask φ i The completion time is determined by the waiting time and calculation time It consists of two parts; Among them, V c Indicates the client vehicle, avail(V c ) represents the time when the client vehicle completes all the scheduled tasks, w i Represents subtask φ i The amount of computation required; f c Indicates the computing power of the client vehicle; For RSU j When executing tasks on the i The completion time is determined by the upload time Waiting time and calculation time It consists of three parts; in, Indicates that the client vehicle arrives at the specified RSU j The transmission rate between i Represents subtask φ i The amount of data, Represents roadside unit RSU j computing power; Similarly, the time to execute the task on the service vehicle is calculated by the transmission time Waiting time and calculation time It consists of three parts; in Represents the client vehicle V c Go to the designated service vehicle V k The transmission rate between k Represents the service-side vehicle V k computing power.

4. The method for optimizing dependent task offloading in the Internet of Vehicles using deep reinforcement learning according to claim 1 is characterized in that Step 3.3) is specifically implemented as follows: Among them, ξ is a coefficient related to the computer chip architecture; Calculate the energy consumption of executing tasks on RSU: Where p represents the upload power, so the roadside unit RSU j Processing subtask φ i The incentive compensation required to be paid is expressed by the following formula: Among them, p j Represents roadside unit RSU j Fees charged per unit time (s); Calculate the energy consumption of executing tasks on the server vehicle: Among them, the service-side vehicle k processes the subtask φ i The incentive compensation required to be paid is expressed by the following formula: Among them, ρ k It represents the fee charged per unit of resource service by vehicle k on the service side.

5. The method for optimizing dependent task offloading in the Internet of Vehicles using deep reinforcement learning according to claim 1 is characterized in that Step 3.4) Execute a subtask φ i The time required t i Energy consumption i and the fees you need to pay i They are: in, Indicates the decision to uninstall the subtask. The values ​​of are all 0 or 1, and only one of the three can be 1; The total incentive reward cost, that is, the incentive reward that the client vehicle needs to pay to complete all tasks, is recorded as: The total energy cost, that is, the energy consumed by the client vehicle to complete all tasks, is recorded as: Where X represents the offloading decision vector and N represents the number of tasks; These formulas describe the relationship between task offloading decisions, execution time, energy consumption, and incentive rewards, and can be used to optimize task offloading schemes to minimize total energy consumption and total incentive rewards.

6. The method for optimizing dependent task offloading in the Internet of Vehicles using deep reinforcement learning according to claim 1 is characterized in that Step S4 is specifically implemented as follows: 4.1) For the start subtask, assuming Rank D (φ ready )=0; Determine the priority of the subsequent subtasks according to the scheduling of the predecessor subtasks, and determine the current subtask φ i ∈φ ready The ranking of , for which the dynamic downward sorting values ​​are calculated as follows: Among them, pred(φ i ) is the subtask φ i The set of direct predecessor nodes of is the dynamic average computation time, which is affected by φ j ∈pred(φ i ) distribution, C VC (φ i ) indicates that the subtask φ can be executed i The set of candidate service-side vehicles or infrastructures, φ ready Represents a collection of preparation subtasks, which are defined as follows: and, represents the dynamic average transmission time, which is also affected by φ j ∈pred(φ i ) is defined as follows: 4.2) Task scheduling depends not only on priority but also on the dependencies between tasks. A subtask can only be executed after all its predecessor subtasks are completed. Therefore, it is necessary to maintain a queue Q of ready subtasks. ready and executing subtask queue Q exe , and there is also a subtask queue Q that has been calculated finished and the set T of all subtasks in the application task ; Initially, φ entry Subtasks from the set T task Put in Q ready In; according to the subtask priority confirmation formula, Q ready The tasks in are sorted in ascending order; take out the head element and put the task into the executing subtask queue Q exe , if the queue Q exe If there are completed tasks in the Q finished queue; this may generate new preparation subtasks if the set T task A subtask φ i All predecessor node subtasks are in queue Q finished , then the subtask φ i Become a preparation subtask and put it into queue Q ready Then repeat the above steps until the set T task Empty; during the entire subtask priority determination process, the preparation subtask is dynamically determined based on the execution results of the subtask predecessor, and then the priority of the preparation subtask is determined.

7. The method for optimizing dependent task offloading in the Internet of Vehicles using deep reinforcement learning according to claim 1 is characterized in that Step S5 is specifically implemented as follows: 5.1) Initialize the experience replay pool and define the maximum number of executions and the initial episode; 5.2) After initialization, initialize the parameters μ and θ of the current actor network and the current critic network respectively; 5.3) Initialize the target actor network parameters μ′←μ and the target critic network parameters θ′←θ respectively; 5.4) Get the initial environment state s t , and prepare the initial random noise n t , in order to conduct motion exploration; 5.5) From time t = 1 to t = n, according to the output of the Actor-C network of the current network and the random noise n t Get action a t : Where n represents the number of indivisible subtasks in the application; 5.6) Execute action a t , get reward r t , the environment state becomes s t+1 ; 5.7) will (s t , a t , r t , s t+1 ) is stored in the experience replay pool, and historical data in the experience replay pool is used for offline training to improve the stability and robustness of the model; 5.8) Sample N tuples {(s t , a t , r t , s t+1 )}, these tuples are used for training the target network and updating the current network; 5.9) Use the target network to calculate the target value for each tuple y t ,y t is a state with the current state s t and action a t The relevant value; y t By the current reward r t and the next state s t+1 The target value is calculated by the following formula: Where γ is the discount factor, is the target critic network, is the target actor network, and these target network parameters are initialized in step 5.3; 5.10) Update the parameters θ of the current critic network by minimizing the target loss L(θ), which is calculated by the mean squared error (MSE) loss function: Where N is the number of tuples sampled from the experience replay pool, y i is the target value, is the value output by the current critic network; 5.11) Update the Actor network of the current network by calculating the sampled policy gradient; where Q(s, a|θ) represents the current critic network's prediction value for state s and action a, represents the action gradient with respect to the predicted value, represents the action output of the current actor network for state s, Represents the parameter gradient of the action output; the calculation formula of the policy gradient is a Monte Carlo method that estimates the expected value using sampled tuple data: 5.12) If the current iteration number t is divisible by C, i.e., t%C == 0, update the network parameters; where θ′ and u′ represent the parameters of the target critic network and the target actor network, respectively, and v is a hyperparameter less than 1 used to smooth the update: θ′=vθ+(1-v)θ′ (33) u′=vu+(1-v)u′ (34) 5.13) t = t + 1, repeat steps 5.5 to 5.12; 5.14) If the current number of iterations t is equal to n, that is, the maximum number of executions is reached, then episode = episode + 1; then repeat steps 5.4 to 5.13 until the training process is completed and the optimal offloading strategy is obtained.

Images