Deep reinforcement learning-based task unloading method in vehicular edge computing environment

Abstract

The present application relates to the technical field of intelligent Internet of Vehicles, and in particular to a deep reinforcement learning-based task unloading method in a vehicular edge computing environment, comprising: acquiring tasks to be executed generated by a task vehicle, uploading the tasks to an RSU, generating a scheduling decision on the basis of the current states of the tasks and available resources, and allocating the tasks to service vehicles or executing the tasks locally; prioritizing the tasks by means of the analytic hierarchy process, and acquiring the current state of each of the tasks to be executed; on the basis of the states and the priorities of the tasks, using a model for scheduling to acquire service vehicle numbers and a task vehicle number for the tasks, and a computing unloading and scheduling policy; acquiring available computing resources of the current service vehicle; and, by means of the Actor-Critic algorithm, training a sequence-to-sequence model, so as to obtain an optimal task partial unloading and scheduling policy. The present application can fully use computing resources of service vehicles and edge servers, so as to allow for shorter execution delays of all tasks throughout the entire time period and higher task execution success rates.

Description

A Task Offloading Method Based on Deep Reinforcement Learning in Vehicle Edge Computing Environment Technical Field

[0001] This application relates to the field of intelligent vehicle networking technology, and in particular to a task offloading method in an in-vehicle edge computing environment based on deep reinforcement learning. Background Technology

[0002] With the continuous development of IoT technology and the increasing business demands, a large number of new intelligent devices have emerged, among which in-vehicle devices occupy a dominant position. As an important component of the IoT, the Internet of Vehicles (IoV) enables data interaction and information sharing between vehicles (V2V), vehicles (V2I), and vehicles (V2X) through wireless communication technology. This plays a crucial role in promoting sustainable urban development and improving people's travel experience. The development of 5G technology has led to many new latency-sensitive and computationally intensive computing tasks in IoV, such as road condition monitoring, map navigation, and route planning. However, the widespread application of these computing tasks has resulted in a significant increase in data volume, causing vehicles with limited computing resources to be unable to handle such a large number of computationally intensive tasks, thus failing to meet the service quality levels required by users.

[0003] In existing technologies, a common solution to the limited computing resources of in-vehicle equipment is to utilize cloud computing (CC), offloading computing tasks to a central cloud for processing. However, due to the long distance between the vehicle and the central cloud, this approach may encounter significant transmission latency during task transfer, and task execution efficiency will further decrease when network links become congested. Therefore, cloud computing may not be able to meet the real-time and low-latency requirements of vehicles in certain latency-sensitive task scenarios. To address this, vehicular edge computing (VEC) is gradually being adopted. VEC provides low-latency and real-time computing services by offloading computing and storage resources to servers or vehicles at the network edge, effectively alleviating the pressure on the central cloud. However, VEC suffers from high infrastructure construction costs, limited coverage of roadside units (RSUs), and its computing resources become scarce as the workload increases.

[0004] However, RSUs have limited computing resources and high infrastructure deployment costs, and their coverage cannot be extended to every task-demanding area. Some latency-sensitive tasks, although computationally less demanding, have high real-time requirements, and offloading them to nearby vehicles may be more efficient than transmitting them to the RSU. Therefore, existing offloading strategies have not fully utilized the idle computing resources of vehicles in vehicular edge computing, resulting in the inability to achieve optimal task processing in some scenarios. Summary of the Invention

[0005] This application provides a task offloading method based on deep reinforcement learning in an in-vehicle edge computing environment. This method fully utilizes the computing resources of the service vehicle and the edge server, resulting in lower execution latency and higher task success rate for all tasks throughout the entire time period. The technical solution provided in this application is as follows:

[0006] Firstly, this application provides a task offloading method based on deep reinforcement learning in an in-vehicle edge computing environment, the method comprising:

[0007] The system acquires pending tasks generated by the task vehicle, uploads the pending tasks to the roadside unit, and the roadside unit generates a scheduling decision based on the current status of the pending tasks and available resources, and assigns the pending tasks to service vehicles or executes them locally.

[0008] The analytic hierarchy process (AHP) is used to prioritize all the tasks to be executed, and the current state of each task is obtained by combining computational complexity, task data volume, and latency tolerance.

[0009] Based on the task status and priority, a calculation model is used to schedule tasks, obtain the service vehicle number, task vehicle number, calculation offloading, and scheduling strategy for each task; leave part of the task to be executed locally and add it to the task vehicle's waiting queue, and offload the other part to a service vehicle and add it to the task vehicle's waiting queue.

[0010] After the task is partially unloaded, the execution status of the task is monitored in real time based on the previous scheduling strategy, and the available computing resources of the currently serving vehicle are obtained.

[0011] Based on the task status and the available computing resources of the service vehicle, an improved sequence-to-sequence model is trained using the Actor-Critic algorithm to obtain the optimal task offloading and scheduling strategy.

[0012] In one specific implementation scheme, the step of acquiring the tasks to be executed generated by the task vehicle, uploading the tasks to be executed to the roadside unit, and having the roadside unit generate a scheduling decision based on the current status of the tasks to be executed and available resources, and assigning the tasks to service vehicles or local execution, includes:

[0013] When a task vehicle generates a task to be performed, the task vehicle uploads the task information to the roadside unit via wireless communication technology. The task information includes the computational complexity of the task, the amount of input data, and the latency tolerance.

[0014] Based on the task information and the current computing resource status of the system, the roadside unit generates a task unloading decision and assigns the task to a service vehicle or executes it locally on the task vehicle.

[0015] The task vehicle is a vehicle with computing tasks within the communication range of the roadside unit, and the service vehicle is a vehicle with remaining computing resources around the task vehicle that can provide services. After the task vehicle sends a task offload request to the roadside unit, the roadside unit evaluates the link duration based on the distance between the task vehicle and the service vehicle, the channel status, and the relative speed, and decides whether to offload the task to the service vehicle.

[0016] The task is broken down into multiple subtasks. Based on the task's computational complexity, input data volume, and latency tolerance, each subtask is offloaded. Some tasks are executed locally by the task vehicle, while others are processed in parallel by the service vehicle.

[0017] After the service vehicle completes the task, it transmits the results to the task vehicle via wireless communication technology. The roadside unit then dynamically adjusts the task unloading strategy based on the task execution status.

[0018] In a specific implementation scheme, prioritizing all the tasks to be executed using the analytic hierarchy process (AHP) and obtaining the current state of each task to be executed by considering computational complexity, task data volume, and latency tolerance includes:

[0019] Factors related to task offloading decisions are divided into target layer, criterion layer and scheme layer, and a priority hierarchical structure model is constructed. The target layer is the priority order of task processing, the criterion layer includes computational complexity, task data volume and latency tolerance, and the scheme layer is all tasks to be processed.

[0020] The three influencing factors in the criterion layer are compared pairwise to construct an evaluation index judgment matrix from the target layer to the criterion layer. The weight of each influencing factor is calculated by the sum-product method. The weight is determined according to the urgency of the task, and the importance of computational complexity, task data volume and time delay tolerance decreases in that order.

[0021] Construct a task judgment matrix from the criterion layer to the scheme layer, calculate the weight of each task under different influencing factors, and obtain the weight vector of each task under each influencing factor by the sum-product method;

[0022] A consistency check is performed on the weight vector to obtain the final task weight matrix. The task priority is determined by the task weight matrix, and tasks with higher weights are processed first when computing resources are limited.

[0023] In a specific feasible implementation, the construction of the evaluation index judgment matrix from the target layer to the criterion layer and the construction of the task judgment matrix from the criterion layer to the solution layer include:

[0024] The evaluation index judgment matrix from the target layer to the criterion layer is constructed as A = (a lr ) 3×3 , l, r∈{1,2,3}, where:

[0025] Compare the M tasks at the solution layer, and construct the judgment matrix B from the three influencing factors at the criterion layer to the solution layer. k =(b ij ) M×M , i,j∈{1,2,...,M}, k∈{1,2,3}; where

[0026] The value of q is obtained from the existing PCM quantization value table.

[0027] In one specific implementation scheme, after the task is partially unloaded, the execution status of the task is monitored in real time based on the previous scheduling strategy, and the available computing resources of the currently serving vehicle are obtained, including:

[0028] Assess the latency of task transmission and processing between task vehicles and service vehicles, and determine whether the total execution latency of the task meets the set latency tolerance.

[0029] When a task is executed locally on the task vehicle, the local execution latency of the task is calculated and compared with the latency tolerance of the task.

[0030] When a task needs to be partially unloaded, the task is split into two sub-tasks: one part is executed locally on the task vehicle, and the other part is unloaded to the service vehicle for execution.

[0031] The task unloading process includes the waiting time before task unloading, the time for task data to be uploaded to the service vehicle, the processing time of the task on the service vehicle, and the time for the task results to be sent back to the task vehicle.

[0032] In one specific implementation scheme, the step of training an improved sequence-to-sequence model using the Actor-Critic algorithm based on the task status and the available computing resources of the service vehicle to obtain the optimal task offloading and scheduling strategy includes:

[0033] Build multiple task queues to manage the execution status of tasks;

[0034] The queue includes a failure queue, a processing queue, and a waiting queue; the highest priority task is selected from the waiting queue for scheduling, and a suitable service vehicle or task vehicle is assigned to it.

[0035] During the allocation process, the required computing resources are compared with the currently available computing resources to determine whether the task has been successfully allocated.

[0036] If successful, the task will be added to the processing queue.

[0037] If the task fails, a service vehicle or a task vehicle will be reassigned to the task.

[0038] During execution, the execution status of the task is monitored in real time, the remaining execution time and total execution time of the task are updated, and tasks that time out are processed and added to the failure queue.

[0039] In one specific implementation scheme, the step of training an improved sequence-to-sequence model using the Actor-Critic algorithm based on the task status and the available computing resources of the service vehicle to obtain the optimal task offloading and scheduling strategy includes:

[0040] An improved sequence-to-sequence model is trained using the Actor-Critic algorithm. The sequence-to-sequence model takes the task state and service vehicle state as input and outputs the service vehicle number that performs each task.

[0041] By interacting with the environment, the Actor and Critic networks are continuously updated to iteratively optimize task unloading strategies and improve task execution success rates.

[0042] The Actor network is trained using an improved Seq2Seq model and incorporates an attention mechanism to focus on the part of the input sequence that is most relevant to the current task when processing longer input sequences.

[0043] In one specific implementation scheme, the task scheduling process is optimized through a defined reward function, specifically including:

[0044] After each scheduling action is executed, the corresponding reward value is calculated based on the task's execution result and status.

[0045] State information and actions are input into the Critic network, and the Critic network is used to estimate the long-term reward for each state.

[0046] The parameters of the Actor and Critic networks are continuously adjusted through iterative optimization during the training process.

[0047] In a second aspect, this application provides an electronic device, the device including a processor and a memory; the memory stores a program, the program being loaded and executed by the processor to implement a task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning as described in the first aspect.

[0048] Thirdly, this application provides a computer-readable storage medium storing a program that, when executed by a processor, is used to implement a task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning as described in the first aspect.

[0049] In summary, the beneficial effects of this application include at least the following:

[0050] 1) By training the task offloading model using the Actor-Critic (AC) reinforcement learning algorithm, the latency during task processing was effectively reduced and the success rate of task execution was significantly improved.

[0051] 2) By comprehensively considering the computational complexity, data volume, and latency tolerance of the tasks to be executed, the task scheduling order is optimized, further reducing task execution time and improving the overall performance of the system.

[0052] 3) It overcomes the limitation of traditional deep reinforcement learning algorithms that rely on a single-threaded architecture, can make efficient use of computing resources, avoid the problem of experience coupling, and significantly improve training efficiency.

[0053] This method optimizes task allocation and scheduling by fully utilizing the computing resources of Service Vehicles (SVs) and Remote Units (RSUs) in an vehicular edge computing system. It employs the Actor-Critic algorithm from deep reinforcement learning, dynamically adjusting the partial offloading strategy based on vehicle status and task priority. Specifically, it splits computational tasks into two sub-tasks: one part remains in the local vehicle for execution, while the other is offloaded to the service vehicle or edge server. Through reasonable task splitting and offloading decisions, it minimizes task execution latency while ensuring high success rates. The core steps of this method include task information uploading, priority allocation, task splitting, scheduling and offloading, and obtaining the optimal scheduling strategy based on deep reinforcement learning. By establishing waiting queues, processing queues, and failure queues to manage task status, and utilizing an improved Seq2Seq model and IndRNN combined with an attention mechanism, it accurately predicts and dynamically schedules task execution positions, further improving the system's parallel processing capabilities. Through deep reinforcement learning, it dynamically adjusts task scheduling based on the real-time status of vehicles and servers, optimizing resource allocation and reducing latency. By employing a reasonable task splitting strategy, computational tasks are refined into multiple sub-tasks for parallel execution, enhancing the system's parallel capabilities and effectively improving execution efficiency. By using the Actor-Critic algorithm, the task allocation strategy is continuously optimized to ensure the success rate of task execution and improve the overall performance of the system.

[0054] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0055] Figure 1 is a schematic diagram of the scalable computing task offloading system architecture used in the embodiments of this application.

[0056] Figure 2 is a schematic diagram of the hierarchical structure model of task priority in the embodiments of this application.

[0057] Figure 3 is a schematic diagram of the task splitting model in an embodiment of this application.

[0058] Figure 4 is a schematic diagram of the task unloading structure in an embodiment of this application.

[0059] Figure 5 is a schematic diagram of the structure of the improved Seq2Seq model in the embodiments of this application.

[0060] Figure 6 is a schematic diagram of the structure of the Actor-Critic algorithm in the embodiment of this application.

[0061] Figure 7 is a block diagram of an electronic device for task offloading in a vehicle-mounted edge computing environment based on deep reinforcement learning, according to an embodiment of this application. Detailed Implementation

[0062] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0063] Optionally, this application uses the task offloading method based on deep reinforcement learning in the vehicle edge computing environment provided in various embodiments as an example for application in an electronic device. The electronic device is a terminal or a server. The terminal can be a computer, tablet computer, etc. This embodiment does not limit the type of electronic device.

[0064] Referring to Figure 1, it is a flowchart illustrating a task offloading method based on deep reinforcement learning in an in-vehicle edge computing environment according to an embodiment of this application. The method includes at least the following steps:

[0065] Step S101: When a task vehicle generates a task to be executed, it uploads the task information to the RSU. The RSU generates a scheduling decision based on the current status of the task and available resources, and assigns the task to a suitable service vehicle or executes it locally.

[0066] Step S102: After the RSU generates the scheduling decision, the hierarchical analysis method is used to prioritize all tasks to be executed, and the current status of each task to be executed is obtained by combining the computational complexity, task data volume and latency tolerance.

[0067] Step S103: Based on task status and priority, perform task scheduling using a computational model to obtain the service vehicle number and task vehicle number for each task, and acquire the current computation offloading and scheduling strategy. Leave a portion of the task running locally and add it to the task vehicle's waiting queue, while offloading the other portion to a service vehicle and adding it to the task vehicle's waiting queue.

[0068] Step S104: After the task is partially unloaded, based on the previous scheduling strategy, monitor the execution status of the task in real time and obtain the available computing resources of the currently serving vehicle.

[0069] Step S105: Based on the task status and the available computing resources of the service vehicles, train the improved sequence-to-sequence model using the Actor-Critic algorithm to obtain the optimal task offloading and scheduling strategy.

[0070] In step S101, when a task vehicle (TV) generates a task to be performed, it first uploads the task information to the roadside unit (RSU) via wireless communication technology. The RSU, as the central node for task scheduling, is responsible for receiving the task information uploaded by the task vehicle, analyzing it based on the task requirements and the current state of the system, and then generating a corresponding task scheduling decision.

[0071] Referring to Figure 1, which is a schematic diagram of the shardable computing task offloading system architecture used in an embodiment of this application, it is assumed that there are many vehicles traveling on a one-way straight road, denoted by the set V = {1,2,3,...,v}. All vehicles are evenly distributed on the road and travel at a constant speed within a certain time. An RSU equipped with a VEC server is deployed on one side of the road, and all vehicles on the road are always within the communication range of this RSU. Wireless communication is possible between vehicles and between vehicles and the RSU. Tasks generated by the TV can be offloaded to the SV for processing through V2V communication technology. After the task is completed, the SV returns the result directly to the TV via a single-hop transmission.

[0072] In implementation, vehicles generating tasks within the current RSU communication range are referred to as task vehicles (TVs). In this context, vehicles with remaining computing resources around the mission vehicle that can provide services are called currently serviceable vehicles (SVs). This indicates that vehicles without remaining computing resources around the task vehicle that cannot provide service, as well as vehicles outside the current RSU communication range, are collectively referred to as "other vehicles." When a TV generates a task that needs to be unloaded, the TV first sends a task unloading request to the RSU. Then, the RSU sends the task unloading decision to both the TV and the SV. The TV then unloads the task to be unloaded to the specific SV. After the task is completed, the SV returns the result to the TV. f l It refers to the TV's computing power, specifically CPU cycles per second. This refers to the transmit power of the TV when sending mission data to the SV. Assume there are N SVs around the TV during mission offloading, where SV = {SV1, SV2, ..., SV}. N},in j∈{1,2,...,N},f j This refers to the computing power of the j-th SV, also meaning CPU cycles per second. Let be the transmit power of the j-th SV. Each vehicle travels at a constant speed v, and the speeds of different vehicles follow a uniform distribution. The channel state is evaluated using the distance between TV and SV, and the link duration is evaluated using the relative positions and relative speeds of TV and SV.

[0073] Furthermore, preferably, the total system time is divided into several time periods. Within a single time period, TV generates M computational tasks, all of which are decomposable computational tasks. The i-th task is defined as a quadruple Ω. i ={C i D i ,κ i ,τ i}, i∈{1,2,...,M}, where C i Indicates completion Ω i Required CPU cycles, D i Represents Ω i The size of the input data, κ i Represents Ω i The computational complexity, τ i Represents Ω i The maximum tolerable latency. Assuming there are no dependencies between all tasks, SV can process multiple tasks in parallel.

[0074] In step S102, when the task vehicle unloads the computing task to a service vehicle with available computing resources, the different attributes of the computing tasks lead to different task types and varying degrees of urgency. The computing task types generated by the vehicles include urgent security tasks, general real-time tasks, and other task types. This application uses priority to represent the urgency of tasks. Given limited computing resources, the higher the priority of a task, the more urgent it is, and it will be processed first. The Analytic Hierarchy Process (AHP) is a multi-criteria decision-making method that combines qualitative and quantitative analysis. This method divides the decision-related components into three levels: the objective level, the criterion level, and the alternative level, thus forming a multi-level structural model. This model is well-suited for handling the weight allocation of task attributes and the division of task priorities.

[0075] In implementation, this application primarily considers the following three influencing factors regarding task attributes: computational complexity, task data volume, and latency tolerance, in the following order of importance: computational complexity, task data volume, and latency tolerance. The hierarchical structure model of task priority is shown in Figure 2. This application uses the AHP method to assign higher weights to tasks with high computational complexity, large data volume, and low latency tolerance, thereby ensuring that under latency constraints, the system can more rationally allocate limited computing resources and improve the task offloading success rate.

[0076] Specifically, the steps of the AHP algorithm are as follows:

[0077] Step S1021: First, compare the three influencing factors at the criterion level and construct the evaluation index judgment matrix (Pairwise Comparison Matrix, PCM) from the target level to the criterion level as A = (a lr ) 3×3 , l, r∈{1,2,3}, where:

[0078] Subsequently, the M tasks at the scheme layer are compared, and a judgment matrix B is constructed from the three influencing factors at the criterion layer to the scheme layer. k =(b ij ) M×M Let i,j∈{1,2,...,M}, and k∈{1,2,3}.

[0079] It should be noted that the values ​​of q in equations (1) and (2) above are obtained from Table 1 below:

[0080] Table 1 PCM Quantization Values

[0081] Step 1022: Subsequently, the weight β of the l-th influencing factor in A is obtained by the sum-product method. l for:

[0082] The weight vector of the three influencing factors of A can be represented as β = [β 1 ,β 2 ,β 3 ] T .

[0083] Step 1023: Obtain B again using the sum-product method. k The weight of the k-th influencing factor in the i-th task for:

[0084] Among them B k The weight vector of the M tasks can be represented as B 1 B 2 B 3 The weight vectors form the criterion layer, and the weight matrix for all tasks in the scheme layer is:

[0085] Step 1024: After verifying the consistency of the weights, the final weight vector S for all tasks is obtained as shown in equation (6). The elements in S consist of the weights of the corresponding computational tasks.

[0086] In step S103, when the scheduling decision generated by the RSU determines that a task needs to be partially offloaded, the task vehicle (TV) will split the computation task into two subtasks according to the RSU allocation result: one subtask will remain on the task vehicle for local execution, and the other subtask will be offloaded to a nearby service vehicle (SV) with available computing resources for execution. The entire computation task is considered successful only when each subtask is successfully executed. It is assumed that there are no dependencies between all computation tasks, each subtask can only select one SV for offloading, and each SV can process multiple subtasks in parallel.

[0087] Specifically, the task partitioning diagram consists of TV, SV sets, and task sets {Task1, Task2, ..., Task...} M The system consists of several parts, all of which are decomposable computational tasks, as shown in Figure 3. To simplify the analysis, we assume that all tasks are bit-independent, meaning that the task data partitions have complete granularity, and therefore tasks can be divided into subsets of any size. Let task Ω... i The task split ratio is φ i (i∈M,φ i ∈[0,1]),φ i Represents task Ω i The proportion of tasks executed locally on the TV to the total number of tasks, the remaining 1-φ i This indicates the proportion of tasks that were offloaded to service vehicles for execution out of the total number of tasks. Since the tasks are independent, the resulting subtasks do not need to be executed in a specific order.

[0088] In step S104, during the task partial unloading process, the execution latency of the task directly affects the overall task scheduling and resource allocation strategy. The task partial unloading process between the task vehicle (TV) and the service vehicle (SV) includes data transmission and processing. The system needs to evaluate the total latency under different task execution modes based on the latency model to ensure that the task execution latency does not exceed its set latency tolerance.

[0089] Specifically, when task Ω i When the execution method is local execution, task Ω i computation delay for

[0090] The computation latency of a task running locally cannot exceed its latency tolerance, therefore the following restrictions apply:

[0091] When task Ω i When the execution method is partial unloading, task Ω i φ i Part of the computation is performed locally on the TV, and its latency is...

[0092] Assuming task Ω i 1-φ i Part of the task is unloaded to the SV for execution. The unloading process is as follows: the TV uploads the task data to the SV, the SV executes the task and sends the result back to the TV. The total task unloading time is divided into four parts: the waiting time before task unloading, the data upload time, the task processing time on the SV, and the result return time. Before task unloading, the task information and SV status need to be uploaded to the RSU via the wireless network, and then the RSU allocates an SV for the task. Using R... l R represents the link transmission rate when the task is offloaded to the j-th SV. j This represents the link transmission rate when the j-th SV sends the result back to the TV.

[0093] In equations (10)-(11) above, B represents the vehicle transmission channel bandwidth, and d j Let δ represent the distance between TV and the j-th SV, and h represent the path loss exponent. j Let N represent the complex Gaussian channel coefficients, N0 represent additive white Gaussian noise, and I represent interference from other vehicles during V2V transmission. The transmission time for uploading the task data to SV is as follows:

[0094] The execution time of the task on the j-th SV is as follows:

[0095] The result return time is as follows:

[0096] In equation (14) above, η represents the ratio between the size of the output data and the size of the input data. Typically, η <= 1 and can be ignored when calculating task time delay. The total delay for task unloading is then as follows:

[0097] In the above formula (15), Represents task Ω i Waiting time before uninstallation. Due to task Ω i φ i Part and 1-φ i Partially parallel processing, so Ω i If the total latency is the larger of the two, then task Ω i Total execution latency t i as follows:

[0098] The total latency during unloading of each task must not exceed its latency tolerance; therefore: t i≤τ i (17)

[0099] By calculating the latency model described above, a suitable local processing or partial offloading strategy can be selected for the task, thereby optimizing the task's execution time and resource allocation.

[0100] In step S105, a deep reinforcement learning algorithm is introduced to optimize the scheduling decision for task offloading. To improve the task execution success rate and reduce the average execution latency, a queue-based scheduling model and an improved Seq2Seq network model are designed, and the Actor-Critic algorithm is used to iteratively optimize the scheduling strategy.

[0101] Specifically, to manage and track the execution status of tasks, three queues are designed in this step:

[0102] (1) Fail Queue, which records tasks that failed to be assigned or timed out;

[0103] (2) Processing Queue: Records tasks that are being executed.

[0104] (3) Waiting Queue: Records tasks that have not yet been scheduled.

[0105] In implementation, the task scheduling process is shown in Algorithm 1 of Listing 2 below. The highest priority task Ω is identified in the Waiting Queue and scheduled. First, an SV or TV is assigned to task Ω to determine whether task Ω should be offloaded or executed locally. Then, the initial computing resources of the SV or TV assigned to task Ω are compared with the computing resources required by task Ω. If the initial computing resources of the SV or TV are less than the computing resources required by task Ω, the task allocation fails, and an SV or TV needs to be reassigned to task Ω. If the task allocation is successful, it is determined whether the currently available computing resources of the SV or TV are sufficient to compute the task. If sufficient, the task is added to the Processing Queue of the current SV or TV, and task Ω is removed from the Waiting Queue; otherwise, task Ω waits. Once the task with the shortest remaining execution time in the Processing Queue of the current SV or TV finishes execution and releases its occupied computing resources, the remaining execution time and total execution time of task Ω and all tasks in the Processing Queue are immediately updated. After the update is completed, it is determined whether task Ω has timed out. If it has timed out, task Ω fails to execute and is added to the Fail Queue. If it has not timed out, task Ω is rescheduled by recursively calling Algorithm 1.

[0106] Table 2 Task Scheduling Pseudocode

[0107] This embodiment aims to maximize the task execution success rate while minimizing the average task execution latency. Therefore, the execution success rate of the computation task is defined as follows:

[0108] In equation (18) above, q represents the number of tasks that timed out and failed, i.e., the number of tasks in the Fail Queue, and M is the total number of tasks. The average latency of all tasks is as follows:

[0109] In practice, deep reinforcement learning methods can iteratively improve through continuous interaction with the environment, gradually approaching the optimal solution. The Critic network in the Actor-Critic algorithm provides a baseline estimate of the policy, reducing the variance of the policy gradient and thus making the training process more stable. The Critic network estimates the long-term reward of each state through a value function, helping the agent better understand the environment and improve its policy. Therefore, this embodiment proposes a computational task offloading and scheduling method based on deep reinforcement learning, and trains the neural network using the Actor-Critic algorithm.

[0110] Specifically, the task unloading diagram is shown in Figure 4. Assume that TV generates three computational tasks: Task1, Task2, and Task3. These tasks are then divided into six subtasks: Subtask1, Subtask2…Subtask6, resulting in a new task sequence. TV is numbered 0, and the number of SVs is N=3, with SVs numbered 1, 2, and 3 respectively. One possible execution decision for these three computational tasks can be represented as {0, 2, 1, 3, 2, 1}, where 0 indicates that Subtask1 is executed locally on TV number 0, 2 indicates that Subtask2 is unloaded to SV number 2 for execution, and 1 indicates that Subtask3 is unloaded to SV number 1 for execution, and so on.

[0111] Sequence-to-sequence models are a type of encoder-decoder architecture, with a basic structure consisting of recurrent neural networks. The problem of assigning tasks to service vehicles can be transformed into mapping a sequence of tasks and service vehicles to a sequence of service vehicle numbers of the same length as the task sequence. This problem is suitable for Seq2Seq. However, commonly used RNN structures in Seq2Seq, such as GRU, employ the hyperbolic tangent activation function, which causes gradient decay in the network layers, leading to vanishing and exploding gradients. Unlike RNNs, IndRNNs introduce parameter constraints, making gradients more stable during backpropagation, effectively solving the vanishing and exploding gradient problems. Furthermore, IndRNNs exhibit better robustness after training. Therefore, this application replaces the RNN in Seq2Seq with IndRNN. In addition, Seq2Seq models require compressing all information from the input sequence, making them difficult to handle long input sequences. To address this, this application incorporates an attention mechanism: after the encoder receives an output, it searches the input sequence for the part most relevant to that output. The model then predicts the unloading vehicle number for the next task based on the context vector corresponding to this part of the input sequence and all previously generated service vehicle numbers. Compared to the original Seq2Seq model, the improved model focuses on the part of the input sequence most relevant to the current task during decoding, and can better handle long input sequences.

[0112] The execution process of the improved Seq2Seq model used in this application at time step t is shown in Figure 5. Task information and service vehicle information serve as the input sequence of the model, and the vehicle number executing each task serves as the output sequence. The input task sequence consists of decomposable computational tasks, which need to be decomposed before scheduling their subtasks. Therefore, a set of decomposition weights φ is defined in the encoder. After the task information is input into the encoder, the decomposable computational tasks are decomposed using these weights, which are iteratively optimized. The model input consists of the task sequence X and SV, along with TV information, obtained through AHP calculation. After decomposing the decomposable computational tasks in the task sequence using the decomposition weights φ, a new task sequence X' is obtained, consisting of 2m subtasks derived from the decomposable computational tasks. The new task sequence X' is used as the input to the encoder. The decoder input is the context vector c. t and the hidden layer output s at time step t-1 t-1 The output is task x. t Corresponding service vehicle number y t This yields the vehicle number for the task, and finally, the Seq2Seq algorithm is used to obtain the coding sequence of all task execution decisions.

[0113] At time step t in the decoding phase, the decoder, given the context vector c t and all previously predicted results {y1,...,y t-1 In the case of}, predict the next result y t The output result is {y1,...,y t The probabilities of} are as follows:

[0114] In equation (20) above, each conditional probability is modeled as follows:

[0115] Then the encoder's hidden layer vector H and the decoder's hidden layer output s at time step t-1 are used. t-1 The input is fed into the Attention module to obtain the attention value. Right now:

[0116] In equation (22) above, s0, as the initial input of the Attention at time step t=0, is obtained by linear transformation of the last hidden layer. As intermediate variables, W1 and v1 are trainable parameters, and attn is a fully connected neural network. This is determined by H and the attention value vector e. t , thus obtaining the context vector c t As follows: c t =e t H (23)

[0117] Then c t s t-1 and the decoder's output y at time step t-1 t-1 The input is fed into the decoder and passes through the tanh activation function to obtain the hidden state s at time step t. t The conditional probability required by equation () is as follows: s t =tanh(W2[y t-1 ,c t ,s t-1 ]) (twenty four)

[0118] In equations (24)-(25) above, W2 and v2 are trainable parameters.

[0119] Specifically, the Actor-Critic network training is shown in Figure 6. The Actor network is the improved Seq2Seq model mentioned in the previous section. State represents the state, including the task state and the service vehicle state. The task state consists of five parts: the task's execution time on the CPU, computational complexity, data volume, latency tolerance, and task priority. The service vehicle state is the computing resources available to the vehicle. Action represents the action, that is, the server number to be unloaded by the task generated by the Actor network. The set of actions is the unloading strategy. Reward represents the reward corresponding to the action. The training process is as follows: the state is input into the Actor network, the corresponding action is generated, and then the interaction with the environment is performed through Algorithm 1 to obtain the reward. The state and the reward after the action are used as the input of the Critic network, and then the Actor network and the Critic network are updated through the loss function. The parameter of the Actor is defined as θ, and reward(a) is the reward generated by the Actor when using strategy a, which is obtained by equation (18). The function J(θ) is defined as follows: J(θ)=E(reward(a)) (26)

[0120] The optimization objective of the Actor is to maximize the reward value. This application uses the Reinforce algorithm to train the Actor network, as shown below:

[0121] In the above formula (27), The baseline function P is obtained from the Critic network. θ (a|s) is the conditional probability of taking policy a in state s, obtained from the Actor network, and is determined by equation (21). Let the batch size in the training dataset be K, then the update formula for parameter θ can be expressed as follows:

[0122] The mean squared error loss function was then used to evaluate the parameters of Critic. Update:

[0123] In the above formula (29), Let reward(a) be the reward value predicted by the Critic network. k |s k The value is the true value. The training process of Actor-Critic is shown in Table 3 below, where the learning rate is 0.0001, the batch size is 32, the optimizer is Adam, T is the total number of epochs, and K is the number of times the state sequence is updated. The Critic network structure consists of an RNN layer, a convolutional layer, and a fully connected network, used to calculate the predicted reward value of the input state.

[0124] Table 3. Pseudocode for Actor-Critic Network Training

[0125] In summary, this application optimizes task allocation and scheduling by fully utilizing the computing resources of Service Vehicles (SVs) and Remote Units (RSUs) in an vehicular edge computing system. The method employs the Actor-Critic algorithm from deep reinforcement learning, dynamically adjusting the partial offloading strategy based on vehicle status and task priority. Specifically, it splits the computational task into two sub-tasks: one part remains in the local vehicle for execution, while the other is offloaded to the service vehicle or edge server. Through reasonable task splitting and offloading decisions, the method can minimize task execution latency while ensuring high success rates. The core steps of this method include task information uploading, priority allocation, task splitting, scheduling and offloading, and obtaining the optimal scheduling strategy based on deep reinforcement learning. By establishing waiting queues, processing queues, and failure queues to manage task status, and utilizing an improved Seq2Seq model and IndRNN combined with an attention mechanism, the execution position of tasks is accurately predicted and dynamically scheduled, further improving the system's parallel processing capabilities. Through deep reinforcement learning, task scheduling can be dynamically adjusted based on the real-time status of vehicles and servers, optimizing resource allocation and reducing latency. By employing a reasonable task decomposition strategy, computational tasks are broken down into multiple subtasks and executed in parallel, enhancing the system's parallel capabilities and effectively improving execution efficiency. Furthermore, the Actor-Critic algorithm continuously optimizes the task allocation strategy, ensuring high task execution success rates and improving overall system performance.

[0126] Figure 7 is a block diagram of an electronic device provided in one embodiment of this application. The device includes at least a processor 401 and a memory 402.

[0127] Processor 401 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0128] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 402 is used to store at least one instruction, which is executed by the processor 401 to implement the task offloading method based on deep reinforcement learning in an in-vehicle edge computing environment provided in the method embodiments of this application.

[0129] In some embodiments, the electronic device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 401, memory 402, and peripheral device interface can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to: radio frequency circuitry, a touch display screen, audio circuitry, and a power supply.

[0130] Of course, electronic devices may also include fewer or more components, and this embodiment does not limit this.

[0131] Optionally, this application also provides a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the task offloading method for a vehicle-mounted edge computing environment based on deep reinforcement learning in the above-described method embodiments.

[0132] Optionally, this application also provides a computer product including a computer-readable storage medium storing a program, which is loaded and executed by a processor to implement the task offloading method for a vehicle-mounted edge computing environment based on deep reinforcement learning in the above-described method embodiments.

[0133] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

A task offloading method based on deep reinforcement learning in an in-vehicle edge computing environment, characterized in that, The method includes: The system acquires pending tasks generated by the task vehicle, uploads these tasks to the roadside unit, and the roadside unit generates a scheduling decision based on the current status of the pending tasks and available resources, and then assigns the pending tasks to a service vehicle or executes them locally. The analytic hierarchy process (AHP) is used to prioritize all the tasks to be executed, and the current state of each task is obtained by combining computational complexity, task data volume, and latency tolerance. Based on the task status and priority, a calculation model is used to schedule tasks, obtain the service vehicle number, task vehicle number, calculation offloading, and scheduling strategy for each task; leave part of the task to be executed locally and add it to the task vehicle's waiting queue, and offload the other part to a service vehicle and add it to the task vehicle's waiting queue. After the task is partially unloaded, the execution status of the task is monitored in real time based on the previous scheduling strategy, and the available computing resources of the currently serving vehicle are obtained. Based on the task status and the available computing resources of the service vehicle, an improved sequence-to-sequence model is trained using the Actor-Critic algorithm to obtain the optimal task offloading and scheduling strategy. The task offloading method for vehicular edge computing environment based on deep reinforcement learning according to claim 1 is characterized in that, The process of acquiring pending tasks generated by the task vehicle, uploading these tasks to the roadside unit, and having the roadside unit generate scheduling decisions based on the current status and available resources of the pending tasks, and then assigning the tasks to service vehicles or for local execution, includes: When a task vehicle generates a task to be performed, the task vehicle uploads the task information to the roadside unit via wireless communication technology. The task information includes the computational complexity of the task, the amount of input data, and the latency tolerance. Based on the task information and the current computing resource status of the system, the roadside unit generates a task unloading decision and assigns the task to a service vehicle or executes it locally on the task vehicle. The task vehicle is a vehicle with computing tasks within the communication range of the roadside unit, and the service vehicle is a vehicle with remaining computing resources around the task vehicle that can provide services. After the task vehicle sends a task offload request to the roadside unit, the roadside unit evaluates the link duration based on the distance between the task vehicle and the service vehicle, the channel status, and the relative speed, and decides whether to offload the task to the service vehicle. The task is broken down into multiple subtasks. Based on the task's computational complexity, input data volume, and latency tolerance, each subtask is offloaded. Some tasks are executed locally by the task vehicle, while others are processed in parallel by the service vehicle. After the service vehicle completes the task, it transmits the results to the task vehicle via wireless communication technology. The roadside unit then dynamically adjusts the task unloading strategy based on the task execution status. The task offloading method for vehicular edge computing environment based on deep reinforcement learning according to claim 1 is characterized in that, The step of prioritizing all the tasks to be executed using the analytic hierarchy process (AHP) and obtaining the current state of each task by considering computational complexity, task data volume, and latency tolerance includes: Factors related to task offloading decisions are divided into target layer, criterion layer and scheme layer, and a priority hierarchical structure model is constructed. The target layer is the priority order of task processing, the criterion layer includes computational complexity, task data volume and latency tolerance, and the scheme layer is all tasks to be processed. The three influencing factors in the criterion layer are compared pairwise to construct an evaluation index judgment matrix from the target layer to the criterion layer. The weight of each influencing factor is calculated by the sum-product method. The weight is determined according to the urgency of the task, and the importance of computational complexity, task data volume and time delay tolerance decreases in that order. Construct a task judgment matrix from the criterion layer to the scheme layer, calculate the weight of each task under different influencing factors, and obtain the weight vector of each task under each influencing factor by the sum-product method; A consistency check is performed on the weight vector to obtain the final task weight matrix. The task priority is determined by the task weight matrix, and tasks with higher weights are processed first when computing resources are limited. The task offloading method for vehicular edge computing environment based on deep reinforcement learning according to claim 3 is characterized in that, The construction of the evaluation index judgment matrix from the target layer to the criterion layer and the construction of the task judgment matrix from the criterion layer to the solution layer include: The evaluation index judgment matrix from the target layer to the criterion layer is constructed as A = (a lr ) 3×3 , l, r∈{1,2,3}, where: Compare the M tasks at the solution layer, and construct the judgment matrix B from the three influencing factors at the criterion layer to the solution layer. k =(b ij ) M×M , i,j∈{1,2,…,M}, k∈{1,2,3}; where The value of q is obtained from the existing PCM quantization value table. The task offloading method for vehicular edge computing environment based on deep reinforcement learning according to claim 1 is characterized in that, After the task is partially unloaded, the execution status of the task is monitored in real time based on the previous scheduling strategy, and the available computing resources of the currently serving vehicle are obtained, including: Assess the latency of task transmission and processing between task vehicles and service vehicles, and determine whether the total execution latency of the task meets the set latency tolerance. When a task is executed locally on the task vehicle, the local execution latency of the task is calculated and compared with the latency tolerance of the task. When a task needs to be partially unloaded, the task is split into two sub-tasks: one part is executed locally on the task vehicle, and the other part is unloaded to the service vehicle for execution. The task unloading process includes the waiting time before task unloading, the time for task data to be uploaded to the service vehicle, the processing time of the task on the service vehicle, and the time for the task results to be sent back to the task vehicle. The task offloading method for vehicular edge computing environment based on deep reinforcement learning according to claim 1 is characterized in that, The optimal task offloading and scheduling strategy, based on the task status and the available computing resources of the service vehicle, is obtained by training an improved sequence-to-sequence model using the Actor-Critic algorithm, including: Build multiple task queues to manage the execution status of tasks; The queue includes a failure queue, a processing queue, and a waiting queue; the highest priority task is selected from the waiting queue for scheduling, and a suitable service vehicle or task vehicle is assigned to it. During the allocation process, the required computing resources are compared with the currently available computing resources to determine whether the task has been successfully allocated. If successful, the task will be added to the processing queue. If the task fails, a service vehicle or a task vehicle will be reassigned to the task. During execution, the execution status of the task is monitored in real time, the remaining execution time and total execution time of the task are updated, and tasks that time out are processed and added to the failure queue. The task offloading method for vehicular edge computing environment based on deep reinforcement learning according to claim 1 is characterized in that, The optimal task offloading and scheduling strategy, based on the task status and the available computing resources of the service vehicle, is obtained by training an improved sequence-to-sequence model using the Actor-Critic algorithm, including: An improved sequence-to-sequence model is trained using the Actor-Critic algorithm. The sequence-to-sequence model takes the task state and service vehicle state as input and outputs the service vehicle number that performs each task. By interacting with the environment, the Actor and Critic networks are continuously updated to iteratively optimize task unloading strategies and improve task execution success rates. The Actor network is trained using an improved Seq2Seq model and incorporates an attention mechanism to focus on the part of the input sequence that is most relevant to the current task when processing longer input sequences. The task offloading method for vehicular edge computing environment based on deep reinforcement learning according to claim 6 is characterized in that, The task scheduling process is optimized through a defined reward function, specifically including: After each scheduling action is executed, the corresponding reward value is calculated based on the task's execution result and status. State information and actions are input into the Critic network, and the Critic network is used to estimate the long-term reward for each state. The parameters of the Actor and Critic networks are continuously adjusted through iterative optimization during the training process. An electronic device, characterized in that, The device includes a processor and a memory; the memory stores a program, which is loaded and executed by the processor to implement a task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning as described in any one of claims 1 to 8. A computer-readable storage medium, characterized in that, The storage medium stores a program, which, when executed by a processor, is used to implement a task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning as described in any one of claims 1 to 8.

Images