Task unloading method in vehicle-mounted edge computing environment based on deep reinforcement learning

Abstract

The invention relates to the technical field of intelligent Internet of Vehicles, in particular to a task unloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning, which comprises the following steps: acquiring a task vehicle to generate a to-be-executed task, uploading the task to an RSU, generating a scheduling decision according to the current state of the task and available resources, and sending the scheduling decision to the RSU. The task is allocated to a service vehicle or locally executed; performing priority division on the tasks by utilizing an analytic hierarchy process, and obtaining the current state of each to-be-executed task; obtaining service vehicle numbers and task vehicle numbers of the tasks based on task states and priority level utilization model scheduling, and calculating unloading and scheduling strategies; obtaining available computing resources of the current service vehicle; and training the sequence to the sequence model by using an Actor-Cr-it ic algorithm to obtain an optimal task part unloading and scheduling strategy. The computing resources of the service vehicle and the edge server can be fully utilized, the execution time delay of all tasks in the whole time period is smaller, and the task execution success rate is higher.

Description

Technical Field

[0001] The present application relates to the field of intelligent vehicle networking technology, and in particular to a task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning. Background Art

[0002] With the continuous development of Internet of Things technology and the increase in business needs, various new intelligent devices have emerged in large numbers, among which vehicle-mounted devices occupy a major position. As an important part of the Internet of Things, the Internet of Vehicles can realize data interaction and information sharing between vehicles (V2V), vehicles and infrastructure (V2I), and vehicles and mobile devices (V2X) through wireless communication technology, which plays an important role in promoting sustainable urban development and improving people's travel experience. The development of 5G technology has led to the emergence of many new delay-sensitive and computationally intensive computing tasks in the Internet of Vehicles, such as road condition monitoring, map navigation, and path planning. However, the widespread application of these computing tasks has brought about a substantial increase in the amount of data, resulting in vehicles with limited computing resources being unable to handle such a large number of computationally intensive tasks, and thus unable to meet the service quality level required by users.

[0003] In the prior art, a common solution to the problem of limited computing resources of vehicle-mounted equipment is to use cloud computing (CC) to offload computing tasks to the central cloud for processing. However, due to the long distance between the vehicle and the central cloud, this solution may encounter a long transmission delay during task transmission, and when the network link is congested, the task execution efficiency will further decrease. Therefore, cloud computing may not meet the vehicle's requirements for real-time and low latency in some delay-sensitive task scenarios. For this reason, the vehicular edge computing (VEC) model has gradually been applied. VEC can provide low-latency and real-time computing services by sinking computing and storage resources to servers or vehicles at the edge of the network, effectively alleviating the pressure on the central cloud. However, due to the high cost of VEC infrastructure construction, the coverage of roadside units (RSUs) is limited, and their computing resources become scarce as the amount of tasks increases.

[0004] However, RSU has limited computing resources, and the infrastructure deployment cost is high. The coverage cannot be extended to every task requirement area. Although some delay-sensitive tasks have a small amount of computing, they have high real-time requirements. Offloading them to neighboring vehicles may be more efficient than transmitting them to RSU. Therefore, the existing offloading strategy has not fully utilized the idle computing resources of vehicles in vehicle-mounted edge computing, resulting in the inability to optimize task processing in some scenarios. Summary of the invention

[0005] This application provides a task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning, which can make full use of the computing resources of the service vehicle and the edge server, so that the execution delay of all tasks in the entire time period is smaller and the task execution success rate is higher. This application provides the following technical solutions:

[0006] In a first aspect, the present application provides a task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning, the method comprising:

[0007] Obtaining pending tasks generated by task vehicles, uploading the pending tasks to roadside units, and having the roadside units generate scheduling decisions based on the current status of the pending tasks and available resources, and assigning the pending tasks to service vehicles or local execution;

[0008] Prioritize all the tasks to be executed by using the hierarchical analysis method, and obtain the current status of each task to be executed by combining the computational complexity, task data volume, and delay tolerance;

[0009] Based on the task status and the priority, the computing model is used to schedule the tasks, and the service vehicle number, task vehicle number, computing offloading and scheduling strategy for executing each task are obtained; a part of the task is left to be executed locally and added to the waiting queue of the task vehicle, and another part is offloaded to the service vehicle and added to the waiting queue of the task vehicle;

[0010] After the task is partially unloaded, the execution status of the task is monitored in real time based on the previous scheduling strategy, and the available computing resources of the current service vehicle are obtained;

[0011] Based on the task status and the available computing resources of the service vehicle, an improved sequence-to-sequence model is trained using the Actor-Critic algorithm to obtain the optimal task partial offloading and scheduling strategy.

[0012] In a specific implementation scheme, the step of acquiring the pending tasks generated by the task vehicle, uploading the pending tasks to the roadside unit, generating a scheduling decision by the roadside unit according to the current status and available resources of the pending tasks, and allocating the pending tasks to the service vehicle or local execution includes:

[0013] When a task vehicle generates a task to be performed, the task vehicle uploads the task information to the roadside unit through wireless communication technology, and the task information includes the computational complexity, input data volume and delay tolerance of the task;

[0014] The roadside unit generates a task offloading decision based on the task information and the current computing resource status of the system, and assigns the task to the service vehicle or has the task vehicle execute it locally;

[0015] The task vehicle is a vehicle with a computing task within the communication range of the roadside unit, and the service vehicle is a vehicle with remaining computing resources around the task vehicle and can provide services. After the task vehicle sends a task offloading request to the roadside unit, the roadside unit evaluates the link duration based on the distance, channel status and relative speed between the task vehicle and the service vehicle, and decides whether to offload the task to the service vehicle;

[0016] Decompose the task into multiple subtasks, and offload each subtask according to the task's computational complexity, input data volume, and delay tolerance. Some tasks are executed locally by the task vehicle, and some tasks are processed in parallel by the service vehicle.

[0017] After the service vehicle completes the task, the results are transmitted to the task vehicle through wireless communication technology, and the roadside unit dynamically adjusts the task unloading strategy according to the task execution status.

[0018] In a specific implementation scheme, the use of the analytic hierarchy process to prioritize all the tasks to be executed, and combining the computational complexity, task data volume, and delay tolerance to obtain the current status of each task to be executed includes:

[0019] The factors related to task offloading decision are divided into target layer, criterion layer and scheme layer, and a priority hierarchy model is constructed, in which the target layer is the priority processing order of tasks, the criterion layer includes computational complexity, task data volume and delay tolerance, and the scheme layer is all pending tasks.

[0020] The three influencing factors in the criterion layer are compared in pairs, and an evaluation index judgment matrix from the target layer to the criterion layer is constructed. The weight of each influencing factor is calculated by the sum-product method. The weight is determined according to the urgency of the task, and the importance of calculation complexity, task data volume and delay tolerance decreases in turn;

[0021] Construct a task judgment matrix from the criterion layer to the solution layer, calculate the weight of each task under different influencing factors, and obtain the weight vector of each task under each influencing factor through the sum-product method;

[0022] The weight vector is checked for consistency to obtain the final task weight matrix. The priority of the task is determined by the task weight matrix. When computing resources are limited, tasks with higher weights are processed first.

[0023] In a specific possible implementation scheme, the construction of the evaluation index judgment matrix from the target layer to the criterion layer and the construction of the task judgment matrix from the criterion layer to the solution layer include:

[0024] Constructing the evaluation index judgment matrix from the target layer to the criterion layer is A = (a lr ) 3×3 , l,r∈{1,2,3}, where:

[0025]

[0026] Compare the M tasks at the solution level and construct the judgment matrix B from the three influencing factors at the criterion level to the solution level. k =(b ij ) M×M , i,j∈{1,2,…,M}, k∈{1,2,3};

[0027]

[0028] The value of q is obtained according to the existing PCM quantization value table.

[0029] In a specific implementation scheme, after the task is partially unloaded, the execution status of the task is monitored in real time based on the previous scheduling strategy, and the available computing resources of the current service vehicle are obtained, including:

[0030] Evaluate the task transmission and processing delay between the task vehicle and the service vehicle, and determine whether the total execution delay of the task meets the set delay tolerance;

[0031] When the task is executed locally in the task vehicle, the local execution delay of the task is calculated and compared with the delay tolerance of the task;

[0032] When a task needs to be partially offloaded, the task is split into two subtasks, one of which is executed locally on the task vehicle and the other is offloaded to the service vehicle for execution;

[0033] The task offloading process includes the waiting time before task offloading, the time for uploading task data to the service vehicle, the processing time of the task on the service vehicle, and the time for transmitting the task result back to the task vehicle.

[0034] In a specific implementation scheme, the improved sequence-to-sequence model is trained using the Actor-Critic algorithm based on the task status and the available computing resources of the service vehicle to obtain the optimal task partial offloading and scheduling strategy, including:

[0035] Build multiple task queues to manage the execution status of tasks;

[0036] The queues include a failure queue, a processing queue and a waiting queue; the task with the highest priority is selected from the waiting queue for scheduling, and a suitable service vehicle or task vehicle is allocated to it;

[0037] During the allocation process, the required computing resources are compared with the currently available computing resources to determine whether the task is successfully allocated;

[0038] If successful, the task is added to the processing queue;

[0039] If it fails, reassign the service vehicle or task vehicle to the task;

[0040] During the execution process, the execution status of the task is monitored in real time, the remaining execution time and total execution time of the task are updated, and the timed tasks are processed and added to the failure queue.

[0041] In a specific implementation scheme, the improved sequence-to-sequence model is trained using the Actor-Critic algorithm based on the task status and the available computing resources of the service vehicle to obtain the optimal task partial offloading and scheduling strategy, including:

[0042] An improved sequence-to-sequence model is trained using an Actor-Critic algorithm, wherein the sequence-to-sequence model receives a task state and a service vehicle state as input and outputs a service vehicle number that performs each task;

[0043] By interacting with the environment, the Actor and Critic networks are continuously updated to iteratively optimize the task offloading strategy and improve the success rate of task execution;

[0044] The Actor network is trained using an improved Seq2Seq model and introduces an attention mechanism to focus on the part of the input sequence that is most relevant to the current task to process longer input sequences.

[0045] In a specific implementation scheme, the task scheduling process is optimized by a defined reward function, specifically including:

[0046] After executing each scheduling action, the corresponding reward value is calculated based on the execution result and status of the task;

[0047] Input state information and actions into the Critic network, and use the Critic network to estimate the long-term return of each state;

[0048] Through iterative optimization during the training process, the parameters of the Actor and Critic networks are continuously adjusted.

[0049] In a second aspect, the present application provides an electronic device, comprising a processor and a memory; the memory stores a program, and the program is loaded and executed by the processor to implement a task offloading method in a vehicle edge computing environment based on deep reinforcement learning as described in the first aspect.

[0050] In a third aspect, the present application provides a computer-readable storage medium, which stores a program. When the program is executed by a processor, it is used to implement a task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning as described in the first aspect.

[0051] In summary, the beneficial effects of this application include at least:

[0052] 1) By using the Actor-Critic (AC) reinforcement learning algorithm to train the task offloading model, the delay in task processing is effectively reduced and the success rate of task execution is significantly improved.

[0053] 2) Comprehensively consider the computational complexity, data volume and delay tolerance of the tasks to be executed, so as to optimize the task scheduling order, further reduce the task execution time and improve the overall performance of the system.

[0054] 3) It overcomes the limitation of traditional deep reinforcement learning algorithms that rely on single-threaded architecture, can efficiently utilize computing resources, avoid experience coupling problems, and significantly improve training efficiency.

[0055] By making full use of the computing resources of the service vehicle (SV) and the edge server (RSU) in the vehicle-mounted edge computing system, the allocation and scheduling of tasks are optimized. This method adopts the Actor-Critic algorithm in deep reinforcement learning to dynamically adjust the partial offloading strategy of the task according to the vehicle status and task priority, that is, the computing task is split into two subtasks, one of which is left in the local vehicle for execution, and the other is offloaded to the service vehicle or edge server for execution. Through reasonable task splitting and offloading decisions, the execution delay of the task can be minimized while ensuring the success rate of task execution. The core steps of this method include task information uploading, priority division, task splitting, scheduling offloading, and obtaining the optimal scheduling strategy based on deep reinforcement learning. The task status is managed by establishing a waiting queue, a processing queue, and a failure queue, and the improved Seq2Seq model and IndRNN combined with the attention mechanism are used to accurately predict and dynamically schedule the execution location of the task, further improving the parallel processing capability of the system. Through the deep reinforcement learning method, the task scheduling can be dynamically adjusted according to the real-time status of the vehicle and the server, optimizing resource allocation and reducing latency. Through a reasonable task splitting strategy, the computing task is refined into multiple subtasks and executed in parallel, enhancing the parallel capability of the system and effectively improving the execution efficiency. Through the Actor-Critic algorithm, the task allocation strategy is continuously optimized to ensure the success rate of task execution and improve the overall performance of the system.

[0056] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the splittable computing task offloading system architecture adopted in the embodiment of the present application.

[0058] Figure 2 It is a structural diagram of the hierarchical model of task priorities in an embodiment of the present application.

[0059] Figure 3 It is a structural diagram of the task splitting model in the embodiment of the present application.

[0060] Figure 4 It is a structural diagram of task offloading in an embodiment of the present application.

[0061] Figure 5 It is a structural diagram of the improved Seq2Seq model in the embodiment of the present application.

[0062] Figure 6 It is a structural diagram of the Actor-Critic algorithm in the embodiment of the present application.

[0063] Figure 7 It is a block diagram of an electronic device for task offloading in a vehicle edge computing environment based on deep reinforcement learning in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0065] Optionally, the present application takes the task offloading method based on deep reinforcement learning in a vehicle-mounted edge computing environment provided in each embodiment as an example of being used in an electronic device, where the electronic device is a terminal or a server. The terminal may be a computer, a tablet computer, etc. This embodiment does not limit the type of electronic device.

[0066] Reference Figure 1 , is a flowchart of a task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning provided by an embodiment of the present application. The method includes at least the following steps:

[0067] Step S101: When a task vehicle generates a task to be executed, it uploads the task information to the RSU, which generates a scheduling decision based on the current status of the task and available resources, and assigns the task to a suitable service vehicle or executes it locally.

[0068] Step S102: After the RSU generates a scheduling decision, it uses the hierarchical analysis method to prioritize all tasks to be executed, and obtains the current status of each task to be executed in combination with the computational complexity, task data volume, and delay tolerance.

[0069] Step S103: Based on the task status and priority, the computing model is used to schedule the task, obtain the service vehicle number and the task vehicle number for each task, and obtain the current computing offloading and scheduling strategy. Part of the task is left to be executed locally and added to the waiting queue of the task vehicle, and the other part is offloaded to the service vehicle and added to the waiting queue of the task vehicle.

[0070] Step S104: After the task is partially unloaded, based on the previous scheduling strategy, the execution status of the task is monitored in real time, and the available computing resources of the current service vehicle are obtained.

[0071] Step S105: Based on the task status and the available computing resources of the service vehicle, the improved sequence-to-sequence model is trained using the Actor-Critic algorithm to obtain the optimal task partial offloading and scheduling strategy.

[0072] In step S101, when a task vehicle (TV) generates a task to be executed, the task information is first uploaded to the roadside unit (RSU) through wireless communication technology. As the central node of task scheduling, RSU is responsible for receiving the task information uploaded by the task vehicle, analyzing it according to the task requirements and the current status of the system, and then generating the corresponding task scheduling decision.

[0073] Reference Figure 1 , is a schematic diagram of the splittable computing task offloading system architecture used in the embodiment of the present application. Assume that there are many vehicles on a one-way straight road, represented by the set V = {1,2,3,...,v}, and all vehicles are evenly distributed on the road and travel at a constant speed within a certain period of time. An RSU equipped with a VEC server is deployed on one side of the road, and all vehicles on the road are always within the communication range of this RSU. Wireless communication can be carried out between vehicles, and wireless communication can also be carried out between vehicles and RSUs. The tasks generated by TV can be offloaded to SV for processing through V2V communication technology. After the task is executed, the SV returns the result directly to TV through single-hop transmission.

[0074] In the implementation, the vehicle that generates the task within the current RSU communication range is called the task vehicle (TV). The vehicles around the task vehicle that have remaining computing resources and can provide services are called current service vehicles (SVs). Indicates that vehicles that have no remaining computing resources around the task vehicle and cannot provide services and vehicles outside the current RSU communication range are collectively referred to as other vehicles. When TV generates a task that needs to be unloaded, it first sends a task unloading request to RSU, and then RSU sends the task unloading decision to TV and SV respectively, and then TV unloads the task to be unloaded to a specific SV, and waits for the task to be completed before SV returns the result to TV. Among them, TV = {f l ,P l s}, f l is the computing power of the TV, which refers to the CPU cycles per second, P l s is the transmission power of TV to send task data to SV. Assume that there are N SVs around TV during task offloading, SV={SV1,SV2,...,SV N},in f j is the computing power of the jth SV, also referred to as CPU cycles per second, is the transmission power of the jth SV. Each vehicle travels at a constant speed v, and the speeds of different vehicles follow a uniform distribution. The distance between TV and SV is used to evaluate the channel state, and the relative position and relative speed of TV and SV are used to evaluate the link duration.

[0075] In addition, as a preferred embodiment, the total system time is divided into several time periods, and TV generates M computing tasks in a single time period, and these M tasks are all splittable computing tasks. The i-th task is defined as a four-tuple Ω i ={C i ,D i ,κ i ,τ i}, i∈{1,2,...,M}, where C i Indicates completion i Required CPU cycles, D i Represents Ω i The size of the input data, κ i Represents Ω i The computational complexity of τ i Represents Ω i Assuming there is no dependency between all tasks, SV can process multiple tasks in parallel.

[0076] In step S102, when the task vehicle unloads the computing task to the service vehicle with idle computing resources, the different computing task attributes lead to different task types and different urgency levels. The computing task types generated by the vehicle include emergency security tasks, general real-time tasks and other task types. This application uses priority to indicate the urgency of the task. When computing resources are limited, the higher the priority of the task and the more urgent the task, the priority is given to it. The analytic hierarchy process is a multi-criteria decision-making method for qualitative and quantitative analysis. This method divides the decision-making-related factors into three levels: the target level, the criterion level and the scheme level, thereby forming a multi-level structural model, which is very suitable for dealing with the weight allocation of task attributes and the division of task priorities.

[0077] In implementation, for the attributes of the task, this application mainly considers the following three influencing factors: computational complexity, task data volume, and delay tolerance, with the order of importance being: computational complexity, task data volume, and delay tolerance. The hierarchical structure model of task priority is as follows: Figure 2 As shown, the present application uses the AHP method to assign higher weights to tasks with high computational complexity, large data volume and low delay tolerance, thereby ensuring that under the constraint of delay, the system can more reasonably allocate limited computing resources and improve the success rate of task offloading.

[0078] Specifically, the steps of the AHP algorithm are as follows:

[0079] Step S1021: First, compare the three influencing factors of the criterion layer, and construct an evaluation index judgment matrix (Pairwise Comparison Matrix, PCM) from the target layer to the criterion layer as A = (a lr ) 3×3 , l,r∈{1,2,3}, where:

[0080]

[0081] Then, the M tasks at the solution level are compared to construct the judgment matrix B from the three influencing factors at the criterion level to the solution level. k =(b ij ) M×M , i,j∈{1,2,...,M}, k∈{1,2,3}.

[0082]

[0083] It should be noted that in the above formula (1) and formula (2), the value of q is obtained according to the following Table 1:

[0084] Table 1 PCM quantization values

[0085]

[0086] Step 1022: Then, the weight β of the lth influencing factor in A is obtained by the sum-product method. l for:

[0087]

[0088] The weight vectors of the three influencing factors of A can be expressed as β = [β 1 ,β 2 ,β 3 ] T .

[0089] Step 1023: Use the sum-product method again to obtain B k The weight of the kth influencing factor of the i-th task for:

[0090]

[0091] Among them B k The weight vector of the M tasks can be expressed as B 1 ,B 2 ,B 3The weight vectors of form the criterion layer, and the weight matrix for all tasks in the solution layer is:

[0092]

[0093] Step 1024: After checking the consistency of the weights, the weight vector S of all tasks is finally obtained as shown in formula (6), where the elements in S are composed of the weights of the corresponding computing tasks:

[0094]

[0095] In step S103, when the scheduling decision generated by the RSU determines that the task needs to be partially unloaded, the task vehicle (TV) will split the computing task into two subtasks according to the allocation result of the RSU: one subtask is executed locally in the task vehicle, and the other subtask is unloaded to a nearby service vehicle (SV) with available computing resources. Only when each subtask is successfully executed can the entire computing task be considered to be successfully executed. Assuming that there is no dependency between all computing tasks, each subtask can only select one SV for unloading when unloading, and each SV can process multiple subtasks in parallel.

[0096] Specifically, the task partition graph consists of TV, SV set and task set {Task1, Task2, …, Task M}, all tasks are splittable computing tasks, such as Figure 3 As shown. To simplify the analysis, it is assumed that tasks are bit-independent, that is, there is complete granularity in the task data partition, so tasks can be divided into subsets of any size. Let task Ω i The task split ratio is φ i (i∈M,φ i ∈[0,1]),φ i Represents the task Ω i The proportion of tasks executed locally on TV to the total number of tasks, and the remaining 1-φ i It indicates the ratio of tasks unloaded to the service vehicle to the total number of tasks. Since the tasks are bit-independent, the subtasks do not need to be executed in a specific order.

[0097] In step S104, during the task partial unloading process, the execution delay of the task directly affects the overall task scheduling and resource allocation strategy. The task partial unloading process between the task vehicle (TV) and the service vehicle (SV) includes data transmission and processing. The system needs to evaluate the total delay under different task execution modes according to the delay model to ensure that the execution delay of the task does not exceed the set delay tolerance.

[0098] Specifically, when the task Ω i When the execution mode is local execution, task Ωi The computational delay for

[0099]

[0100] The computational latency of a task when it is left to execute locally cannot exceed its latency tolerance, so the restrictions are as follows:

[0101]

[0102] When the task i When the execution mode is partial unloading, the task Ω i φ i Part of it is executed locally on TV, and its computational delay is

[0103]

[0104] Assume that the task Ω i 1-φ i Part of the task is offloaded to the SV for execution. The offloading process is as follows: TV uploads the task data to SV, SV executes the task and transmits the result back to TV. The total time of task offloading is divided into four parts: waiting time before task offloading, data uploading time, task processing time in SV and result transmission time. Before task offloading, the task information and SV status to be offloaded need to be uploaded to RSU through wireless network, and then RSU assigns SV to the task. l represents the link transmission rate when offloading the task to the jth SV, R j It represents the link transmission rate when the jth SV transmits the result back to the TV.

[0105]

[0106] In the above equations (10)-(11), B represents the vehicle transmission channel bandwidth, d j represents the distance between TV and the jth SV, δ represents the path loss exponent, and h j represents the complex Gaussian channel coefficient, N0 represents additive Gaussian white noise, and I represents the interference caused by other vehicles during V2V transmission. Then the transmission time for uploading the mission data to the SV is as follows:

[0107]

[0108] The execution time of the task on the jth SV is as follows:

[0109]

[0110] The result return time is as follows:

[0111]

[0112] In the above formula (14), η represents the ratio between the size of the output data volume and the size of the input data volume. Usually η <= 1, which can be ignored when calculating the task time delay. The total delay of task offloading is as follows:

[0113]

[0114] In the above formula (15), Represents the task Ω i Waiting time before uninstallation. Due to task Ω i φ i Partial sum 1-φ i Part of it is parallel processing, so Ω i The total delay of task Ω is the larger of the two. i The total execution delay t i as follows:

[0115]

[0116] The total delay of each task when unloading cannot exceed its delay tolerance, so:

[0117] t i ≤τ i (17)

[0118] By calculating the above delay model, an appropriate local processing or partial offloading strategy is selected for the task, thereby optimizing the task execution time and resource allocation.

[0119] In step S105, the scheduling decision of task offloading is optimized by introducing a deep reinforcement learning algorithm. In order to improve the success rate of task execution and reduce the average execution delay of tasks, a queue-based scheduling model and an improved Seq2Seq network model are designed, and the scheduling strategy is iteratively optimized using the Actor-Critic algorithm.

[0120] Specifically, in order to manage and track the execution status of tasks, three queues are designed in this step:

[0121] (1) Failure Queue, which records tasks that have failed to be allocated or have timed out;

[0122] (2) Processing Queue, which records the tasks being executed;

[0123] (3) Waiting Queue, which records tasks that have not yet been scheduled.

[0124] In implementation, the task scheduling process is shown in Algorithm 1 in Listing 2 below. Find the task Ω with the highest priority in the Waiting Queue and schedule it. First, assign SV or TV to task Ω to determine whether task Ω is offloaded or executed locally. Then compare the initial computing resources of the SV or TV assigned to task Ω with the computing resources required by task Ω. If the initial computing resources of the SV or TV are less than the computing resources required by task Ω, the task allocation fails, and it is necessary to reassign SV or TV to task Ω. If the task allocation is successful, determine whether the current available computing resources of the SV or TV are sufficient to calculate the task. If so, add the task to the Processing Queue of the current SV or TV, and delete task Ω from the Waiting Queue. Otherwise, task Ω waits. After the task with the shortest remaining execution time in the Processing Queue of the current SV or TV is completed and the computing resources it occupies are released, the remaining execution time and total execution time of task Ω and all tasks in the Processing Queue are immediately updated. After the update is completed, determine whether the task Ω has timed out. If so, the task Ω fails to execute and is added to the Fail Queue. If not, recursively call Algorithm 1 to reschedule the task Ω.

[0125] Table 2 Task scheduling pseudo code

[0126]

[0127] This embodiment aims to improve the success rate of task execution as much as possible while reducing the average delay of task execution. Therefore, the execution success rate of the calculation task is defined as follows:

[0128]

[0129] In the above formula (18), q represents the number of tasks that have timed out, that is, the number of tasks in the Fail Queue, and M is the total number of tasks. The average delay of all tasks is as follows:

[0130]

[0131] In practice, the deep reinforcement learning method can be iteratively improved through continuous interaction with the environment, gradually approaching the optimal solution. The Critic network in the Actor-Critic algorithm provides a baseline estimate of the policy, which can reduce the variance of the policy gradient, making the training process more stable. The Critic network estimates the long-term return of each state through the value function, which helps the intelligent agent to better understand the environment and improve the strategy. Therefore, this embodiment proposes a computing task offloading scheduling method based on deep reinforcement learning, and trains the neural network through the Actor-Critic algorithm.

[0132] Specifically, the task offloading diagram is as follows: Figure 4 As shown, assume that TV generates three computing tasks, namely Task1, Task2 and Task3. The computing tasks generated by TV are split into six subtasks, Subtask1, Subtask2...Subtask6, to obtain a new task sequence. TV is numbered 0, the number of SVs is N=3, and the SVs are numbered 1, 2, and 3 in sequence. One of the execution decisions of these three computing tasks can be expressed as {0, 2, 1, 3, 2, 1}, where 0 means that Subtask1 is executed locally on TV numbered 0, 2 means that Subtask2 is offloaded to SV numbered 2 for execution, and 1 means that Subtask3 is offloaded to SV numbered 1 for execution, and the rest are similar.

[0133] Among them, the sequence-to-sequence model is a type of encoder-decoder structure, and its basic structure consists of a recurrent neural network. The problem of assigning tasks to service vehicles can be transformed into: the problem of mapping the task and service vehicle sequence into a service vehicle number sequence with the same length as the task sequence, which is suitable for Seq2Seq to solve. However, the commonly used RNN structure in Seq2Seq, such as GRU, uses a hyperbolic tangent activation function, which will cause the gradient of the network layer to decay, resulting in the problem of gradient disappearance and gradient explosion. Unlike RNN, a parameter constraint mechanism is introduced in IndRNN, which makes the gradient more stable during the back propagation process, thereby effectively solving the problem of gradient disappearance and gradient explosion, and IndRNN has better robustness after training. Therefore, this application replaces the RNN in Seq2Seq with IndRNN. In addition, the Seq2Seq model needs to compress all the information of the input sequence, so it is difficult to process longer input sequences. In this regard, the present application adds an attention mechanism. When the Encoder obtains an output, it searches for the part of the input sequence that is most correlated with this output. The model then predicts the unloading vehicle number of the next task based on the context vector corresponding to this part of the input sequence and all previously generated service vehicle numbers. Compared with the original Seq2Seq model, the improved model focuses on the part of the input sequence that is most relevant to the current task when decoding, and can better handle long input sequences.

[0134] The execution process of the improved Seq2Seq model used in this application at the tth time step is as follows Figure 5 As shown in the figure, the task information and service vehicle information are used as the input sequence of the model, and the vehicle number that executes each task is used as the output sequence of the model. The input task sequence is a divisible computing task, which needs to be split first and then its subtasks are scheduled. Therefore, a set of split weights φ is defined in the encoder. After the task information is input into the encoder, the split weights are used to split the divisible computing tasks. The split weights are continuously iteratively optimized. The input of the model is the information of the task sequence X, SV and TV to be scheduled obtained by AHP calculation. After splitting the divisible computing tasks in the task sequence by the split weight φ, a new task sequence X' is obtained. X' consists of 2m subtasks obtained by splitting the divisible computing tasks. The new task sequence X' is used as the input of the encoder. The input of the decoder is the context vector c t and the hidden layer output s at the t-1th time step t-1 , the output is task x t Corresponding service vehicle number y t , thereby obtaining the vehicle number for executing the task, and finally obtaining the coding sequence of all task execution decisions through Seq2Seq.

[0135] At the tth time step in the decoding phase, the decoder is given a context vector c t and all previous predictions {y1,...,y t-1}, predict the next result y t , then the output is {y1,...,y t The probability of} is as follows:

[0136]

[0137] In the above formula (20), each conditional probability is modeled as follows:

[0138]

[0139] Then the encoder's hidden layer vector H and the decoder's hidden layer output s at the t-1th time step t-1 Input into Attention to get the attention value Right now:

[0140]

[0141] In the above formula (22), s0 is the initial input of Attention at time step t = 0, which is obtained by linear transformation of the last hidden layer. is an intermediate variable, W1 and v1 are trainable parameters, and attn is a fully connected neural network. t , get the context vector c t as follows:

[0142] c t =e t H (23)

[0143] Then c t ,s t-1 and the decoder output y at the t-1th time step t-1 Input into the decoder, and after the tanh activation function, the hidden state s at the tth time step is obtained t The conditional probability required by formula () is as follows:

[0144] s t =tanh(W2[y t-1 ,c t ,s t-1 ]) (twenty four)

[0145]

[0146] In the above equations (24)-(25), W2, v2 are trainable parameters.

[0147] Specifically, the Actor-Critic network training is as follows Figure 6 As shown. Among them, the Actor network is the improved Seq2Seq model mentioned in the previous section. State represents the state, including the task state and the service vehicle state. The task state consists of five parts: the execution time of the task on the CPU, the computational complexity, the amount of data, the delay tolerance and the task priority. The service vehicle state is the computing resources available to the vehicle. Action represents the action, that is, the server number of the task to be offloaded generated by the Actor network. The set of actions is the offloading strategy. Reward represents the corresponding reward for executing the action. The training process is: input the state into the Actor network, generate the corresponding action, and then interact with the environment through Algorithm 1 to obtain the reward. The state and the reward after executing the action are used as the input of the Critic network, and then the Actor network and the Critic network are updated through the loss function. The parameter of the Actor is defined as θ, and reward(a) is the reward generated by the Actor when the strategy a is adopted, which is obtained by formula (18). The function J(θ) is defined as follows:

[0148] J(θ)=E(reward(a)) (26)

[0149] The optimization goal of Actor is to maximize the reward value. This application uses the Reinforce algorithm to train the Actor network, which is expressed as follows:

[0150]

[0151] In the above formula (27), is the baseline function obtained by the Critic network, P θ (a|s) is the conditional probability of adopting strategy a in state s obtained by the Actor network, which is determined by formula (21). Assuming the batch size in the training data set is K, the update formula of parameter θ can be expressed as follows:

[0152]

[0153] Then the mean square error loss function is used to adjust the parameters of the Critic To update:

[0154]

[0155] In the above formula (29), is the reward value predicted by the Critic network, reward(a k |s k) is the true value. The training process of Actor-Critic is shown in Table 3 below, where the learning rate is 0.0001, the batch size is 32, the optimizer uses Adam, T is the total number of epochs, and K is the number of times the state sequence is updated. The Critic network structure consists of an RNN layer, a convolutional layer, and a fully connected network to calculate the predicted reward value of the input state.

[0156] Table 3 Actor-Critic network training pseudo code

[0157]

[0158] In summary, the present application optimizes the allocation and scheduling of tasks by making full use of the computing resources of the service vehicle (SV) and the edge server (RSU) in the vehicle-mounted edge computing system. The method adopts the Actor-Critic algorithm in deep reinforcement learning to dynamically adjust the partial offloading strategy of the task according to the vehicle status and task priority, that is, to split the computing task into two subtasks, one part of which is left in the local vehicle for execution, and the other part is offloaded to the service vehicle or edge server for execution. Through reasonable task splitting and offloading decisions, the execution delay of the task can be minimized while ensuring the success rate of task execution. The core steps of the method include task information uploading, priority division, task splitting, scheduling offloading, and obtaining the optimal scheduling strategy based on deep reinforcement learning. The task status is managed by establishing a waiting queue, a processing queue, and a failure queue, and the improved Seq2Seq model and IndRNN are combined with the attention mechanism to accurately predict and dynamically schedule the execution position of the task, further improving the parallel processing capability of the system. Through the deep reinforcement learning method, the task scheduling can be dynamically adjusted according to the real-time status of the vehicle and the server, resource allocation can be optimized, and latency can be reduced. Through a reasonable task splitting strategy, the computing task is refined into multiple subtasks and executed in parallel, which enhances the parallel capability of the system and effectively improves the execution efficiency. Through the Actor-Critic algorithm, the task allocation strategy is continuously optimized to ensure the success rate of task execution and improve the overall performance of the system.

[0159] Figure 7 4 is a block diagram of an electronic device provided by an embodiment of the present application. The device at least includes a processor 401 and a memory 402.

[0160] The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0161] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, which is used to be executed by the processor 401 to implement the task offloading method in the vehicle-mounted edge computing environment based on deep reinforcement learning provided in the method embodiment of the present application.

[0162] In some embodiments, the electronic device may further optionally include: a peripheral device interface and at least one peripheral device. The processor 401, the memory 402 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0163] Of course, the electronic device may also include fewer or more components, which is not limited in this embodiment.

[0164] Optionally, the present application also provides a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning of the above-mentioned method embodiment.

[0165] Optionally, the present application also provides a computer product, which includes a computer-readable storage medium, in which a program is stored, and the program is loaded and executed by a processor to implement the task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning of the above-mentioned method embodiment.

[0166] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0167] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning, characterized in that: The method comprises: Obtaining pending tasks generated by task vehicles, uploading the pending tasks to roadside units, and having the roadside units generate scheduling decisions based on the current status of the pending tasks and available resources, and assigning the pending tasks to service vehicles or local execution; Prioritize all the tasks to be executed by using the hierarchical analysis method, and obtain the current status of each task to be executed by combining the computational complexity, task data volume, and delay tolerance; Based on the task status and the priority, the computing model is used to schedule the tasks, and the service vehicle number, task vehicle number, computing offloading and scheduling strategy for executing each task are obtained; a part of the task is left to be executed locally and added to the waiting queue of the task vehicle, and another part is offloaded to the service vehicle and added to the waiting queue of the task vehicle; After the task is partially unloaded, the execution status of the task is monitored in real time based on the previous scheduling strategy, and the available computing resources of the current service vehicle are obtained; Based on the task status and the available computing resources of the service vehicle, an improved sequence-to-sequence model is trained using the Actor-Critic algorithm to obtain the optimal task partial offloading and scheduling strategy.

2. The task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning according to claim 1 is characterized in that: The acquiring of the pending tasks generated by the task vehicle, uploading the pending tasks to the roadside unit, the roadside unit generating a scheduling decision according to the current state of the pending tasks and available resources, and allocating the pending tasks to the service vehicle or local execution includes: When a task vehicle generates a task to be performed, the task vehicle uploads the task information to the roadside unit through wireless communication technology, and the task information includes the computational complexity, input data volume and delay tolerance of the task; The roadside unit generates a task offloading decision based on the task information and the current computing resource status of the system, and assigns the task to the service vehicle or has the task vehicle execute it locally; The task vehicle is a vehicle with a computing task within the communication range of the roadside unit, and the service vehicle is a vehicle with remaining computing resources around the task vehicle and can provide services. After the task vehicle sends a task offloading request to the roadside unit, the roadside unit evaluates the link duration based on the distance, channel status and relative speed between the task vehicle and the service vehicle, and decides whether to offload the task to the service vehicle; Decompose the task into multiple subtasks, and offload each subtask according to the task's computational complexity, input data volume, and delay tolerance. Some tasks are executed locally by the task vehicle, and some tasks are processed in parallel by the service vehicle. After the service vehicle completes the task, the results are transmitted to the task vehicle through wireless communication technology, and the roadside unit dynamically adjusts the task unloading strategy according to the task execution status.

3. The task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning according to claim 1 is characterized in that: The method of using the hierarchical analysis method to prioritize all the tasks to be executed, and combining the computational complexity, task data volume, and delay tolerance to obtain the current status of each task to be executed includes: The factors related to task offloading decision are divided into target layer, criterion layer and scheme layer, and a priority hierarchy model is constructed, in which the target layer is the priority processing order of tasks, the criterion layer includes computational complexity, task data volume and delay tolerance, and the scheme layer is all pending tasks. The three influencing factors in the criterion layer are compared in pairs, and an evaluation index judgment matrix from the target layer to the criterion layer is constructed. The weight of each influencing factor is calculated by the sum-product method. The weight is determined according to the urgency of the task, and the importance of calculation complexity, task data volume and delay tolerance decreases in turn; Construct a task judgment matrix from the criterion layer to the solution layer, calculate the weight of each task under different influencing factors, and obtain the weight vector of each task under each influencing factor through the sum-product method; The weight vector is checked for consistency to obtain the final task weight matrix. The priority of the task is determined by the task weight matrix. When computing resources are limited, tasks with higher weights are processed first.

4. The task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning according to claim 3 is characterized in that: The construction of the evaluation index judgment matrix from the target layer to the criterion layer and the construction of the task judgment matrix from the criterion layer to the solution layer include: Constructing the evaluation index judgment matrix from the target layer to the criterion layer is A = (a lr ) 3×3 , l,r∈{1,2,3}, where: Compare the M tasks at the solution level and construct the judgment matrix B from the three influencing factors at the criterion level to the solution level. k =(b ij ) M×M , i,j∈{1,2,…,M}, k∈{1,2,3}; The value of q is obtained according to the existing PCM quantization value table.

5. The task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning according to claim 1 is characterized in that: After the task is partially unloaded, the execution status of the task is monitored in real time based on the previous scheduling strategy, and the available computing resources of the current service vehicle are obtained, including: Evaluate the task transmission and processing delay between the task vehicle and the service vehicle, and determine whether the total execution delay of the task meets the set delay tolerance; When the task is executed locally in the task vehicle, the local execution delay of the task is calculated and compared with the delay tolerance of the task; When a task needs to be partially offloaded, the task is split into two subtasks, one of which is executed locally on the task vehicle and the other is offloaded to the service vehicle for execution; The task offloading process includes the waiting time before task offloading, the time for uploading task data to the service vehicle, the processing time of the task on the service vehicle, and the time for transmitting the task result back to the task vehicle.

6. The task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning according to claim 1 is characterized in that: The improved sequence-to-sequence model is trained using the Actor-Critic algorithm based on the task status and the available computing resources of the service vehicle to obtain the optimal task partial offloading and scheduling strategy, including: Build multiple task queues to manage the execution status of tasks; The queues include a failure queue, a processing queue and a waiting queue; the task with the highest priority is selected from the waiting queue for scheduling, and a suitable service vehicle or task vehicle is allocated to it; During the allocation process, the required computing resources are compared with the currently available computing resources to determine whether the task is successfully allocated; If successful, the task is added to the processing queue; If it fails, reassign the service vehicle or task vehicle to the task; During the execution process, the execution status of the task is monitored in real time, the remaining execution time and total execution time of the task are updated, and the timed tasks are processed and added to the failure queue.

7. The task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning according to claim 1 is characterized in that: The improved sequence-to-sequence model is trained using the Actor-Critic algorithm based on the task status and the available computing resources of the service vehicle to obtain the optimal task partial offloading and scheduling strategy, including: An improved sequence-to-sequence model is trained using an Actor-Critic algorithm, wherein the sequence-to-sequence model receives a task state and a service vehicle state as input and outputs a service vehicle number that performs each task; By interacting with the environment, the Actor and Crit ic network is continuously updated to iteratively optimize the task offloading strategy and improve the success rate of task execution; The Actor network is trained using an improved Seq2Seq model and introduces an attention mechanism to focus on the part of the input sequence that is most relevant to the current task to process longer input sequences.

8. The task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning according to claim 6 is characterized in that: The task scheduling process is optimized through the defined reward function, which specifically includes: After executing each scheduling action, the corresponding reward value is calculated based on the execution result and status of the task; Input state information and actions into the Critic network, and use the Critic network to estimate the long-term return of each state; Through iterative optimization during the training process, the parameters of the Actor and Critic networks are continuously adjusted.

9. An electronic device, characterized in that: The device includes a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement a task offloading method in a vehicle edge computing environment based on deep reinforcement learning as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, which, when executed by the processor, is used to implement a task offloading method in a vehicle-mounted edge computing environment based on deep reinforcement learning as described in any one of claims 1 to 8.

Images