Vehicle edge cooperative computing task unloading method and device based on deep reinforcement learning

Through the vehicle-side collaborative computing task unloading method based on deep reinforcement learning, the vehicle-vehicle and vehicle-server collaboration model is used to solve the problem of insufficient computing resources of the vehicle-mounted equipment, and the task is processed at low latency and high success rate, which improves the Calculate the efficiency and resource utilization of the offload system.

CN119052273AActive Publication Date: 2024-11-29SHANGHAI UNIV OF ENG SCI

Patent Information

Application Number
CN202411010835.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-29
Estimated Expiration
2044-07-26

AI Technical Summary

Technical Problem

When handling computing-intensive tasks, the time delay is increased due to the limitation of computing resources. The remote location of the cloud computing center leads to the task transmission time too long, affecting vehicle security and privacy. Vehicle-intensive areas serve too much vehicle load or resources When there is insufficient, the task cannot be processed in time.

Method used

Using the vehicle-side collaborative computing task unloading method based on deep reinforcement learning, through vehicle-vehicle collaboration and vehicle-server collaborative computing model, the IndRNN-AM sequence-to-sequence model and advantageous action value algorithm are used to formulate task unload and scheduling strategies to realize tasks local execution, overall uninstall or partial uninstallation, optimize computing resource utilization, and ensure low latency and high success rate.

Benefits of technology

Effectively utilize the computing resources serving vehicles and edge servers to meet the needs of different types of computing tasks for low latency and high success rates, improve the stability and efficiency of the computing and offloading system, shorten the task execution time, reduce task offloading costs, and increase system resources Utilization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119052273A_ABST
    Figure CN119052273A_ABST
Patent Text Reader

Abstract

The invention relates to a vehicle edge cooperative computing task unloading method and device based on deep reinforcement learning. The method comprises the steps of obtaining a to-be-executed task sequence, an initial service vehicle sequence and an initial server sequence; dividing task priorities of the to-be-executed task sequence, and obtaining a current task sequence in combination with attributes of the to-be-executed task sequence; based on the current task sequence, the service vehicle sequence and the server sequence, task scheduling is performed by using a pre-constructed vehicle-vehicle cooperative calculation model and a vehicle-server cooperative calculation model, and a task unloading strategy is acquired by using an IndRNN-AM-based sequence-to-sequence model; and obtaining a current task sequence, a service vehicle sequence and a server sequence based on the current task unloading and scheduling strategy, and training a sequence-to-sequence model based on IndRNN-AM by using a dominant action value algorithm to obtain a final task unloading and scheduling strategy. Compared with the prior art, the method has the advantages of improving the stability and high efficiency of the whole computing unloading system and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of vehicle networking technology, and in particular to a method and device for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning. Background Art

[0002] The development of technologies such as artificial intelligence and big data analysis has made in-vehicle Internet more practical, but due to the limitation of massive data and transmission channels, such as road condition monitoring, map navigation and route planning, it has brought huge pressure and high latency to the crowded network. The widespread application of these computing tasks has brought a substantial increase in data volume, resulting in vehicles with limited computing resources being unable to handle such a large number of computing-intensive tasks, thus affecting the quality of user service.

[0003] Currently, cloud computing (CC) can be used to offload computing tasks to the central cloud for processing. Although cloud computing infrastructure has existed for many years and can be used to process a large number of computing-intensive tasks, cloud computing centers are placed in relatively remote locations, far away from vehicles, which may cause problems such as long task transmission time and link congestion, which in turn affects the execution time of tasks and cannot guarantee vehicle safety and privacy. To this end, vehicular edge computing (VEC) has been introduced, which can sink computing and storage capabilities to servers and vehicles at the edge of the network. Its low latency, high bandwidth and real-time computing capabilities can better meet the needs of edge devices and end users, and provide computing and storage services for vehicles with limited computing resources at the edge of the network, greatly reducing the processing delay of computing tasks.

[0004] Since the computing resources of vehicle-mounted equipment are limited, when the computing tasks generated by them cannot be processed by themselves, these tasks can be transferred to nearby vehicles with idle resources for processing. However, due to the high-speed mobility of vehicles themselves, when there are too many vehicles in the same section, vehicles will gather, and more computing tasks with offloading requirements will be generated in densely populated areas, resulting in excessive load on the service vehicles, which cannot meet the latency requirements of the tasks; and when there are too few service vehicles with idle computing resources in the same section, the tasks cannot be processed in time. In this regard, the collaborative computing offloading between the vehicle side and the server side in the vehicular edge network (VEN) is introduced, so that when the task vehicle is in a densely populated area or there are too few service vehicles, the task can be offloaded to the VEN server with richer computing resources within the communication range of the current roadside infrastructure (Road Side Unit, RSU). Through this vehicle-side collaborative offloading method, not only can the low latency requirements of latency-constrained computing tasks be guaranteed, but also the tasks can be processed in time when computing resources are tight. However, how to make full use of the computing resources of service vehicles and VEN servers to meet the requirements of low latency and high success rate for the two types of computing tasks requires the formulation of a reasonable and efficient offloading strategy. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a vehicle-side collaborative computing task unloading method and device based on deep reinforcement learning, which can formulate reasonable and efficient unloading and scheduling strategies, and make full use of the computing resources of service vehicles and servers to meet the requirements of different types of computing tasks for low latency and high success rate.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, a method for vehicle-side collaborative computing task unloading based on deep reinforcement learning is provided, comprising the following steps: S1, obtaining a sequence of tasks to be executed, an initial service vehicle sequence and an initial server sequence; S2, dividing the task priorities of the sequence of tasks to be executed, and obtaining the current task sequence in combination with the attributes of the sequence of tasks to be executed, wherein the attributes of the task sequence include CPU execution time, computational complexity, data volume, delay tolerance, whether the task can be split and task priority; S3, based on the current task sequence, service vehicle sequence and server sequence, using a pre-built vehicle-to-vehicle collaborative computing model and a vehicle-to-server collaborative computing model to schedule tasks, and using a sequence-to-sequence model based on IndRNN-AM to obtain a corresponding task unloading strategy; S4, based on the current task unloading and scheduling strategy, obtaining the current task sequence, service vehicle sequence and server sequence, and using the dominant action value algorithm to train the sequence-to-sequence model based on IndRNN-AM to obtain the final task unloading and scheduling strategy.

[0008] As a preferred technical solution, the scheduling mode of each task includes local execution, overall offloading and partial offloading.

[0009] As a preferred technical solution, when the scheduling mode is partial unloading, based on the vehicle-to-vehicle cooperative calculation model, the corresponding total task execution delay is:

[0010]

[0011] and

[0012]

[0013] In the formula, t i is the total execution delay of the i-th task, is the φ of the i-th task i The latency of some tasks being performed locally in the mission vehicle, is 1-φ of the i-th task i The total delay from partial unloading to service vehicle execution, C i The CPU execution time required to complete the i-th task, f l The computing power of the mission vehicle, represents the waiting time before the i-th task is unloaded, represents the 1-φ of the i-th task i Transmission time for partial upload to the service vehicle, represents the execution time of the subtask of the ith task on the service vehicle, Indicates the result return time, D i is the size of the input data, f jis the computing capacity of the jth service vehicle, η represents the ratio of the output data volume to the input data volume, R l is the link transmission rate when offloading the task to the service vehicle, R j The link transmission rate when the service vehicle transmits the results back to the TV.

[0014] As a preferred technical solution, in the process of task scheduling, a failure queue, a processing queue, a waiting queue and a task sequence are used to jointly record different task states.

[0015] As a preferred technical solution, a pre-built vehicle-to-vehicle collaborative computing model and a vehicle-to-server collaborative computing model are used for task scheduling. The specific process includes: selecting the highest priority task from the task sequence; allocating a service vehicle, task vehicle or server to the highest priority task, and adding it to the waiting queue of the allocated service vehicle, task vehicle or server; when the initial computing resources of the allocated service vehicle, task vehicle or server are greater than or equal to the first computing resources required by the highest priority task, the task allocation is successful, and it is determined whether the current available computing resources of the allocated service vehicle, task vehicle or server are greater than or equal to the first computing resources: if yes, the allocated service vehicle, task vehicle or server is allocated to the waiting queue of the allocated service vehicle, task vehicle or server; The highest priority task is added to the processing queue of the current service vehicle, task vehicle or server and deleted from the waiting queue; if not, the highest priority task enters the waiting queue; when the task with the shortest remaining execution time in the processing queue of the current service vehicle, task vehicle or server has been executed and the corresponding computing resources have been released, the remaining execution time and total execution time of the highest priority task and all tasks in the processing queue are updated; after the update is completed, it is determined whether the highest priority task has timed out: if yes, the highest priority task is added to the failure queue; if not, the service vehicle, task vehicle or server is continued to be reallocated to the highest priority task.

[0016] As a preferred technical solution, a pre-built vehicle-to-vehicle collaborative computing model and a vehicle-to-server collaborative computing model are used for task scheduling. The specific process also includes: when the initial computing resources are less than the first computing resources, the task allocation fails, and the service vehicle, task vehicle or server is reallocated to the highest priority task; when the task with the shortest remaining execution time in the processing queue of the current service vehicle, task vehicle or server has not been completed and the corresponding computing resources have not been released, continue waiting.

[0017] As a preferred technical solution, the sequence-to-sequence model based on IndRNN-AM is used as the policy network of the dominant action value algorithm, the state space of the dominant action value algorithm includes task sequence, service vehicle sequence and server sequence, the action space of the dominant action value algorithm includes the number of the task vehicle, service vehicle or server that performs each task and is generated using the policy network, and the reward function of the dominant action value algorithm is the inverse of the sum of the average delay of task execution and the number of weighted failed tasks.

[0018] As a preferred technical solution, the average delay is:

[0019]

[0020] Where M is the total number of tasks, is the φ of the i-th task i The latency of some tasks being performed locally in the mission vehicle, is 1-φ of the i-th task i Total delay of partial offloading to the service vehicle or server execution, t i is the total execution delay of the i-th task.

[0021] As a preferred technical solution, the division of task priorities is achieved using hierarchical analysis method.

[0022] According to a second aspect of the present invention, there is provided a device for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning, comprising a memory, a processor, and a program stored in the memory, wherein the method described above is implemented when the processor executes the program.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. Aiming at the problem of offloading hybrid computing tasks in the vehicle-side collaborative scenario, the present invention proposes a vehicle-side collaborative computing task offloading method based on deep reinforcement learning. This method takes into account the collaborative computing offloading between the vehicle and the server when the computing resources of the service vehicle are insufficient. Delay computing models are designed for vehicle-to-vehicle collaboration and vehicle-to-server collaboration to perform task scheduling, and the corresponding task offloading strategy is obtained using a sequence-to-sequence model based on IndRNN-AM. On this basis, the model is trained using the Advantage Actor-Critic (A2C) algorithm to obtain a better task offloading and scheduling strategy. This method can make full use of the computing resources of the service vehicle and the edge server, meet the requirements of the two types of computing tasks for low latency and high success rate, and improve the stability and efficiency of the entire computing offloading system.

[0025] 2. In the method provided by the present invention, the scheduling modes of each task include local execution, overall unloading and partial unloading. When the computing resources of the vehicle are insufficient, the task is unloaded to the server for processing, and the server has stronger execution capability. When unloading computing tasks, for tasks that fail to be allocated, service vehicles, servers or local resources are also reallocated for them. Through this vehicle-side collaborative unloading mode, the low latency requirements of latency-constrained tasks can be guaranteed, and in the case of tight computing resources and an increase in the number of tasks, tasks can also be processed in a timely manner, thereby improving the execution success rate of tasks and shortening the execution time of tasks;

[0026] 3. The method provided by the present invention uses the sequence-to-sequence model based on IndRNN-AM as the policy network of the A2C algorithm, and then uses the A2C algorithm to train the policy network. It adopts a multi-threaded parallel architecture, breaking the single-threaded architecture of most deep reinforcement learning algorithms, and can efficiently utilize computing resources, break the coupling between experiences, and improve the efficiency of training;

[0027] 4. In the method provided by the present invention, the attributes of the task sequence include CPU execution time, computational complexity, data volume, delay tolerance, whether the task can be split, and task priority. The attribute of whether the task can be split means that all tasks to be executed can be divided into indivisible computing tasks and divisible computing tasks. Divisible computing tasks can be split into multiple subtasks to be executed. On this basis, according to the CPU execution time, computational complexity, data volume, and delay tolerance of the task, the hierarchical analysis method is used to prioritize all tasks to be executed, so that each task to be executed is independent and has no dependencies, and the unloading and scheduling strategies can be better obtained, thereby realizing the optimization of computing resources and maximization of the utility of computing task unloading;

[0028] 5. The method and device provided in this embodiment are also applicable to other IoT devices with edge computing requirements, have good application prospects, and can effectively reduce task offloading costs and improve the utilization of system resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the architecture of a vehicle-side collaborative computing offloading system provided in an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of one of the task offloading methods provided in an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of the strategy network model structure provided in an embodiment of the present invention;

[0032] Figure 4 The figure is a flowchart of the dominant action value algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0034] Example

[0035] This embodiment provides a method for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning. Figure 1 The vehicle-side collaborative computing offloading system architecture is shown in the figure. This method takes into account the collaborative computing offloading between the vehicle and the server when the computing resources of the service vehicle are insufficient, and divides the task offloading strategy into vehicle-vehicle collaboration and vehicle-server collaboration, and designs the computing model of the task for each of them; based on this, a scheduling model for tasks in local vehicles, service vehicles and servers is also proposed, and the A2C algorithm is used to train the policy network. The steps of the method include:

[0036] Step S1, obtain the sequence of tasks to be executed, the initial service vehicle sequence and the initial server sequence. The sequences can be obtained from the vehicle-side collaborative computing unloading system, which includes the task vehicle (TV) responsible for generating tasks to be executed, the service vehicle (SV) responsible for executing the tasks unloaded from the TV, and the server.

[0037] Step S2, using the analytic hierarchy process (AHP) to divide the task priorities of the task sequence to be executed, and combining the attributes of the task sequence to be executed to obtain the current task sequence, the attributes of the task sequence include CPU execution time, computational complexity, data volume, delay tolerance, whether the task can be split, and task priority.

[0038] Step S3, based on the current task sequence, service vehicle sequence and server sequence, use the pre-built vehicle-vehicle collaborative computing model and vehicle-server collaborative computing model to perform task scheduling, and use the sequence-to-sequence model based on IndRNN-AM to obtain the corresponding task offloading strategy. In order to efficiently implement task scheduling, this embodiment designs a total of four queues, namely, a failure queue, a processing queue, a waiting queue and a task sequence, to record different task states together; the sequence-to-sequence model based on IndRNN-AM introduces a parameter constraint mechanism on the basis of the recurrent neural network (RNN) to obtain IndRNN (Independent Recurrent Neural Network), and adds an attention mechanism (AM) to obtain the final model, which can better handle long input sequences.

[0039] Step S4, based on the current task offloading and scheduling strategy, obtain the current task sequence, service vehicle sequence and server sequence, and use the dominant action value algorithm to train the sequence-to-sequence model based on IndRNN-AM to obtain the final task offloading and scheduling strategy. RSU will send the scheduling decision to the corresponding vehicle.

[0040] The specific contents of the above steps are as follows:

[0041] 1. Task Offloading System

[0042] 1. Computation offloading system architecture

[0043] like Figure 1 As shown, in the vehicle-side collaborative computing unloading system of this embodiment: suppose there are many vehicles traveling on a one-way straight road, represented by the set V = {1,2,3,...,v}, and all vehicles are evenly distributed on the road and travel at a constant speed within a certain period of time. An RSU equipped with a VEN server is deployed on one side of the road, and all vehicles on the road are always within the communication range of this RSU. Wireless communication can be carried out between vehicles, and wireless communication can also be carried out between vehicles and RSUs and servers. The computing tasks generated by TV can be offloaded to the service vehicle going straight through V2V communication technology, or they can be offloaded to the server for execution through V2I communication technology. After the task is executed, the SV or server returns the result directly to TV through single-hop transmission.

[0044] When TV generates a task that needs to be offloaded, TV will first send a task offloading request to RSU, and then RSU will make a task offloading decision to determine whether to offload the task to the server or SV, and send the task offloading decision to TV and the corresponding SV respectively. TV will then offload the task to be offloaded to the server or specific SV, and wait for the task to be completed, and the server or SV that executes the task will return the result to TV.

[0045] Assume that during task offloading, there are N SVs around TV, SV = {SV1, SV2, ..., SV N}, where SV j ={f j ,P j}, j∈{1,2,...,N}, f j is the computing power of the jth SV, which refers to the CPU cycles per second, P j is the transmission power of the jth SV. TV = {f l ,P l}, f l is the computing power of TV, P l It is the transmission power of TV to send task data to SV or server. VEN server is represented by Sever, Sever = {f s ,P s}, f s is the computing power of the server, P s is the transmission power of the server. Each vehicle travels at a constant speed v, and the speeds of all vehicles follow a uniform distribution. The distances between TV and SV and the server are used to evaluate the channel states of V2V and V2I respectively, and the relative positions and relative speeds of TV and SV and the server are used to evaluate the link duration. The total system time is divided into several time periods. In a single time period, TV generates a total of M computing tasks. These M tasks can be divided into two categories: non-separable computing tasks and separable computing tasks. The i-th task is defined as a six-tuple Ω = {C i ,D i ,κ i ,τ i ,μ i ,ι i}, i∈{1,2,...,M}, where C i Indicates completion i Required CPU cycles, D i Represents Ω i The size of the input data, κ i Represents Ω i The computational complexity of τ i Represents Ω i The delay tolerance, μ iIndicates the splittable attribute of the task, ι i Represents Ω i priority.

[0046] There are three execution modes for tasks: local execution, full offloading, and partial offloading, and each task can only choose one of these execution modes. Specifically:

[0047] 1) Local execution: add the task to the waiting queue of TV;

[0048] 2) Overall offloading: offloading the task to the SV or server and adding it to the waiting queue of the TV or server;

[0049] 3) Partial offloading: a part of the task is left to be executed locally and added to the waiting queue of the TV, and the other part is offloaded to the SV or server and added to the waiting queue of the TV or server.

[0050] When the task is offloaded in whole or in part, there are two options: one is to offload to a nearby SV with available computing resources, and the other is to offload to a server. Before the task is offloaded, the task information and SV status to be offloaded need to be uploaded to the RSU via the wireless network, and then the RSU will make an execution decision. Assuming that there is no dependency between all tasks, each indivisible computing task or the subtask after the splitting of the divisible computing task can only select one SV or server for offloading when offloading, and each SV and server can process multiple tasks in parallel.

[0051] (II) Computational Model

[0052] 1. Vehicle-to-Vehicle Collaborative Computing Model

[0053] 1) Calculation model 1

[0054] When the task i When the execution mode is local execution, task Ω i The computational delay for:

[0055]

[0056] The computational delay of the task when it is left to execute locally cannot exceed its delay tolerance, so

[0057]

[0058] 2) Calculation Model 2

[0059] When the task i When the execution mode is overall unloading, the task Ω i Upload the data to SV j The transmission time is

[0060]

[0061] In the formula, R l To offload tasks to SV j The link transmission rate at that time.

[0062] Mission Ω i In SV j The execution time on

[0063]

[0064] The result will be returned at

[0065]

[0066] In formula (5), R j For SV j The link transmission rate when the result is sent back to TV, η represents the ratio between the output data size and the input data size, usually η≤1, which can be ignored when calculating the task time delay. Represents the task Ω i Waiting time before unloading, then task Ω i Total delay of unloading for

[0067]

[0068] The total delay of each task when offloading cannot exceed its delay tolerance, so

[0069]

[0070] 3) Calculation model 3

[0071] When the task i When the execution mode is partial unloading, the task Ω i φ i Part of it is executed locally on TV, and its computational delay is

[0072]

[0073] Then the task Ω i 1-φ i Partially offloaded to SV j Execute and upload the task data to SV j The transmission time is

[0074]

[0075] Mission Ω i The subtasks in SVj The execution time on

[0076]

[0077] The result will be returned at

[0078]

[0079] Then the task Ω i 1-φ i The total delay of partial unloading is

[0080]

[0081] In formula (12), Represents the task Ω i Waiting time before uninstallation. Due to task Ω i φ i Partial sum 1-φ i Part of it is parallel processing, so Ω i The total delay of task Ω is the larger of the two. i The total execution delay t i for

[0082]

[0083] The total delay of each task when offloading cannot exceed its delay tolerance, so

[0084] t i ≤τ i (14)

[0085] 2. Vehicle-Server Collaborative Computing Model

[0086] If the task Ω i Offload to the server for execution. The offloading process is as follows: TV uploads the task data to the server through RSU, and the server executes the taskΩ i After the execution is completed, the result is transmitted back to TV through RSU. The total offloading time of the task is divided into four parts: waiting time before task offloading, data uploading time, task execution time on the server and result transmission time. s represents the link transmission rate when offloading tasks to the server, R' s Indicates the link transmission rate when the server sends the result back to the TV.

[0087]

[0088] In equations (15) and (16), B represents the bandwidth of the transmission channel from the vehicle to the server, d srepresents the distance between TV and server, δ represents the path loss exponent, h represents the complex Gaussian channel coefficient, and N0 represents additive Gaussian white noise.

[0089] Computational model 1: When task Ω i When the execution mode is local execution, task Ω i The computational delay for

[0090]

[0091] The computational delay of the task when it is left to execute locally cannot exceed its delay tolerance, so

[0092]

[0093] Computational Model 2: When task Ω i When the uninstallation mode is overall uninstallation, the task Ω i The transmission time for uploading data to the server is

[0094]

[0095] Mission Ω i The execution time on the server is

[0096]

[0097] The result will be returned at

[0098]

[0099] In formula (21), η represents the ratio between the size of the output data and the size of the input data. Usually η≤1, which can be ignored when calculating the task time delay. Represents the task Ω i Waiting time before unloading, then task Ω i Total delay of overall unloading for

[0100]

[0101] The total delay of each task when offloading cannot exceed its delay tolerance, so

[0102]

[0103] Computational Model 3: When task Ω i When the uninstallation mode is partial uninstallation, the task Ω i φ i Part of it is executed locally on TV, and its computational delay is

[0104]

[0105] Then the task Ω i 1-φ i Part of it is offloaded to the server for execution, and the transmission time for uploading the task data to the server is

[0106]

[0107] The execution time of the subtask on the server is

[0108]

[0109] The result will be returned at

[0110]

[0111] In formula (27), Represents the task Ω i Waiting time before uninstallation. Then, task Ω i 1-φ i The total delay of partial unloading is

[0112]

[0113] Due to the task i φ i Partial sum 1-φ i Part of it is parallel processing, so Ω i The total delay of task Ω is the larger of the two. i The total execution delay t i for

[0114]

[0115] The total delay of each task when offloading cannot exceed its delay tolerance, so

[0116] t i ≤τ i (30)

[0117] (III) Task Scheduling Model

[0118] This embodiment designs four queues to record the status of task scheduling:

[0119] (1) Failure Queue: records the tasks that failed to be allocated and timed out.

[0120] (2) Processing Queue, which records the tasks being executed.

[0121] (3) Waiting Queue, which records tasks that have not yet been scheduled.

[0122] (4) Task sequence X, records the original task information.

[0123] The detailed task scheduling process is shown in Algorithm 1 in Table 1. Specifically:

[0124] First, find the task Ω with the highest priority in the task sequence X and schedule it.

[0125] Secondly, assign SV, TV or server to task Ω, and add task Ω to the WaitingQueue of the SV, TV or server to determine whether task Ω is offloaded to SV, offloaded to server or kept for local execution.

[0126] Again, compare the initial computing resources of the SV, TV or server allocated to task Ω with the computing resources required by task Ω: if the initial computing resources of the SV, TV or server are less than the computing resources required by task Ω, then the task allocation fails, and it is necessary to reallocate the SV, TV or server to task Ω; if the task allocation is successful, determine whether the currently available computing resources of the SV, TV or server are sufficient to calculate the task. If sufficient, add the task to the Processing Queue of the current SV, TV or server, and delete task Ω from the Waiting Queue; otherwise, task Ω waits.

[0127] Next, after the task with the shortest remaining execution time in the Processing Queue of the current SV, TV or server is completed and the computing resources it occupies are released, the remaining execution time and total execution time of task Ω and all tasks in the Processing Queue are immediately updated.

[0128] Finally, after the update is completed, it is determined whether the task Ω has timed out. If so, the task Ω fails to execute and is added to the Fail Queue. If not, Algorithm 1 is recursively called to reschedule the task Ω.

[0129] Table 1 Task scheduling model Algorithm 1 Process

[0130]

[0131]

[0132]

[0133] This embodiment aims to improve the success rate of task execution as much as possible while reducing the average delay of task execution. Therefore, the execution success rate of the calculation task is defined as

[0134]

[0135] In formula (31), q represents the number of tasks that have timed out, that is, the number of tasks in the Fail Queue, and M represents the total number of tasks. The average delay of all tasks is

[0136]

[0137] 2. Computation offloading method based on deep reinforcement learning

[0138] Most deep reinforcement learning algorithms, like the Actor-Critic algorithm, are single-threaded, that is, experience is generated through the interaction of an agent with the environment. However, when the environment is fixed and the agent generates continuous actions, the generated experience usually has a strong temporal correlation. In order to improve this problem, the Advantage Actor-Critic (A2C) algorithm adopts a multi-threaded parallel architecture. Through this parallel architecture, the A2C algorithm can efficiently utilize computing resources, break the coupling between experiences, and improve the efficiency of training. Therefore, this embodiment uses the A2C method to train the policy network.

[0139] (I) Network design

[0140] Figure 2 A task offloading method is shown. Specifically:

[0141] Assume that the task vehicle TV generates M=4 computing tasks, namely Task1, Task2, Task3 and Task4, among which Task1 and Task3 are splittable computing tasks, and Task2 and Task4 are inseparable computing tasks. Task1 and Task3 are split into four subtasks Subtask1, Subtask2 and Subtask3, Subtask4 respectively, and then these four subtasks together with Task2 and Task4 form a new task sequence, and generate the execution decision of the new task sequence. The number of service vehicles SV is N=3, the number of servers is 1, TV is numbered 0, SV is numbered 1, 2, 3 in sequence, and the server is numbered 4. One of the execution decisions of these four computing tasks can be expressed as {0, 2, 4, 3, 0, 1}, 0 means that Subtask1 is executed locally on TV numbered 0, 2 means that Subtask2 is offloaded to SV numbered 2 for execution, 4 means that Task2 is offloaded to server numbered 4 for execution, and the rest are similar.

[0142] Sequence to Sequence (Seq2Seq) is a type of encoder-decoder structure, and its basic structure consists of a recurrent neural network (RNN). The problem of assigning tasks to service vehicles can be transformed into the problem of mapping the task and service vehicle sequence into a service vehicle number sequence with the same length as the task sequence, which is suitable for Seq2Seq. However, the commonly used RNN structure in Seq2Seq, such as GRU, uses a hyperbolic tangent activation function, which will cause the gradient of the network layer to decay, resulting in the problem of gradient vanishing and gradient exploding. Unlike RNN, IndRNN (Independent Recurrent Neural Network) introduces a parameter constraint mechanism to make the gradient more stable during the back propagation process, thereby effectively solving the problem of gradient vanishing and gradient exploding, and IndRNN has better robustness after training. Therefore, this paper replaces the RNN in Seq2Seq with IndRNN. In addition, the Seq2Seq model needs to compress all the information of the input sequence, so it is difficult to process longer input sequences. In this regard, this paper adds an attention mechanism (AM). When the encoder gets an output, it searches for the part of the input sequence that is most relevant to the output. The model then predicts the unloading vehicle number for the next task based on the context vector corresponding to this part of the input sequence and all previously generated service vehicle numbers. Compared to the original Seq2Seq model, the improved model focuses on the part of the input sequence that is most relevant to the current task when decoding, and can better handle long input sequences.

[0143] This embodiment uses the above improved Seq2Seq model as the A2C policy network to solve the computational offloading problem of hybrid tasks in the vehicle-side collaboration scenario. Figure 3 As shown in the figure, in addition to the task information and service vehicle information, the server information is also added to the input sequence, and the output sequence is the vehicle number or server number that executes each task. The input of the model is the task sequence X and SV, TV to be scheduled obtained by the analytic hierarchy process (AHP) and the server information, and the server is represented by Server.

[0144] (II) Model training

[0145] The policy network is trained by the advantage action value algorithm, and the state space, action space and reward are defined as follows:

[0146] (1) State space: It consists of task sequence, service vehicle sequence and server sequence. The task sequence has six attributes, namely, the execution time of the task on the CPU, computational complexity, data volume, delay tolerance, task splitting attribute and task priority. The attribute of the service vehicle is the computing resources available to the vehicle, and the attribute of the server is also the computing resources available to the server.

[0147] (2) Action space: The number of the task vehicle TV or service vehicle SV or server that performs each task, generated by the policy network. The set of actions is the offloading policy.

[0148] (3) Reward: The reward corresponding to the execution action. In order to meet the research goal of reducing the average delay of task execution and improving the success rate of task execution, the reward function is set to the inverse of the sum of the average delay of task execution and the number of failed tasks q multiplied by a weight factor λ, as shown in formula (33).

[0149]

[0150] In formula (34), average_time is the average delay of all tasks, that is,

[0151]

[0152] The multi-threaded parallel architecture of the A2C algorithm is as follows Figure 4 As shown in the figure, the agent consists of two parts: the global network and several independent and parallel worker threads. Each worker interacts with its own environment independently, collects independent sample experience, and uses this experience to update the local Actor-Critic network. After all worker threads complete each episode, the parameters of the global network are updated synchronously.

[0153] During network training, the advantage function can be expressed as

[0154] A(s t ,a t )=E[r t+1 +γV(s t+1 )-V(s t )] (35)

[0155] In the formula, A(·) represents the advantage function, E(·) represents the value function, and V(·) represents the state s t The value function of s t represents the hidden state at the tth time step, a t represents the action at the tth time step, r t+1represents the reward of the action corresponding to the t+1th time step, and γ represents the discount rate.

[0156] Initialize the parameters θ of the policy network and the parameters of the value function network The calculation formula of the policy gradient can be expressed as

[0157]

[0158] Where T represents the batch size in the training dataset.

[0159] Set the learning rate of the Actor network to α, and the update formula of the parameter θ of the Actor network can be expressed as

[0160]

[0161] Set the learning rate of the Critic network to β, and the parameters of the Critic network The update formula is

[0162]

[0163] The network training process is shown in Table 2, where N is the total number of episodes.

[0164] Table 2 A2C model training process

[0165]

[0166]

[0167] Further, the present embodiment also provides a vehicle-side collaborative computing task offloading device based on deep reinforcement learning, including a memory, a processor, and a program stored in the memory, and the processor implements the aforementioned method when executing the program. The device processor includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or computer program instructions loaded from a storage unit to a random access memory (RAM). In RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. The input / output (I / O) interface is also connected to the bus. Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks. The processing unit performs the various methods and processes described above, such as the aforementioned method. For example, in some embodiments, the aforementioned method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the aforementioned method can be performed. Alternatively, in other embodiments, the CPU can be configured to perform the aforementioned method in any other appropriate manner (for example, by means of firmware). The functions described above can be performed at least in part by one or more hardware logic components. For example, without limitation, the hardware logic components of exemplary types that can be used include: field programmable gate array (FPGA), application specific integrated circuit (ASIC), application specific standard product (ASSP), system on chip (SOC), complex programmable logic device (CPLD), etc.

[0168] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning. It is characterized in that The following steps are involved: S1, obtaining a sequence of tasks to be executed, an initial service vehicle sequence and an initial server sequence; S2, dividing the task priorities of the task sequence to be executed, and combining the attributes of the task sequence to be executed to obtain the current task sequence, the attributes of the task sequence including CPU execution time, computational complexity, data volume, delay tolerance, whether the task can be split, and task priority; S3, based on the current task sequence, service vehicle sequence and server sequence, the pre-built vehicle-vehicle collaborative computing model and vehicle-server collaborative computing model are used to schedule tasks, and the sequence-to-sequence model based on IndRNN-AM is used to obtain the corresponding task offloading strategy; S4, based on the current task offloading and scheduling strategy, obtain the current task sequence, service vehicle sequence and server sequence, and use the dominant action value algorithm to train the sequence-to-sequence model based on IndRNN-AM to obtain the final task offloading and scheduling strategy.

2. According to the method for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning in claim 1, It is characterized in that The scheduling modes for each task include local execution, overall offloading, and partial offloading.

3. According to claim 2, the method for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning, It is characterized in that When the scheduling mode is partial unloading, based on the vehicle-to-vehicle cooperative calculation model, the corresponding total task execution delay is: and Where, t i is the total execution delay of the ith task, is the φ of the i-th task i The latency of some tasks being performed locally in the mission vehicle, is 1-φ of the i-th task i The total delay from partial unloading to service vehicle execution, C i The CPU execution time required to complete the i-th task, f l The computing power of the mission vehicle, represents the waiting time before the i-th task is unloaded, represents the 1-φ of the i-th task i Transmission time for partial upload to the service vehicle, represents the execution time of the subtask of the ith task on the service vehicle, Indicates the result return time, D i is the size of the input data, f j is the computing capacity of the jth service vehicle, η represents the ratio of the output data volume to the input data volume, R l is the link transmission rate when offloading the task to the service vehicle, R j The link transmission rate when the service vehicle transmits the results back to the TV.

4. According to claim 1, the method for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning, It is characterized in that In the process of task scheduling, a failure queue, a processing queue, a waiting queue and a task sequence are used to record different task states.

5. According to claim 4, the method for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning, It is characterized in that The pre-built vehicle-to-vehicle collaborative computing model and vehicle-to-server collaborative computing model are used to schedule tasks. The specific process includes: Selecting a highest priority task from the task sequence; Allocate a service vehicle, task vehicle or server to the highest priority task, and add the task to a waiting queue of the allocated service vehicle, task vehicle or server; When the initial computing resources of the assigned service vehicle, task vehicle or server are greater than or equal to the first computing resources required by the highest priority task, the task is successfully assigned, and it is determined whether the currently available computing resources of the assigned service vehicle, task vehicle or server are greater than or equal to the first computing resources: if yes, the highest priority task is added to the processing queue of the current service vehicle, task vehicle or server, and deleted from the waiting queue; if no, the highest priority task enters the waiting queue; When the task with the shortest remaining execution time in the processing queue of the current service vehicle, task vehicle or server has been completed and the corresponding computing resources have been released, the remaining execution time and the total execution time of the highest priority task and all tasks in the processing queue are updated; After the update is completed, determine whether the highest priority task has timed out: if yes, add the highest priority task to the failure queue; if no, continue to reallocate the service vehicle, task vehicle or server for the highest priority task.

6. According to claim 5, the vehicle-side collaborative computing task offloading method based on deep reinforcement learning, It is characterized in that The pre-built vehicle-to-vehicle collaborative computing model and vehicle-to-server collaborative computing model are used to schedule tasks. The specific process also includes: When the initial computing resources are less than the first computing resources, task allocation fails, and a service vehicle, a task vehicle or a server is reallocated to the highest priority task; When the task with the shortest remaining execution time in the processing queue of the current service vehicle, task vehicle or server has not been completed and the corresponding computing resources have not been released, the waiting continues.

7. The vehicle-side collaborative computing task offloading method based on deep reinforcement learning according to claim 1, It is characterized in that The sequence-to-sequence model based on IndRNN-AM is used as the policy network of the dominant action value algorithm, the state space of the dominant action value algorithm includes task sequence, service vehicle sequence and server sequence, the action space of the dominant action value algorithm includes the number of the task vehicle, service vehicle or server that performs each task and is generated using the policy network, and the reward function of the dominant action value algorithm is the inverse of the sum of the average delay of task execution and the number of weighted failed tasks.

8. The vehicle-side collaborative computing task offloading method based on deep reinforcement learning according to claim 7, It is characterized in that The average delay is: Where M is the total number of tasks, is the φ of the i-th task i The latency of some tasks being performed locally in the mission vehicle, is 1-φ of the i-th task i Total delay of partial offloading to the service vehicle or server execution, t i is the total execution delay of the i-th task.

9. The vehicle-side collaborative computing task offloading method based on deep reinforcement learning according to claim 1, It is characterized in that The task priority is divided by using the hierarchical analysis method.

10. A vehicle-side collaborative computing task offloading device based on deep reinforcement learning, comprising a memory, a processor, and a program stored in the memory, It is characterized in that When the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

Citation Information

Patent Citations

  • Energy-saving automatic interconnected vehicle service unloading method based on deep reinforcement learning

    CN114528042A

  • Calculation unloading method and system for task with dependency relationship in edge calculation

    CN116755882A

  • Multi-task learning for end-to-end automatic speech recognition confidence and deletion estimation

    CN117099157A

  • Internet of vehicles computing task unloading method and device

    CN117793665A

  • Intelligent warehousing task unloading method based on mobile edge computing

    CN118377556A

Cited By

  • Task unloading method in vehicle-mounted edge computing environment based on deep reinforcement learning

    CN119440659A