A method and apparatus for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning

Abstract

This invention relates to a method and apparatus for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning. The method acquires a sequence of tasks to be executed, an initial sequence of service vehicles, and an initial sequence of servers; prioritizes the tasks in the sequence of tasks to be executed and obtains the current task sequence by combining the attributes of the task sequences; based on the current task sequence, service vehicle sequence, and server sequence, performs task scheduling using pre-built vehicle-to-vehicle collaborative computing models and vehicle-to-server collaborative computing models, and obtains a task offloading strategy using a sequence-to-sequence model based on IndRNN-AM; based on the current task offloading and scheduling strategy, acquires the current task sequence, service vehicle sequence, and server sequence, and trains the sequence-to-sequence model based on IndRNN-AM using the Advantageous Action Value algorithm to obtain the final task offloading and scheduling strategy. Compared with existing technologies, this invention has advantages such as improving the stability and efficiency of the entire computing offloading system.

Description

Technical Field

[0001] This invention relates to the field of vehicle networking technology, and in particular to a method and apparatus for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning. Background Technology

[0002] The development of technologies such as artificial intelligence and big data analytics has made in-vehicle internet more practical. However, limitations imposed by massive amounts of data and transmission channels, such as traffic monitoring, map navigation, and route planning, have placed enormous pressure on congested networks and resulted in high latency. The widespread application of these computing tasks has led to a significant increase in data volume, causing vehicles with limited computing resources to be unable to handle such a large number of computationally intensive tasks, thereby affecting the quality of user service.

[0003] Currently, cloud computing (CC) can be used to offload computing tasks to a central cloud for processing. While cloud computing infrastructure has existed for many years and can handle a large number of computationally intensive tasks, cloud computing centers are often located in relatively remote locations, far from vehicles. This can lead to problems such as long task transmission times and link congestion, affecting task execution time and compromising vehicle safety and privacy. To address this, vehicular edge computing (VEC) has been introduced. This technology pushes computing and storage capabilities down to servers and vehicles at the network edge. Its low latency, high bandwidth, and real-time computing capabilities better meet the needs of edge devices and end users, providing computing and storage services to resource-constrained vehicles at the network edge, significantly reducing processing latency for computing tasks.

[0004] Because onboard equipment has limited computing resources, when it generates computational tasks that it cannot handle itself, these tasks can be transferred to nearby vehicles with available resources. However, due to the high-speed mobility of vehicles, excessive traffic on the same road segment can lead to vehicle aggregation. Densely populated areas generate an even greater number of computational tasks requiring offloading, resulting in overloaded service vehicles that cannot meet the latency requirements of the tasks. Conversely, when there are too few service vehicles with available computing resources on the same road segment, tasks cannot be processed in a timely manner. To address this, a collaborative computational offloading mechanism between the vehicle and server sides is introduced in a Vehicular Edge Network (VEN). This allows task vehicles to offload tasks to VEN servers with richer computing resources within the communication range of the Road Side Unit (RSU) when in densely populated areas or when there are too few service vehicles. This vehicle-side collaborative offloading method not only ensures low latency requirements for latency-constrained computational tasks but also enables timely processing of tasks under conditions of limited computing resources. However, a reasonable and efficient offloading strategy is still needed to fully utilize the computing resources of service vehicles and VEN servers to meet the low latency and high success rate requirements of both types of computing tasks. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a vehicle-side collaborative computing task offloading method and apparatus based on deep reinforcement learning. This method can formulate reasonable and efficient offloading and scheduling strategies, and make full use of the computing resources of service vehicles and servers to meet the requirements of different types of computing tasks for low latency and high success rate.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, a vehicle-side collaborative computing task offloading method based on deep reinforcement learning is provided, comprising the following steps: S1, obtaining a sequence of tasks to be executed, an initial sequence of service vehicles, and an initial sequence of servers; S2, dividing the task priorities of the sequence of tasks to be executed, and combining the attributes of the sequence of tasks to be executed to obtain the current task sequence, wherein the attributes of the task sequence include CPU execution time, computational complexity, data volume, latency tolerance, whether the task is divisible, and task priority; S3, based on the current task sequence, the sequence of service vehicles, and the sequence of servers, performing task scheduling using a pre-built vehicle-to-vehicle collaborative computing model and a vehicle-to-server collaborative computing model, and obtaining a corresponding task offloading strategy using a sequence-to-sequence model based on IndRNN-AM; S4, based on the current task offloading and scheduling strategy, obtaining the current task sequence, the sequence of service vehicles, and the sequence of servers, and training the sequence-to-sequence model based on IndRNN-AM using the Advantage Action Value algorithm to obtain the final task offloading and scheduling strategy.

[0008] As a preferred technical solution, the scheduling methods for each task include local execution, overall unloading, and partial unloading.

[0009] As a preferred technical solution, when the scheduling method is partial unloading, based on the vehicle-to-vehicle cooperative computing model, the corresponding total task execution latency is:

[0010]

[0011] and

[0012]

[0013] In the formula, t i Let be the total execution delay of the i-th task. For the i-th task, φ i The latency of some tasks being executed locally on the mission vehicle. For the i-th task, 1-φ i The total latency of partially unloading to the service vehicle is C. i The CPU execution time required to complete the i-th task, f l The computing power of the mission vehicle. This represents the waiting time before the i-th task is unloaded. Represents 1-φ of the i-th task i The transmission time for some parts uploaded to the service vehicle, This represents the execution time of the subtask of the i-th task on the service vehicle. D represents the return time of the result. i f represents the size of the input data. jLet R represent the computing power of the j-th service vehicle, η represent the ratio of output data volume to input data volume, and R be the computing power of the j-th service vehicle. l To determine the link transmission rate when offloading tasks to service vehicles, R j The link transmission rate when the service vehicle sends the results back to the TV.

[0014] As a preferred technical solution, during the task scheduling process, a failure queue, a processing queue, a waiting queue, and a task sequence are used to record different task states.

[0015] As a preferred technical solution, task scheduling is performed using a pre-built vehicle-to-vehicle collaborative computing model and a vehicle-to-server collaborative computing model. The specific process includes: selecting the highest priority task from the task sequence; allocating a service vehicle, task vehicle, or server to the highest priority task and adding it to the waiting queue of the allocated service vehicle, task vehicle, or server; when the initial computing resources of the allocated service vehicle, task vehicle, or server are greater than or equal to the first computing resources required by the highest priority task, the task allocation is successful, and it is determined whether the currently available computing resources of the allocated service vehicle, task vehicle, or server are greater than or equal to the first computing resources: if so, the task is scheduled to proceed as planned. The highest priority task is added to the processing queue of the current service vehicle, task vehicle, or server, and removed from the waiting queue; if not, the highest priority task enters the waiting state; when the task with the shortest remaining execution time in the processing queue of the current service vehicle, task vehicle, or server has been completed and released its corresponding computing resources, the remaining execution time and total execution time of the highest priority task and all tasks in the processing queue are updated; after the update, it is determined whether the highest priority task has timed out: if yes, the highest priority task is added to the failure queue; if no, the highest priority task continues to be reassigned to a service vehicle, task vehicle, or server.

[0016] As a preferred technical solution, task scheduling is performed using a pre-built vehicle-to-vehicle collaborative computing model and a vehicle-to-server collaborative computing model. The specific process further includes: when the initial computing resources are less than the first computing resources, the task allocation fails, and a service vehicle, task vehicle, or server is reallocated to the highest priority task; when the task with the shortest remaining execution time in the current service vehicle, task vehicle, or server's processing queue has not been completed and the corresponding computing resources have been released, the process continues to wait.

[0017] As a preferred technical solution, the sequence-to-sequence model based on IndRNN-AM is used as the policy network of the dominant action value algorithm. The state space of the dominant action value algorithm includes a task sequence, a service vehicle sequence, and a server sequence. The action space of the dominant action value algorithm includes the number of the task vehicle, service vehicle, or server executing each task and is generated using the policy network. The reward function of the dominant action value algorithm is the reciprocal of the sum of the average delay of task execution and the number of weighted failed tasks.

[0018] As a preferred technical solution, the average latency is:

[0019]

[0020] In the formula, M represents the total number of tasks. For the i-th task, φ i The latency of some tasks being executed locally on the mission vehicle. For the i-th task, 1-φ i The total latency of partially unloading to the service vehicle or server is t. i Let be the total execution delay of the i-th task.

[0021] As a preferred technical solution, the task priority division is achieved using the analytic hierarchy process (AHP).

[0022] According to a second aspect of the present invention, a vehicle-side collaborative computing task offloading device based on deep reinforcement learning is provided, comprising a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method as described above.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. This invention addresses the problem of offloading hybrid computing tasks in vehicle-side collaborative scenarios. It proposes a vehicle-side collaborative computing task offloading method based on deep reinforcement learning. This method considers the offloading of collaborative computing between the vehicle and the server when the computing resources of the service vehicle are insufficient. It designs latency computing models for vehicle-to-vehicle collaboration and vehicle-to-server collaboration for task scheduling, and uses a sequence-to-sequence model based on IndRNN-AM to obtain the corresponding task offloading strategy. On this basis, the model is trained using the Advantage Actor-Critic (A2C) algorithm to obtain a better task offloading and scheduling strategy. This method can make full use of the computing resources of the service vehicle and the edge server, meet the requirements of low latency and high success rate for both types of computing tasks, and improve the stability and efficiency of the entire computing offloading system.

[0025] 2. In the method provided by this invention, the scheduling mode of each task includes three types: local execution, overall unloading, and partial unloading. When the computing resources of the vehicle are insufficient, the task is unloaded to the server for processing, and the server has stronger execution capabilities. When unloading computing tasks, for tasks that fail to be allocated, service vehicles, servers, or local resources are also reallocated. Through this vehicle-side collaborative unloading method, the low latency requirements of latency-constrained tasks can be guaranteed. In addition, when computing resources are scarce and the number of tasks increases, tasks can be processed in a timely manner, improving the success rate of task execution and shortening the time of task execution.

[0026] 3. The method provided by this invention uses the sequence-to-sequence model based on IndRNN-AM as the policy network of the A2C algorithm, and then uses the A2C algorithm to train the policy network. It adopts a multi-threaded parallel architecture, which breaks the single-threaded architecture of most deep reinforcement learning algorithms. It can efficiently utilize computing resources, break the coupling between experiences, and improve training efficiency.

[0027] 4. In the method provided by this invention, the attributes of the task sequence include CPU execution time, computational complexity, data volume, latency tolerance, whether the task is divisible, and task priority. The attribute of whether the task is divisible means that all tasks to be executed can be divided into indivisible computing tasks and divisible computing tasks. Divisible computing tasks can be divided into multiple subtasks to be executed. Based on this, according to the CPU execution time, computational complexity, data volume, and latency tolerance of the task, the hierarchical analysis method is used to prioritize all tasks to be executed, so that each task to be executed is independent and has no dependency relationship, which can better obtain the unloading and scheduling strategy, thereby achieving the optimization of computing resources and maximizing the efficiency of computing task unloading.

[0028] 5. The method and apparatus provided in this embodiment are also applicable to other IoT devices with edge computing needs, and have good application prospects. They can effectively reduce task offloading costs and improve the utilization rate of system resources. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the vehicle-side collaborative computing unloading system architecture provided in an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of one of the task unloading methods provided in an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the policy network model structure provided in the embodiments of the present invention;

[0032] Figure 4 This is a schematic diagram of the advantageous action value algorithm provided in the embodiments of the present invention. Detailed Implementation

[0033] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0034] Example

[0035] This embodiment provides a method for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning, which can be achieved through... Figure 1 The illustrated vehicle-side collaborative computing offloading system architecture is implemented. This method considers the collaborative computing offloading between the vehicle and server sides when the service vehicle's computing resources are insufficient. It divides the task offloading strategy into vehicle-to-vehicle cooperation and vehicle-to-server cooperation, designing computational models for each. Based on this, it also proposes a scheduling model for tasks in the local vehicle, service vehicle, and server, and uses the A2C algorithm to train the policy network. The steps of the method include:

[0036] Step S1: Obtain the sequence of tasks to be executed, the initial service vehicle sequence, and the initial server sequence. All sequences can be obtained from the vehicle-side collaborative computing unloading system, which includes the task vehicle (TV) responsible for generating the tasks to be executed, the service vehicle (SV) responsible for executing the tasks unloaded from the TV, and the server.

[0037] Step S2: Use the Analytic Hierarchy Process (AHP) to divide the task priority of the task sequence to be executed, and combine it with the attributes of the task sequence to be executed to obtain the current task sequence. The attributes of the task sequence include CPU execution time, computational complexity, data volume, latency tolerance, whether the task is divisible, and task priority.

[0038] Step S3: Based on the current task sequence, service vehicle sequence, and server sequence, task scheduling is performed using pre-built vehicle-to-vehicle cooperative computing models and vehicle-to-server cooperative computing models. The corresponding task offloading strategy is obtained using a sequence-to-sequence model based on IndRNN-AM. To achieve efficient task scheduling, this embodiment designs four queues—a failure queue, a processing queue, a waiting queue, and a task sequence—to record different task states. The sequence-to-sequence model based on IndRNN-AM introduces a parameter constraint mechanism on top of a Recurrent Neural Network (RNN) to obtain an IndRNN (Independent Recurrent Neural Network). An attention mechanism (AM) is also added to obtain the final model, which can better handle long input sequences.

[0039] Step S4: Based on the current task offloading and scheduling strategy, obtain the current task sequence, service vehicle sequence, and server sequence, and train a sequence-to-sequence model based on IndRNN-AM using the Advantageous Action Value algorithm to obtain the final task offloading and scheduling strategy. The RSU will send the made scheduling decision to the corresponding vehicle.

[0040] The specific details of each of the above steps are as follows:

[0041] I. Task Unloading System

[0042] (I) Computational Unloading System Architecture

[0043] like Figure 1 As shown, in the vehicle-side collaborative computing unloading system of this embodiment: Assume there are many vehicles traveling on a one-way straight road, denoted by the set V = {1,2,3,...,v}. All vehicles are evenly distributed on the road and travel at a constant speed within a certain time period. A Remote Unit (RSU) equipped with a VEN server is deployed on one side of the road, and all vehicles on the road are always within the communication range of this RSU. Wireless communication is possible between vehicles, as well as between vehicles and the RSU and the server. The computing tasks generated by the TV can be unloaded to the serving vehicle via V2V communication technology, or to the server via V2I communication technology. After the task is completed, the SV or the server directly returns the result to the TV via a single-hop transmission.

[0044] When a TV generates a task that needs to be unloaded, the TV first sends a task unload request to the RSU. Then, the RSU makes a task unload decision, which determines whether to unload the task to the server or the SV. The RSU then sends the task unload decision to the TV and the corresponding SV. The TV then unloads the task to be unloaded to the server or the specific SV. After the task is completed, the server or SV that executed the task returns the result to the TV.

[0045] Assume that during task unloading, there are N SVs around TV, where SV = {SV1, SV2, ..., SV}. N}, where SV j ={f j ,P j}, j∈{1,2,...,N}, f j The computing power of the j-th SV refers to CPU cycles per second, P. j It is the transmit power of the j-th SV. TV = {f} l ,P l}, f l It's the computing power of TV, P l This refers to the transmission power at which the TV sends task data to the SV or server. The VEN server is denoted by Sever, where Sever = {f s ,P s}, f s It refers to the server's computing power, P. s This is the server's transmit power. Each vehicle travels at a constant speed v, and the speeds of all vehicles follow a uniform distribution. The channel states for V2V and V2I are evaluated using the distances between TV, SV, and the server, respectively. The link duration is evaluated using the relative positions and relative speeds of TV, SV, and the server. The total system time is divided into several time periods. Within a single time period, TV generates M computational tasks, which can be divided into two categories: indivisible computational tasks and divisible computational tasks. The i-th task is defined as a six-tuple Ω = {C}. i D i ,κ i ,τ i ,μ i ,ι i}, i∈{1,2,...,M}, where C i Indicates completion Ω i Required CPU cycles, D i Represents Ω i The size of the input data, κ i Represents Ω i The computational complexity, τ i Represents Ω i The latency tolerance, μ iIndicates the detachable attribute of a task, ι i Represents Ω i Priority.

[0046] There are three execution methods for tasks: local execution, full uninstallation, and partial uninstallation, and each task can only choose one of these methods. Specifically:

[0047] 1) Local execution: Add the task to the TV's waiting queue;

[0048] 2) Complete Unload: Unload the task to SV or server and add it to the waiting queue of TV or server;

[0049] 3) Partial Unloading: Leave part of the task running locally and add it to the TV's waiting queue, while unloading the other part to the SV or server and adding it to the TV or server's waiting queue.

[0050] When a task is to be completely or partially unloaded, there are two options: one is to unload it to a nearby Storage Module (SV) with available computing resources, and the other is to unload it to a server. Before unloading a task, the information of the task to be unloaded and the SV status need to be uploaded to the Resource Unit (RSU) via a wireless network, and then the RSU generates an execution decision. Assuming that there are no dependencies between all tasks, each indivisible computing task or its subtasks can only select one SV or server for unloading, and each SV and server can process multiple tasks in parallel.

[0051] (II) Computational Model

[0052] 1. Vehicle-to-vehicle cooperative computing model

[0053] 1) Calculation Model 1

[0054] When task Ω i When the execution method is local execution, task Ω i computation delay for:

[0055]

[0056] The computation latency of a task running locally cannot exceed its latency tolerance.

[0057]

[0058] 2) Calculation Model 2

[0059] When task Ω i When the execution method is a complete uninstallation, task Ω will be executed. i Data uploaded to SV j The transmission time is

[0060]

[0061] In the formula, R l To unload the task to SV j The link transmission rate at that time.

[0062] Task Ω i In SV j The execution time is

[0063]

[0064] The result return time is

[0065]

[0066] In equation (5), R j For SV j The link transmission rate when the result is sent back to the TV is η, which represents the ratio between the size of the output data and the size of the input data. Usually, η≤1, which can be ignored when calculating the task time delay. Represents task Ω i The waiting time before uninstallation is for task Ω. i Total uninstallation time for

[0067]

[0068] The total latency during the unloading of each task must not exceed its latency tolerance, therefore

[0069]

[0070] 3) Calculation Model 3

[0071] When task Ω i When the execution method is partial unloading, task Ω i φ i Part of the computation is performed locally on the TV, and its latency is...

[0072]

[0073] So, task Ω i 1-φ i Some were unloaded to SV j Execute the task and upload the task data to SV. j The transmission time is

[0074]

[0075] Task Ω i Subtasks in SVj The execution time is

[0076]

[0077] The result return time is

[0078]

[0079] Then task Ω i 1-φ i The total latency of partial uninstallation is

[0080]

[0081] In equation (12), Represents task Ω i Waiting time before uninstallation. Due to task Ω i φ i Part and 1-φ i Partially parallel processing, so Ω i If the total latency is the larger of the two, then task Ω i Total execution latency t i for

[0082]

[0083] The total latency during the unloading of each task must not exceed its latency tolerance, therefore

[0084] t i ≤τ i (14)

[0085] 2. Vehicle-Server Collaborative Computing Model

[0086] If task Ω i The task is then unloaded to the server for execution. The unloading process is as follows: TV uploads the task data to the server via RSU, and the server executes the task Ω. i After execution, the results are sent back to TV via RSU. The total unloading time of the task is divided into four parts: the waiting time before task unloading, the data upload time, the task execution time on the server, and the result return time. Using R... s R' represents the link transfer rate when the task is offloaded to the server. s This indicates the link transmission rate when the server sends the result back to the TV.

[0087]

[0088] In equations (15) and (16), B represents the transmission channel bandwidth from the vehicle to the server, and d sδ represents the distance between the TV and the server, h represents the complex Gaussian channel coefficient, and N0 represents additive white Gaussian noise.

[0089] Computational Model 1: When task Ω i When the execution method is local execution, task Ω i computation delay for

[0090]

[0091] The computation latency of a task running locally cannot exceed its latency tolerance.

[0092]

[0093] Computational Model 2: When task Ω i When the uninstallation method is a complete uninstallation, task Ω i The data upload time to the server is

[0094]

[0095] Task Ω i The execution time on the server is

[0096]

[0097] The result return time is

[0098]

[0099] In equation (21), η represents the ratio between the size of the output data and the size of the input data. Usually, η≤1, which can be ignored when calculating the task time delay. Represents task Ω i The waiting time before uninstallation is for task Ω. i Total latency of overall uninstallation for

[0100]

[0101] The total latency during the unloading of each task must not exceed its latency tolerance, therefore

[0102]

[0103] Computational Model 3: When task Ω i When the uninstallation method is partial uninstallation, task Ω i φ i Part of the computation is performed locally on the TV, and its latency is...

[0104]

[0105] So, task Ω i 1-φ i Part of the task was unloaded and executed on the server; the transfer time for uploading the task data to the server was [time missing].

[0106]

[0107] The execution time of the subtask on the server is

[0108]

[0109] The result return time is

[0110]

[0111] In equation (27), Represents task Ω i Waiting time before uninstallation. Then, task Ω i 1-φ i The total latency of partial uninstallation is

[0112]

[0113] Due to task Ω i φ i Part and 1-φ i Partially parallel processing, so Ω i The total delay is the larger of the two, therefore task Ω i Total execution latency t i for

[0114]

[0115] The total latency during the unloading of each task must not exceed its latency tolerance, therefore

[0116] t i ≤τ i (30)

[0117] (III) Task Scheduling Model

[0118] This embodiment designs four queues to record the state during task scheduling:

[0119] (1) Fail Queue: Records tasks that failed to be assigned or timed out.

[0120] (2) Processing Queue: Records tasks that are being executed.

[0121] (3) Waiting Queue: Records tasks that have not yet been scheduled.

[0122] (4) Task sequence X, which records the original task information.

[0123] The detailed task scheduling process is shown in Algorithm 1 in Table 1. Specifically:

[0124] First, identify the highest priority task Ω in the task sequence X and schedule task Ω.

[0125] Next, assign an SV, TV, or server to task Ω, and add task Ω to the WaitingQueue of that SV, TV, or server to determine whether task Ω should be unloaded to the SV, unloaded to the server, or remain to be executed locally.

[0126] Next, the initial computing resources allocated to the SV, TV, or server for task Ω are compared with the computing resources required by task Ω: if the initial computing resources of the SV, TV, or server are less than the computing resources required by task Ω, then the task allocation fails, and the SV, TV, or server needs to be reallocated to task Ω; if the task allocation is successful, then it is determined whether the currently available computing resources of the SV, TV, or server are sufficient to compute the task. If they are sufficient, the task is added to the Processing Queue of the current SV, TV, or server, and task Ω is removed from the Waiting Queue; otherwise, task Ω waits.

[0127] Next, once the task with the shortest remaining execution time in the Processing Queue of the current SV, TV, or server has finished executing and released the computing resources it occupies, the remaining execution time and total execution time of task Ω and all tasks in the Processing Queue are immediately updated.

[0128] Finally, after the update is complete, it is determined whether task Ω has timed out. If it has timed out, task Ω fails to execute and is added to the Fail Queue; if it has not timed out, task Ω is rescheduled by recursively calling Algorithm 1.

[0129] Table 1. Task Scheduling Model Algorithm 1 Flowchart

[0130]

[0131]

[0132]

[0133] This embodiment aims to maximize the task execution success rate while minimizing the average task execution latency. Therefore, the execution success rate of the computation task is defined as...

[0134]

[0135] In equation (31), q represents the number of tasks that timed out and failed, i.e., the number of tasks in the Fail Queue, and M is the total number of tasks. The average latency of all tasks is...

[0136]

[0137] II. Computational Offloading Methods Based on Deep Reinforcement Learning

[0138] Most deep reinforcement learning algorithms, like the Actor-Critic algorithm, are single-threaded, meaning they generate experience through an agent's interaction with the environment. However, when the environment is fixed and the agent performs consecutive actions, the generated experience often exhibits strong temporal correlation. To address this issue, the Advantage Actor-Critic (A2C) algorithm employs a multi-threaded parallel architecture. This parallel architecture allows the A2C algorithm to efficiently utilize computational resources, break down the coupling between experiences, and improve training efficiency. Therefore, this embodiment uses the A2C method to train the policy network.

[0139] (I) Network Design

[0140] Figure 2 This demonstrates one method of task unloading. Specifically:

[0141] Assume that the task vehicle TV generates M = 4 computational tasks, namely Task1, Task2, Task3, and Task4. Task1 and Task3 are divisible computational tasks, while Task2 and Task4 are indivisible computational tasks. Task1 and Task3 are each divided into four subtasks: Subtask1, Subtask2, Subtask3, and Subtask4. These four subtasks, along with Task2 and Task4, form a new task sequence, and an execution decision for this new task sequence is generated. The number of service vehicles (SVs) is N = 3, and the number of servers is 1. The TV is numbered 0, the SVs are numbered 1, 2, 3, and the server is numbered 4. One possible execution decision for these four computational tasks can be represented as {0, 2, 4, 3, 0, 1}, where 0 indicates that Subtask1 is executed locally on TV number 0, 2 indicates that Subtask2 is offloaded to SV number 2 for execution, 4 indicates that Task2 is offloaded to server number 4 for execution, and so on.

[0142] Sequence-to-Sequence (Seq2Seq) models are a type of encoder-decoder architecture, with its basic structure consisting of a Recurrent Neural Network (RNN). The problem of assigning tasks to service vehicles can be transformed into mapping a sequence of tasks and service vehicles to a sequence of service vehicle numbers of the same length as the task sequence. This problem is well-suited for Seq2Seq. However, commonly used RNN structures in Seq2Seq, such as GRU, employ the hyperbolic tangent activation function, which causes gradient decay in the network layers, leading to vanishing and exploding gradients. Unlike RNNs, IndRNNs (Independent Recurrent Neural Networks) introduce parameter constraints, making gradients more stable during backpropagation, effectively solving the vanishing and exploding gradient problems. Furthermore, IndRNNs exhibit better robustness after training. Therefore, this paper replaces the RNN in Seq2Seq with IndRNNs. In addition, Seq2Seq models require compressing all information from the input sequence, making them difficult to handle long input sequences. To address this, this paper incorporates an attention mechanism (AM). Once the Encoder receives an output, it searches the input sequence for the part most relevant to that output. Then, the model predicts the unloading vehicle number for the next task based on the context vector corresponding to this part of the input sequence and all previously generated service vehicle numbers. Compared to the original Seq2Seq model, the improved model focuses on the part of the input sequence most relevant to the current task during decoding, allowing it to handle long input sequences better.

[0143] This embodiment uses the improved Seq2Seq model described above as the A2C policy network to solve the computation offloading problem of hybrid tasks in vehicle-side collaborative scenarios. The network structure is as follows: Figure 3 As shown, in addition to task information and service vehicle information, server information is added to the input sequence, and the output sequence is the vehicle number or server number that executes each task. The model's input consists of the task sequence X and SV to be scheduled, TV, and server information calculated using the Analytic Hierarchy Process (AHP), with the server denoted as Server.

[0144] (II) Model Training

[0145] The policy network is trained using the advantage action value algorithm, and the state space, action space, and reward are defined as follows:

[0146] (1) State space: It consists of task sequence, service vehicle sequence and server sequence. The task sequence has six attributes, namely the execution time of the task on the CPU, computational complexity, data volume, latency tolerance, task divisibility, and task priority. The attribute of the service vehicle is the computing resources available to the vehicle. The attribute of the server is also the computing resources available to the server.

[0147] (2) Action Space: The number of the task vehicle (TV) or service vehicle (SV) or server that executes each task. It is generated by the policy network. The set of actions is the unloading policy.

[0148] (3) Reward: The reward corresponding to the action performed. In order to meet the research goal of reducing the average latency of task execution and improving the success rate of task execution, the reward function is set as the reciprocal of the sum of the average latency of task execution and the number of failed tasks q multiplied by a weight factor λ, as shown in Equation (33).

[0149]

[0150] In equation (34), average_time is the average latency of all tasks, i.e.

[0151]

[0152] The multi-threaded parallel architecture of the A2C algorithm is as follows: Figure 4 As shown, the agent consists of two parts: a global network and several independent and parallel worker threads. Each worker independently interacts with its own environment, collects independent sample experience, and uses this experience to update its local Actor-Critic network. After all worker threads have completed their tasks, each episode synchronously updates the parameters of the global network.

[0153] During network training, the advantage function can be expressed as:

[0154] A(s t ,a t )=E[r t+1 +γV(s t+1 )-V(s t (35)

[0155] In the formula, A(·) represents the dominance function, E(·) represents the value function, and V(·) represents the state s. t The value function, s t Let a represent the hidden state at time step t. t Let r represent the action at time step t. t+1Let γ represent the reward for the action at time step t+1, and let γ represent the discount rate.

[0156] Initialize the parameters θ of the policy network and the parameters of the value function network. The formula for calculating the policy gradient can be expressed as follows:

[0157]

[0158] In the formula, T represents the batch size in the training dataset.

[0159] With the learning rate of the Actor network set to α, the update formula for the parameters θ of the Actor network can be expressed as:

[0160]

[0161] Set the learning rate of the Critic network to β, and set the parameters of the Critic network... The update formula is

[0162]

[0163] The network training process is shown in Table 2, where N is the total number of episodes.

[0164] Table 2 A2C Model Training Process

[0165]

[0166]

[0167] Furthermore, this embodiment also provides a vehicle-side collaborative computing task offloading device based on deep reinforcement learning, including a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements the aforementioned methods. The device processor includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from the memory unit into random access memory (RAM). Various programs and data required for device operation can also be stored in the RAM. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus. Multiple components in the device are connected to the I / O interfaces, including: input units, such as a keyboard, mouse, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network interface cards, modems, wireless transceivers, etc. The communication units allow the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks. The processing unit executes the various methods and processes described above, such as the methods described above. For example, in some embodiments, the aforementioned methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the aforementioned methods can be performed. Alternatively, in other embodiments, the CPU can be configured to perform the aforementioned methods by any other suitable means (e.g., by means of firmware). The functions described above can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SOCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0168] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning, characterized in that, Includes the following steps: S1, obtain the sequence of tasks to be executed, the initial service vehicle sequence, and the initial server sequence; S2, divide the task priority of the task sequence to be executed, and combine it with the attributes of the task sequence to be executed to obtain the current task sequence. The attributes of the task sequence include CPU execution time, computational complexity, data volume, latency tolerance, whether the task can be split, and task priority. S3, based on the current task sequence, service vehicle sequence, and server sequence, performs task scheduling using pre-built vehicle-to-vehicle collaborative computing models and vehicle-to-server collaborative computing models, and obtains the corresponding task offloading strategy using a sequence-to-sequence model based on IndRNN-AM. S4. Based on the current task offloading and scheduling strategy, obtain the current task sequence, service vehicle sequence, and server sequence, and use the Advantage Action Value algorithm to train the sequence-to-sequence model based on IndRNN-AM to obtain the final task offloading and scheduling strategy.

2. The method for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning according to claim 1, characterized in that, The scheduling methods for each task include local execution, complete uninstallation, and partial uninstallation.

3. The method for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning according to claim 2, characterized in that, When the scheduling method is partial unloading, based on the vehicle-to-vehicle cooperative computing model, the corresponding total task execution latency is: and In the formula, t i Let be the total execution delay of the i-th task. For the i-th task The latency of some tasks being executed locally on the mission vehicle. For the i-th task, 1- The total latency of partially unloading to the service vehicle is C. i The CPU execution time required to complete the i-th task, f l The computing power of the mission vehicle. This represents the waiting time before the i-th task is unloaded. Represents the 1- of the i-th task The transmission time for some parts uploaded to the service vehicle, This represents the execution time of the subtask of the i-th task on the service vehicle. D represents the return time of the result. i f represents the size of the input data. j Let R represent the computing power of the j-th service vehicle, η represent the ratio of output data volume to input data volume, and R be the computing power of the j-th service vehicle. l To determine the link transmission rate when offloading tasks to service vehicles, R j The link transmission rate when the service vehicle sends the results back to the TV.

4. The method for offloading vehicle-side collaborative computing tasks based on deep reinforcement learning according to claim 1, characterized in that, During the task scheduling process, a failure queue, a processing queue, a waiting queue, and a task sequence are used to record different task states.

5. The vehicle-side collaborative computing task offloading method based on deep reinforcement learning according to claim 4, characterized in that, Task scheduling is performed using pre-built vehicle-to-vehicle cooperative computing models and vehicle-to-server cooperative computing models. The specific process includes: Select the highest priority task from the task sequence; Assign a service vehicle, task vehicle, or server to the highest priority task and add it to the waiting queue of the assigned service vehicle, task vehicle, or server. When the initial computing resources of the assigned service vehicle, task vehicle, or server are greater than or equal to the first computing resources required by the highest priority task, the task allocation is successful. Then, it is determined whether the current available computing resources of the assigned service vehicle, task vehicle, or server are greater than or equal to the first computing resources. If yes, the highest priority task is added to the processing queue of the current service vehicle, task vehicle, or server and removed from the waiting queue. If no, the highest priority task enters the waiting state. When the task with the shortest remaining execution time in the current service vehicle, task vehicle, or server processing queue has been completed and the corresponding computing resources have been released, update the remaining execution time and total execution time of the highest priority task and all tasks in the processing queue. After the update is completed, determine whether the highest priority task has timed out: if yes, add the highest priority task to the failure queue; if no, continue to reassign the highest priority task to a service vehicle, task vehicle, or server.

6. The vehicle-side collaborative computing task offloading method based on deep reinforcement learning according to claim 5, characterized in that, Task scheduling is performed using pre-built vehicle-to-vehicle cooperative computing models and vehicle-to-server cooperative computing models. The specific process also includes: When the initial computing resources are less than the first computing resources, the task allocation fails, and the service vehicle, task vehicle, or server is reassigned to the highest priority task. If the task with the shortest remaining execution time in the current service vehicle, task vehicle, or server's processing queue has not been completed and released its corresponding computing resources, then continue to wait.

7. The vehicle-side collaborative computing task offloading method based on deep reinforcement learning according to claim 1, characterized in that, The sequence-to-sequence model based on IndRNN-AM is used as the policy network for the Advantage Action Value Algorithm. The state space of the Advantage Action Value Algorithm includes a task sequence, a service vehicle sequence, and a server sequence. The action space of the Advantage Action Value Algorithm includes the numbers of the task vehicles, service vehicles, or servers executing each task and is generated using the policy network. The reward function of the Advantage Action Value Algorithm is the reciprocal of the sum of the average delay of task execution and the number of weighted failed tasks.

8. The vehicle-side collaborative computing task offloading method based on deep reinforcement learning according to claim 7, characterized in that, The average delay is: In the formula, M represents the total number of tasks. For the i-th task The latency of some tasks being executed locally on the mission vehicle. For the i-th task, 1- The total latency of partially unloading to the service vehicle or server is t. i Let be the total execution delay of the i-th task.

9. The vehicle-side collaborative computing task offloading method based on deep reinforcement learning according to claim 1, characterized in that, The task priority division is achieved using the analytic hierarchy process (AHP).

10. A vehicle-side collaborative computing task offloading device based on deep reinforcement learning, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-9.

Images