Deep reinforcement learning-based cooperative parallel task unloading method and system in vehicle-mounted edge computing environment

Abstract

The invention discloses a collaborative parallel task unloading method and system based on deep reinforcement learning in a vehicle-mounted edge computing environment, and belongs to the cross technical field of reinforcement learning, intelligent traffic and the like. An Internet of Vehicles environment comprising a vehicle and roadside unit (RSU) is constructed, the position of the vehicle is tracked in real time through a mobile detection model, the residence time of the vehicle in the coverage range of the RSU is determined, and meanwhile, the priority is defined according to task attributes so as to generate an ordered task sequence. Afterwards, a single-RSU or multi-RSU calculation model is selected by integrating the residence time and task information, and the problem of cooperative parallel task unloading between RSUs is modeled as an optimization problem by taking the minimum total delay and total energy consumption of task unloading as an optimization target; and finally, establishing a Markov decision process model in combination with the task sequence and the optimization problem, and solving an optimal cooperative task unloading strategy for implementation. According to the method, the task unloading performance and efficiency are effectively improved.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of reinforcement learning and intelligent transportation, and specifically to a collaborative parallel task offloading method and system based on deep reinforcement learning in an in-vehicle edge computing environment. Background Technology

[0002] In recent years, the deep integration of deep reinforcement learning and edge computing technologies has driven the rapid development of research on task offloading in the Internet of Vehicles (IoV). Existing research mainly revolves around two paradigms: task offloading based on a single roadside unit (RSU) and cooperative task offloading based on multiple roadside units. Single-RSU task offloading methods rely on a single RSU to process tasks, but the high-speed movement of vehicles can easily lead to interruptions in task processing, and the resources of a single RSU are very limited, making it difficult to handle high-concurrency tasks. Although cooperative task offloading methods based on multiple RSUs solve the problem of insufficient resources in a single RSU by sharing resources among multiple RSUs, they generally do not consider the impact of parallel task execution on task latency. Furthermore, existing research also ignores the impact of task urgency on resource scheduling, lacking a task priority scheduling mechanism, resulting in resource allocation imbalances and task offloading failures.

[0003] At the key technology level, algorithms based on deep Q-networks (DQN) (such as QRMP-DQN) can optimize single-node task queues, but they face the problem of low policy convergence efficiency in high-dimensional action spaces; greedy algorithms can achieve fast decision-making, but they are limited to local optima and cannot achieve global resource optimization; traditional mobility detection models (such as LSTM) can capture the temporal features of vehicle trajectories, but they are insufficient in modeling the complex spatiotemporal dependencies between vehicles, resulting in prediction accuracy and real-time performance that cannot meet the requirements of dynamic unloading decision-making. Summary of the Invention

[0004] The main purpose of this invention is to address the shortcomings of existing methods by proposing a collaborative parallel task offloading method and system based on deep reinforcement learning in an in-vehicle edge computing environment.

[0005] This invention constructs a cooperative task offloading model in a vehicle-to-everything (V2X) environment. By designing a vehicle position detection module to detect the communication time between the vehicle and the RSU (Roadside Unit), and based on this communication time, parallel task processing is achieved, significantly reducing task processing time. Simultaneously, the task offloading problem in a dynamic V2X environment is modeled as an optimization problem with system efficiency as the objective, and this optimization problem is formulated as a Markov decision process. A deep reinforcement learning algorithm is introduced to achieve adaptive task offloading decision-making. This algorithm requires no prior knowledge; it directly learns the optimal strategy through interaction with the environment, thereby adapting to the complex and ever-changing V2X environment and task requirements.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, this invention proposes a collaborative parallel task offloading method based on deep reinforcement learning in an in-vehicle edge computing environment, comprising the following steps:

[0008] S1: Establish a vehicle-to-everything (V2X) environment that includes vehicles and roadside units (RSUs) based on the vehicle's mobility, the tasks generated by the vehicle, and the task unloading process.

[0009] S2: Using a mobile detection model, the vehicle's position is detected in real time in the vehicle-to-everything (V2X) environment of step S1, and the dwell time of the vehicle within the RSU coverage area is determined based on the vehicle's position.

[0010] S3: Define the priority of vehicle tasks based on task attributes, and sort the tasks to obtain a task sequence from high to low priority;

[0011] S4: Based on the dwell time and task information of the vehicle within the RSU coverage area, select a single RSU calculation model or a multi-RSU calculation model. Calculate the total latency and total energy consumption of the vehicle unloading task when it is executed on a single RSU or multiple RSUs, and model the collaborative parallel task unloading problem between RSUs as an optimization problem with the total latency and total energy consumption of task unloading as the optimization objective.

[0012] S5: Based on the vehicle's task sequence and the optimization problem described in step S4, establish a Markov decision process model to optimize the decision process;

[0013] S6: Using the Markov decision process model described in step S5, a reinforcement learning algorithm is employed to determine and implement the cooperative task offloading strategy.

[0014] Furthermore, the vehicle network environment in step S1 includes N vehicles and M roadside units (RSUs). Each RSU is equipped with an edge server to process vehicle tasks. Tasks are transmitted between RSUs via wireless communication. Each RSU has multiple vehicles within its coverage area, and each vehicle generates certain computing tasks. The task types include, but are not limited to, perception tasks, path planning tasks, and control tasks.

[0015] Furthermore, the motion detection model in step S2 consists of a multilayer perceptron, a graph attention network, a GRU encoder, and a GRU decoder, and the training process is as follows:

[0016] S2.1: Collect the coordinate information of the target vehicle over a period of time as input to the motion detection model, and expand the input dimension using a multilayer perceptron;

[0017] S2.2: Using the expanded results of the target vehicle and the expanded results of surrounding vehicles as input to the graph attention network, the spatial information of the target vehicle is extracted to obtain the spatial features e={e 1 ,e 2 ,...,e T}, where e T This represents the spatial information of the target vehicle at time T in history;

[0018] S2.3: Utilize the GRU encoder to encode the temporal information of spatial features, and then use the GRU decoder to predict the target vehicle position in the next K time steps;

[0019] S2.4: Update the motion detection model based on the prediction error loss function.

[0020] Furthermore, the priority formula for vehicle tasks is as follows:

[0021]

[0022] Among them, T stay P indicates the time a vehicle spends in the RSU. i Let represent the priority of the i-th task, where α and β are weighting coefficients and α + β = 1. d represents the CPU cycles required for task i generated by vehicle n at time t. n,i (t) represents the maximum tolerance time for the task generated by vehicle n at time t. The larger the CPU cycle requirement of the task and the shorter the maximum tolerance time of the task, the higher the priority of the task.

[0023] Furthermore, in step S4, the single RSU calculation model refers to the unloading task being executed on a single RSU. The vehicle needs to unload the task to the corresponding RSU, and after the RSU completes the calculation, it sends the calculation result back to the RSU. This process will generate communication delay and calculation delay. In vehicle edge computing, multiple vehicles unload tasks to the same RSU, which will generate corresponding queuing delay.

[0024] The multi-RSU computation model refers to the collaborative execution of offloading tasks across multiple RSUs. Communication and queuing latency are the same as for a single RSU, but due to vehicle movement, the current vehicle may be unable to complete the task computation. Therefore, part of the task needs to be wirelessly transmitted to another RSU for computation. Task execution and transmission are treated as parallel tasks to reduce computation latency. The computation latency is expressed as... Where t1 represents the time the vehicle spends within the current RSU coverage area. This represents the transmission delay between RSU m and RSU m' for task i generated by vehicle n at time t. This represents the computation delay in RSU m' for the remaining data of task i generated by vehicle n at time t. Indicates computation delay;

[0025] In both the single-RSU and multi-RSU computation models, the total delay of task i unloading at time t is the sum of communication delay, computation delay, and queuing delay.

[0026] Furthermore, the total energy consumption includes transmission energy consumption. And calculate energy consumption;

[0027] Computational energy consumption of single-RSU and multi-RSU computational models The formula is:

[0028]

[0029] Transmission energy consumption of a single RSU computational model The formula is:

[0030]

[0031] Transmission energy consumption of multi-RSU computational models The formula is:

[0032]

[0033] in, This is the power consumption for RSU m calculation. It is the computation delay of task i. It is the communication delay of task i. It is the energy consumption for transmission between vehicle n and RSU m. It is the transmission delay of task i generated at time t between RSU m and RSU m'. This represents the energy consumption transmitted between RSU m and RSU m'.

[0034] Furthermore, the optimization problem is as follows:

[0035]

[0036] The constraints are as follows:

[0037]

[0038] Where v(t) is the vehicle speed at time t, v max For maximum speed, T i all Let d be the execution time of task i. i N represents the maximum tolerance time for task i. queueFor the computational cost of the task on RSU, C RSU This represents the maximum computing power of RSU. Calculate the energy consumption generated by task i, where I represents the number of tasks, and E... max This represents the maximum energy consumption. Let t be the amount of data processed by the task at time t. Let U be the maximum amount of data for task i, T represent time, σ be the weight, and U be the weight. m (t) represents the average delay and energy consumption of tasks generated by vehicles within the coverage area of ​​RSU m at time t.

[0039] Furthermore, in the Markov decision process model of step S5, the state space is... Used to represent environmental information for in-vehicle edge computing, where v∈V represents the position coordinates of vehicle v at time t, where V represents a set of vehicles within the coverage area of ​​an RSU. This indicates that at time t, the vehicle currently contains task-related information including task priority and task data volume. This indicates the location and load status of the RSU where the vehicle is located at time t;

[0040] The action space is a(t) = {0, 1, ..., M}, which represents the decision of whether the RSU where the current vehicle is located should cooperate to complete the task at time t and the RSU object to cooperate with. Here, 0 means no cooperation and M means the RSU number to cooperate.

[0041] The reward function is the negative of the sum of the total delay and total energy consumption of completing all tasks up to time t.

[0042] Furthermore, in step S6, the Markov decision process model described in step S5 is used to train a global policy network using the A3C algorithm and deploy it on each RSU to generate the optimal task unloading decision in real time in the vehicle dynamic environment.

[0043] Secondly, this invention proposes a collaborative parallel task offloading system based on deep reinforcement learning in an in-vehicle edge computing environment, which is used to implement the aforementioned collaborative parallel task offloading method based on deep reinforcement learning in an in-vehicle edge computing environment.

[0044] The advantages of this invention are:

[0045] 1) Task completion rate optimization: Based on the vehicle movement detection model of GAT and GRU, the vehicle position is tracked in real time and the communication time is predicted to realize dynamic task collaboration between RSUs. At the same time, combined with a multi-dimensional task priority ranking mechanism, the rapid execution of urgent tasks is prioritized, which significantly improves the task completion rate and meets the needs of high timeliness scenarios.

[0046] 2) Delay and Energy Consumption Co-optimization: With minimizing task latency and energy consumption as the core objectives, a multi-objective optimization model was constructed, encompassing task offloading, execution latency, RSU load status, and resource allocation. Through intelligent decision-making algorithms, an optimal balance between task processing efficiency and energy consumption was achieved while ensuring service quality, providing users with a more efficient and energy-saving service experience. Attached Figure Description

[0047] Figure 1 This is a flowchart of the method of the present invention;

[0048] Figure 2 This is a diagram of the vehicle-mounted edge computing structure designed for this invention;

[0049] Figure 3 This is a schematic diagram of the collaborative parallel task delay calculation of the present invention;

[0050] Figure 4 This is an architecture diagram of the cooperative parallel task offloading method based on reinforcement learning of the present invention. Detailed Implementation

[0051] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0052] To make the objectives, technical solutions, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below.

[0053] Reference Figure 1-Figure 4 This invention provides a method for collaborative parallel task offloading in vehicle edge computing based on deep reinforcement learning. The structural diagram is shown below. Figure 1 As shown, the method includes:

[0054] S1: As Figure 2 As shown, a vehicle-to-everything (V2X) environment is considered, consisting of N vehicles and M roadside units (RSUs). Each RSU is connected to the vehicles and RSUs via wireless communication. Each RSU is represented as m∈{1,2,...,M}, and each vehicle is represented as n∈{1,2,...,N}. Each vehicle generates a computational task s in each time slot. n , where n represents the vehicle number.

[0055] S2: Considering the dynamic changes of vehicles in an vehicular edge computing environment, during task unloading, the vehicle may leave the current RSU's communication range before the task is completed, causing task unloading failure. This invention proposes a vehicle movement detection model, such as... Figure 3As shown. This model can detect the vehicle's position in real time, avoiding task unloading failures caused by vehicle movement during the unloading process. The specific steps are as follows:

[0056] S2.1: First, the motion detection model uses the vehicle's coordinate information over a historical period. As input to the model, the dimensionality of the input is expanded through two layers of Multilayer Perceptron (MLP), and the expanded output is given by the formula. The conclusion is as follows. Among them, This represents the extended encoding result of vehicle n at time t. Represents the coordinates of vehicle n at time t;

[0057] S2.2: Output after MLP processing The MLP processing results of the vehicle and its surrounding vehicles are used as input to a Graph Attention Network (GAT) to extract spatial information of vehicle n and obtain its spatial features. In GAT, each node learns the importance of nodes dynamically based on the features of its neighbors. After GAT processing, the feature representation of each vehicle not only integrates its own historical location information but also dynamically weights and fuses the important features of its neighboring vehicles, thereby capturing the spatial dependencies between vehicles.

[0058] S2.3: After processing by MLP and GAT, an input sequence containing T historical time points is obtained. The encoder processes the input features at each time step by step through a series of gated recurrent units (GRUs), ultimately outputting the hidden state. Where the hidden state at time t is This hidden state contains time-series information about the entire historical location.

[0059] S2.4: The decoder GRU uses the encoder's output. As contextual information, the hidden state is used to detect the vehicle's position over several future time intervals. Each time the decoder outputs a new position, it uses the current hidden state as input for the next time interval. In this way, the decoder can detect the vehicle's position over a certain period of time. The vehicle's position is determined by the formula... Output, where h T+k This represents the hidden state at future time steps k. Let W1 and b1 represent the learnable matrix and bias of the vertical coordinate position, respectively, and W0 and b0 represent the learnable matrix and bias of the horizontal coordinate position, respectively.

[0060] S2.5: After obtaining the vehicle's coordinates, the mean square error between the detected coordinates and the true coordinates is used as the loss, allowing the model to continuously optimize network parameters and gradually improve the accuracy of future position detection. The loss calculation formula is: Where N represents the number of vehicles and K represents the future time step.

[0061] S3: Each task has task attributes such as task data size and maximum tolerance time. Based on these task attributes and the dwell time of the vehicle in the RSU detected by the motion detection module, a priority is defined for each vehicle task.

[0062] S3.1: The task priority algorithm mainly determines the priority of tasks based on the following key rules, thereby ensuring that high-priority tasks are processed first, improving the overall efficiency and reliability of task unloading:

[0063] 1. Tasks with stricter time constraints have higher priority.

[0064] 2. Tasks with larger amounts of data have higher priority.

[0065] 3. Vehicles with longer dwell times have higher task priority.

[0066] The priority of task i is determined by the formula Determined, where T stay This represents the vehicle's dwell time in the RSU, as determined by the mobile monitoring module. A longer dwell time indicates a higher task priority. α and β are weighting coefficients, and α + β = 1. d represents the CPU cycles required for task i generated by vehicle n at time t. n,i (t) represents the maximum tolerance time for the task generated by vehicle n at time t. This means that the greater the CPU cycle requirement of the task and the shorter the maximum tolerance time, the higher the priority of the task.

[0067] S4: Before the task unloading is completed, if the vehicle does not leave the communication range of the current RSU, the unloading task is calculated on a single RSU; if the vehicle leaves the communication range of the current RSU and enters the communication range of another RSU, the unloading task is calculated collaboratively across multiple RSUs. Therefore, task unloading can be divided into two cases: single RSU calculation and multi-RSU collaborative calculation. Figure 4 As shown, the specific steps are as follows:

[0068] S4.1: When an unloaded task is executed on a single RSU, the RSU provides the computing resources to complete the task computation; the vehicle must first unload the task to the corresponding RSU. During this process, the communication delay caused by task i generated by vehicle n at time t is... Where ρ n,i (t) represents the amount of data generated by vehicle n for task i at time t, r V2R This indicates the transmission rate between the vehicle and the RSU;

[0069] S4.2: Considering that other tasks may be queued and waiting to be processed before the current task arrives at the corresponding RSU, the current task cannot start execution immediately. Therefore, queuing delay needs to be considered, specifically the queuing delay T of RSU m. wait Defined as Where N m queue This represents the total amount of data for the RSU m-queued tasks. This represents the computational power of RSU m;

[0070] S4.3: The execution of task i generated by vehicle n at time t in RSU m will result in a corresponding computational delay. From the formula The calculation yields the following result. Therefore, the total computational delay is... Because the size of the calculated result is much smaller than the original data, the delay in sending the result back can be ignored.

[0071] S4.4: When an unloading task is computed collaboratively across multiple RSUs, the computational resources provided by the multiple RSUs are used to jointly complete the task computation; the vehicle first unloads the task to the corresponding RSU, and the communication delays generated during this process... The calculation formula is the same as that in S4.1;

[0072] S4.5: S2 provides the vehicle's dwell time t1 within the corresponding RSU. Based on the RSU's computing power and dwell time, the amount of task data the corresponding RSU can complete and the remaining data to be computed can be calculated. After calculating the amount of unfinished task data, the corresponding transmission delay can be calculated. and processing delays in This represents the remaining data of task i generated by vehicle n at time t. The computation delay performed in RSU m' RSU m' represents the computational power, r R2R This indicates the transmission rate between RSUs. This represents the transmission delay of task i generated by vehicle n at time t between RSU m and RSU m'.

[0073] S4.6: The communication latency and queuing latency for collaborative computation between multiple RSUs are calculated in the same way as in S4.1 and S4.2. The difference is that when the current RSU is computing task data, the transmission of remaining data and the unloading of the RSU to compute the remaining data can be performed in parallel. Therefore, the total task computation latency is reduced. This parallelization strategy not only reduces latency but also improves task reliability, especially in dynamically changing connected vehicle environments. Therefore, the total latency of collaborative computation among multiple RSUs is reduced.

[0074] S4.7: After obtaining the various delays of the unloading task, the energy consumption for completing the unloading task can be calculated. The energy consumption is mainly divided into two parts: transmission energy consumption and computation energy consumption.

[0075] Transmission energy consumption refers to the energy required to transmit data between the vehicle and the RSU, and between RSUs themselves. Computational energy consumption refers to the energy required to complete a task. Computational energy consumption is expressed by the formula... The results, of which This refers to the computational power consumption of RSU m. The energy consumption for computational transmission is also related to whether the tasks are completed collaboratively. The calculation formula is shown below:

[0076]

[0077]

[0078] in This represents the energy consumption for transmission between vehicle n and RSU m. This represents the energy consumption transmitted between RSU m and RSU m', x i,r This represents the collaborative action of task i, with 0 indicating no collaboration. Therefore, the total energy consumption for task i can be defined as:

[0079] S4.8: Based on S4.6 and S4.7, the task unloading optimization problem is determined. The objective of the optimization problem is to minimize the average latency and energy consumption. We define the average latency and energy consumption of the tasks generated by the vehicle in Unity as follows:

[0080]

[0081] Then, the above problem is formalized as follows:

[0082]

[0083] The constraints are as follows:

[0084]

[0085] Where v(t) is the vehicle speed at time t, v max C1 represents the maximum speed, indicating that each vehicle's speed cannot exceed the maximum speed limit; T i all Let d be the execution time of task i.i Let C1 be the maximum tolerable time for task i, and C2 represent the execution time of each task, which cannot exceed the maximum tolerable time of the task. queue For the computational cost of the task on RSU, C RSU C3 represents the maximum computing power of the RSU, indicating that the computational load of tasks on the RSU cannot exceed the maximum computing power of the RSU; E i all Calculate the energy consumption generated by task i, where I represents the number of tasks, and E... max C4 represents the maximum energy consumption, indicating that the energy consumption generated by the task computation cannot exceed the maximum energy consumption. Let t be the amount of data processed by the task at time t. Let T represent the maximum data size for task i, T represent time, and C5 represent the data size processed by the task at each time step, which will not exceed the maximum data size for the task; σ represents the weight, and U... m (t) represents the average delay and energy consumption of tasks generated by vehicles within the coverage area of ​​RSU m at time t.

[0086] S5: Using the optimization problem described in S4, establish a Markov decision process model and optimize the decision process. The specific steps are as follows:

[0087] S5.1: Determine the state space

[0088] At time slot t, the entire vehicle-to-everything (V2X) network includes the vehicle, the RSU (Responsible Unit), and the task information generated by the vehicle, so it uses... Let the state space at time t be defined, where v∈V represents the vehicle's position coordinates at time t. This indicates that at time t, the vehicle currently contains task-related information such as task priority and task data volume. This indicates the location and load status of the RSU where the vehicle is located at time t.

[0089] S5.2: Determine the motion space

[0090] At time slot t, the action space represents the decision of whether the RSU to which the current vehicle belongs should cooperate to complete the task and the object to cooperate with. Therefore, the action space a(t) is defined as a(t)={0,1,...,M}, where 0 indicates no cooperation and M indicates the RSU number to cooperate with.

[0091] S5.3: Define the reward function

[0092] It is defined as the negative of the sum of delays and energy consumption for completing all tasks up to time t, denoted by the symbol r(t), r(t) = -(T total +E total ).

[0093] S6: Using the Markov decision process model mentioned in S5, the A3C algorithm is employed to determine the task offloading collaboration decision and implement it, such as... Figure 3 As shown. The specific steps are as follows:

[0094] S6.1: To address the real-time decision optimization problem of task offloading in dynamic vehicle-to-everything (V2X) scenarios, this embodiment employs the Asynchronous Advantage Actor-Critic (A3C) algorithm to construct a distributed training framework. Through the collaborative optimization of the policy network (Actor) and the value network (Critic), as well as a multi-threaded asynchronous parameter update mechanism, efficient learning of vehicle task offloading strategies is achieved.

[0095] S6.2: First, initialize the parameters θ of the global policy network and the global value function θ. V Establish a state space and action space that cover dynamic scenarios of vehicle networking.

[0096] S6.3: Create multiple asynchronous worker threads, each thread synchronously copies parameters θ and θ from the global network. v The thread receives the current environment state s from the local Actor network and Critic network. In each time slot t, the thread receives the current environment state s from the local Actor network. t The probability distribution of generating task unloading actions is π = (a t |s t ;θ), where θ represents the parameters of the local Actor network, and action a is selected based on probability sampling. t And execute it. After the action is executed, the environment will provide timely feedback, including a reward and the next state. t+1 .

[0097] S6.4: Each thread records the trajectory {s} t ,a t ,r t ,s t+1}, by calculating cumulative discount rewards The subsequent rewards are weighted and summed according to the discount factor γ, where γ∈[0,1] is used to adjust the weight of future rewards.

[0098] S6.5: Calculate the dominance function A = R t -V(s t ,θ v ), quantify the selected action a t Additional benefits for the average strategy. Based on the advantage function, minimize the Actor network loss function L. π (θ)=-log π (a t |s t ;θ)(Rt -V(s t ;θ v This drives the probability of actions to shift towards a higher advantage.

[0099] S6.6: Minimize the loss function L of the Critic network V (θ v )=(R t -V(s t ;θ v )) 2 To optimize the accuracy of the Critic network in estimating state value.

[0100] S6.7: Each thread will compute the policy gradient locally. and Asynchronous upload to the global network. The global network uses a gradient accumulation mechanism to synchronously update parameters according to the learning rate α.

[0101] S6.8: Repeat the training process multiple times until the average reward fluctuation of the policy network stabilizes or the preset maximum number of iterations is reached. Finally, deploy the converged global network parameters to the edge nodes of the vehicle network to enable vehicles to generate optimal task unloading decisions in real time in dynamic environments, maximizing long-term system rewards.

[0102] The principle of the A3C algorithm described above for solving the cooperative task unloading strategy problem is a well-known approach in the field. Those skilled in the art can also use other algorithms to solve the above optimization problem.

[0103] In autonomous driving scenarios, perception, path planning, and control tasks are the core of vehicle decision-making and operation. Their real-time requirements and inter-task dependencies directly determine the task offloading strategy, as detailed below:

[0104] Perception tasks require data acquisition, object detection and classification, and environmental perception. These tasks typically require a large amount of computing resources and cannot be completed by the vehicle itself. They are best offloaded to edge servers for processing. When such tasks occur, the vehicle will make an offloading decision based on the offloading method described above.

[0105] The path planning problem involves a vehicle determining a safe and optimal driving route. It considers various factors such as road conditions, traffic rules, obstacles, and dynamic environmental changes. Path planning tasks include global path planning and local path planning. Global path planning has lower real-time requirements and is suitable for offloading to the edge or cloud for processing; while local path planning, due to its need to frequently respond to environmental changes, may be more suitable for execution on the vehicle's computing unit. When such tasks arise, the vehicle will make appropriate decisions based on the offloading method described above.

[0106] Control tasks involve adjusting the vehicle's status in real time based on the path planning results, including speed control and steering control. These control tasks directly impact vehicle safety and must respond within a very short timeframe. Due to their extremely high real-time requirements, control tasks are typically processed locally on the vehicle. When such tasks occur, the vehicle will make a decision to execute them locally, based on the offloading method described above.

[0107] This invention is not limited to the three types of tasks mentioned above.

[0108] Based on the same inventive concept, this embodiment also provides a collaborative parallel task offloading system based on deep reinforcement learning in an in-vehicle edge computing environment, the system comprising:

[0109] The vehicle network environment establishment module is used to establish a vehicle network environment including the vehicle and the roadside unit (RSU) based on the vehicle's mobility, the tasks generated by the vehicle, and the task offloading process.

[0110] The mobile detection model module is used to detect the location of vehicles in real time in a vehicle-to-everything (V2X) environment and determine the dwell time of the vehicle within the RSU coverage area based on the vehicle's location.

[0111] The task priority sorting module is used to define the priority of vehicle tasks according to task attributes and sort the tasks to obtain a task sequence from high to low priority.

[0112] The task unloading modeling module is used to select a single RSU calculation model or a multi-RSU calculation model based on the vehicle's dwell time and task information within the RSU coverage area. It calculates the total latency and total energy consumption of the vehicle unloading task when it is executed on a single RSU or multiple RSUs, and models the collaborative parallel task unloading problem between RSUs as an optimization problem with the total latency and total energy consumption of task unloading as the optimization objective.

[0113] The decision module is used to build a Markov decision process model based on the optimization problem constructed by the vehicle's task sequence and task unloading model, and to optimize the decision process. It uses reinforcement learning algorithms to determine and implement the cooperative task unloading strategy.

[0114] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments; the implementation methods of the remaining modules will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0115] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0116] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A collaborative parallel task offloading method based on deep reinforcement learning in an in-vehicle edge computing environment, characterized in that, Includes the following steps: S1: Establish a vehicle-to-everything (V2X) environment that includes vehicles and roadside units (RSUs) based on the vehicle's mobility, the tasks generated by the vehicle, and the task unloading process. S2: Using a mobile detection model, the vehicle's position is detected in real time in the vehicle-to-everything (V2X) environment of step S1, and the dwell time of the vehicle within the RSU coverage area is determined based on the vehicle's position. S3: Define the priority of vehicle tasks based on task attributes, and sort the tasks to obtain a task sequence from high to low priority; S4: Based on the dwell time and task information of the vehicle within the RSU coverage area, select a single RSU calculation model or a multi-RSU calculation model. Calculate the total latency and total energy consumption of the vehicle unloading task when it is executed on a single RSU or multiple RSUs, and model the collaborative parallel task unloading problem between RSUs as an optimization problem with the total latency and total energy consumption of task unloading as the optimization objective. S5: Based on the vehicle's task sequence and the optimization problem described in step S4, establish a Markov decision process model to optimize the decision process; S6: Using the Markov decision process model described in step S5, a reinforcement learning algorithm is employed to determine and implement the cooperative task offloading strategy.

2. The collaborative parallel task offloading method based on deep reinforcement learning in a vehicle-mounted edge computing environment according to claim 1, characterized in that, The vehicle network environment in step S1 is as follows: it includes N vehicles and M roadside units (RSUs). Each RSU is equipped with an edge server to process vehicle tasks. RSUs transmit tasks to each other via wireless communication. Each RSU has multiple vehicles within its coverage area, and each vehicle generates a certain amount of computing tasks.

3. The collaborative parallel task offloading method based on deep reinforcement learning in a vehicle-mounted edge computing environment according to claim 1, characterized in that, The motion detection model in step S2 consists of a multilayer perceptron, a graph attention network, a GRU encoder, and a GRU decoder. The training process is as follows: S2.1: Collect the coordinate information of the target vehicle over a period of time as input to the motion detection model, and expand the input dimension using a multilayer perceptron; S2.2: Using the expanded results of the target vehicle and the expanded results of surrounding vehicles as input to the graph attention network, the spatial information of the target vehicle is extracted to obtain the spatial features e={e 1 ,e 2 ,...,e T }, where e T This represents the spatial information of the target vehicle at time T in history; S2.3: Utilize the GRU encoder to encode the temporal information of spatial features, and then use the GRU decoder to predict the target vehicle position in the next K time steps; S2.4: Update the motion detection model based on the prediction error loss function.

4. The collaborative parallel task offloading method based on deep reinforcement learning in a vehicle-mounted edge computing environment according to claim 1, characterized in that, The priority formula for vehicle tasks is as follows: Among them, T stay P indicates the time a vehicle spends in the RSU. i Let represent the priority of the i-th task, where α and β are weighting coefficients and α + β = 1. d represents the CPU cycles required for task i generated by vehicle n at time t. n,i (t) represents the maximum tolerance time for the task generated by vehicle n at time t. The larger the CPU cycle requirement of the task and the shorter the maximum tolerance time of the task, the higher the priority of the task.

5. The collaborative parallel task offloading method based on deep reinforcement learning in a vehicle-mounted edge computing environment according to claim 1, characterized in that, In step S4, the single RSU calculation model refers to the unloading task being executed on a single RSU. The vehicle needs to unload the task to the corresponding RSU. After the RSU completes the calculation, it sends the calculation result back to the RSU. This process will generate communication delay and calculation delay. In vehicle-mounted edge computing, when multiple vehicles offload tasks to the same RSU, queuing delays will occur. The multi-RSU computation model refers to the collaborative execution of offloading tasks across multiple RSUs. Communication and queuing latency are the same as for a single RSU, but due to vehicle movement, the current vehicle may be unable to complete the task computation. Therefore, part of the task needs to be wirelessly transmitted to another RSU for computation. Task execution and transmission are treated as parallel tasks to reduce computation latency. The computation latency is expressed as... Where t1 represents the time the vehicle spends within the current RSU coverage area. This represents the transmission delay between RSU m and RSU m' for task i generated by vehicle n at time t. This represents the computation delay in RSU m' for the remaining data of task i generated by vehicle n at time t. Indicates computation delay; In both the single-RSU and multi-RSU computation models, the total delay of task i unloading at time t is the sum of communication delay, computation delay, and queuing delay.

6. The collaborative parallel task offloading method based on deep reinforcement learning in a vehicle-mounted edge computing environment according to claim 1 or 5, characterized in that, The total energy consumption mentioned includes transmission energy consumption. And calculate energy consumption; Computational energy consumption of single-RSU and multi-RSU computational models The formula is: Transmission energy consumption of a single RSU computational model The formula is: Transmission energy consumption of multi-RSU computational models The formula is: in, This is the computational power consumption of RSU m, T i compute T is the computation delay of task i. i upload It is the communication delay of task i. It is the energy consumption for transmission between vehicle n and RSU m. It is the transmission delay of task i generated at time t between RSU m and RSU m'. This represents the energy consumption transmitted between RSU m and RSU m'.

7. The collaborative parallel task offloading method based on deep reinforcement learning in a vehicle-mounted edge computing environment according to claim 1, characterized in that, The optimization issues are as follows: The constraints are as follows: C1:v(t)≤v max C2:T i all ≤d i C3:N queue ≤C RSU C6:σ∈(0,1) Where v(t) is the vehicle speed at time t, v max For maximum speed, T i all Let d be the execution time of task i. i N represents the maximum tolerance time for task i. queue For the computational cost of the task on RSU, C RSU This represents the maximum computing power of RSU. Calculate the energy consumption generated by task i, where I represents the number of tasks, and E... max This represents the maximum energy consumption. Let t be the amount of data processed by the task at time t. Let U be the maximum amount of data for task i, T represent time, σ be the weight, and U be the weight. m (t) represents the average delay and energy consumption of tasks generated by vehicles within the coverage area of ​​RSU m at time t.

8. The collaborative parallel task offloading method based on deep reinforcement learning in a vehicle-mounted edge computing environment according to claim 1, characterized in that, In the Markov decision process model of step S5, the state space is: Used to represent environmental information for in-vehicle edge computing, where v∈V represents the position coordinates of vehicle v at time t, where V represents a set of vehicles within the coverage area of ​​an RSU. This indicates that at time t, the current vehicle contains task-related information including task priority and task data volume, s t RSU This indicates the location and load status of the RSU where the vehicle is located at time t; The action space is a(t) = {0, 1, ..., M}, which represents the decision of whether the RSU where the current vehicle is located should cooperate to complete the task at time t and the RSU object to cooperate with. Here, 0 means no cooperation and M means the RSU number to cooperate. The reward function is the negative of the sum of the total delay and total energy consumption of completing all tasks up to time t.

9. The collaborative parallel task offloading method based on deep reinforcement learning in a vehicle-mounted edge computing environment according to claim 1, characterized in that, In step S6, the Markov decision process model described in step S5 is used to train a global policy network using the A3C algorithm and deploy it on each RSU to generate the optimal task unloading decision in real time in the vehicle dynamic environment.

10. A collaborative parallel task offloading system based on deep reinforcement learning in an in-vehicle edge computing environment, used to implement the collaborative parallel task offloading method based on deep reinforcement learning in an in-vehicle edge computing environment as described in claim 1, characterized in that, The system includes: The vehicle network environment establishment module is used to establish a vehicle network environment including the vehicle and the roadside unit (RSU) based on the vehicle's mobility, the tasks generated by the vehicle, and the task offloading process. The mobile detection model module is used to detect the location of vehicles in real time in a vehicle-to-everything (V2X) environment and determine the dwell time of the vehicle within the RSU coverage area based on the vehicle's location. The task priority sorting module is used to define the priority of vehicle tasks according to task attributes and sort the tasks to obtain a task sequence from high to low priority. The task unloading modeling module is used to select a single RSU calculation model or a multi-RSU calculation model based on the vehicle's dwell time and task information within the RSU coverage area. It calculates the total latency and total energy consumption of the vehicle unloading task when it is executed on a single RSU or multiple RSUs, and models the collaborative parallel task unloading problem between RSUs as an optimization problem with the total latency and total energy consumption of task unloading as the optimization objective. The decision module is used to build a Markov decision process model based on the optimization problem constructed by the vehicle's task sequence and task unloading model, and to optimize the decision process. It uses reinforcement learning algorithms to determine and implement the cooperative task unloading strategy.

Images