A service offloading and migration method for vehicle edge computing

Abstract

The present invention relates to the field of wireless communication technology, and specifically to a service offloading and migration method for vehicle edge computing, including building a system model, specifically including the following processes: establishing a network model to simulate the scenario in which a vehicle offloads computing tasks to different edge servers and performs real-time computing during movement; establishing a task model to manage and schedule vehicle computing tasks to be executed; establishing a communication model to describe the process of a vehicle communicating with a base station through a wireless channel. This technical solution studies the solution to the joint computing offloading and migration optimization problem for multiple vehicles and multiple edge server scenarios in an ultra-dense vehicle edge computing environment. This solution uses Lyapunov function optimization technology to convert the complex collaborative computing offloading and migration joint optimization problem into an average delay optimization problem under a long-term migration cost constraint; then, a COMA-based multi-agent deep reinforcement learning algorithm is used to obtain the optimal solution.

Description

Technical Field

[0001] The present invention relates to the field of vehicle networking technology, and in particular to a service unloading and migration method for vehicle edge computing. Background Art

[0002] The Internet of Vehicles (IoV) refers to a dynamic mobile communication system that enables vehicles to communicate with the public network through interactions between vehicles, roads, people, and sensor devices. This has spawned a series of computationally intensive and delay-sensitive vehicle applications, such as autonomous driving, pedestrian recognition, and path planning. These applications are mainly used to enhance driving safety, improve traffic efficiency, and provide entertainment services. However, due to the limited computing and storage resources of the vehicle itself, it is difficult to meet the needs of the above applications. Cloud computing uses advanced communication technology and has abundant computing resources to assist in task offloading. Vehicles can either process computing tasks locally or offload tasks to the cloud. However, as the number of connected devices increases, the number of computing tasks also increases, which inevitably leads to network congestion. Therefore, offloading computing tasks to the cloud center for processing is difficult to meet the needs of delay-sensitive tasks.

[0003] Vehicle Edge Computing (VEC) sinks storage and computing resources to the vicinity of the vehicle, which can effectively reduce the response time of tasks and reduce the reverse network load. Although VEC has great advantages in processing ultra-intensive and delay-sensitive applications, it still faces many challenges in the face of highly dynamic IoV environments: (1) Due to the high mobility of vehicles, the vehicle may drive out of the coverage of the current edge server (ES) before receiving the task processing results, and the continuity of task processing cannot be guaranteed; (2) The computing resources of ES are relatively abundant compared to the local vehicle, but the computing resources of ES are limited and unevenly distributed, and can only perform a limited number of computing tasks; (3) When the overall task is offloaded, because some ES are too far away or have low computing power, there are fewer resources available for IoV applications, which cannot meet their timeliness, resulting in limited overall performance optimization.

[0004] On the one hand, vehicle mobility poses considerable challenges to VEC. As vehicles move between different geographical locations, migration services may be required to track the vehicles to ensure service continuity. Deciding whether to migrate services and where to migrate to is a problem that needs to be solved in each time slot. Failure to migrate services may result in service interruptions when vehicles are moving, and frequent migration of services may increase migration costs. Due to the mobility of vehicles and the complex trade-off between migration costs and transmission delays, making optimal decisions is a challenge.

[0005] On the other hand, computation offloading strategies have been widely studied. For some complex in-vehicle applications such as pedestrian recognition, such applications can be divided into a set of interdependent subtasks. The order of subtask execution is described as a directed acyclic graph (DAG). The nodes of the DAG represent subtasks, and the directed edges represent the dependencies between subtasks. By dividing the latency-sensitive IoV tasks, the task granularity becomes smaller and the demand for computing resources is reduced, thus providing more resource allocation options for computing tasks. However, most of the current work offloads part of the computing tasks to the same ES for execution without considering the collaborative processing of multiple ESs, which has a significant impact on latency-sensitive tasks. This is because: (1) The computing resources of the ES are limited, and offloading to one ES will increase the response time of the task. Subtasks with the same priority can be offloaded to multiple edge servers for execution; (2) The priorities of subtasks are different, and subtasks with higher priorities can be assigned to ESs with more abundant computing resources.

[0006] Therefore, in the IoV environment, efficient computing task offloading and migration strategies are very important for resource utilization as well as the reliability and stability of task execution. Summary of the invention

[0007] In order to overcome the above-mentioned technical problems, the purpose of the present invention is to provide a service offloading and migration method for vehicle edge computing, which solves the problem of joint computing offloading and migration optimization in the current multi-vehicle and multi-edge server scenarios.

[0008] The purpose of the present invention can be achieved through the following technical solutions:

[0009] A service unloading and migration method for vehicle edge computing includes the following steps:

[0010] S1. Build a system model, including the following processes:

[0011] S11. Establish a network model to simulate the scenario where the vehicle offloads computing tasks to different edge servers and performs real-time computing during movement;

[0012] S12, establishing a task model for managing and scheduling vehicle computing tasks to be executed;

[0013] S13, establishing a communication model for describing a process in which a vehicle communicates with a base station through a wireless channel;

[0014] S14, establishing a computing delay model for evaluating the delay of executing computing tasks locally in the vehicle and offloading the computing tasks to the edge server for execution;

[0015] S15, calculating the actual completion time of the task;

[0016] S16. Set task constraints to ensure the order and timeliness of computing tasks during execution;

[0017] S2. For the above system model, plan the optimization problem and introduce the Lyapunov function to transform the optimization problem, which specifically includes the following processes:

[0018] S21. Calculate the boundary of Lyapunov drift;

[0019] S22, calculating the drift penalty boundary;

[0020] S3. Combine the Lyapunov function and the COMA architecture to obtain the Ly-COMA model. Based on the Ly-COMA model, the vehicle computing tasks are distributed to multiple edge servers for processing, realizing a high-efficiency, low-latency distributed service offloading and migration strategy.

[0021] Furthermore, the establishment of the network model in step S11 specifically includes:

[0022] Set N BSs (base stations), N = {1, 2, ..., n, ..., N}, n∈N; M ESs (edge ​​servers) M = {1, 2, 3, ..., m, ..., M}, m∈M; V vehicles, V = {1, 2, 3, ..., v, ... V}, v∈V; and add a time slot model T, T = {1, 2, 3, ..., t, ..., T}, and perform continuous time sampling at the same time interval;

[0023] The vehicle computing task includes multiple subtasks. In each time slot t, the vehicle offloads the subtasks to different ESs for execution. As the BS load changes, the available resources of the ES will also change. Each subtask can be executed locally in the vehicle or offloaded to the ES for execution.

[0024] Furthermore, in step S12, establishing the task model specifically includes:

[0025] Assume that each vehicle has a computationally intensive application that contains a series of subtasks with time series dependencies. At time t, the vehicle v The application can be represented using a directed acyclic graph (DAG) as follows: in, represents the set of I subtasks for the v-th vehicle, Representation Node and The dependencies between them are that the initial task has no predecessor task, and the terminal task has no successor task;

[0026] Each subtask It can be expressed as in Representation Task The data size, Representation Task The number of CPU cycles required, Representation Task The result size of

[0027] vehicle v The task has a delay tolerance time d v ,The task must be completed within the specified delay tolerance time, otherwise the task execution will fail.

[0028] Furthermore, the establishment of the communication model in step S13 specifically includes:

[0029] The vehicle communicates with the BS through a wireless channel, and the wireless channel follows the Rayleigh distribution during the wireless communication process;

[0030] Vehicle at time t v Connect BS n The signal-to-noise ratio (SINR) can be expressed as:

[0031]

[0032] in, Indicates BS n The transmission power, Indicates ES n The distance between the vehicle and the vehicle, α represents the path loss coefficient, Indicates that the fast fading factor follows a Gaussian distribution, represents the channel thermal noise power, represents the interference set of other BSs;

[0033] Vehicle at time t v With BS n The transmission rate between It can be expressed as:

[0034]

[0035] Among them, B represents the bandwidth of the channel, and the task at time t is Wireless transmission to BS n Delay It can be expressed as:

[0036]

[0037] Different ESs are connected by wires. The wired transmission delay is calculated based on the static hop count. The transmission delay between different ESs It can be expressed as:

[0038]

[0039] Among them, ξ represents the delay between single hops, It represents the number of hops between BSn through wireless access and ESm providing computing services;

[0040] if yes The direct parent node of The calculation results are transmitted to If two nodes execute at different locations, additional communication overhead will be incurred No distinction is made between uplink and downlink bandwidth. It can be expressed as:

[0041]

[0042] Indicates vehicle v The delay of returning the exit task result to the vehicle. If the end task is executed locally in the vehicle, no transmission delay is required. Otherwise, It can be expressed as:

[0043]

[0044] Furthermore, the establishment of the calculation delay model in step S11 specifically includes:

[0045] Calculate tasks separately The latency of executing tasks locally in the vehicle The latency of offloading to ES, where:

[0046] Task The computation latency performed locally in the vehicle is expressed as:

[0047]

[0048] in, It is a vehicle v CPU frequency;

[0049] Task In ES m The computation delay on It is expressed as:

[0050]

[0051] in, represents ES at time t m The number of CPU cycles required for the queue on Representation Task Migration strategy

[0052] when When indicating a task Migrated to ES m′ ,m≠m',m'∈M;when When The remaining tasks are still in ES m Execute on; if the task The remaining tasks are migrated to ES m′ , the computation delay is expressed as:

[0053]

[0054] Furthermore, the actual completion time of the task is calculated in step S15, which specifically includes:

[0055] vehicle v Multiple subtasks can be offloaded to different ES for execution, while computing tasks The actual completion time must meet the following conditions:

[0056] ① If the task is executed on ES, the data must have been transferred to the ES;

[0057] ② All directly preceding task nodes are completed;

[0058] ③、The task queue on ES is executed;

[0059] Therefore the task Start time It can be expressed as:

[0060]

[0061] when Time Task The transmission time is 0;

[0062] Representation Task The results of all predecessor tasks are passed to The latest completion time:

[0063]

[0064] Among them, pre v,i Representation Task The set of immediate predecessor nodes, Indicates the predecessor task End time of

[0065] Task The completion time is the actual start time plus the task processing time. According to formulas (7)(8)(9)(10), It can be expressed as:

[0066]

[0067] Therefore, task K v The actual completion time is:

[0068]

[0069] Furthermore, the successful execution of the vehicle application in step S16 must satisfy the following constraints:

[0070] ①. The time it takes to transmit the result of the exit task back to the vehicle cannot be greater than the vehicle's delay tolerance d. v , the constraint can be expressed as:

[0071]

[0072] ②If the task yes The direct parent node of Execution can only be executed after completion That is to say has a higher priority than Priority ω v,i It can be expressed as:

[0073]

[0074] Among them, succ v,i It's a task The set of immediate successor tasks;

[0075] Recursively exit from a task Traverse the DAG to calculate the priority of each node and exit the task Priority ω v,I for:

[0076]

[0077] Therefore, task G v Must satisfy ω during execution v,i >ω v,j .

[0078] Furthermore, the boundary of calculating the Lyapunov drift in step S21 specifically includes:

[0079] Constructing a vehicle using Lyapunov functions v Virtual migration cost insufficient queue Now assume that the first time slot y v (1) = 0, the migration cost is insufficient and the queue backlog evolves as follows:

[0080]

[0081] Among them, y v,i (t) represents the task at time t The distance between the actual migration cost deviation and the available migration cost; v The length of (t) is defined as the deviation between the actual migration cost and the available migration cost, which can be understood as the degree to which the constraint condition is satisfied; βψ represents the migration cost allowed in a time slot;

[0082] To jointly control offloading and migration, the total queue backlog is expressed as:

[0083]

[0084] Definition 1: The Lyapunov function L(Y(t)) and the Lyapunov drift ΔL(Y(t)) are defined as follows:

[0085]

[0086] ΔL(Y(t))=Ε[L(Y(t+1))-L(Y(t))|Y(t)] (20)

[0087] Definition 2: The goal of the Lyapunov algorithm is to minimize the bound of the drift penalty expression ΘΔL(Y(t)) for each time slot, which is defined as follows:

[0088]

[0089] Among them, Z>0 is the importance weight for measuring the penalty;

[0090] Lemma 1: The bound of the Lyapunov drift ΔL(Y(t)) can be expressed as:

[0091]

[0092] Proof: By Taking the queues of V on both sides at the same time, we can get:

[0093]

[0094] in,

[0095] Furthermore, the boundary of the drift penalty calculated in step S22 specifically includes:

[0096] Lemma 2: The bound of the drift penalty ΘΔL(Y(t)) can be expressed as:

[0097]

[0098] Proof: Substituting formula (22) of Lemma 1 into formula (21) yields:

[0099]

[0100] Furthermore, the distributed service unloading and migration method based on Ly-COMA in step S3 specifically includes:

[0101] Based on the Actor-Critic network, the Counterfactual Multi-Agent (COMA) policy gradient algorithm is introduced;

[0102] Actor modules are deployed in each ES. At each time slot t, the input of the Actor network includes the status of vehicles, edge servers, communications, migration loss queues, and vehicle mission information. At this time, a probability distribution is output according to the state of the environment, and then the sampled actions are applied to the environment. The environment gives the agent (ES m )A reward and the next state At this time, the Critic network evaluates the environment and gives back the reward to the Actor;

[0103] In the single-user scenario, the parameter θ of the Actor network is m,actor TD error update calculated by the Critic network;

[0104] For multi-user scenarios, the Q value Q(S t ,A t ) contains the global state and behavior; the global state S t It can be expressed as in represents ES at time t m The state of the global behavior A t It can be expressed as in Moment ES m behavior; for ES m In terms of keeping ES m Other than unchanged, then ES m The advantage function at time slot t can be expressed as:

[0105]

[0106] in, Indicates ESm The strategy function of the Actor network;

[0107] according to To calculate ES m The calculated baseline can reflect the current agent (ES m ) Contribution to the global reward;

[0108] Each Actor module selects unloading and migration decisions based on the local state in a distributed manner, and then the centralized Critic network guides the Actor module to learn the optimal strategy by calculating the contribution of the Actor module to the global reward. In order to improve the stability of the algorithm, the centralized Critic network also introduces a target Q value network. At the same time, in order to improve learning efficiency, all Actor networks share network parameters, use the same network structure, and adopt batch input for training. In each training stage, ε, ES m With parameters Actor network selection action and will Store it in the experience replay buffer D;

[0109] Calculate the reward for step j according to the variant TD algorithm -TD(λ):

[0110]

[0111] Therefore, the target Q value of the Critic network can be expressed as:

[0112]

[0113] Among them, γ∈[0,1] represents the discount factor of the reward, V(S t ) represents the value function of the centralized Critic network; the parameters θ of the Critic network are updated using the gradient descent method critic , the Loss function L of the Critic network t (θ cr itic) is represented as follows:

[0114] L t (θ critic )=(y (λ) -Q(S t ,A t )) 2 (28)

[0115] Parameters of the target critic network Every C steps and θ critic Synchronize once;

[0116] Then ESm According to the advantage function of formula (25), the policy gradient algorithm is used to update the parameters θ of the Actor network. m,actor :

[0117]

[0118] Beneficial effects of the present invention:

[0119] Compared with traditional methods, this technical solution studies the joint computing offloading and migration optimization problems for multi-vehicle and multi-edge server scenarios in an ultra-dense vehicle edge computing environment. This solution uses Lyapunov function optimization technology to convert the complex collaborative computing offloading and migration joint optimization problem into an average delay optimization problem under a long-term migration cost constraint; then, a COMA-based multi-agent deep reinforcement learning algorithm is used to obtain the optimal solution, thereby achieving a more efficient and stable vehicle computing task offloading and migration strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] The present invention will be further described below in conjunction with the accompanying drawings.

[0121] Figure 1 It is a system model diagram in the present invention;

[0122] Figure 2 It is an algorithm convergence diagram of the distributed service offloading and migration method based on Ly-COMA in the present invention;

[0123] Figure 3 is a graph showing the relationship between the number of different tasks and the average delay in the present invention;

[0124] Figure 4 It is a relationship diagram between different ES numbers and average delay in the present invention;

[0125] Figure 5 is a graph showing the relationship between different numbers of vehicles and average delay in the present invention;

[0126] Figure 6 It is a flow chart of the distributed service unloading and migration method based on Ly-COMA in the present invention. DETAILED DESCRIPTION

[0127] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0128] like Figure 6The figure shows a flow chart of a service offloading and migration method for vehicle edge computing. The implementation of this method in this embodiment is based on the following environment: This method is implemented using Pytorch 1.12.1 and Python 3.7. The hardware environment is a GPU-based server with 32GB 1600MHz DDR3, 2.8GHz Intel Core i9 and 2T memory. In the VEC ultra-dense computing scenario, multiple vehicles and base stations supporting MEC are included, with a density of [10,50] BSs per square kilometer. The BS can provide wireless access for vehicles within a radius of 150m. Set λ in the TD(λ) algorithm to 0.8, and use mini-batch for training. The size of each batch is 32, the learning rate of the Actor network is 0.0001, and the parameters of the Critic network are set to 0.001.

[0129] The service offloading and migration method for vehicle edge computing in this solution mainly includes the following steps:

[0130] S1. Build a system model, such as Figure 1 As shown, the specific process includes the following:

[0131] S11. Establish a network model to simulate the scenario where the vehicle offloads computing tasks to different edge servers and performs real-time computing during movement;

[0132] S12, establishing a task model for managing and scheduling vehicle computing tasks to be executed;

[0133] S13, establishing a communication model for describing a process in which a vehicle communicates with a base station through a wireless channel;

[0134] S14, establishing a computing delay model for evaluating the delay of executing computing tasks locally in the vehicle and offloading the computing tasks to the edge server for execution;

[0135] S15, calculating the actual completion time of the task;

[0136] S16. Set task constraints to ensure the order and timeliness of computing tasks during execution;

[0137] S2. For the above system model, we plan the optimization problem and introduce the Lyapunov function to transform the optimization problem. The Lyapunov optimization method can transform the original complex problem into a simpler and easier to handle problem. This transformation not only helps us better understand the dynamic characteristics of the system, but also provides an effective strategy to optimize system performance, thus providing a solid theoretical foundation for subsequent service unloading and migration strategies. The specific process includes the following:

[0138] S21. Calculate the boundary of Lyapunov drift. Lyapunov drift refers to the speed at which the system state evolves over time. In this step, we need to calculate the Lyapunov drift value of the system model under different states, which helps us understand the dynamic behavior and stability of the system model under different conditions.

[0139] S22. Calculate the boundary of drift penalty. Drift penalty means that in Lyapunov optimization, we will impose additional penalties on those states that deviate far from the optimal path to encourage the system model to return to the optimal state as soon as possible. The purpose of this step is to determine the threshold of this penalty, that is, when the deviation of the system model state exceeds this threshold, we need to take measures to adjust it.

[0140] S3. Combine the Lyapunov function and the COMA architecture to obtain the Ly-COMA model. Based on the Ly-COMA model, the vehicle computing tasks are distributed to multiple edge servers for processing, realizing a high-efficiency, low-latency distributed service offloading and migration strategy.

[0141] As an example, establishing a network model in step S11 specifically includes:

[0142] A network model is established based on a VEC ultra-dense computing scenario with fine-grained task collaborative offloading and migration. The network model includes N BSs (base stations), N = {1, 2, ..., n, ..., N}, n∈N; M ESs (edge ​​servers), M = {1, 2, 3, ..., m, ..., M}, m∈M; V vehicles, V = {1, 2, 3, ..., v, ... V}, v∈V. Vehicles offload computing tasks to ESs for real-time computing during movement; and a time slot model T is added, T = {1, 2, 3, ..., t, ..., T}, which performs continuous time sampling at the same time interval.

[0143] The vehicle computing task includes multiple subtasks. In order to improve the computing efficiency of the vehicle task, in each time slot t, the vehicle v The subtasks will be offloaded to different ES for execution; and due to the dynamic characteristics of ultra-dense edge computing, the communication resources of the BS will change with the change of the number of connections, which will cause the load of the BS to change, and the available resources of the ES will also change according to the change of the BS load. v Different BSs and ESs should be selected to provide computing offloading services during the mobile process;

[0144] Different ES are connected through high-speed optical fiber networks. The computing power of ES can be expressed by CPU frequency. m The computing power is expressed as (GHz), the computing power of all ES can be expressed as Each subtask can be executed locally in the vehicle or offloaded to the ES. The uninstall decision is defined as when express Execute locally, m∈M represents In ES m Executed on vehicles v The connected BS will change with the time slot. A vehicle can only connect to one BS in time slot t. represents the BS to which vehicle v is connected at time t.

[0145] As an example, establishing the task model in step S12 specifically includes:

[0146] Assume that each vehicle has a computationally intensive application that contains a series of subtasks with time series dependencies. At time t, the vehicle v The application can be represented using a directed acyclic graph (DAG) as follows: in, represents the set of I subtasks for the v-th vehicle, Representation Node and The dependency relationship between them is that the initial task has no predecessor task, and the terminal task has no successor task, that is, if yes The direct parent node of Must be in Completed before;

[0147] Each subtask It can be expressed as in Representation Task The data size, Representation Task The number of CPU cycles required, Representation Task The result size of

[0148] vehicle v The task has a delay tolerance time d v ,The task must be completed within the specified delay tolerance time, otherwise the task execution will fail.

[0149] As an example, establishing the communication model in step S13 specifically includes:

[0150] The vehicle communicates with the BS through a wireless channel, and the wireless channel follows the Rayleigh distribution during the wireless communication process;

[0151] Vehicle at time t v Connect BS n The signal-to-noise ratio (SINR) can be expressed as:

[0152]

[0153] in, Indicates BS n The transmission power, Indicates ES n The distance between the vehicle and the vehicle, α represents the path loss coefficient, Indicates that the fast fading factor follows a Gaussian distribution, represents the channel thermal noise power, represents the interference set of other BSs;

[0154] Vehicle at time t v With ES n The transmission rate between It can be expressed as:

[0155]

[0156] Wherein, B represents the bandwidth of the channel;

[0157] Task at time t Wireless transmission to BS n Delay It can be expressed as:

[0158]

[0159] Since different ESs are connected by wires, for simplicity, the wired transmission delay is calculated based on the static hop count. It can be expressed as:

[0160]

[0161] Among them, ξ represents the delay between single hops, It represents the number of hops between BSn through wireless access and ESm providing computing services;

[0162] if yes The direct parent node of The calculation results are transmitted to If two nodes execute at different locations, additional communication overhead will be incurred No distinction is made between uplink and downlink bandwidth. It can be expressed as:

[0163]

[0164] Indicates vehicle v The delay of returning the exit task result to the vehicle. If the end task is executed locally in the vehicle, no transmission delay is required. Otherwise, It can be expressed as:

[0165]

[0166] As an example, a computational delay model is established. The model aims to comprehensively consider factors such as vehicle mobility, task execution status, and task queue status, provide an estimate of computational delay for decision-making task execution location (local or ES) and task migration, and optimize task scheduling strategies through delay analysis. The computational delay model established in step S11 specifically includes:

[0167] Calculate tasks separately The latency of executing tasks locally in the vehicle The latency of offloading to ES, where:

[0168] Task The computation latency performed locally in the vehicle is expressed as:

[0169]

[0170] in, It is a vehicle v The CPU frequency of the vehicle, i.e., for the case of local execution of tasks, the computational delay depends on the vehicle’s CPU frequency and the number of CPU cycles required for the task;

[0171] Taking into account the mobility of the vehicle, when the vehicle is away from the current ES, the vehicle decides whether to migrate the unfinished computing tasks to other ESs around the vehicle to perform the tasks, which will bring additional transmission delays; and switching between different ESs will also incur migration costs, ψ represents the cost of one migration.

[0172] Tasks in ES m The execution delay is affected by two main factors: the tasks that have not yet been completed The total number of CPU cycles required and the current ES m The length of the task queue; according to the first-come-first-served principle, the currently unloaded task can only be executed after the tasks in the queue are completed.

[0173] Therefore, the task In ES m The computation delay on It is expressed as:

[0174]

[0175] in, represents ES at time t m The number of CPU cycles required for the queue on Representation Task Migration strategy

[0176] when When indicating a task Migrated to ES m′ ,m≠m',m'∈M;when When The remaining tasks are still in ES m Execute on; if the task The remaining tasks are migrated to ES m′ , the computation delay is expressed as:

[0177]

[0178] As an example, the actual completion time of the task is calculated in step S15, which specifically includes:

[0179] For Vehicles v For vehicles v Multiple subtasks can be offloaded to different ES for execution, thereby greatly improving the parallelism of vehicle-mounted tasks and improving the efficiency of task execution. Start executing and calculating tasks The actual completion time step must meet the following conditions:

[0180] ① If the task is executed on ES, the data must have been transferred to the ES;

[0181] ② All directly preceding task nodes are completed;

[0182] ③、The task queue on ES is executed;

[0183] Therefore the task Start time It can be expressed as:

[0184]

[0185] when Time Task The transmission time is 0;

[0186] Representation Task The results of all predecessor tasks are passed to The latest completion time:

[0187]

[0188] Among them, pre v,i Representation Task The set of immediate predecessor nodes, Indicates the predecessor task End time of

[0189] Task The completion time is the actual start time plus the task processing time. According to formulas (7)(8)(9)(10), It can be expressed as:

[0190]

[0191] Therefore, task K v The actual completion time is:

[0192]

[0193] As an example, the successful execution of the vehicle application in step S16 must satisfy the following constraints:

[0194] ①. The time it takes to transmit the result of the exit task back to the vehicle cannot be greater than the vehicle's delay tolerance d. v , the constraint can be expressed as:

[0195]

[0196] ②If the task yes The direct parent node of Execution can only be executed after completion That is to say has a higher priority than Priority ω v,i It can be expressed as:

[0197]

[0198] Among them, succ v,i It's a task The set of immediate successor tasks;

[0199] Recursively exit from a task Traverse the DAG to calculate the priority of each node and exit the task Priority ω v,I It is expressed as:

[0200]

[0201] Therefore, task Gv Must satisfy ω during execution v,i >ω v,j .

[0202] As an example, the boundary of calculating the Lyapunov drift in step S21 specifically includes:

[0203] Building a vehicle using Lyapunov functions v Virtual migration cost insufficient queue Now assume that the first time slot y v (1) = 0, the migration cost is insufficient and the queue backlog evolves as follows:

[0204]

[0205] Among them, y v,i (t) represents the task at time t The distance between the actual migration cost deviation and the available migration cost; v The length of (t) is defined as the deviation between the actual migration cost and the available migration cost, which can be understood as the degree to which the constraint condition is satisfied; βψ represents the migration cost allowed in a time slot;

[0206] To jointly control offloading and migration, the total queue backlog is expressed as:

[0207]

[0208] Definition 1: The Lyapunov function L(Y(t)) and the Lyapunov drift ΔL(Y(t)) are defined as follows:

[0209]

[0210] ΔL(Y(t))=Ε[L(Y(t+1))-L(Y(t))Y(t)] (20)

[0211] Definition 2: The goal of the Lyapunov algorithm is to minimize the bound of the drift penalty expression ΘΔL(Y(t)) for each time slot, which is defined as follows:

[0212]

[0213] Among them, Z>0 is the importance weight for measuring the penalty;

[0214] Lemma 1: The bound of the Lyapunov drift ΔL(Y(t)) can be expressed as:

[0215]

[0216] Proof: By Taking the queues of V on both sides at the same time, we can get:

[0217]

[0218] in,

[0219] As an example, the boundary of drift penalty calculation in step S22 specifically includes:

[0220] Lemma 2: The bound of the drift penalty ΘΔL(Y(t)) can be expressed as:

[0221]

[0222] Proof: Substituting formula (22) of Lemma 1 into formula (21) yields:

[0223]

[0224] In the traditional multi-agent policy gradient, each agent (BS m ) It is the simplest to independently learn its own state and behavior based on its own Actor-Critic network; although this method is easy to implement, for the joint collaborative offloading and migration optimization problem of multiple vehicles and multiple edge servers, it is necessary to consider the interaction between multiple agents, and independent Actor-Critic strategies are difficult to learn collaborative strategies; therefore, this solution introduces a policy gradient algorithm based on counterfactual multi-agent (COMA) to solve the above problems.

[0225] As an example, the Ly-COMA-based distributed service unloading and migration method in step S3 specifically includes:

[0226] Based on the Actor-Critic network, a counterfactual multi-agent (COMA) policy gradient algorithm is introduced. Actor modules are deployed in each ES. At each time slot t, the input of the Actor network includes the status of the vehicle, edge server, communication, migration loss queue and vehicle task information. At this time, a probability distribution is output according to the state of the environment, and then the sampled action is applied to the environment. The environment gives the agent (ES m )A reward and the next state At this time, the Critic network evaluates the environment and gives back the reward to the Actor;

[0227] In the single-user scenario, the parameter θ of the Actor network is m,actor TD (Temporal Difference) error update calculated by the Critic network;

[0228] However, the above method is not suitable for multi-user environments, because the centralized critic calculates the TD error based on the global reward and ignores the contribution of a single actor to the global reward. The counterfactual baseline proposed by COMA can effectively solve this problem. For multi-user scenarios, the Q value Q(S t ,A t ) contains the global state and behavior; the global state S t It can be expressed as in represents ES at time t m The state of the global behavior A t It can be expressed as in Moment ES m behavior; for ES m In terms of keeping ES m Other than unchanged, then ES m The advantage function at time slot t can be expressed as:

[0229]

[0230] in, Indicates ES m The strategy function of the Actor network;

[0231] according to To calculate ES m The calculated baseline can reflect the current agent (ES m ) Contribution to the global reward;

[0232] Each Actor module selects unloading and migration decisions based on the local state in a distributed manner, and then the centralized Critic network guides the Actor module to learn the optimal strategy by calculating the contribution of the Actor module to the global reward. In order to improve the stability of the algorithm, the centralized Critic network also introduces a target Q value network. At the same time, in order to improve learning efficiency, all Actor networks share network parameters, use the same network structure, and adopt batch input for training. In each training stage, ε, ES m With parameters Actor network selection action and will Store it in the experience replay buffer D;

[0233] Calculate the reward for step j according to the variant TD algorithm -TD(λ):

[0234]

[0235] Therefore, the target Q value of the Critic network can be expressed as:

[0236]

[0237] Among them, γ∈[0,1] represents the discount factor of the reward, V(S t ) represents the value function of the centralized Critic network; the parameters θ of the Critic network are updated using the gradient descent method critic , the Loss function L of the Critic network t (θ critic ) is expressed as follows:

[0238] L t (θ critic )=(y (λ) -Q(S t ,A t )) 2 (28)

[0239] Parameters of the target critic network Every C steps and θ critic Synchronize once;

[0240] Then ES m According to the advantage function of formula (25), the policy gradient algorithm is used to update the parameters θ of the Actor network. m,actor :

[0241]

[0242] This solution studies the joint computing offloading and migration optimization problem for multiple vehicles and multiple edge servers in the ultra-dense vehicle edge computing environment. This solution uses Lyapunov function optimization technology to convert the complex collaborative computing offloading and migration joint optimization problem into an average delay optimization problem under the constraint of long-term migration cost; then, a COMA-based multi-agent deep reinforcement learning algorithm is used to obtain the optimal solution, thereby achieving a more efficient and stable vehicle computing task offloading and migration strategy.

[0243] As an example, in this embodiment, the algorithm performance is verified according to simulation experiments, including the following algorithms: Ly-COMA, DSMA, ATEE, RORM, ATEL, among which;

[0244] DSMA is a reinforcement learning algorithm based on distributed learning and task allocation, which is used to deal with task allocation and collaboration problems in multi-agent systems;

[0245] ATEE is a variant of reinforcement learning algorithm that uses a target Q-value network. It introduces a target Q-value network into the centralized critic network to speed up the learning process.

[0246] RORM is a replay-first reinforcement learning algorithm that improves the learning ability of the agent by replaying important replay priorities;

[0247] ATEL is an adaptive temporal difference learning reinforcement learning algorithm that makes improvements and innovations on issues such as continuity and speed.

[0248] It should also be noted that the Actor network in this scheme is based on a Recurrent Neural Network (RNN), which contains 128-bit Gated Recurrent Units (GRUS), and the Critic network is a feed-forward network with multiple ReLU layers and fully connected layers.

[0249] The experiment consists of two parts:

[0250] ① Experiment A: Convergence of Ly-COMA algorithm.

[0251] ② Experiment B: Compare the performance of the five algorithms, Ly-COMA, DSMA, ATEE, RORM, and ATEL, in different scenarios.

[0252] The experimental parameters are shown in Table 1 below:

[0253] Table 1 Experimental parameters

[0254]

[0255] The simulation results of Experiment A are as follows:

[0256] This embodiment mainly considers the convergence of the Ly-COMA algorithm, that is, the speed and stability of the convergence of the Ly-COMA algorithm as the number of training times increases.

[0257] like Figure 2 The figure shows the average reward of Ly-COMA. The COMA algorithm uses an Actor-Critic structure. After a period of iteration, the Actor and Critic gradually approach the optimal strategy and finally converge. At the beginning of training, the average reward value of the Ly-COMA algorithm increases slowly and has large jitters. This is because the algorithm is more of an exploratory training in the early stages of training. As the number of training increases, the number of explorations decreases, and the average reward of Ly-COMA gradually increases and converges to the optimal solution.

[0258] Throughout the training process, the Ly-COMA algorithm is able to achieve fast and stable convergence, reaching a stable state after about 500 iterations. This is because the COMA algorithm uses a counterfactual baseline to solve the problem that the global rewards generated by joint actions are difficult to accurately assign to individual agents. This allows the algorithm to more accurately evaluate the action value of individual agents, thereby improving learning efficiency and final performance, thereby improving the convergence speed and stability of Ly-COMA.

[0259] The simulation results of Experiment B are as follows:

[0260] This embodiment mainly considers the average delay of the above five algorithms in different scenarios. While ensuring that other parameters remain unchanged, the average delay of the five algorithms in scenarios with different numbers of subtasks, different ES computing capabilities, and different numbers of vehicles is compared.

[0261] like Figure 3 As shown in the figure, when the number of subtasks of a single in-vehicle application increases from 8 to 12, the average delay of all algorithms increases, but the Ly-COMA algorithm is significantly better than other algorithms and can effectively reduce the delay. This is because Ly-COMA can convert long-term optimization problems into optimal problems for each time slot according to Lyapunov. By introducing the COMA strategy that combines centralized training with distributed execution, the algorithm enhances the efficiency of collaboration between agents in the environment. This enables each agent to make the best decision based on the information it observes, thereby improving the operational efficiency of the entire system. The DSMA algorithm optimizes both the unloading and migration strategies. It is closer to the Ly-COMA algorithm proposed in the present invention in terms of average delay, but it uses a global network to train the network of each thread without considering the contribution of each agent in the global network. In actual application scenarios, it is difficult for each agent to achieve fast and stable convergence, and the delay increases significantly when the number of tasks increases.

[0262] like Figure 4The figure shows the effect of the number of ES per square kilometer on latency. When the number of ES increases from 10 to 50, it can be seen from the figure that the average latency of Ly-COMA, DSMA, RORM, and ATEE decreases with the increase in the number of ES. It can also be seen from the figure that the changes in the average latency of the above four algorithms gradually become gentle. The reason is that with the increase in the number of ES, more computing resources can be used, which increases the parallelism of task execution. When the number of ES reaches 40 per square kilometer, it can already meet the requests of all vehicles in the scene. At this time, increasing the number of ES has little effect on latency. It is worth noting that with the increase in the number of ES, Ly-COMA still has the lowest average latency. Ly-COMA can find the optimal unloading and migration strategy when the number of ES increases. All tasks of ATEL are processed locally, so the change in the number of ES has no effect on the ATEL algorithm.

[0263] like Figure 5 The figure shows the impact of different numbers of vehicles on the performance of the algorithms. The number of vehicles increases from 20 to 60, and as the number of vehicles increases, the average delay of all algorithms increases. This is because the increase in the number of vehicles served by each ES leads to intensified channel contention, longer transmission time, and more computing resources required. At the same time, the increase in the number of vehicles will also increase the number of tasks that need to be migrated, so the migration delay will also increase. ATEL will cause higher computing delays, and offloading tasks to ES can reduce delays. However, as the number of users increases, the transmission delay of vehicles will continue to increase, so the average delay of ATEE will gradually become higher than the Ly-COMA and DSMA algorithms. The experimental results show that compared with other algorithms, Ly-COMA exhibits superior performance, can more effectively reduce the average delay, and achieves near-optimal performance.

[0264] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0265] The above contents are merely examples and explanations of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the invention or exceed the scope defined by the claims, they shall all fall within the protection scope of the present invention.

Claims

1. A service unloading and migration method for vehicle edge computing, characterized in that: The following steps are involved: S1. Build a system model, which includes the following processes: S11. Establish a network model to simulate the scenario where the vehicle offloads computing tasks to different edge servers and performs real-time computing during movement; S12, establishing a task model for managing and scheduling vehicle computing tasks to be executed; S13, establishing a communication model for describing a process in which a vehicle communicates with a base station through a wireless channel; S14, establishing a computing delay model for evaluating the delay of executing computing tasks locally in the vehicle and offloading the computing tasks to the edge server for execution; S15, calculating the actual completion time of the task; S16. Set task constraints to ensure the order and timeliness of computing tasks during execution; S2. For the above system model, plan the optimization problem and introduce the Lyapunov function to transform the optimization problem, which specifically includes the following processes: S21. Construct a virtual migration cost-deficient queue, calculate the boundary of Lyapunov drift, and define the total queue backlog and Lyapunov function expression; S22. Combine the lemma to derive the bounds of drift penalty, and minimize the bounds of drift penalty by jointly controlling unloading and migration; S3. Combining the Lyapunov function and the COMA architecture to obtain the Ly-COMA model, based on which the vehicle computing tasks are distributed to multiple edge servers for processing, specifically including: S31, deploy the Actor module on each edge server, generate probabilistic distributed actions based on vehicle, server status and task information, and evaluate the global reward through the Critic network; S32, using the counterfactual multi-agent policy gradient algorithm, optimizing the Actor network parameters through the advantage function, and the centralized Critic network calculates the target Q value and updates the network parameters; S33, through the experience replay buffer and variant TD algorithm for training, synchronize the target critic network parameters to achieve high-efficiency, low-latency task allocation and migration.

2. A service unloading and migration method for vehicle edge computing according to claim 1, characterized in that: The establishment of the network model in step S11 specifically includes: Set up N base stations (BS), , ; Edge Servers (ES) , ; A car, , ; and add the time slot model T, , continuous time sampling is performed at the same time interval; The vehicle computing task consists of multiple subtasks. In the process, the vehicle offloads subtasks to different ESs for execution; and as the load on the BS changes, the available resources of the ES will also change. Each subtask can be executed locally in the vehicle or offloaded to the ES for execution.

3. A service unloading and migration method for vehicle edge computing according to claim 2, characterized in that: The establishment of the task model in step S12 specifically includes: Assume that each vehicle has a computationally intensive application that consists of a series of subtasks with time series dependencies. At time t, The application can be represented using a directed acyclic graph (DAG) as: ,in, Indicates Car A set of subtasks, Represents a subtask and The dependencies between them are that the initial task has no predecessor task, and the terminal task has no successor task; Each subtask It can be expressed as ,in Represents a subtask The data size, Represents a subtask The number of CPU cycles required, Represents a subtask The result size of The task has a delay tolerance time ,The task must be completed within the specified delay tolerance time, otherwise the task execution will fail.

4. A service unloading and migration method for vehicle edge computing according to claim 3, characterized in that: The establishment of the communication model in step S13 specifically includes: The vehicle communicates with the BS through a wireless channel, and the wireless channel follows the Rayleigh distribution during the wireless communication process; time connect The signal-to-noise ratio (SINR) can be expressed as: （1） in, express The transmission power, express The distance between the vehicle and represents the path loss coefficient, Indicates that the fast fading factor follows a Gaussian distribution, represents the channel thermal noise power, represents the interference set of other BSs; time and The transmission rate between It can be expressed as: （2） in, represents the bandwidth of the channel, Time Mission Wireless transmission to Delay It can be expressed as: （3） in, express Time Vehicle The BS to which it is connected; Different ESs are connected by wires. The wired transmission delay is calculated based on the static hop count. The transmission delay between different ESs It can be expressed as: （4） in, represents the delay between single hops, Indicates wireless access and providing computing services The number of hops between Time Mission The uninstall decision is defined as ,when ,express Execute locally, , express exist Execute on; if yes The direct parent node of The calculation results are transmitted to ; If the two nodes execute at different locations, additional communication consumption will be incurred , does not distinguish between uplink and downlink bandwidth, It can be expressed as: （5） express The delay of returning the exit task result to the vehicle. If the end task is executed locally in the vehicle, no transmission delay is required. Otherwise, It can be expressed as: （6）。 5. A service unloading and migration method for vehicle edge computing according to claim 4, characterized in that: The establishment of the calculation delay model in step S14 specifically includes: Calculate tasks separately The latency of executing tasks locally in the vehicle The latency of offloading to ES, where: Task The computation latency performed locally in the vehicle is expressed as: （7） in, yes CPU frequency; Task exist The computation delay on It is expressed as: (8) in, express time The number of CPU cycles required for the queue on Representation Task Migration strategy , express computing power; when When indicating a task Migrated to , ;when When The remaining tasks are still Execute on; if the task The remaining tasks are moved to , the computation delay is expressed as: （9）。 6. A service unloading and migration method for vehicle edge computing according to claim 5, characterized in that: The actual completion time of the task in step S15 is calculated as follows: Multiple subtasks can be offloaded to different ES for execution, while computing tasks The actual completion time must meet the following conditions: ① If the task is executed on ES, the data must have been transferred to the ES; ② All directly preceding task nodes are completed; ③、The task queue on ES is executed; Therefore the task Start time It can be expressed as: （10） when Time Task The transmission time is 0; Representation Task The results of all predecessor tasks are passed to The latest completion time: （11） in, Representation Task The set of immediate predecessor nodes of Indicates the predecessor task End time of Task The completion time is the actual start time plus the task processing time. According to formulas (7)(8)(9)(10), It can be expressed as: （12） Therefore the task The actual completion time is: （13）。 7. A service unloading and migration method for vehicle edge computing according to claim 6, characterized in that: The following constraints must be met to achieve successful execution of the vehicle application in step S16: ①. The time it takes to transmit the result of the exit task back to the vehicle cannot be greater than the vehicle's delay tolerance , the constraint can be expressed as: (14) ②If the task yes The direct parent node of Execution can only be executed after completion , that is to say has a higher priority than , Priority It can be expressed as: (15) in, It's a task The set of immediate successor tasks; Recursively exit from a task Traverse the DAG to calculate the priority of each node and exit the task Priority for: (16) Therefore the task Must be satisfied during execution .

8. A service unloading and migration method for vehicle edge computing according to claim 7, characterized in that: The boundaries for calculating the Lyapunov drift in step S21 specifically include: Using Lyapunov function to construct Virtual migration cost insufficient queue , assuming that the first time slot , the migration cost insufficient queue backlog evolves as follows: (17) in, express Time Mission The distance between the actual migration cost deviation and the available migration cost; The length of is defined as the deviation between the actual migration cost and the available migration cost, which can be understood as the degree to which the constraint condition is satisfied; represents the migration cost allowed in a time slot, Indicates the cost of a migration; To jointly control offloading and migration, the total queue backlog is expressed as: (18) Definition 1: Lyapunov function and Lyapunov drift The definition is as follows: (19) (20) Definition 2: The goal of the Lyapunov algorithm is to minimize the drift plus penalty expression for each time slot: The borders of The definition is as follows: (21) in, is a weight that measures the importance of punishment; Lemma 1: Lyapunov drift The boundary of can be expressed as: (22) Proof: By , both sides take The queue and can be obtained: (23) in, .

9. A service unloading and migration method for vehicle edge computing according to claim 8, characterized in that: The boundary of drift penalty calculation in step S22 specifically includes: Lemma 2: Drift Penalty The boundary of can be expressed as: (24) Proof: Substituting formula (22) of Lemma 1 into formula (21) yields: 。 10. A service unloading and migration method for vehicle edge computing according to claim 9, characterized in that: The distributed service unloading and migration method based on Ly-COMA in step S3 specifically includes: Based on the Actor-Critic network, the Counterfactual Multi-Agent (COMA) policy gradient algorithm is introduced; Deploy Actor modules on each ES in each time slot , the input of the Actor network includes the status of vehicles, edge servers, communications, migration loss queues, and vehicle mission information; at this time, a probability distribution is output according to the state of the environment, and then the sampled actions are applied to the environment, which gives the agent ( )A reward and the next state , and at this time the Critic network evaluates the reward fed back to the Actor by the environment; In a single-user scenario, the parameters of the Actor network are TD error update calculated by the Critic network; For multi-user scenarios, the Q value estimated by the centralized critic Contains global state and behavior; global state It can be expressed as ,in express time state; global behavior It can be expressed as ,in time behavior; In terms of keeping Other than If unchanged, In time slot The advantage function can be expressed as: (25) in, express The strategy function of the Actor network; according to To calculate The calculated baseline can reflect the current proxy ( ) Contribution to the global reward; Each Actor module selects the unloading and migration decision based on the local state in a distributed manner, and then the centralized Critic network guides the Actor module to learn the optimal strategy by calculating the contribution of the Actor module to the global reward; the centralized Critic network also introduces the target Q value network; and all Actor networks share network parameters, use the same network structure, and adopt batch input for training; in each training stage , With parameters Actor network selection action , and Stored in the experience replay buffer middle; According to the variant TD algorithm calculate Step Rewards: (26) Therefore, the target Q value of the Critic network can be expressed as: (27) in, represents the discount factor of the reward, Represents the value function of the centralized Critic network; uses gradient descent to update the parameters of the Critic network , Loss function of the Critic network It is expressed as follows: (28) Parameters of the target critic network Every Step and Synchronize once; Then According to the advantage function of formula (25), the policy gradient algorithm is used to update the parameters of the Actor network. : (29)。

Images