Distributed task collaborative unloading method for vehicle edge calculation

Abstract

The invention relates to the technical field of Internet of Things, in particular to a vehicle edge calculation-oriented distributed task collaborative unloading method, which comprises the following steps of: constructing a system model; the system model comprises a network model, an edge server unloading model, a V2V unloading model, a V2V link state model, a task migration model and a vehicle excitation model; the multi-agent-based distributed task unloading method comprises the steps of calculating V2V link quality and vehicle unloading willingness, selecting an optimal service vehicle and selecting an optimal edge node based on an anti-fact multi-agent reinforcement learning method. The calculation delay of the whole system is effectively reduced, the average delay and migration cost of the system are reduced, and the edge node with the highest cost benefit is accurately selected for task unloading.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a distributed task collaborative offloading method for vehicle edge computing. Background Art

[0002] In recent years, VEC has received extensive attention from academia and industry, and has promoted the development of IoV. The development of the Internet of Vehicles has spawned a large number of computationally intensive and latency-sensitive applications (path planning, pedestrian recognition, audio and video entertainment, etc.), which require continuous real-time processing of computing tasks. However, such as pedestrian recognition, this application requires a large amount of computing resources to train and infer models, and the limited local resources of the vehicle make it difficult to meet the above requirements. The traditional solution is to offload these computing tasks to a remote cloud center for execution, but the cloud center is generally far away from the vehicle, which will bring long transmission delays and is not suitable for latency-sensitive tasks. VEC provides an effective solution to "sink" cloud resources near the vehicle, thereby shortening the delay of task transmission. In VEC, each roadside unit (RSU) is equipped with an edge server (ES). ES creates a series of containers to provide services to vehicles within its communication range to achieve low computing and transmission delays. However, when multiple vehicles simultaneously offload computing services to the same ES during peak traffic hours, it will also cause network congestion and high computing delays.

[0003] V2V, as a promising model, can effectively alleviate the above problems. In V2V, short-distance V2V communication is established between vehicles. V2V can expand the coverage of communication. The vehicle's computing tasks are not only offloaded to nearby ESs for execution, but can also be transmitted to farther ESs for execution, thereby achieving load balancing between ESs (Edge Servers) and solving the network congestion problem. At the same time, vehicles can also offload computing tasks to nearby vehicles with rich computing resources through V2V communication, thereby reducing the burden on ESs and effectively improving the execution efficiency of vehicle-mounted applications.

[0004] To realize the V2V-assisted VEC collaborative offloading strategy, several challenges need to be overcome: 1. Unstable mobility of vehicles. When a vehicle moves, it moves within the coverage of different edge nodes (ESs and vehicles providing services). At this time, if the computing task is not migrated with the vehicle, it will lead to long transmission delays or even service interruptions. Frequent migration of services will also incur additional migration costs (delays in starting containers and delays in data transmission between edge nodes); 2. Incomplete information. To achieve collaborative offloading of tasks, it is necessary to obtain complete information about the environment. However, for highly dynamic scenarios such as the Internet of Vehicles, the wireless channels, network topology, and computing resources of edge nodes are difficult to predict before task offloading, which also brings great challenges to collaborative offloading; 3. The selfishness of vehicles. The selfish tendency of vehicle behavior stems from the limited communication and computing resources, which makes them more inclined to take selfish action patterns when interacting with the environment. This trend makes vehicles reserved about shared resources, which is not conducive to the implementation of V2V communication. Therefore, how to design an effective incentive mechanism to encourage vehicles to share their resources has become a key challenge to be solved. Summary of the invention

[0005] The purpose of the present invention is to provide a distributed task collaborative offloading method based on vehicle edge computing to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A distributed task collaborative offloading method for vehicle edge computing, the method comprising:

[0008] Constructing a system model; the system model includes: a network model, an edge server offloading model, a V2V offloading model, a V2V link status model, a task migration model and a vehicle incentive model;

[0009] A distributed task offloading method based on multi-agent includes selecting the optimal service vehicle and selecting the optimal edge node.

[0010] Preferably, the construction of the network model includes:

[0011] Acquire edge servers and vehicle information in the network; the vehicle information includes customer vehicles and service vehicles;

[0012] Set a time slot model Ψ = {1, 2, 3, … t, …, T}, and continuously sample the vehicle information at the same time interval. Then, in each time slot, the vehicle information generates a task offloading decision;

[0013] Select to transmit the task offloading decision to the ES server or the service vehicle SV for execution, and execute the task offloading decision generated by the vehicle information in each time slot in sequence;

[0014] Get all task offloading decisions of vehicle v at a certain time t, where the nth task offloading decision of vehicle v is: in M is the task offloading decision set M of the ES server ES Or the task offloading decision set M of the service vehicle SV SV Composition, where M = M ES ∪M SV ;

[0015] It is assumed that within each time slot, the edge node selected by the vehicle to offload the task does not change, but it can change across time slots.

[0016] Preferably, the construction of the edge server offloading model includes:

[0017] when , it means that the nth task offloading decision of vehicle v at a certain time t is executed on the ES server; then the communication rate between vehicle v and ES server at a certain time t is:

[0018]

[0019] in, and denote the transmission power and channel power gain between vehicle v and ES server, respectively. and B ES denote the noise power and wireless bandwidth respectively, and are assumed to remain unchanged during T time slots;

[0020] The transmission delay between the nth task offloading decision of vehicle v and the ES server at a certain time t is obtained as:

[0021] in represents the size of the nth task offloading decision in time slot t, represents the communication rate between vehicle v and ES server;

[0022] According to the M / M / 1 queuing model, the queuing delay of task n in the network during task offloading is:

[0023] Among them, t exp represents the delay of sending and receiving data without network congestion, and c0 represents the normalized computing task processed by ES collaboration;

[0024] The computation delay of computing task n on the ES server is: in, represents the computational requirements of task n, represents the computing resources allocated by ES to the nth task of vehicle v;

[0025] The total delay of computing task offloading on the ES server is:

[0026]

[0027] Preferably, the construction of the V2V offloading model includes:

[0028] when , which means that the nth task offloading decision of vehicle v at a certain time t is executed on the service vehicle SV, where CV communicates with SV through the V2V link;

[0029] At a certain time t, the communication rate between CV and service vehicle SV is:

[0030]

[0031] in, and denote the transmission power and channel power gain between vehicle v and service vehicle SV, respectively, and B ES represent noise power and wireless bandwidth respectively;

[0032] And the transmission delay between the nth task of vehicle v and the service vehicle SV is obtained as:

[0033] in represents the communication rate between CV and service vehicle SV;

[0034] At the same time, the computational delay of task n on the service vehicle SV is: in, represents the computing resources allocated to task n by the service vehicle SV;

[0035] Then the total delay of offloading the computing task to the service vehicle SV is:

[0036]

[0037] Preferably, the construction of the V2V link state model includes:

[0038] S1. Use the change in relative displacement to represent the stability of the link between vehicle v and service vehicle SV:

[0039] The distance between vehicle v and its one-hop neighbor m at a certain time t for:

[0040] Where (x v ,y v ) represents the coordinates of vehicle v, (x m ,y m ) represents the coordinates of one-hop neighbor m;

[0041] At this time, the link stability between vehicle v and its one-hop neighbor, vehicle m, is obtained as:

[0042] in represents the relative displacement between vehicle v and its one-hop neighbor, vehicle m, r vehicle Indicates the communication radius of the vehicle, which is a fixed value;

[0043] S2. The signal energy received by each vehicle is:

[0044]

[0045] Among them, μ represents the path attenuation factor, τ represents the wavelength of signal transmission, β represents the ground reflection coefficient, and H represents the height of the antenna;

[0046] Then the signal-to-noise ratio of the signal received by the vehicle is: in Represents all the energy received by the vehicle, including noise;

[0047] The bit error rate of data transmission is:

[0048] in

[0049] The packet error rate is Wherein, L represents the length of the data packet;

[0050] S3. According to Calculate the link maintenance time between vehicles v and m Where V represents the relative speed of vehicles v and m;

[0051] Get the transmission delay between task n and service vehicle SV at a certain time t Then when When the link is unstable and easily interrupted, the link maintenance time is normalized: in Indicates the maximum duration;

[0052] According to the link maintenance time and packet error rate The link quality between vehicles v and m is obtained as: in represents the weight value; at this time, the link state between vehicles v and m is:

[0053] Preferably, the construction of the task migration model includes:

[0054] According to the formula Get the migration cost of the nth task of vehicle v at a certain time t

[0055] Among them, Γ(x) represents the indicator function, φ represents the migration cost between different edge nodes;

[0056] When x is true, Γ(x) = 1; at this time, the nth task of vehicle v is migrated to the edge node m'; conversely, when x is false, Γ(x) = 0; at this time, the remaining tasks are still executed on the edge node m.

[0057] Preferably, the construction of the vehicle incentive model includes:

[0058] Get service vehicle M SV The willingness to uninstall task n:

[0059] in, represents the incentive factor of task n at a certain time t, α is the normalization coefficient, and Both represent the performance factors of the service vehicle SV.

[0060] Preferably, choose the best service vehicle, including:

[0061] According to the service vehicle M SV The willingness to offload task n and the link status between vehicles v and m

[0062] According to the link stability between vehicles within the communication range of vehicle v and service vehicle M SV Uninstall intention To select the optimal service vehicle SVm * :

[0063] in, When SVm * When selected as a service vehicle, its incentive factor Add 1 and add the selected optimal service vehicle to the ES server.

[0064] Preferably, selecting the optimal edge node includes:

[0065] Actor modules are deployed in each vehicle information. In each time slot, the input of the Actor network includes vehicle task information, optimal SVm * Communication status and load information with the ES server;

[0066] Based on the input and the current state of the environment, the Actor outputs a probability distribution, applies the sampled action to the environment, and the environment gives the vehicle a reward and the next state

[0067] In a single-user scenario, the parameters of the Actor network are TD error update calculated by the Critic network;

[0068] The Q value Q(S) estimated by the centralized critic t , A t ), which contains the global state and behavior, where the global state Global Behavior

[0069] For vehicle v, keep the global behavior except vehicle v If it remains unchanged, the advantage function of vehicle v in time slot t is:

[0070] in, The policy function of the Actor network representing vehicle v;

[0071] According to To calculate the baseline of vehicle v, the calculated baseline can reflect the contribution of vehicle v to the global reward. Each actor selects an unloading strategy based on the local state in a distributed manner. Then the centralized Critic guides the actor to learn the optimal strategy by calculating the actor's contribution to the global reward.

[0072] In order to improve the stability of the algorithm, the centralized Critic network also introduces a target Q value network. At the same time, in order to improve learning efficiency, all Actor networks are set to share network parameters, use the same network structure, and use batch input for training. Then, a training stage k is set, and the vehicle v has parameters Actor network selection action and will Stored in the experience replay buffer D, the reward for step j is calculated according to the variant TD algorithm - TD(λ):

[0073] At this time, the target Q value of the Critic network is:

[0074] Among them, ζ∈[0,1] represents the discount factor of the reward, V(S t+j ) represents the value function of the centralized Critic network;

[0075] Use gradient descent to update the parameters θ of the Critic network Critic , then the Loss function of the Critic network is t (θ Critic )for:

[0076] Among them, the parameters of the Critic network Every C steps and θ Critic Synchronize once; then the vehicle v is based on the advantage function Update the parameters θ of the Actor network using the policy gradient algorithm Critic :

[0077]

[0078] Where χ represents the learning rate of the Actor network.

[0079] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0080] This paper mainly designs a distributed task collaborative offloading method for vehicle edge computing, in which computationally intensive vehicle-mounted tasks can be collaboratively completed through ES or V2V offloading, thereby effectively reducing the computational delay of the entire system; in the collaborative offloading framework, the optimal service vehicle is selected by considering vehicle mobility, V2V communication link quality and vehicle offloading willingness. The COMA multi-agent reinforcement learning method is used to implement the collaborative offloading strategy to reduce the average system delay and migration cost; this strategy enables vehicles to observe and learn system costs among many candidate edge nodes, so as to accurately select the most cost-effective edge node for task offloading; experimental results show that the method proposed in this invention performs better in terms of delay and has certain practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0082] Figure 1 It is a flow chart of a distributed task collaborative offloading method for vehicle edge computing of the present invention;

[0083] Figure 2 It is a system model diagram of a distributed task collaborative offloading method for vehicle edge computing according to the present invention.

[0084] Figure 3 It is an algorithm convergence diagram of a distributed task collaborative offloading method for vehicle edge computing of the present invention;

[0085] Figure 4 is a graph of the relationship between different vehicle speeds and average delay;

[0086] Figure 5 is a graph of the relationship between the number of different vehicles and the average delay;

[0087] Figure 6 This is a graph showing the relationship between the number of edge servers and average latency. DETAILED DESCRIPTION

[0088] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] See also Figure 1-Figure 6 , the present invention provides a technical solution:

[0090] The method designed in this example uses Python 3.7 environment and PyTorch 1.12.1 framework. The hardware environment is a GPU-based server with 32GB 1600MHz DDR3, 2.8GHz Intel Core i9 and 2T memory. The experiment simulates an IoV environment, which contains [10-50] vehicles and [4-12] ESs. The ES can provide wireless access services to vehicles within a radius of 300 meters. The CPU frequency of the vehicle is set to [1-3] GHz, and the bandwidth between V2V and vehicle to ES is set to 10MHz according to the dedicated short-range communication (DSRC). In the simulation experiment, the TD algorithm is set to 0.8, and the mini-batch training method is used, with each mini-batch containing 32 samples. For the Actor network, a small learning rate of 0.0001 is selected to promote its stable learning. For the Critic network, a learning rate of 0.001 is used to ensure the rapid convergence of its parameters.

[0091] A distributed task collaborative offloading method for vehicle edge computing, the method comprising:

[0092] Constructing a system model; the system model includes: a network model, an edge server offloading model, a V2V offloading model, a V2V link status model, a task migration model and a vehicle incentive model;

[0093] A distributed task offloading method based on multi-agents includes calculating the V2V link quality and vehicle offloading willingness, selecting the optimal service vehicle, and selecting the optimal edge node based on the counterfactual multi-agent reinforcement learning method.

[0094] Preferably, the construction of the network model includes:

[0095] Acquire edge servers and vehicle information in the network; the vehicle information includes customer vehicles and service vehicles;

[0096] Set a time slot model Ψ = {1, 2, 3, … t, …, T}, and continuously sample the vehicle information at the same time interval. Then, in each time slot, the vehicle information generates a task offloading decision;

[0097] Select to transmit the task offloading decision to the ES server or the service vehicle SV for execution, and execute the task offloading decision generated by the vehicle information in each time slot in sequence;

[0098] Get all task offloading decisions of vehicle v at a certain time t, where the nth task offloading decision of vehicle v is: in M is the task offloading decision set M of the ES server ES Or the task offloading decision set M of the service vehicle SV SV Composition, where M = M ES ∪M SV ;

[0099] It is assumed that within each time slot, the edge node selected by the vehicle to offload the task does not change, but it can change across time slots.

[0100] Preferably, the construction of the edge server offloading model includes:

[0101] when , it means that the nth task offloading decision of vehicle v at a certain time t is executed on the ES server, where the vehicle and the ES server communicate in an orthogonal frequency division multiplexing manner; then the communication rate between vehicle v and the ES server at a certain time t is:

[0102]

[0103] in, and denote the transmission power and channel power gain between vehicle v and ES server, respectively. and B ES denote the noise power and wireless bandwidth respectively, and are assumed to remain unchanged during T time slots;

[0104] The transmission delay between the nth task offloading decision of vehicle v and the ES server at a certain time t is obtained as:

[0105] in represents the size of the nth task offloading decision in time slot t, represents the communication rate between vehicle v and ES server;

[0106] According to the M / M / 1 queuing model, the queuing delay of task n in the network during task offloading is:

[0107] Among them, t exp represents the delay of sending and receiving data without network congestion, and c0 represents the normalized computing task processed by ES collaboration;

[0108] Since the ES server can provide computing services for multiple vehicles at time t, the computing delay of computing task n on the ES server is: in, represents the computational requirements of task n, represents the computing resources allocated by ES to the nth task of vehicle v;

[0109] Since the size of the output result is much smaller than the size of the input task, both ES offloading and V2V offloading ignore the transmission delay of the returned result, and the total delay of the computing task offloading on the ES server is:

[0110]

[0111] Preferably, the construction of the V2V offloading model includes:

[0112] when , which means that the nth task offloading decision of vehicle v at a certain time t is executed on the service vehicle SV, where CV communicates with SV through the V2V link;

[0113] At a certain time t, the communication rate between CV and service vehicle SV is:

[0114]

[0115] in, and denote the transmission power and channel power gain between vehicle v and service vehicle SV, respectively, and B ES represent noise power and wireless bandwidth respectively;

[0116] And the transmission delay between the nth task of vehicle v and the service vehicle SV is obtained as:

[0117] in represents the communication rate between CV and service vehicle SV;

[0118] Different from offloading to the ES server, due to the limited computing resources of the vehicle and the smaller communication coverage of the V2V link, it is assumed that each service vehicle SV can only provide computing services to one CV at most, and the computing delay of task n on the service vehicle SV is: in, represents the computing resources allocated to task n by the service vehicle SV;

[0119] Then the total delay of offloading the computing task to the service vehicle SV is:

[0120]

[0121] Preferably, the construction of the V2V link state model includes:

[0122] In VEC scenarios, due to the fast-moving characteristics of vehicles, the connection time between vehicles is relatively short, which may have an adverse effect on the data transmission efficiency of V2V communication and even cause the transmission process to fail. In addition, the quality of the link between vehicles is also an important factor in determining whether data can be fully transmitted. Therefore, the status of the link is crucial to achieving efficient and reliable V2V communication;

[0123] S1. Definition 1 Link Stability:

[0124] The stability of the link between vehicle v and service vehicle SV is expressed using the change in relative displacement:

[0125] The distance between vehicle v and its one-hop neighbor m at a certain time t for:

[0126] Where (x v ,y v ) represents the coordinates of vehicle v, (x m ,y m ) represents the coordinates of one-hop neighbor m;

[0127] At this time, the link stability between vehicle v and its one-hop neighbor, vehicle m, is obtained as:

[0128] in represents the relative displacement between vehicle v and its one-hop neighbor, vehicle m, r vehicle Indicates the communication radius of the vehicle, which is a fixed value;

[0129] By evaluating the relative displacement changes between vehicles, it can be found that the smaller the displacement change, the higher the stability of the link;

[0130] S2, define 2 packet error rate:

[0131] Considering that the data packets transmitted by the sending vehicle far exceed the data volume of the beacon signal, the video transmission model is used to analyze the error rate of the data packets received by the receiving vehicle to measure the link quality between vehicles;

[0132] The signal energy received by each vehicle is:

[0133]

[0134] Among them, μ represents the path attenuation factor, τ represents the wavelength of signal transmission, β represents the ground reflection coefficient, and H represents the height of the antenna;

[0135] Then the signal-to-noise ratio of the signal received by the vehicle is: in Represents all the energy received by the vehicle, including noise;

[0136] The bit error rate of data transmission is:

[0137] in

[0138] The packet error rate is Wherein, L represents the length of the data packet;

[0139] S3. Definition 3 link maintenance time:

[0140] Considering that the differences in vehicle speed and driving direction may cause dynamic changes in the network topology, this may lead to failure in data transmission during the unloading process. Therefore, the link maintenance time is crucial to ensure the successful unloading of the task;

[0141] according to Calculate the link maintenance time between vehicles v and m Where V represents the relative speed of vehicles v and m;

[0142] in,

[0143] v v and v m denote the speeds of vehicles v and m respectively;

[0144] Get the transmission delay between task n and service vehicle SV at a certain time t Then when When the link is unstable and easily interrupted, the link maintenance time is normalized: in Indicates the maximum duration;

[0145] According to the link maintenance time and packet error rate The link quality between vehicles v and m is obtained as: in represents the weight value; at this time, the link state between vehicles v and m is:

[0146] Preferably, the construction of the task migration model includes:

[0147] In order to ensure the continuity of service, the mobility of vehicles between different ESs must be taken into account. This involves the dynamic migration of computing tasks to adapt to the real-time location changes of the vehicle. When vehicle v enters the coverage area of ​​another edge node, there are two options: one is to continue to execute the remaining computing tasks on the original edge node, but this may cause a large data transmission delay; the other is to migrate the computing tasks to the new edge node for execution, thereby reducing the data transmission delay. However, the latter will introduce additional migration costs, including the re-creation of business containers and the time delay of business interruption. Therefore, deciding whether to migrate computing tasks as the vehicle moves is crucial to achieving efficient and collaborative task offloading. In order to balance transmission delay and migration cost, it is necessary to design an intelligent task migration strategy that can optimize decisions based on the vehicle's mobility model, network conditions, and the processing power of the edge server. In this way, it can be ensured that a high-quality service experience is maintained even when the vehicle crosses different edge nodes, and the resource utilization efficiency of the entire system is maximized;

[0148] According to the formula Get the migration cost of the nth task of vehicle v at a certain time t

[0149] Among them, Γ(x) represents the indicator function, φ represents the migration cost between different edge nodes;

[0150] When x is true, Γ(x) = 1; at this time, the nth task of vehicle v is migrated to the edge node m'; conversely, when x is false, Γ(x) = 0; at this time, the remaining tasks are still executed on the edge node m.

[0151] Preferably, the construction of the vehicle incentive model includes:

[0152] Vehicles tend to show a tendency to prioritize their own resources when performing tasks, and are usually reserved about sharing computing resources. This is because tasks for other vehicles will take up a lot of computing resources, and in resource-constrained scenarios, the primary concern of vehicles is to maintain their business performance. However, this self-centered resource allocation strategy may lead to unnecessary waste of resources, and this is particularly evident when vehicles face task execution delays due to insufficient resources. It is worth noting that in the Internet of Vehicles environment, SV may also play the role of CV at different time slots and need to rely on task offloading services provided by other devices. Therefore, if the vehicle continues to put its own interests first, it may inadvertently damage its own performance and ultimately affect the efficiency of the entire system; in order to incentivize vehicles to share resources, the present invention designs a model that incentivizes sharing of resources between vehicles;

[0153] Get service vehicle M SV The willingness to uninstall task n:

[0154] in, represents the incentive factor of task n at a certain time t, which indicates the frequency of vehicle v as a service vehicle SV providing computing services to other vehicles at time slot t. α is the normalization coefficient, which is used to convert the unloading willingness Normalized to the range of 0 to 1, and All represent the performance factors of the service vehicle SV;

[0155] When the local computing task demand of the service vehicle SV increases, Decrease, reduce the willingness to provide services to other CVs; when vehicle v has a larger incentive factor When , it indicates that vehicle v has a greater willingness to become a service vehicle SV to provide computing offloading services to other vehicles; on the contrary, when the local computing task requirements of the service vehicle SV are reduced, the computing resources are large, and it will show a greater willingness to handle the computing tasks of CV.

[0156] Preferably, choose the best service vehicle, including:

[0157] The distributed task collaborative offloading method for vehicle edge computing proposed in this invention includes two stages, namely SV selection and collaborative offloading decision. In the SV selection stage, the link quality between vehicles and the offloading willingness of the vehicles are comprehensively evaluated to select the optimal service vehicle. This not only improves the success rate of data packet transmission, but also increases the success rate of task offloading.

[0158] According to the service vehicle M SV The willingness to offload task n and the link status between vehicles v and m

[0159] According to the link stability between vehicles within the communication range of vehicle v and service vehicle M SV Uninstall intention To select the optimal service vehicle SVm * : m,m * ∈M SV ;

[0160] in, When SVm * When selected as a service vehicle, its incentive factor Add 1 and add the selected optimal service vehicle to the ES server.

[0161] Preferably, selecting the optimal edge node includes:

[0162] Actor modules are deployed in each vehicle information. In each time slot, the input of the Actor network includes vehicle task information, optimal SVm * Communication status and load information with the ES server;

[0163] Based on the input and the current state of the environment, the Actor outputs a probability distribution, applies the sampled action to the environment, and the environment gives the vehicle a reward and the next state Critic evaluates the rewards given by the environment to the Actor;

[0164] In a single-user scenario, the parameters of the Actor network are TD error update calculated by the Critic network;

[0165] However, this method is not suitable for multi-user environments, because the centralized critic calculates the TD error based on the global reward and ignores the contribution of a single actor to the global reward. The counterfactual baseline proposed by COMA can effectively solve this problem;

[0166] The Q value Q(S) estimated by the centralized critic t ,A t ), which contains the global state and behavior, where the global state Global Behavior

[0167] For vehicle v, keep the global behavior except vehicle v If it remains unchanged, the advantage function of vehicle v in time slot t is:

[0168] in, The policy function of the Actor network representing vehicle v;

[0169] According to To calculate the baseline of vehicle v, the calculated baseline can reflect the contribution of vehicle v to the global reward. Each actor selects an unloading strategy based on the local state in a distributed manner. Then the centralized Critic guides the actor to learn the optimal strategy by calculating the actor's contribution to the global reward.

[0170] In order to improve the stability of the algorithm, the centralized Critic network also introduces a target Q value network. At the same time, in order to improve learning efficiency, all Actor networks are set to share network parameters, use the same network structure, and use batch input for training. Then, a training stage k is set, and the vehicle v has parameters Actor network selection action and will Stored in the experience replay buffer D, the reward for step j is calculated according to the variant TD algorithm - TD(λ):

[0171] At this time, the target Q value of the Critic network is:

[0172] Among them, ζ∈[0,1] represents the discount factor of the reward, V(S t+j ) represents the value function of the centralized Critic network;

[0173] Use gradient descent to update the parameters θ of the Critic network Critic , then the Loss function of the Critic network is t (θ Critic )for:

[0174] Among them, the parameters of the Critic network Every C steps and θ Critic Synchronize once; then the vehicle v is based on the advantage function Update the parameters θ of the Actor network using the policy gradient algorithm Critic :

[0175]

[0176] Where χ represents the learning rate of the Actor network.

[0177] 1) Experiment A: Convergence of the algorithm.

[0178] 2) Experiment B: Performance of the four algorithms under different parameters, increasing the vehicle speed from 20 to 100, the number of vehicles from 10 to 50, and the number of edge servers from 4 to 12.

[0179] The experimental parameters are shown in Table 1:

[0180] Table 1 Experimental parameters

[0181]

[0182]

[0183] This example will consider two performance indicators, namely:

[0184] 1. Convergence of the algorithm. As the number of training times increases, the speed and stability of the algorithm convergence in different scenarios.

[0185] 2. Average delays of the four algorithms in different scenarios. While ensuring that other parameters remain unchanged, the average delays of the four algorithms in scenarios with different vehicle speeds, different numbers of vehicles, and different numbers of edge servers are compared.

[0186] The simulation results of this example are as follows:

[0187] 1. Convergence of the algorithm

[0188] Attached Figure 3 The characteristics of reward convergence in scenarios with different numbers of vehicles are shown; scenario A consists of 15 vehicles and 4 ESs, while scenario B contains 25 vehicles and the same number of 4 ESs. As can be seen from the figure, in scenario B, both the initial reward and the final stable reward level are significantly higher than scenario A. This is because scenario B has more service vehicles to choose from, which provides more optimization opportunities. In addition, although scenario A has a smaller number of vehicles, its reward convergence speed is faster. This is because in scenario A, due to the fewer available service vehicles, the system can explore and converge to a feasible solution faster. However, due to resource constraints, the learning process of scenario A faces the challenge of data scarcity, which limits its ability to discover the optimal strategy.

[0189] 2. Comparison of average latency of four algorithms in different scenarios

[0190] Attached Figure 4 The relationship between the average system delay and speed is given. Figure 4 It can be seen that when the vehicle speed increases from 20km / h to 100km / h, the average delay of all algorithms increases. Due to the high-speed movement of the vehicle, the stability of the communication link between V2V and the vehicle and ES will be affected, and the frequency of task migration will increase. Figure 4It can be seen that the delay of the collaborative unloading algorithm based on COMA is relatively stable and lower than that of the other three algorithms. TONS has the fastest delay growth rate and the highest delay, which is mainly because the high vehicle speed causes a significant decrease in the stability of the V2V link, which in turn reduces the number of available service vehicles. The algorithm designed in this patent can comprehensively consider the link quality between vehicles and the unloading willingness of the vehicles to select the optimal service vehicle. Then, multi-agent reinforcement learning is used to achieve the optimal collaborative unloading decision to improve the overall system performance.

[0191] Attached Figure 5 The effect of the number of vehicles on the average delay is shown in Figure 5 It can be seen that as the number of vehicles increases, the average delays of COMA, ROM and TONS all decrease, while the average delay of TOE shows an upward trend. This is because the TOE algorithm offloads all tasks to ES for processing only. When the number of vehicles increases, the transmission delay and calculation delay of the task will increase. Therefore, the average delay of TOE will increase with the increase in the number of vehicles. It can also be seen from the figure that when the number of vehicles increases, the delay of TONS gradually becomes lower than ROM. This is because TONS has greater flexibility in the selection of service vehicles and has more abundant computing resources. At the same time, as the vehicle density increases, the V2V communication link becomes more stable. As can be seen from the figure, the average delay of the distributed offloading strategy based on COMA designed in this patent is still the lowest.

[0192] Attached Figure 6 It represents the impact of the number of ES on the average delay. It can be seen from the figure that as the number of ES increases, the average delays of COMA, ROM and TOE decrease. It can also be seen from the figure that the changes in the average delay of the above three algorithms gradually become gentle. The reason is that as the number of ES increases, more computing resources can be used, and the current computing resources are sufficient to handle the current tasks. It is worth noting that as the number of ES increases, the average delay and global task completion rate based on the COMA collaborative offloading strategy designed in this patent still perform optimally, and it can find the optimal offloading and migration strategy when the number of ES increases. The computing tasks of the TONS algorithm can only be offloaded to the nearest SV for execution through V2V communication, so changes in the number of ES have no effect on the TONS algorithm.

[0193] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A distributed task collaborative offloading method for vehicle edge computing, characterized by: The method comprises: Constructing a system model; the system model includes: a network model, an edge server offloading model, a V2V offloading model, a V2V link status model, a task migration model and a vehicle incentive model; A distributed task offloading method based on multi-agent includes selecting the optimal service vehicle and selecting the optimal edge node.

2. A distributed task collaborative offloading method for vehicle edge computing according to claim 1, characterized in that: The construction of the network model includes: Acquire edge servers and vehicle information in the network; the vehicle information includes customer vehicles and service vehicles; Set a time slot model Ψ = {1, 2, 3, … t, …, T}, and continuously sample the vehicle information at the same time interval. Then, in each time slot, the vehicle information generates a task offloading decision; Select to transmit the task offloading decision to the ES server or the service vehicle SV for execution, and execute the task offloading decision generated by the vehicle information in each time slot in sequence; Get all task offloading decisions of vehicle v at a certain time t, where the nth task offloading decision of vehicle v is: in M is the task offloading decision set M of the ES server ES Or the task offloading decision set M of the service vehicle SV SV Composition, where M = M ES ∪M SV ; It is assumed that within each time slot, the edge node selected by the vehicle to offload the task does not change, but it can change across time slots.

3. A distributed task collaborative offloading method for vehicle edge computing according to claim 1, characterized in that: The construction of the edge server offloading model includes: when , it means that the nth task offloading decision of vehicle v at a certain time t is executed on the ES server; then the communication rate between vehicle v and ES server at a certain time t is: in, and denote the transmission power and channel power gain between vehicle v and ES server, respectively. and B ES denote the noise power and wireless bandwidth respectively, and are assumed to remain unchanged during T time slots; The transmission delay between the nth task offloading decision of vehicle v and the ES server at a certain time t is obtained as: in represents the size of the nth task offloading decision in time slot t, represents the communication rate between vehicle v and ES server; According to the M / M / 1 queuing model, the queuing delay of task n in the network during task offloading is: Among them, t exp represents the delay of sending and receiving data without network congestion, and c0 represents the normalized computing task processed by ES collaboration; The computation delay of computing task n on the ES server is: in, represents the computational requirements of task n, represents the computing resources allocated by ES to the nth task of vehicle v; The total delay of computing task offloading on the ES server is:

4. A distributed task collaborative offloading method for vehicle edge computing according to claim 1, characterized in that: The construction of the V2V offloading model includes: when , which means that the nth task offloading decision of vehicle v at a certain time t is executed on the service vehicle SV, where CV communicates with SV through the V2V link; At a certain time t, the communication rate between CV and service vehicle SV is: in, and denote the transmission power and channel power gain between vehicle v and service vehicle SV, respectively, and B ES represent noise power and wireless bandwidth respectively; And the transmission delay between the nth task of vehicle v and the service vehicle SV is obtained as: in represents the communication rate between CV and service vehicle SV; At the same time, the computational delay of task n on the service vehicle SV is: in, represents the computing resources allocated to task n by the service vehicle SV; Then the total delay of offloading the computing task to the service vehicle SV is:

5. A distributed task collaborative offloading method for vehicle edge computing according to claim 1, characterized in that: The construction of the V2V link state model includes: S1. Use the change in relative displacement to represent the stability of the link between vehicle v and service vehicle SV: The distance between vehicle v and its one-hop neighbor m at a certain time t for: Where (x v ,y v ) represents the coordinates of vehicle v, (x m ,y m ) represents the coordinates of one-hop neighbor m; At this time, the link stability between vehicle v and its one-hop neighbor, vehicle m, is obtained as: in represents the relative displacement between vehicle v and its one-hop neighbor, vehicle m, r vehicle Indicates the communication radius of the vehicle, which is a fixed value; S2. The signal energy received by each vehicle is: Among them, μ represents the path attenuation factor, τ represents the wavelength of signal transmission, β represents the ground reflection coefficient, and H represents the height of the antenna; Then the signal-to-noise ratio of the signal received by the vehicle is: in Represents all the energy received by the vehicle, including noise; The bit error rate of data transmission is: in The packet error rate is Wherein, L represents the length of the data packet; S3. According to Calculate the link maintenance time between vehicles v and m Where V represents the relative speed of vehicles v and m; Get the transmission delay between task n and service vehicle SV at a certain time t Then when When the link is unstable and easily interrupted, the link maintenance time is normalized: in Indicates the maximum duration; According to the link maintenance time and packet error rate The link quality between vehicles v and m is obtained as: in represents the weight value; at this time, the link state between vehicles v and m is:

6. A distributed task collaborative offloading method for vehicle edge computing according to claim 1, characterized in that: The construction of the task migration model includes: According to the formula Get the migration cost of the nth task of vehicle v at a certain time t Among them, Γ(x) represents the indicator function, φ represents the migration cost between different edge nodes; When x is true, Γ(x) = 1; at this time, the nth task of vehicle v is migrated to the edge node m'; conversely, when x is false, Γ(x) = 0; at this time, the remaining tasks are still executed on the edge node m.

7. A distributed task collaborative offloading method for vehicle edge computing according to claim 1, characterized in that: The construction of the vehicle incentive model includes: Get service vehicle M SV The willingness to uninstall task n: in, represents the incentive factor of task n at a certain time t, α is the normalization coefficient, and Both represent the performance factors of the service vehicle SV.

8. A distributed task collaborative offloading method for vehicle edge computing according to claim 1, characterized in that: The selecting of the optimal service vehicle comprises: According to the service vehicle M SV The willingness to offload task n and the link status between vehicles v and m According to the link stability between vehicles within the communication range of vehicle v and service vehicle M SV Uninstall intention To select the optimal service vehicle SVm * : Among them, γ+θ=1, when SVm * When selected as a service vehicle, its incentive factor Add 1 and add the selected optimal service vehicle to the ES server.

9. A distributed task collaborative offloading method for vehicle edge computing according to claim 1, characterized in that: The selecting the optimal edge node comprises: Actor modules are deployed in each vehicle information. In each time slot, the input of the Actor network includes vehicle task information, optimal SVm * Communication status and load information with the ES server; Based on the input and the current state of the environment, the Actor outputs a probability distribution, applies the sampled action to the environment, and the environment gives the vehicle a reward and the next state In a single-user scenario, the parameters of the Actor network are TD error update calculated by the Critic network; The Q value Q(S) estimated by the centralized critic t ,A t ), which contains the global state and behavior, where the global state Global Behavior For vehicle v, keep the global behavior except vehicle v If it remains unchanged, the advantage function of vehicle v in time slot t is: in, The policy function of the Actor network representing vehicle v; According to To calculate the baseline of vehicle v, the calculated baseline can reflect the contribution of vehicle v to the global reward. Each actor selects an unloading strategy based on the local state in a distributed manner. Then the centralized Critic guides the actor to learn the optimal strategy by calculating the actor's contribution to the global reward. In order to improve the stability of the algorithm, the centralized Critic network also introduces a target Q value network. At the same time, in order to improve learning efficiency, all Actor networks are set to share network parameters, use the same network structure, and use batch input for training. Then, a training stage k is set, and the vehicle v has parameters Actor network selection action and will Stored in the experience replay buffer D, the reward for step j is calculated according to the variant TD algorithm - TD(λ): At this time, the target Q value of the Critic network is: Among them, ζ∈[0,1] represents the discount factor of the reward, V(S t+j ) represents the value function of the centralized Critic network; Use gradient descent to update the parameters θ of the Critic network Critic , then the Loss function of the Critic network is t (θ Critic )for: Among them, the parameters of the Critic network Every C steps and θ Critic Synchronize once; then the vehicle v is based on the advantage function Update the parameters θ of the Actor network using the policy gradient algorithm Critic : Where χ represents the learning rate of the Actor network.

Images