A method for computing offloading and power optimization in Internet of Vehicles based on potential game

Abstract

The present invention proposes a method for offloading and optimizing computing power of Internet of Vehicles based on potential game. Vehicles with surplus computing resources can be used as on-board cloud servers to provide computing resources for other vehicles. V2I links and V2V links are connected to different slices through 5G slicing technology without interfering with each other. First, the offloading decision of the task request vehicle is obtained through potential game, that is, the task request vehicle decides to perform local computing, offload to the MEC server, and offload to the on-board cloud server closest to the task request vehicle. For the task request vehicle that decides to offload the task, a distributed resource allocation method is adopted based on deep reinforcement learning. The base station does not need to centrally schedule channel state information. Each task request vehicle that decides to offload the task is regarded as an intelligent body, and the transmission power is selected based on the local observation state information. The algorithm can minimize the system delay under the limitation of the maximum transmission power. The present invention uses a combination of potential game and DDPG algorithm to effectively solve the joint problem of offloading decision and power optimization in the scenario of collaborative computing offloading between mobile edge servers and on-board cloud servers, and can be stable in the optimization of a series of continuous state spaces.

Description

Technical Field

[0001] The present invention relates to a vehicle networking technology, and in particular to a vehicle networking computing offloading and power optimization method, and more specifically to a vehicle networking computing offloading and power optimization method based on potential game. Background Art

[0002] Vehicular networks (V2X) are a new networking paradigm that has emerged in recent years, aiming to enable data transmission between vehicles and, in turn, control and manage urban traffic. In this new paradigm, information transmission and network access are achieved through vehicle-to-everything (V2X) communication, including vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications. The development of V2X has driven the growth of the Internet of Vehicles (IoV), giving rise to a variety of vehicular applications such as route planning, autonomous driving, and infotainment. These applications not only ensure travel safety but also provide on-the-go entertainment and connectivity. However, most of these applications are latency-sensitive and resource-intensive, characterized by complex computations and high energy requirements. Currently, many vehicles have limited storage capacity and insufficient computing resources to meet the stringent latency constraints of these applications. Mobile edge computing (MEC) can provide low-latency computing services for vehicles by deploying computing and storage resources at the edge of the network. As vehicles develop in terms of enhanced edge intelligence, they are equipped with more and more computing and storage resources. People have proposed the concept of on-board cloud, that is, vehicles can act as on-board cloud servers by sharing their remaining computing resources, and other vehicles, pedestrians and other users can offload computing tasks to them.

[0003] The highly dynamic vehicle environment introduces further uncertainty into computation offloading decisions and resource allocation. First, due to vehicle mobility, the topology of the in-vehicle network and the state of the wireless channels change rapidly over time. Second, the computational resources of the in-vehicle cloud server are heterogeneous and fluctuate over time. These factors are often difficult to predict but can significantly impact the latency of computational tasks.

[0004] Therefore, the present invention proposes a method for Internet of Vehicles computing offloading and power optimization based on potential game. This method targets the scenario of collaborative computing offloading between mobile edge servers and on-board cloud servers, and takes minimizing system latency as the optimization goal of computing migration and resource allocation, achieving a good balance between complexity and performance. Summary of the Invention

[0005] Purpose of the Invention: To address the aforementioned issues with existing technologies, a potential game-based approach for offloading and optimizing computing power in connected vehicles is proposed. Vehicles with excess computing resources can act as onboard cloud servers to provide computing resources to other vehicles. This approach utilizes hybrid spectrum access technology for transmission, minimizing system latency.

[0006] Technical Solution: For scenarios where mobile edge servers and onboard cloud servers collaborate on computation offloading, system latency is minimized through efficient computation offloading decisions and power allocation. To reduce system latency while improving spectrum utilization, hybrid spectrum access technology is employed for transmission. V2I and V2V links access different slices using 5G slicing technology, ensuring no interference. First, a potential game is used to determine the offloading decision of the task-requesting vehicle. The task-requesting vehicle decides to offload local computation to the MEC server, which is then the onboard cloud server closest to the task-requesting vehicle. For the task-requesting vehicle that decides to offload, a distributed resource allocation approach is employed, eliminating the need for central scheduling of channel state information by the base station. Each task-requesting vehicle that decides to offload is treated as an intelligent agent, and transmit power is selected based on locally observed state information. A deep reinforcement learning model is established and optimized using the Deep Deterministic Policy Gradient (DDPG) algorithm. Based on the optimized DDPG model, the transmit power for each task-requesting vehicle that decides to offload is determined. The above invention is achieved through the following technical solution: a method for computing offloading and power optimization of Internet of Vehicles based on potential game, comprising the following steps:

[0007] (1) Deploy mobile edge computing (MEC) servers on the roadside unit (RSU). Vehicles that can share their remaining computing resources can be used as on-board cloud servers. The computing tasks of task requesting vehicles can be processed locally and offloaded to the on-board cloud servers and MEC servers.

[0008] (2), establish N mission request vehicles, N sev The communication model and computing model of an on-board cloud server and a roadside unit equipped with an MEC server are used to establish a computing offloading and power optimization model.

[0009] (3),Each task requesting vehicle obtains the computing resource occupancy of the surrounding onboard cloud servers and MEC as well as task information;

[0010] (4) Based on the potential game, the offloading decision of each vehicle is obtained, that is, the task requesting vehicle decides to perform local computation, offload to the MEC server, and offload to the vehicle-mounted cloud server closest to the task requesting vehicle. According to the offloading decision, a deep reinforcement learning model is established for the vehicle that decides to offload the task with the goal of reducing the system latency;

[0011] (5) Deep reinforcement learning model training based on DDPG;

[0012] (6) In the execution phase, the task requesting vehicle obtains the task offloading decision through potential game, and the task requesting vehicle n0 that decides to offload obtains the current state according to the local observation For all tasks where offloading decisions are not locally computed, the vehicle is requested to obtain transmit power using the trained deep reinforcement learning model;

[0013] Furthermore, the step (2) includes the following specific steps:

[0014] (2a), establish the communication model of vehicle network computing offloading, the system includes V task request vehicles, N sev There are two on-board cloud servers and one roadside unit deploying MEC servers. The task requesting vehicles are represented by the set N = {1, 2…, N}. The set N sev ={1, 2..., N sev} represents the vehicle-mounted cloud server, and the task requesting vehicle n can be expressed as c n Indicates the number of CPU cycles required for user n to complete the task, s n Indicates the amount of task data uninstalled by user n, represents the maximum tolerable delay of task execution by user n. Assuming that in each time slot, the task request vehicle generates a task, and the task unloading decision of task request vehicle n is A={a1,a2,...,a n ,...,a N}, Indicates that a n = 0 means the task requests vehicle n to perform the computing task locally, a n =1 means that the task requesting vehicle n offloads the task to the MEC server through the V2I link, a n =2 means that the task requesting vehicle n offloads the task to the onboard cloud server closest to the task requesting vehicle through the V2V link for calculation. V2V communication and V2I communication do not interfere with each other using 5G slicing technology;

[0015] (2b), the task requests vehicle n to offload the task to the onboard cloud server n sev The signal-to-interference-and-noise ratio (SINR) is expressed as:

[0016]

[0017] Task request vehicle n offloads the task to the onboard cloud server n sev The transmission rate is expressed as:

[0018]

[0019] in, Indicates that the task is assigned to the requesting vehicle n to offload the task to the onboard cloud server n sev The transmission bandwidth, P n represents the transmission power of the task request vehicle n, σ 2 represents the noise power, Indicates a task request from vehicle n to the onboard cloud server n sev The channel gain, Indicates that the task requesting vehicles other than the task requesting vehicle n offload the task to the vehicle-mounted cloud server n′ sev Interference with the task request vehicle n

[0020]

[0021] in a n′ =2, J(a n′ =2)=1, otherwise J(a n′ =2)=0, Indicates whether the task requesting vehicle n′ offloads the task to the vehicle-mounted cloud server n′ sev , Indicates that the task requests vehicle n′ to offload the task to the vehicle-mounted cloud server n′ sev , Indicates that the task requesting vehicle n′ has not offloaded the task to the onboard cloud server n′ sev , P n′ represents the transmission power of the mission request vehicle n′, Indicates a task request from vehicle n′ to the onboard cloud server n′ sev The channel gain of

[0022] (2c), similarly, the signal-to-interference-and-noise ratio (SINR) of the task requesting vehicle n to offload the task to the MEC server is expressed as:

[0023]

[0024] Task request vehicle n offloads the task to the onboard cloud server n sev The transmission rate is expressed as:

[0025]

[0026] Among them, P n represents the transmission power of the task request vehicle n, σ 2 represents the noise power, h n，mec represents the channel gain from the task request vehicle n to the MEC server, Indicates the interference of task requesting vehicles other than task requesting vehicle n in offloading tasks to the MEC server to task requesting vehicle n

[0027]

[0028] in a n′ =1, J(a n′ =1)=1, otherwise J(a n′ =1)=0, Indicates that the task requests vehicle n′ to offload the task to the MEC server, Indicates that the task requesting vehicle n′ does not offload the task to the MEC server, P n′ represents the transmission power of the task request vehicle n′, h n′，mec represents the channel gain from the task request vehicle n′ to the MEC server;

[0029] (2d), establish a computational model, a n =0 means the task requests vehicle n to perform the computing task locally, represents the local computing capability of the task request vehicle n, and the local computing latency is:

[0030]

[0031] a n =1 means that the task request vehicle n offloads the task to the MEC server for calculation through the V21 link. The upload delay of the task request vehicle n uploading the task to the MEC server is:

[0032]

[0033] The computational delay of the task request vehicle n uploading the task to the MEC server is:

[0034]

[0035] The computing capacity of the task assigned by the MEC server to the task request vehicle n.

[0036] a n =2 means that the task request vehicle n offloads the task to the vehicle cloud server through the V2R link, and the task request vehicle n uploads the task to the vehicle cloud server n sev The upload delay is

[0037]

[0038] Task request vehicle n uploads the task to the vehicle cloud server n sev The computational delay is

[0039]

[0040] For vehicle-mounted cloud server n sev The computing power of the task assigned to the task request vehicle n. Many studies have shown that the amount of data downloaded from the final calculation result is very small compared to the amount of data uploaded, so the delivery delay can be ignored. Therefore, the delay generated by the task request vehicle n to offload the task to the MEC server is

[0041]

[0042] Task request vehicle n offloads the task to the onboard cloud server n sev The resulting delay is:

[0043]

[0044] (2e), in summary, the following objective function and constraints can be established:

[0045]

[0046] Among them, the constraint condition Cl indicates that the task requesting vehicle n performs the computing task locally, or offloads the task to the MEC server for computing, or offloads the task to the on-board cloud server closest to the task requesting vehicle for computing. Each computing task can only choose one computing method. The constraint condition C2 indicates the local computing capacity range of the task requesting vehicle n. is the local maximum computing capacity of the task requesting vehicle n. Constraints C3 and C4 mean that the computing capacity allocated by the MEC server and the vehicle-mounted cloud server to the task requesting vehicle n is non-negative. Constraints C5 and C6 indicate that the computing capacity allocated by the MEC server and the vehicle-mounted cloud server to the task requesting vehicle cannot exceed its maximum computing capacity. mec is the maximum computing capacity of the MEC server, F mec is the maximum computing capacity of the onboard cloud server; constraints C7 and C8 indicate that the task request vehicle n performs the task to meet its maximum delay constraint; the constraint condition indicates that the task request vehicle n's transmission power is non-negative and meets its maximum transmission power constraint;

[0047] Furthermore, the step (4) includes the following specific steps:

[0048] (4a), based on the potential game, the decision of whether to unload each vehicle is obtained. The unloading decision of the task vehicle is modeled as a potential game, expressed as G = {N, (a n ) n∈N ,(u n ) n∈N}, where N is the set of vehicles requesting the task, a nRequest the unloading decision of vehicle n for the task, u n Cost function for requesting vehicle n for a task.

[0049] In the game model, each task requesting vehicle is a resource competitor, so there are N vehicles competing for limited resources in the network. Each vehicle can choose to offload the computation or perform the task computation locally, where a n ∈{0, 1, 2} is the unloading decision of the task request vehicle n, A represents the unloading decision set of all task request vehicles, a n =1 means that the task requests vehicle n to offload the task to the MEC server for calculation, a n = 0 indicates that the task requests vehicle n to perform the computing task locally. n =2 means that the task request vehicle n offloads the task to the vehicle-mounted cloud server closest to the task request for calculation. n When the cost function is expressed as u n (a n , a -n ), where a -n represents the set of unloading decisions of all task request vehicles except task request vehicle n. Each task request vehicle can hope to minimize its own cost by finding the optimal unloading decision, that is,

[0050]

[0051] The potential game converges to the Nash equilibrium, that is, the unloading decision is found through optimal response iteration. All task request vehicles do not exist and changing the current unloading decision can minimize their own costs.

[0052] (4b), according to the offloading decision Use set N o = {1, 2, ..., N0} represents the unloading decision in the task request vehicle The mission request vehicle, N0 represents The number of task request vehicles, the state space S is defined as the observation information related to the transmission power and the low-dimensional fingerprint information, including the task request vehicle n0 to the vehicle cloud server N sev ={1, 2, ..., N sev}Channel state information Mission request vehicle, channel status information to MEC Task request vehicle, to the vehicle cloud server N sev ={1, 2, .., N sev} received The interference received by the mission request vehicle n0 to the MEC Task request vehicle n0's task information The number of training rounds e and the random exploration variable ε in the ε-greedy algorithm, that is,

[0053]

[0054] Will The task request vehicle is regarded as an intelligent agent. Each time the vehicle is based on the current state Select the transmit power;

[0055] (4c), define action A: the action of each task request vehicle that decides to unload is the selected transmission power For continuous variables ranging from 0 to Indicates the mission request vehicle and maximum transmission power;

[0056] (4d), define the reward function r, and the optimization goal is to optimize the offloading decision under the constraint of maximum transmission power The task request vehicle's selected transmission power is minimized by the offloading decision The task request vehicle's task processing delay, so the reward function can be expressed as:

[0057]

[0058] Where b is a fixed value used to adjust the value of the reward function;

[0059] (4e) Based on the established state, action and reward functions, a deep reinforcement learning model is established, and each agent trains the corresponding action network and evaluation network;

[0060] Furthermore, the step (5) includes the following specific steps:

[0061] (5a), initialize the number of training rounds P;

[0062] (5b), initialize the time step t in P rounds;

[0063] (5c), each agent is the online Actor strategy network of the vehicle that decides to unload the task request according to the input state Output Action After all agents perform the corresponding actions, they obtain the reward rt and each agent goes to the next state. Thus, we can obtain the training data of each agent.

[0064] (5d), the training data of each agent Deposit into the corresponding experience replay pool;

[0065] (5e), Randomly sample m training data from the experience replay pool for each agent to form a data set and send it to the online Actor policy network, the online Critic evaluation network, the target Actor policy network, and the target Critic evaluation network;

[0066] (5f), Define the loss function of the online Critic evaluation network Update all parameters θ of the current Critic network through the gradient backpropagation of the neural network;

[0067] (5g), Define the sampling policy gradient of the online Actor policy network Update all parameters δ of the current Actor network through the gradient backpropagation of the neural network;

[0068] (5h), When the number of online training times reaches the target network update frequency, update the target network parameters δ′ and θ′ according to the online network parameters δ and θ respectively;

[0069] (5i), Determine whether t < K is satisfied, where K is the total number of time steps in p rounds. If so, t = t + 1 and go to step 5c; otherwise, go to step 5j;

[0070] (5j), Determine whether p < I is satisfied, where I is the set threshold of the number of training rounds. If so, p = p + 1 and go to step 5b; otherwise, the optimization ends and the optimized deep reinforcement learning model is obtained.

[0071] Furthermore, the step (6) includes the following specific steps:

[0072] (6a), Based on the occupancy of the surrounding vehicle-mounted cloud servers and MEC computing resources and the task information obtained when each vehicle migrates tasks, obtain the vehicle's offloading decision through potential game, and obtain the state information at this moment for all task request vehicles whose offloading decisions are not local computing

[0073] (6b), For each task request vehicle n0 that decides to offload the task, use the trained deep reinforcement learning model and input the state information

[0074] (6c), The task request vehicle n0 that decides to offload outputs the optimal action policy That is, the optimal transmission power of the task request vehicle is obtained Beneficial Effects: This invention proposes a method for offloading computation and optimizing power in connected vehicles (IoVs) based on potential game theory. This method targets scenarios where computation is collaboratively offloaded between mobile edge servers and onboard cloud servers. It utilizes hybrid spectrum access technology for transmission. V2V and V2I links access different slices based on 5G slicing technology, preventing interference. Potential game theory is used to determine the offloading decision of the task-requesting vehicle. Deep dual-Q learning is then used to optimize the transmit power of the task-requesting vehicle, minimizing system latency and achieving task computation. The algorithm, combining potential game theory and deep dual-Q learning, effectively solves the joint optimization problem of offloading decisions and transmit power for task-requesting vehicles, achieving a good balance between complexity and performance.

[0075] In summary, in the scenario of collaborative computing offloading between mobile edge servers and vehicle-mounted cloud servers, the potential game-based vehicle network computing offloading and power optimization method proposed in the present invention is superior in minimizing system latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A flowchart of a method for computing offloading and power optimization of an Internet of Vehicles based on potential game theory provided by an embodiment of the present invention;

[0077] Figure 2 A schematic diagram of a model for collaborative computing offloading between a mobile edge server and an in-vehicle cloud server provided in an embodiment of the present invention;

[0078] Figure 3 A schematic diagram of the deep deterministic policy gradient algorithm framework provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0079] The core concept of this invention is to use hybrid spectrum access technology for transmission in scenarios where mobile edge servers and vehicle-mounted cloud servers collaborate on computing offload. The V2V link and V21 access different slices based on 5G slicing technology, without interfering with each other. The task request vehicle's offload decision is obtained through potential game theory. The task request vehicle that decides to offload is considered an intelligent agent. A deep reinforcement learning model is established and optimized using a deep deterministic policy gradient algorithm. Based on the optimized deep reinforcement learning model, the optimal transmission power of the offloaded task request vehicle is obtained, thereby minimizing system latency.

[0080] The present invention is described in further detail below.

[0081] Step (1) deploys a mobile edge computing (MEC) server on the roadside unit (RSU), and uses the vehicle that can share its remaining computing resources as an on-board cloud server. The computing tasks of the task requesting vehicle can be processed locally and offloaded to the on-board cloud server or MEC server.

[0082] Step (2): Create a task request vehicle with N tasks. sev The communication model and computing model of a vehicle-mounted cloud server and a roadside unit are constructed, and then a joint computing migration and resource allocation model is established, including the following steps:

[0083] (2a), establish the communication model of vehicle network computing offloading, the system includes N task request vehicles, N sev There are two on-board cloud servers and one roadside unit deploying MEC servers. The task requesting vehicles are represented by the set N = {1, 2…, N}. The set N sev ={1, 2..., N sev} represents the vehicle-mounted cloud server, and the task requesting vehicle n can be expressed as c n Indicates the number of CPU cycles required for user n to complete the task, s n Indicates the amount of task data uninstalled by user n, represents the maximum tolerable delay of task execution by user n. Assuming that in each time slot, the task request vehicle generates a task, and the task unloading decision of task request vehicle n is A={a1,a2,...,a n ,...,a N}, Indicates that a n = 0 means the task requests vehicle n to perform the computing task locally, a n =1 means that the task requesting vehicle n offloads the task to the MEC server through the V2I link, a n =2 means that the task requesting vehicle n offloads the task to the onboard cloud server closest to the task requesting vehicle through the V2V link for calculation. V2V communication and V2I communication do not interfere with each other using 5G slicing technology;

[0084] (2b), the task requests vehicle n to offload the task to the onboard cloud server n sev The signal-to-interference-and-noise ratio (SINR) is expressed as:

[0085]

[0086] Task request vehicle n offloads the task to the onboard cloud server n sev The transmission rate is expressed as:

[0087]

[0088] in, Indicates that the task is assigned to the requesting vehicle n to offload the task to the onboard cloud server n sev The transmission bandwidth, P n represents the transmission power of the task request vehicle n, σ 2 represents the noise power, Indicates a task request from vehicle n to the onboard cloud server n sev The channel gain, Indicates that the task requesting vehicles other than the task requesting vehicle n offload the task to the vehicle-mounted cloud server n′ sev Interference with the task request vehicle n

[0089]

[0090] in a n′ =2, J(a n′ =2)=1, otherwise J(a n′ =2)=0, Indicates whether the task requesting vehicle n′ offloads the task to the vehicle-mounted cloud server n′ sev , Indicates that the task requests vehicle n′ to offload the task to the vehicle-mounted cloud server n′ sev , Indicates that the task requesting vehicle n′ has not offloaded the task to the onboard cloud server n′ sev , P n′ represents the transmission power of the mission request vehicle n′, Indicates a task request from vehicle n′ to the onboard cloud server n′ sev The channel gain of

[0091] (2c), similarly, the signal-to-interference-and-noise ratio (SINR) of the task requesting vehicle n to offload the task to the MEC server is expressed as:

[0092]

[0093] Task request vehicle n offloads the task to the onboard cloud server n sev The transmission rate is expressed as:

[0094]

[0095] Among them, P n represents the transmission power of the task request vehicle n, σ 2 represents the noise power, h n，mec represents the channel gain from the task request vehicle n to the MEC server, Indicates the interference of task requesting vehicles other than task requesting vehicle n in offloading tasks to the MEC server to task requesting vehicle n

[0096]

[0097] in a n′ =1, J(a n′=1)=1, otherwise J(a n′ =1)=0, Indicates that the task requests vehicle n′ to offload the task to the MEC server, Indicates that the task requesting vehicle n′ does not offload the task to the MEC server, P n′ represents the transmission power of the task request vehicle n′, h n′，mec represents the channel gain from the task request vehicle n′ to the MEC server;

[0098] (2d), establish a computational model, a n =0 means the task requests vehicle n to perform the computing task locally, represents the local computing capability of the task request vehicle n, and the local computing latency is:

[0099]

[0100] a n =1 means that the task requesting vehicle n offloads the task to the MEC server for calculation through the V2R link. The upload delay of the task requesting vehicle n uploading the task to the MEC server is:

[0101]

[0102] The computational delay of the task request vehicle n uploading the task to the MEC server is:

[0103]

[0104] The computing capacity of the task assigned by the MEC server to the task request vehicle n.

[0105] a n =2 means that the task request vehicle n offloads the task to the vehicle cloud server through the V2R link, and the task request vehicle n uploads the task to the vehicle cloud server n sev The upload delay is

[0106]

[0107] Task request vehicle n uploads the task to the vehicle cloud server n sev The computational delay is

[0108]

[0109] For vehicle-mounted cloud server n sevThe computing power of the task assigned to the task request vehicle n. Many studies have shown that the amount of data downloaded from the final calculation result is very small compared to the amount of data uploaded, so the delivery delay can be ignored. Therefore, the delay generated by the task request vehicle n to offload the task to the MEC server is

[0110]

[0111] Task request vehicle n offloads the task to the onboard cloud server n sev The resulting delay is:

[0112]

[0113] (2e), in summary, the following objective function and constraints can be established:

[0114]

[0115] Among them, constraint C1 indicates that the task requesting vehicle n performs the computing task locally, or offloads the task to the MEC server for computing, or offloads the task to the on-board cloud server closest to the task requesting vehicle for computing. Each computing task can only choose one computing method. Constraint C2 indicates the local computing capacity range of the task requesting vehicle n. is the local maximum computing capacity of the task requesting vehicle n. Constraints C3 and C4 mean that the computing capacity allocated by the MEC server and the vehicle-mounted cloud server to the task requesting vehicle n is non-negative. Constraints C5 and C6 indicate that the computing capacity allocated by the MEC server and the vehicle-mounted cloud server to the task requesting vehicle cannot exceed its maximum computing capacity. mec is the maximum computing capacity of the MEC server, F mec is the maximum computing capacity of the onboard cloud server; constraints C7 and C8 indicate that the task request vehicle n performs the task to meet its maximum delay constraint; the constraint condition indicates that the task request vehicle n's transmission power is non-negative and meets its maximum transmission power constraint;

[0116] In step (3), each task requesting vehicle obtains the occupancy of surrounding vehicle-mounted cloud servers and MEC computing resources as well as task information;

[0117] Step (4) is to obtain the unloading decision of each vehicle based on the potential game. According to the unloading decision, a deep reinforcement learning model is established for the vehicle that decides to unload the task with the goal of reducing the system delay, which includes the following specific steps:

[0118] (4a), based on the potential game, the decision of whether to unload each vehicle is obtained. The unloading decision of the task vehicle is modeled as a potential game, expressed as G = {N, (a n ) n∈N ,(u n )n∈N}, where N is the set of vehicles requesting the task, a n Request the unloading decision of vehicle n for the task, u n Cost function for requesting vehicle n for a task.

[0119] In the game model, each task requesting vehicle is a resource competitor, so there are V vehicles competing for limited resources in the network. Each vehicle can choose to offload the computation or perform the task computation locally, where a n ∈{0, 1, 2} is the offloading decision of the task request vehicle n, A represents the offloading decision set of all task request vehicles, and a = 1 means that the task request vehicle n offloads the task to the MEC server for calculation. n = 0 indicates that the task requests vehicle n to perform the computing task locally. n =2 means that the task request vehicle n offloads the task to the vehicle-mounted cloud server closest to the task request for calculation. n When the cost function is expressed as u n (a n , a -n ), where a -n represents the set of unloading decisions of all task request vehicles except task request vehicle n. Each task request vehicle can hope to minimize its own cost by finding the optimal unloading decision, that is,

[0120]

[0121] The potential game converges to the Nash equilibrium, that is, the unloading decision is found through optimal response iteration. All task request vehicles do not exist and changing the current unloading decision can minimize their own costs.

[0122] (4b), according to the offloading decision Use set N o = {1, 2, ..., N0} represents the unloading decision in the task request vehicle The mission request vehicle, N0 represents The number of task request vehicles, the state space S is defined as the observation information related to the transmission power and the low-dimensional fingerprint information, including the task request vehicle n0 to the vehicle cloud server N sev ={1, 2, ..., N sev}Channel state information Mission request vehicle, channel status information to MEC Task request vehicle, to the vehicle cloud server N sev ={1, 2, ..., N sev} received The interference received by the mission request vehicle n0 to the MEC Task request vehicle n0's task information The number of training rounds e and the random exploration variable ε in the ε-greedy algorithm, that is,

[0123]

[0124] Will The task request vehicle is regarded as an intelligent agent, and each time the vehicle is based on the current state Select the offloading node and transmit power;

[0125] (4c), define action A: the action of each task request vehicle that decides to unload is the selected transmission power For continuous variables ranging from 0 to Indicates the mission request vehicle and maximum transmission power;

[0126] (4d), define the reward function r, and the optimization goal is to optimize the offloading decision under the maximum transmission power limit The task request vehicle's selected transmission power is minimized by the offloading decision The task request vehicle's task processing delay, so the reward function can be expressed as:

[0127]

[0128] Where b is a fixed value used to adjust the value of the reward function,

[0129] (4e) Based on the established state, action and reward functions, a deep reinforcement learning model is established, and each agent trains the corresponding action network and evaluation network;

[0130] Step (5), considering the joint optimization problem in the continuous action space, the DDPG algorithm including deep learning fitting, soft update, and replay mechanism is used to optimize the deep reinforcement learning model;

[0131] Deep learning fitting refers to the DDPG algorithm based on the Actor-Critic framework, which uses deep neural networks with parameters θ and δ to fit the deterministic strategy a = μ(s|θ) and action value function Q(s, a|δ).

[0132] Soft update means that the parameters of the action value network are frequently updated while being used to calculate the gradient of the policy network. This makes the learning process of the action value network likely to be unstable. Therefore, it is proposed to use soft update to update the network.

[0133] Create two neural networks, online network and target network, for the policy network and action value network respectively:

[0134]

[0135] During the training process, the network is continuously updated using gradient descent. The target network is updated as follows

[0136] θ′=τθ+(1-τ)θ Expression 18

[0137] δ′=τδ+(1-τ)δ Expression 19

[0138] The experience replay mechanism refers to the temporal correlation of state transition sample data generated during interaction with the environment, which can easily lead to deviations in the action-value function fit. Therefore, drawing on the experience replay mechanism of the deep Q-learning algorithm, the collected samples are first placed in a sample pool, and then a small batch of samples are randomly selected from the sample pool for network training. This process removes correlation and dependency between samples, solves the problem of data correlation and non-stationary distribution, and makes the algorithm more likely to converge.

[0139] Optimizing deep reinforcement learning models using the DDPG algorithm, which includes deep learning fitting, soft update, and replay mechanisms, involves the following steps:

[0140] (5a), initialize the number of training rounds P;

[0141] (5b), initialize the time step t in P rounds;

[0142] (5c), each agent is the online Actor strategy network of the vehicle that decides to unload the task request according to the input state Output Action After all agents perform the corresponding actions, they obtain the reward rt and each agent goes to the next state. Thus, we can obtain the training data of each agent.

[0143] (5d), the training data of each agent Deposit into the corresponding experience replay pool;

[0144] (5e), ​​randomly sample m training data from the experience replay pool for each agent Construct a data set and send it to the online actor policy network, online critic evaluation network, target actor policy network and target critic evaluation network;

[0145] (5f), setting the Q estimate to

[0146] y i =r i +γQ′(s i+1 , μ′(s i+1|θ′)|δ′) Expression 20

[0147] Define the loss function of the online Critic evaluation network as

[0148]

[0149] Update all parameters θ of the current Critic network through the gradient backpropagation of the neural network;

[0150] (5g), define the sampled policy gradient of the online Actor policy network as

[0151]

[0152] Update all parameters δ of the current Actor network through the gradient backpropagation of the neural network;

[0153] (5h), when the number of online training times reaches the target network update frequency, update the target network parameters δ′ and θ′ according to the online network parameters δ and θ respectively;

[0154] (5i), judge whether t < K is satisfied, where K is the total time steps in p rounds. If so, t = t + 1, enter step 5c, otherwise, enter step 5j;

[0155] (5j), judge whether p < I is satisfied, where I is the set threshold of the training round number. If so, p = p + 1, enter step 5b, otherwise, the optimization ends and the optimized deep reinforcement learning model is obtained.

[0156] Step (6), in the execution phase, the task request vehicle obtains the task offloading decision through potential game, determines the task request vehicle for offloading, and obtains the current state according to the local observation For all task request vehicles whose offloading decisions are not local computing, use the trained deep reinforcement learning model to obtain the transmission power, specifically:

[0157] (6a), according to each task request vehicle, obtain the occupancy of the surrounding vehicle-mounted cloud server and MEC computing resources and task information, obtain the vehicle's offloading decision through potential game, and obtain the state information at this moment for all task request vehicles whose offloading decisions are not local computing

[0158] (6b), for each task request vehicle n0 that decides to offload the task, use the trained deep reinforcement learning model and input the state information

[0159] (6c), the task request vehicle n0 outputs the optimal action policy That is, the optimal transmission power of the task request vehicle is obtained

[0160] exist Figure 1 In the paper, a flowchart of the vehicle network computing offloading and power optimization method based on potential game is described. For each task request vehicle, its offloading decision is obtained based on the potential game, and for the task request vehicle that decides to offload, its transmission power is obtained based on the deep reinforcement learning model.

[0161] exist Figure 2 In

[15] , a schematic diagram of the model for collaborative computing offloading between mobile edge servers and on-board cloud servers is described, that is, the task-requesting vehicle can perform the computing task locally, or offload the task to the MEC server for computing, or offload the task to the on-board cloud server closest to the task-requesting vehicle for computing.

[0162] exist Figure 3 In the paper, the algorithm framework of the deep deterministic policy gradient algorithm is described, which includes four networks: online actor policy network, online critic evaluation network, target actor policy network and target critic evaluation network.

[0163] Based on the description of the present invention, it should not be difficult for those skilled in the art to see that the Internet of Vehicles computing offloading and power optimization method based on potential game of the present invention can effectively reduce system latency.

[0164] The contents not described in detail in the present application belong to the prior art known to professional and technical personnel in this field.

Claims

1. A method for computing offloading and power optimization of Internet of Vehicles based on potential game, characterized in that: The steps include: (1) Deploy mobile edge computing (MEC) servers on roadside units (RSUs), and use vehicles that can share their remaining computing resources as on-board cloud servers. The computing tasks of task requesting vehicles can be processed locally and offloaded to the on-board cloud servers or MEC servers. (2) Establish a task request vehicle including N, N sev The communication model and computing model of a vehicle-mounted cloud server and a roadside unit equipped with a MEC server are used to establish a computing offloading and power optimization model. (3) Each task requesting vehicle obtains the computing resource usage of the surrounding on-board cloud servers and MEC as well as task information; (4) Based on the potential game, the offloading decision of each vehicle is obtained, that is, the task requesting vehicle decides to perform local computation, offloads to the MEC server, and then offloads to the vehicle-mounted cloud server closest to the task requesting vehicle. According to the offloading decision, a deep reinforcement learning model is established for the vehicle that decides to offload the task with the goal of reducing system latency; (5) Training deep reinforcement learning model based on DDPG; (6) In the execution phase, the task requesting vehicle obtains the task offloading decision through potential game. The task requesting vehicle n0 that decides to offload obtains the current state based on local observations. For all tasks where the offloading decision is not locally computed, the vehicle is requested to obtain the transmission power using the trained deep reinforcement learning model; Further, the step (2) comprises the following specific steps: (2a), establish the communication model of vehicle network computing offloading. The system includes N task request vehicles, N sev The vehicle-mounted cloud server and a roadside unit deploying the MEC server are used to request the task. Represents, collection represents the on-board cloud server, and the task requesting vehicle n can be expressed as τ n ={c n ,s n , T n max },c n Indicates the number of CPU cycles required for user n to complete the task, s n represents the amount of task data uninstalled by user n, T n max represents the maximum tolerable delay of task execution by user n, and is regarded as one task requesting vehicle generating a task in each time slot. The task requesting vehicle n has a task unloading decision function. Indicates that a n = 0 means the task requests vehicle n to perform the computing task locally, a n =1 means that the task requesting vehicle n offloads the task to the MEC server for calculation via the V2I link, a n =2 means that the task requesting vehicle n offloads the task to the on-board cloud server closest to the task requesting vehicle through the V2V link for calculation. V2V communication and V2I communication do not interfere with each other using 5G slicing technology; (2b), the task requests vehicle n to offload the task to the onboard cloud server n sev The signal to interference and noise ratio (SINR) is expressed as: Task request vehicle n offloads the task to the onboard cloud server n sev The transmission rate is expressed as: in, Indicates that the task request vehicle n is assigned to offload the task to the onboard cloud server n sev The transmission bandwidth, P n represents the transmission power of the task requesting vehicle n, σ 2 represents the noise power, Indicates a task request from vehicle n to the onboard cloud server n sev The channel gain, Indicates that the task requesting vehicles other than the task requesting vehicle n offload the task to the vehicle-mounted cloud server n′ sev Interference with the task request vehicle n in a n′ =2, J(a n′ =2) = 1, otherwise J(a n′ =2)=0, Indicates whether the task requesting vehicle n′ offloads the task to the vehicle-mounted cloud server n′ sev , Indicates that the task requests vehicle n′ to offload the task to the onboard cloud server n′ sev , Indicates that the task requesting vehicle n′ has not offloaded the task to the onboard cloud server n′ sev , P n′ represents the transmission power of the mission request vehicle n′, Indicates the task request vehicle n′ to the vehicle cloud server n′ sev The channel gain of (2c), Similarly, the signal-to-interference-and-noise ratio (SINR) of the task requesting vehicle n to offload the task to the MEC server is expressed as: Task request vehicle n offloads the task to the onboard cloud server n sev The transmission rate is expressed as: Among them, P n represents the transmission power of the task requesting vehicle n, σ 2 represents the noise power, h n，mec represents the channel gain from the task request vehicle n to the MEC server, represents the interference of task requesting vehicles other than task requesting vehicle n in offloading tasks to the MEC server to task requesting vehicle n in a n′ =1, J(a n′ =1) = 1, otherwise J(a n′ =1)=0, Indicates that the task requests vehicle n′ to offload the task to the MEC server, indicates that the task requesting vehicle n′ does not offload the task to the MEC server, P n′ represents the transmission power of the task requesting vehicle n′, h n′,mec represents the channel gain from the task requesting vehicle n′ to the MEC server; (2d), establish a computational model, a n =0 means the task requests vehicle n to perform the computing task locally, represents the local computing capability of the task request vehicle n, then the local computing delay is: a n =1 means that the task requesting vehicle n offloads the task to the MEC server for calculation through the V2I link. The upload delay of the task requesting vehicle n uploading the task to the MEC server is: The computational delay of the task request vehicle n uploading the task to the MEC server is: f n mec The computing capacity of the task assigned by the MEC server to the task request vehicle n; a n =2 indicates that the task request vehicle n offloads the task to the vehicle-mounted cloud server through the V2R link, and the task request vehicle n uploads the task to the vehicle-mounted cloud server n sev The upload delay is Task request vehicle n uploads the task to the vehicle cloud server n sev The computational delay is f n mec For vehicle-mounted cloud server sev The computing power of the task assigned to the task request vehicle n, and the delay incurred by the task request vehicle n to offload the task to the MEC server is Task request vehicle n offloads the task to the onboard cloud server n sev The resulting delay is: (2e), in summary, the following objective function and constraints can be established: Among them, constraint C1 indicates that the task requesting vehicle n performs the computing task locally, or offloads the task to the MEC server for computing, or offloads the task to the on-board cloud server closest to the task requesting vehicle for computing. Each computing task can only choose one computing method. Constraint C2 indicates the local computing capacity range of the task requesting vehicle n. is the local maximum computing capacity of the task requesting vehicle n. Constraints C3 and C4 mean that the computing capacity allocated by the MEC server and the vehicle-mounted cloud server to the task requesting vehicle n is non-negative. Constraints C5 and C6 indicate that the computing capacity allocated by the MEC server and the vehicle-mounted cloud server to the task requesting vehicle cannot exceed its maximum computing capacity. mec is the maximum computing capacity of the MEC server, F mec is the maximum computing capacity of the onboard cloud server; constraints C7 and C8 indicate that the task request vehicle n performs the task to meet its maximum delay constraint; constraint C9 indicates that the task request vehicle n transmits a non-negative power and meets its maximum transmit power constraint; Further, the step (4) comprises the following specific steps: (4a) Based on the potential game, the decision of whether to unload each vehicle is obtained. The unloading decision of the task vehicle is modeled as a potential game, expressed as in The set of vehicles requested for the task, a n Request the unloading decision of vehicle n for the task, u n is the cost function of task request vehicle n. In the game model, each task request vehicle is a resource competitor. Therefore, there are N vehicles competing for limited resources in the network. Each vehicle can choose to offload the calculation or perform the task calculation locally. n ∈{0, 1, 2} is the unloading decision of the task request vehicle n, represents the unloading decision set of all task request vehicles, a n = 0 means the task requests vehicle n to perform the computing task locally, a n =1 means that the task requests vehicle n to offload the task to the MEC server for calculation, a n =2 means that the task requesting vehicle n offloads the task to the on-board cloud server closest to the task requesting vehicle n for calculation. n When the cost function is expressed as u n (a n , a -n ), where a -n represents the unloading decision set of all task request vehicles except task request vehicle n. Each task request vehicle can hope to minimize its own cost by finding the optimal unloading decision, that is, in The latency of the task requesting vehicle n to calculate the task locally, is the latency for task requesting vehicle n to offload the task to the MEC server, Request vehicle n to offload the task to the nearest onboard cloud server n sev The potential game converges to the Nash equilibrium, that is, the unloading decision is found through the optimal response iteration. All task request vehicles do not exist to change the current unloading decision to minimize their own costs; (4b) Based on the uninstall decision Use collection Represents the unloading decision in the task request vehicle The task request vehicle, N0 represents The number of task request vehicles, each task request vehicle n0 that decides to unload is regarded as an intelligent agent, and the state space S is defined as the observation information related to the transmission power and the low-dimensional fingerprint information, including the task request vehicle n0 to the vehicle-mounted cloud server Channel state information Task request vehicle n0 to MEC channel state information Task request vehicle n0 to the vehicle cloud server Received The interference received by the mission request vehicle n0 to the MEC Task request vehicle n0's task information The number of training rounds e and the random exploration variable ε in the ε-greedy algorithm, that is, Will The task request vehicle is regarded as an intelligent agent. Each time vehicle n0 is based on the current state Select the transmit power; (4c) Define action p: The action of each task request vehicle n0 that decides to unload is the selected transmission power For continuous variables between 0 and Indicates the maximum transmission power of vehicle n0 requested by the task; (4d) Define the reward function r. The optimization goal is to optimize the offloading decision under the constraint of maximum transmission power. The task requests the vehicle's transmission power to minimize the offloading decision The task request vehicle's task processing delay, so the reward function can be expressed as: Where b is a fixed value used to adjust the value of the reward function. hour, otherwise hour, otherwise The latency of requesting vehicle n0 to offload the task to the MEC server for the task decided to be offloaded, For the task decided to be unloaded, request vehicle n0 to unload the task to the nearest vehicle cloud server n sev Delay (4e) Based on the established state, action and reward function, a deep reinforcement learning model is established. Each intelligent agent decides to unload the task and requests the vehicle to train the corresponding action network and evaluation network.

Images