Vehicle edge computing system resource management method based on P-DQN

Through parameterization deep reinforcement learning and Markov decision-making process, the resource management of vehicle edge computing system is optimized, and the problems of delay, energy consumption and cost in vehicle edge computing are solved, efficient task offloading and resource allocation are achieved, and dynamic vehicle on-boarding is adapted to Network environment.

CN118885286APending Publication Date: 2024-11-01HARBIN INST OF TECH AT WEIHAI
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410777405.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-17
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

The prior art is difficult to effectively manage latency, energy consumption and costs in vehicle edge computing. Especially in dynamic vehicle network environments, traditional convex optimization and game theory methods have high computational complexity and complexity, which cannot meet the application of intelligent vehicle Low latency and high efficiency requirements.

Method used

Using a method based on parameterized deep reinforcement learning, a three-layer task offload system is built, and the resource management of vehicle edge computing system is optimized through Markov decision-making process and parameterized Q network (P-DQN), combining local, edge and cloud servers. Task unload decisions, design state space and reward functions, and handle hybrid action spaces for optimal unload decisions and resource allocation.

Benefits of technology

It realizes the minimization of the long-term delay, energy consumption and cost of the system in the vehicle edge computing environment, improves resource utilization and system performance, adapts to dynamic vehicle network environments, and provides more efficient task offloading and resource allocation strategies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118885286A_ABST
    Figure CN118885286A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of vehicle system resource management, in particular to a vehicle edge computing system resource management method based on P-DQN, which utilizes parameterized deep reinforcement learning to solve a real-time unloading decision problem so as to minimize long-term delay, energy consumption and cost of a system. Constructing a three-layer task unloading system comprising a local server, an edge server and a cloud layer; according to the method, an optimization problem which jointly considers delay, energy consumption and cost is constructed, the problem is framed into a Markov decision process MDP, a state space and a reward function are designed, a parameterized behavior space is introduced, and the mixed behavior space is effectively managed; in addition, a method based on parameterized Q network P-DQN is provided to solve the optimization problem.
Need to check novelty before this filing date? Find Prior Art

Description

Technical field:

[0001] The present invention relates to the technical field of vehicle system resource management, and specifically to a vehicle edge computing system resource management method based on P-DQN, which utilizes parameterized deep reinforcement learning to solve real-time unloading decision problems to minimize the long-term delay, energy consumption and cost of the system. Background technology:

[0002] The Internet of Vehicles is expected to become a key component of future Intelligent Transportation Systems (ITS). By connecting numerous network entities such as vehicles, roadside units (RSUs), and pedestrians within the transportation system to establish communication links, IoV is able to provide safe and high-quality services to drivers and passengers. The advancement of smart vehicles has driven the innovation of many smart vehicle applications, including augmented reality, real-time navigation, voice recognition, and autonomous driving. The above applications are closely related to safety and generate computationally intensive tasks that require high reliability and low latency. However, the limited computing power of on-board devices can hardly meet the computing needs of these applications.

[0003] Mobile Edge Computing (MEC) technology integrates computing resources and intelligence to the edge of the network to provide fast and efficient computing services to vehicles, thus providing a promising solution to these challenges. Offloading computing tasks to MEC servers close to the terminals effectively alleviates the problem of insufficient onboard computing resources. Despite the many advantages of edge computing technology, it faces challenges in multiple application areas. For example, equipment operators need to pre-deploy roadside units and edge servers. Designing efficient computing offloading and resource allocation algorithms is essential to improving resource utilization and edge computing system performance. In short, vehicle users' offloading decisions, balancing energy consumption and efficiency, and allocating computing resources are all crucial to the system's Quality of Service (QoS).

[0004] In order to cope with the challenges in mobile edge computing, a large number of studies have emerged in recent years. A large number of papers focus on developing optimal strategies for task offloading and resource management under various requirements, aiming to optimize latency, energy consumption or quality of service (QoS) through systematic algorithm design. Most of the existing technical solutions rely on convex optimization and game theory. Algorithms based on convex optimization usually exhibit high computational complexity, while game theory-based solutions require multiple iterations to reach Nash equilibrium, which increases the complexity of their practical application in dynamic vehicular networks. Summary of the invention:

[0005] In view of the shortcomings and deficiencies of the prior art, the present invention proposes a vehicle edge computing system resource management method based on P-DQN, which utilizes parameterized deep reinforcement learning to solve real-time unloading decisions to minimize the long-term delay, energy consumption and cost of the system.

[0006] The present invention is achieved by the following measures:

[0007] A resource management method for a vehicle edge computing system based on P-DQN is characterized by constructing a three-layer task offloading system including local, edge server and cloud layers; constructing an optimization problem that jointly considers delay, energy consumption and cost, and framing the problem as a Markov decision process MDP, designing the state space and reward function, and introducing a parameterized behavior space to effectively manage the mixed behavior space; in addition, a method based on parameterized Q network P-DQN is proposed to solve the optimization problem.

[0008] The present invention constructs a three-layer task computing model including local processing, edge offloading and cloud offloading. The system includes N vehicles, M evenly distributed roadside units RSUs and a cloud server. The vehicle set is represented as Each RSU has the same coverage $R$, and the RSU set is expressed as These RSUs are connected to the remote cloud server through wired connections and core networks. The model divides the road into M segments, and vehicles are randomly distributed on these segments. The total length of the road is represented as L. Each RSU is equipped with a MEC server with a computing power of F. The collection of these MEC servers is represented as RSUs equipped with MEC servers can be regarded as edge nodes, providing reliable and rapid task offloading services to vehicles within their coverage area. These edge nodes are interconnected through wired connections. When the access node is overloaded and cannot process computing tasks, the vehicle can offload tasks to adjacent nodes or cloud servers, thereby alleviating the imbalance of edge computing resources. Cloud servers have huge computing resources and can handle highly complex tasks or serve as supplementary resources during periods of insufficient network computing resources. In addition, time is divided into time periods of equal length, represented as t = {1, 2, …, T}. It is assumed that each vehicle generates at most one task in each time period, and the duration of each time period is Δt.

[0009] The three-layer task calculation model of the present invention also takes into account the mobility of the vehicle, assuming that the vehicle speed follows a truncated Gaussian distribution, reflecting the randomness of the speed while excluding negative values. The probability density function of the truncated Gaussian distribution is as follows:

[0010]

[0011] in represents the probability density function of Gaussian distribution, μ and σ represent the average speed and standard deviation of the vehicle, respectively, V max =μ+3σ and V min =μ-3σ represent the maximum speed and minimum speed of the vehicle respectively, is the error function.

[0012] In order to describe the computing tasks of the vehicle, the present invention adopts the following task model: for vehicle i, the computing task generated in time slot t is usually represented as a triple, which includes the data size of the task, the required computing resources and the deadline of the task, which is expressed as follows:

[0013]

[0014] Among them, d i (t) represents the computational task The input data size; ω i (t) = λ i (t)·d i (t) represents the task The computational intensity of the task, specifically, the number of CPU cycles required to complete the task, where λ i (t) is a coefficient that describes d i (t) and ω i (t), called the service coefficient, is determined by the computational complexity of the task; Representation Task Maximum tolerable delay; Based on the described framework, vehicles can choose to process computing tasks locally or offload them to MEC servers or cloud servers. For each computing task, a vehicle can offload to at most one server. Although vehicles can only communicate directly with the RSU in their area, they can offload tasks to the MEC servers of neighboring RSUs through this RSU. Therefore, for vehicle i, the task in time period t The possible unloading decision set includes M+2 options, denoted as a i (t) = {0, 1, 2, ..., M + 1}. i (t) = 0 means local execution, a i (t) = M + 1 represents cloud offloading, a i (t) = i (1≤i≤M) represents offloading to MEC server i. In addition, for vehicle i, the offloading symbol is defined as Represents a set of uninstallation policies. Specifically, when a i When (t) = 0, Indicates local processing, otherwise when a i When (t) = i (1≤i≤M), Indicates offloading to MEC server i, otherwise when a i When (t) = M + 1, Indicates cloud uninstallation, otherwise

[0015] In the present invention, when a vehicle decides to unload a task, it first needs to upload the input data of the task to the corresponding RSU, and receive the output data from the RSU after the task is completed. The vehicle-RSU communication model based on vehicle-to-infrastructure V2I communication is as follows: Assuming that the uplink channel from vehicle i to RSU is a Rayleigh fading channel, and inter-channel interference is not considered, the channel gain is expressed as follows:

[0016]

[0017] where g im represents the small-scale fast fading power component, which follows an exponential distribution with a mean of one, A represents the path loss constant, and L im is the distance between vehicle i and RSU m, γ represents the attenuation index, β im is a log-normal shadow fading random variable with standard deviation ξ. Assume that the total uplink bandwidth of RSU is B, and at a given time slot t, there are N(t) tasks that need to be offloaded to RSU m. Under these conditions, the uplink transmission rate from vehicle i to RSU m can be calculated as follows:

[0018] where p i (t) is the transmission power of vehicle i in time slot t, and the channel gain from vehicle i to RSU m is h im (t), the background noise power is σ 2 .

[0019] The task processing models under various task execution strategies selected by the vehicle in the present invention include:

[0020] (1) Local execution: The vehicle uses the onboard computing resources to process computing tasks. Assume that the computing capacity of vehicle i is Indicates that local tasks are completed The time and energy consumption are calculated as follows:

[0021]

[0022] Among them, κ is an energy consumption parameter that depends on the chip architecture. When the task is processed locally, the vehicle does not need to pay for the computing resources of the server operator, so the cost of vehicle i is expressed as

[0023] (2) MEC server execution: When a vehicle offloads a computing task to a MEC server, it must first upload its input data through the uplink of the RSU. The task is then executed by the MEC server connected to the RSU. Considering the close distance between the RSU and the MEC server, the transmission time of the input data between them can be ignored. The time required for vehicle i to upload input data to MEC server m can be given by the following formula:

[0024]

[0025] Assuming that computing tasks are executed in parallel on MEC servers, the computing resources requested by vehicle i from MEC server m are expressed as Therefore, the task The execution time and cost of offloading to MEC server m is:

[0026]

[0027]

[0028] where ρ E Represents the unit price of the computing resources of the MEC server; after the task is executed on the MEC server, the calculation result is transmitted back to the vehicle through the downlink of the RSU. Given that the output data volume of the task is relatively small, the delay and energy consumption of transmitting the result back to the vehicle can be ignored. However, considering the dynamic nature of high-speed vehicles, if vehicle i leaves the coverage area of ​​the corresponding RSU, the calculation result must first be transmitted to the RSU of the current road segment through edge node communication, and then the result is downloaded. Therefore, the total transmission time of the calculation result back to the vehicle can be expressed as:

[0029] Where Δx represents the number of road segments between vehicle i and edge node m, t inter represents the transmission delay of the calculation result data between adjacent edge nodes. Let x i Let k denote the position of vehicle i on the road, and let k denote the vehicle is located in the coverage area of ​​the kth RSU, then Δx = |km|, where Indicates that x is rounded down. Finally, the energy consumption and cost of the MEC server execution solution can be calculated as:

[0030]

[0031] (3) Cloud server execution: When offloading computing tasks to the cloud server, the input data is first uploaded from the vehicle to the corresponding roadside unit (RSU), and then uploaded from the RSU to the cloud server. However, since the data transmission between the cloud server and the RSU involves long-distance communication, the upload time from the RSU to the cloud server and the data transmission time from the cloud server to the RSU must be considered. Therefore, the total delay of the cloud server processing decision is as follows:

[0032] in represents the computing resources allocated by the cloud server to vehicle i, t c represents the constant time of data transmission between RSU and cloud server, assuming ρ c represents the unit price of cloud server computing resources. The energy consumption and cost of the cloud server processing model are expressed as:

[0033]

[0034] The present invention constructs an optimization problem in the vehicle computing unloading scenario, task The execution delay, energy consumption and cost of are expressed as follows:

[0035]

[0036]

[0037]

[0038] The utility function of vehicle i is defined as:

[0039]

[0040] Where w1, w2, w3 are the weight parameters of latency, energy consumption and cost respectively, and w1+w2+w3=1. The maximum tolerable delay is However, if the vehicle drives out of the entire road section, the result of the unloading task will not be transmitted back through the RSU. is the remaining time of vehicle i on the road; therefore, the task The actual delay tolerance is Based on the above system model analysis, the unloading of all vehicles and their decisions is expressed as and Furthermore, services are defined as the computational resources and transmission power allocated to vehicles, respectively and In order to maximize the long-term utility of the system, the optimization problem is formulated as follows:

[0041]

[0042]

[0043]

[0044] a i (t)∈{0,1,…,M+1}(23),

[0045] T i (t)≤T max ,i∈N(24),

[0046]

[0047]

[0048] P min ≤p i (t)≤P max (27),

[0049] The above optimization problem aims to determine the optimal unloading decision and resource allocation and Minimize the long-term delay, energy consumption and cost of the system. Constraints (21) and (22) ensure that each vehicle chooses only one offloading option: local, MEC server or cloud server. (23) Decision variables are defined, indicating that the task may be offloaded to any server or executed locally. (24) Time constraints for task execution are specified. (25) The resources allocated to the vehicle by the MEC server must not exceed its total computing capacity. (26) An upper limit is set on the resources that the cloud server can allocate to each vehicle without limiting the total capacity of the cloud server. (27) The transmission power of each vehicle is limited to not exceed the maximum allowed value.

[0050] The present invention proposes an unloading algorithm based on deep reinforcement learning to solve the above optimization problem, which specifically includes:

[0051] First, the above optimization problem is transformed into a Markov decision process, MDP can be represented by a four-tuple Characterization, among which, Represents the set of system states, i.e., the state space; represents the action space; r(s|a) is the reward function that depends on the state and action, which is simplified to r(s) when the reward is only affected by the state. P(s′|s,a) is the state transition function, which represents the probability of transferring to state s′ after executing action a in state s. However, due to the lack of prior environmental knowledge, the state transition function is usually unknown, which makes it complicated to obtain the optimal solution through the dynamic programming of MDP. Therefore, deep reinforcement learning is applied to solve the proposed optimization problem. The goal of the agent is to maximize the long-term cumulative reward. Initially, the agent observes state s from the environment. t And select action a according to the current strategy. After executing the action, the system transfers to state s t+1 , and returns a reward to the agent, which then updates its strategy based on this feedback and eventually converges to the optimal strategy.

[0052] The present invention constructs the state space, action space and reward of the deep reinforcement learning algorithm based on the proposed model:

[0053] (1) State space: For task offloading decisions and resource allocation, we must first determine the server’s computing resource distribution, the variable sizes of all computing tasks, and the vehicle’s position change information. Therefore, in time period t, the state It consists of four components:

[0054] s t ={F m (t),X i (t),V i (t),D i (t)} (28), for s t The detailed explanation is as follows:

[0055] F m (t)=[f1(t),f2(t),…,f M (t)] represents the current computing resource distribution of edge nodes. This array contains the computing resource configuration of each edge node at time t, where f i (t) represents the amount of available computing resources of the i-th edge node. This information is crucial for understanding the load and resource usage of each edge node at a specific point in time, and then more efficient resource allocation and task offloading strategies can be formulated based on this data;

[0056] X i (t) and V i (t) describes the movement of all vehicles, where X i (t) records the vehicle position at time slot t in detail, while V i (t) represents the speed of the vehicle. This information affects the urgency of the computing task and the choice of offloading to the edge node. The state vector of the vehicle position change is defined as:

[0057] X i (t)=[x1(t),x2(t),…,x N (t)] (29),

[0058] V i (t)=[v1(t),v2(t),…,v N (t)] (30),

[0059] Vector D i (t)=[d1(t),d2(t),…,d N (t)] records the input data size of each task waiting to be executed. The input data size directly affects the decision of task offloading, including the location and method of offloading, which also affects the data transmission time and the computing resources required for computing tasks;

[0060] (2) Action space: When the state s is observed t When , vehicle i needs to make an unloading decision. The dimension of the action vector depends on the unloading decision and can be expressed as:

[0061]

[0062] Among them, a i (t) represents the unloading decision of vehicle i. As mentioned above, a i (t) = 0 means local execution; a i (t)>0 means offloading to the server, defined in space It covers M+2 dimensions. and represent the resources of MEC server m and cloud server allocated to vehicle i, respectively, and p i (t) represents the transmission power of the vehicle,

[0063] In order to handle heterogeneous action vectors, this paper introduces a parameterized action space with a hierarchical structure. Vehicles first determine their discrete actions a i (t), and then select the associated continuous parameter in is a vector of variable dimension, defined as For all Continuous set of discrete-continuous mixed action spaces It can be expressed as:

[0064]

[0065] Reward: The agent adjusts its strategy based on the reward received. According to the objective function (20), the reward at time t is defined as follows:

[0066] Where R p represents the penalty introduced to ensure that the agent's actions comply with constraint (24). In addition, the calculation formula for the long-term cumulative discounted reward is:

[0067]

[0068] The vehicle network unloading decision and resource allocation algorithm using P-DQN in the present invention includes: first, the deterministic strategy network μ(θ) is based on the current state s t Generate all continuous parameters, then the state s t The values ​​of the continuous parameters are input into the value network Q(ω), and the algorithm then selects the discrete action corresponding to the highest Q value. Finally, the algorithm combines the selected discrete actions and their corresponding continuous parameters into an action vector and outputs it.

[0069] In the P-DQN algorithm, the Q network is used to approximate the action-value function based on the Bellman equation. In the mixed action space, the equation can be expressed as follows:

[0070]

[0071] Furthermore, when the Q function is fixed, the process of determining continuous parameters can be expressed as a function y(s), and y(s) can be approximated by a deterministic policy network μ(s; θ), whose weight is represented by θ. Therefore, the approximation of the Bellman equation can be expressed as follows:

[0072] Among them, ω * represents the optimal value of the network weight ω. Similar to DDPG and DQN, P-DQN adopts the experience replay mechanism and uses the target network to calculate the temporal difference TD target. The network weight ω is optimized by minimizing the TD loss. The loss functions for ω and θ are defined by equations (37) and (38), respectively:

[0073]

[0074]

[0075] where y t is the TD target, which is defined as shown in formula (39):

[0076]

[0077] The resource allocation algorithm based on P-DQN in the present invention generates computing task requirements at the beginning of each time slot with a specific probability. The base station senses the tasks and location information of all vehicles, aggregates these data into environmental state information, and then broadcasts them. Each agent makes decisions based on the state of the environment. In addition, the transitions (state changes) are stored in the experience replay buffer. The agent updates its network weights using the data from the replay buffer and eventually converges to the optimal strategy. After the training phase and in subsequent deployment, the DRL-based algorithm can be based on s t Respond quickly, thereby achieving dynamic task offloading and resource allocation in the vehicle edge computing environment. Description of the drawings:

[0078] Figure 1 It is a schematic diagram of the system model of the present invention.

[0079] Figure 2 It is a schematic diagram of the basic framework of reinforcement learning in the present invention.

[0080] Figure 3 It is a schematic diagram of the basic architecture of the offloading strategy based on P-DQN in the present invention.

[0081] Figure 4 It is the network structure of P-DQN in the present invention.

[0082] Figure 5 It is the convergence curve of the algorithm corresponding to different value network learning rates in the embodiment of the present invention.

[0083] Figure 6 It is the convergence curve of the algorithm corresponding to different parameter network learning rates in the embodiment of the present invention.

[0084] Figure 7 It is the convergence curve of different algorithms in the embodiment of the present invention.

[0085] Figure 8 It is the reward of the algorithm under different uplink total bandwidths in the embodiment of the present invention.

[0086] Fig. 9 It is the reward of the algorithm under different task generation probabilities in the embodiment of the present invention.

[0087] Fig.10 It is the reward of the algorithm under different numbers of vehicles in the embodiment of the present invention.

[0088] Fig.11 It is a curve showing the change of algorithm reward with service coefficient in the embodiment of the present invention.

[0089] Fig.12 are the rewards, average delays, and average costs obtained by the algorithm under different weights in the embodiment of the present invention, where Fig.12 (a) is the reward obtained by the algorithm under different weights, Fig.12 (b) Average delay of the algorithm under different weights, Fig.12 (c) is the average cost of the algorithm under different weights. Specific implementation method:

[0090] The present invention focuses on the binary unloading problem in the vehicle edge computing network. These optimization problems are usually formulated as mixed integer nonlinear programming (MINLP). These non-convex problems are of high complexity, and when using deep reinforcement learning, it is necessary to deal with a mixed action space containing discrete and continuous actions. Existing studies usually deal with mixed action spaces by discretizing continuous behaviors or relaxing discrete action spaces. However, such an approach increases the complexity of the problem and reduces the accuracy of the solution. To address the above challenges, the present invention uses parameterized deep reinforcement learning to solve the real-time unloading decision problem. The goal of the present invention is to minimize the long-term delay, energy consumption and cost of the system. Specifically, the contributions of the present invention are as follows:

[0091] The task offloading problem in cloud-assisted vehicle edge computing is addressed by building a three-layer task offloading architecture consisting of local, edge server, and cloud layers. This architecture alleviates the challenges of edge server resource limitations and unbalanced computing resources in vehicle edge computing environments.

[0092] This paper constructs an optimization problem that jointly considers latency, energy consumption, and cost, and frames the problem as a Markov decision process (MDP). We design the state space and reward function, and introduce a parameterized behavior space to effectively manage the mixed behavior space. In addition, we propose an algorithm based on a parameterized Q network (P-DQN) to solve the optimization problem.

[0093] The effectiveness of the proposed scheme is verified through extensive numerical simulations. The simulation results show that the proposed scheme has the best performance compared with the benchmark method.

[0094] The present invention will then provide a detailed introduction to the network model, as well as a description of the local computing and task offloading model. Finally, the task offloading decision and resource allocation optimization problem are defined based on the model, with the goal of minimizing the long-term delay, energy consumption and cost of the system.

[0095] like Figure 1 As shown in Figure 2, the network consists of N vehicles, M evenly distributed roadside units (RSUs), and a cloud server. The vehicle set is represented as Each RSU has the same coverage $R$, and the RSU set is expressed as These RSUs are connected to the remote cloud server through wired connections and core networks. The model divides the road into M segments, and vehicles are randomly distributed on these segments. The total length of the road is represented as L. Each RSU is equipped with a MEC server with a computing power of F. The collection of these MEC servers is represented as RSUs equipped with MEC servers can be regarded as edge nodes, providing reliable and rapid task offloading services to vehicles within their coverage area. These edge nodes are interconnected through wired connections. In the case where the access node is overloaded and cannot process computing tasks, the vehicle can offload tasks to adjacent nodes or cloud servers, thereby alleviating the imbalance of edge computing resources. Cloud servers have huge computing resources and are capable of handling highly complex tasks or serving as supplementary resources during periods of insufficient network computing resources. In addition, time is divided into time periods of equal length, denoted as t = {1, 2, ..., T}, assuming that each vehicle generates at most one task in each time period, and the duration of each time period is Δt.

[0096] The mobility of the vehicle is also taken into account in the model. It is assumed that the vehicle speed follows a truncated Gaussian distribution, reflecting the randomness of the speed while excluding negative values. The probability density function of the truncated Gaussian distribution is as follows.

[0097]

[0098] in represents the probability density function of Gaussian distribution, μ and σ represent the average speed and standard deviation of the vehicle respectively. max =μ+3σ and V min =μ-3σ represent the maximum speed and minimum speed of the vehicle respectively, is the error function.

[0099] In order to describe the computing tasks of the vehicle, the present invention adopts the following task model: for vehicle i, the computing task generated in time slot t is usually represented as a triple. This triple includes the data size of the task, the required computing resources and the deadline of the task, which is expressed as follows:

[0100]

[0101] Among them, d i (t) represents the computational task The input data size; ω i (t) = λ i (t)·d i (t) represents the task The computational intensity of a task, specifically, the number of CPU cycles required to complete the task. Here, λ i (t) is a coefficient that describes d i (t) and ωi (t), called the service coefficient, is determined by the computational complexity of the task; Representation Task The maximum delay that can be tolerated.

[0102] Based on the described framework, the present invention constructs a three-layer task computing model including local processing, edge offloading, and cloud offloading. Vehicles can choose to process computing tasks locally or offload them to MEC servers or cloud servers. For each computing task, the vehicle can offload to at most one server. Although vehicles can only communicate directly with the RSU in their area, they can offload tasks to the MEC server of the neighboring RSU through this RSU. Therefore, for vehicle i, task i in time period t The possible unloading decision set includes M+2 options, denoted as a i (t) = {0, 1, 2, ..., M + 1}. i (t) = 0 means local execution, a i (t) = M + 1 represents cloud offloading, a i (t) = i (1≤i≤M) represents offloading to MEC server i. In addition, for vehicle i, the offloading symbol is defined as represents a set of uninstallation policies. Specifically, when a i When (t) = 0, Indicates local processing, otherwise when a i When (t) = i (1≤i≤M), Indicates offloading to MEC server i, otherwise when a i When (t) = M + 1, Indicates cloud uninstallation, otherwise

[0103] When a vehicle decides to unload a task, it first needs to upload the input data of the task to the corresponding RSU, and receive the output data from the RSU after the task is completed. Below, we will introduce the vehicle-RSU communication model based on vehicle-to-infrastructure (V2I) communication. It is assumed that the uplink channel from vehicle i to RSU is a Rayleigh fading channel, and inter-channel interference is not considered. The channel gain is expressed as follows:

[0104]

[0105] where g im represents the small-scale fast fading power component, which follows an exponential distribution with a mean of one. A represents the path loss constant, and L imis the distance between vehicle i and RSU m. γ represents the attenuation exponent, β im is a log-normal shadow fading random variable with standard deviation ξ. Assume that the total uplink bandwidth of RSU is B, and at a given time slot t, there are N(t) tasks that need to be offloaded to RSUm. Under these conditions, the uplink transmission rate from vehicle i to RSU m can be calculated as follows:

[0106]

[0107] where p i (t) is the transmission power of vehicle i in time slot t, and the channel gain from vehicle i to RSU m is h im (t), the background noise power is σ 2 .

[0108] Based on the network and task models described above, this summary analyzes the task processing models under various task execution strategies selected by the vehicle.

[0109] 1) Local execution: The vehicle uses the onboard computing resources to process computing tasks. Assume that the computing power of vehicle i is Indicates that local tasks are completed The time and energy consumption are calculated as follows:

[0110]

[0111]

[0112] Among them, κ is an energy consumption parameter that depends on the chip architecture. In addition, the present invention also evaluates the cost of offloading tasks in the vehicle edge computing environment. When the task is processed locally, the vehicle does not need to pay for the computing resources of the server operator, so the cost of vehicle i is expressed as

[0113] 2) MEC server execution: When a vehicle offloads a computing task to a MEC server, it must first upload its input data through the uplink of the RSU, and then the task is executed by the MEC server connected to the RSU. Considering the close distance between the RSU and the MEC server, the transmission time of the input data between them can be ignored. The time required for vehicle i to upload input data to MEC server m can be given by the following formula:

[0114]

[0115] Assuming that computing tasks are executed in parallel on MEC servers, the computing resources requested by vehicle i from MEC server m are expressed as Therefore, the task The execution time and cost of offloading to MEC server m is:

[0116]

[0117]

[0118] where ρ E Indicates the unit price of MEC server computing resources.

[0119] After the task is executed on the MEC server, the calculation results are transmitted back to the vehicle through the downlink of the RSU. Given that the output data volume of the task is relatively small, the delay and energy consumption of transmitting the results back to the vehicle can be ignored. However, considering the dynamic nature of high-speed vehicles, if vehicle i leaves the coverage area of ​​the corresponding RSU, the calculation results must first be transmitted to the RSU of the current road segment through edge node communication, and then the results are downloaded. Therefore, the total transmission time for the calculation results to be transmitted back to the vehicle can be expressed as:

[0120]

[0121] Where Δx represents the number of road segments between vehicle i and edge node m, t inter represents the transmission delay of the calculation result data between adjacent edge nodes. Let x i Let represent the position of vehicle i on the road, and let k represent the vehicle is located in the coverage area of ​​the kth RSU. Δx = |km|, where Indicates that x is rounded down. Finally, the energy consumption and cost of the MEC server execution solution can be calculated as:

[0122]

[0123] Cloud server execution: When offloading computational tasks to a cloud server, input data is first uploaded from the vehicle to the corresponding roadside unit (RSU), and then uploaded from the RSU to the cloud server. However, since data transmission between the cloud server and the RSU involves long-distance communication, the upload time from the RSU to the cloud server and the data transmission time from the cloud server to the RSU must be considered. Therefore, the total latency of the cloud server processing decision is as follows:

[0124]

[0125] in represents the computing resources allocated by the cloud server to vehicle i, t c represents the constant time of data transmission between RSU and cloud server. Assume ρ c represents the unit price of cloud server computing resources. The energy consumption and cost of the cloud server processing model are expressed as:

[0126]

[0127]

[0128] This paper constructs the optimization problem in the vehicle computing unloading scenario. Based on the previous analysis, the task The execution delay, energy consumption and cost can be expressed as follows:

[0129]

[0130]

[0131]

[0132] The goal of this paper is to determine the optimal unloading and resource allocation strategy to minimize the delay, energy consumption and cost of vehicle tasks. Therefore, the utility function of vehicle i is defined as:

[0133]

[0134] Where w1, w2, w3 are weight parameters of latency, energy consumption and cost respectively, and w1+w2+w3=1.

[0135] Despite the task The maximum tolerable delay is However, if the vehicle drives out of the entire road section, the result of the unloading task will not be transmitted back through the RSU. is the remaining time of vehicle i on the road; therefore, the task The actual delay tolerance is Based on the above system model analysis, the unloading of all vehicles and their decisions is expressed as and Furthermore, services are defined as the computational resources and transmission power allocated to vehicles, respectively and In order to maximize the long-term utility of the system, the optimization problem is formulated as follows:

[0136]

[0137]

[0138]

[0139] a i (t)∈{0,1,...,M+1}(23),

[0140] T i (t)≤T max ,i∈N(24),

[0141]

[0142]

[0143] P min ≤p i (t)≤P max (27),

[0144] The above optimization problem aims to determine the optimal unloading decision and resource allocation and Minimize the long-term delay, energy consumption, and cost of the system. Constraints (21) and (22) ensure that each vehicle chooses only one offloading option: local, MEC server, or cloud server. (23) Decision variables are defined, indicating that the task may be offloaded to any server or executed locally. (24) Time constraints for task execution are specified. (25) The resources allocated to the vehicle by the MEC server must not exceed its total computing capacity. (26) An upper limit is set on the resources that the cloud server can allocate to each vehicle without limiting the total capacity of the cloud server. Finally, (27) The transmission power of each vehicle is limited to not exceed the maximum allowed value.

[0145] Obviously, (20) proposes a mixed integer nonlinear programming problem, which is difficult to solve using traditional methods. Traditional convex optimization solutions have extremely high computational complexity and cannot adapt to the rapidly changing environment of vehicle networks, thus failing to achieve the purpose of adaptive task offloading. Therefore, this paper proposes an offloading algorithm based on deep reinforcement learning to meet this challenge. In addition, in order to overcome the limitation of previous studies that variables need to be relaxed or discretized, we introduce a parameterized action space to handle mixed discrete-continuous mixed action spaces.

[0146] The present invention first transforms the aforementioned optimization problem into a Markov decision process. MDP can be composed of a four-tuple Characterization. Among them, Represents the set of system states, i.e., the state space; represents the action space; r(s|a) is the reward function that depends on the state and action, which is simplified to r(s) when the reward is only affected by the state. P(s′|s,a) is the state transition function, which represents the probability of transitioning to state s′ after executing action a in state s. However, due to the lack of prior environmental knowledge, the state transition function is usually unknown, which makes it complicated to obtain the optimal solution through dynamic programming of MDP. Therefore, deep reinforcement learning is applied to solve the proposed optimization problem. Figure 2 The basic framework of the DRL method is described. The goal of the agent is to maximize the long-term cumulative reward. Initially, the agent observes the state s from the environment tAnd select action a according to the current strategy. After executing the action, the system transfers to state s t+1 , and returns a reward to the agent, which then updates its strategy based on this feedback and eventually converges to the optimal strategy.

[0147] Next, we will construct the state space, action space, and reward of the deep reinforcement learning algorithm based on the proposed model.

[0148] 1) State space: For task offloading decisions and resource allocation, we must first determine the server's computing resource distribution, the variable size of all computing tasks, and the vehicle's position change information. Therefore, at time period t, the state It consists of four components:

[0149] s t ={F m (t),X i (t),V i (t),D i (t)}(28),

[0150] For t The detailed explanation is as follows:

[0151] F m (t)=[f1(t),f2(t),...,f M (t)] represents the current computing resource distribution of edge nodes. This array contains the computing resource configuration of each edge node at time t, where f i (t) represents the amount of available computing resources of the i-th edge node. This information is crucial for understanding the load and resource usage of each edge node at a specific point in time, and then more efficient resource allocation and task offloading strategies can be formulated based on this data.

[0152] X i (t) and V i (t) describes the movement of all vehicles, where X i (t) records the vehicle position at time slot t in detail, while V i (t) represents the speed of the vehicle. This information affects the urgency of the computational task and the choice of offloading to the edge node. The state vector of the vehicle position change is defined as:

[0153] X i (t)=[x1(t),x2(t),...,x N (t)] (29),

[0154] V i (t)=[v1(t),v2(t),...,v N (t)] (30),

[0155] Vector D i (t)=[d1(t),d2(t),...,d N (t)] records the input data size of each task waiting to be executed. The input data size directly affects the decision of task offloading, including the location and method of offloading, which also affects the data transfer time and the computing resources required for computing the task.

[0156] 2) Action space: When the state s is observed t When , vehicle i needs to make an unloading decision. The dimension of the action vector depends on the unloading decision and can be expressed as:

[0157]

[0158] Among them, a i (t) represents the unloading decision of vehicle i. As mentioned above, a i (t) = 0 means local execution; a i (t)>0 means offloading to the server, defined in space It covers M+2 dimensions. and represent the resources of MEC server m and cloud server allocated to vehicle i, respectively, and p i (t) represents the transmission power of the vehicle.

[0159] In order to handle heterogeneous action vectors, this paper introduces a parameterized action space with a hierarchical structure. Vehicles first determine their discrete actions a i (t), and then select the associated continuous parameter in is a vector of variable dimension. Definition For all Continuous set of discrete-continuous mixed action spaces It can be expressed as:

[0160]

[0161] 3) Reward: The agent adjusts its strategy based on the reward received. According to the objective function (20), the reward at time t is defined as follows:

[0162]

[0163] Where R p represents the penalty introduced to ensure that the agent's actions comply with constraint (24). In addition, the calculation formula for the long-term cumulative discounted reward is:

[0164]

[0165] The following introduces a vehicle network unloading decision and resource allocation algorithm using P-DQN. Its architecture is as follows Figure 3 P-DQN is a deep reinforcement learning algorithm for discrete-continuous mixed action space, which can effectively solve the optimization problem outlined in this invention. Figure 4 The basic network structure of P-DQN is shown in detail, which combines the characteristics of DQN and DDPG. First, the deterministic policy network μ(θ) is based on the current state s t Generate all continuous parameters. Then, the state s t The values ​​of the continuous parameters are input into the value network Q(ω), and the algorithm then selects the discrete action corresponding to the highest Q value. Finally, the algorithm combines the selected discrete actions and their corresponding continuous parameters into an action vector and outputs it.

[0166] In the P-DQN algorithm, the Q network is used to approximate the action-value function based on the Bellman equation. In the mixed action space, the equation can be expressed as follows:

[0167]

[0168] Furthermore, when the Q function is fixed, the process of determining continuous parameters can be expressed as a function y(s), and y(s) can be approximated by a deterministic policy network μ(s; θ), whose weights are represented by θ. Therefore, the approximation of the Bellman equation can be expressed as follows:

[0169]

[0170] Among them, ω * Represents the optimal value of the network weight ω. Similar to DDPG and DQN, P-DQN adopts the experience replay mechanism and uses the target network to calculate the temporal difference (TD) target. The network weight ω is optimized by minimizing the TD loss. The loss functions for ω and θ are defined by equations (37) and (38), respectively.

[0171]

[0172]

[0173] where y t is the TD target, which is defined as shown in formula (39).

[0174]

[0175] Algorithm 1 describes the resource allocation algorithm based on P-DQN. At the beginning of each time slot, each vehicle generates a computational task demand with a specific probability. The base station senses the task and location information of all vehicles, aggregates this data into environmental state information, and then broadcasts it. Each agent makes decisions based on the state of the environment. In addition, the transitions (state changes) are stored in the experience replay buffer. The agent uses the data from the replay buffer to update its network weights and eventually converge to the optimal strategy. After the training phase and in subsequent deployments, the DRL-based algorithm is able to respond to the optimal strategy based on s t Respond quickly, thereby achieving dynamic task offloading and resource allocation in the vehicle edge computing environment.

[0176]

[0177] Example:

[0178] In order to verify the effectiveness of the resource management algorithm based on P-DQN in the present invention, this example is simulated and analyzed in the cloud-assisted vehicle edge computing scenario. The simulation results include evaluating the convergence and performance of the P-DQN algorithm in the training and inference stages. First, we analyzed the training curves under various parameter configurations to determine the optimal settings. Then, we compared the performance indicators of the proposed algorithm and the baseline algorithm in the same scenario. The parameter settings of the simulation are listed in Table 1.

[0179] This example was simulated in Python 3.11 and PyTorch 2.0. The simulation environment consists of a 1000-meter one-way road with 20 vehicles on it. Each vehicle has 1GHz of computing power and a transmission power range of 1 to 2 watts. Four RSUs are evenly distributed along the road. Each RSU is equipped with a MEC server with 10GHz computing power and an uplink bandwidth of 30MHz. At the beginning of each time period, there is a 50% probability (P=0.5) that the vehicle will initiate a computing task. The service factor of the computing task is λ=100Kilocycles / b, the input data size of the task is distributed between 8,000bit and 12,000bit, and the maximum delay of each task is selected from the range of 0.9 to 1 second. The experience replay buffer size of the P-DQN algorithm is set to 20,000, N batch =128, and the Adam optimization algorithm is used. The soft update parameters of the target action network and the target parameter network are set to τ a = 0.1 and τ p =0.001.

[0180] Table 1 Simulation parameter settings

[0181]

[0182] This example first studies the effect of different learning rates on convergence efficiency in the P-DQN algorithm, where lra and lrp represent the learning rates of the Actor network (i.e., the value network) and the parameter network, respectively. Figure 5 The results shown highlight how different lra values ​​affect the reward outcomes of the algorithm in various episodes. The learning rate of the parameter network is fixed at 10 -6 , and the learning rate lra of the Actor network was adjusted. When the actor learning rate is 10 -5 When , the algorithm performs best, which is manifested by faster convergence and increased maximum reward. Similarly, Figure 6 Shown at a fixed lra of 10 -5 , and adjust lrp to 10 -5 , 10 -6 and 10 -7 The total reward changes with the number of training episodes when . -5 promotes faster convergence, but the lower 10 -6 This indicates that a higher learning rate may cause the algorithm to converge to a local optimum prematurely. -7 When the learning rate is too low, it significantly hinders the training process, thus affecting the ability of the algorithm to converge effectively. Based on the above results, this study fixed the values ​​of lra and lrp at 10 in subsequent simulations. -5 and 10 -6 .

[0183] In order to verify the performance of the proposed P-DQN-based algorithm, the following algorithms are selected as benchmark algorithms for comparative analysis:

[0184] Double DQN: Double DQN (also called DDQN) is an enhanced form of the DQN algorithm that solves the Q value overestimation problem in the original DQN. Given that the output of DDQN is discrete, and the vehicle transmission power p i and compute resource variables is continuous, so it is necessary to approximate the continuous parameters as discrete actions. In this study, and is quantified into ten levels from 0 to F, and p i It is quantified into three levels.

[0185] DDPG: Since DDPG cannot directly handle discrete-continuous mixed action spaces, we use the same method described in the literature (Xue Z, Liu C, Liao C, et al. Joint service caching and computation offloading schemebased on deep reinforcement learning in vehicular edge computing systems [J]. IEEE Transactions on Vehicular Technology, 2023) and the literature (Hausknecht M, Stone P. Deep reinforcement learning in parameterized action space [J]. arXiv preprintarXiv: 1511.04143, 2015) to relax the action space. This relaxation enables DDPG to output offloading decisions and resource management actions simultaneously.

[0186] AllCloud (Cloud Offloading Algorithm): This method offloads all generated computing tasks to cloud servers through RSUs and allocates fixed computing resources to them. In addition, the transmission power between the vehicle and the RSU is fixed at P max .

[0187] Figure 7 The reward curves of P-DQN and three baseline algorithms (DDPG, DDQN, and AllCloud) are shown. It is observed that in the three deep reinforcement learning-based algorithms, the agent improves its strategy through interaction with the environment and eventually reaches convergence. After convergence, all DRL-based algorithms show better performance than the AllCloud strategy, although they differ in convergence speed, stability, and the final reward value. Specifically, the DDQN algorithm shows fast convergence, stabilizes around 2000 rounds, and achieves full convergence at 5000 rounds; DDPG reaches a relatively stable reward value at 2000 rounds, but the performance fluctuates greatly after convergence. P-DQN achieves convergence at around 6500 rounds and has the highest stability and reward. In addition, after convergence, the rewards obtained by the P-DQN-based algorithm increase by 0.29 and 0.23 compared with the DDPG and DDQN algorithms, respectively.

[0188] like Figure 8 As shown, when the number of vehicles N = 20 and the weight Figure 2. The relationship between the episode reward and the total uplink bandwidth B of the RSU when . It can be observed that as B increases, all curves show an upward trend. This is because as B increases, the transmission rate of the vehicle uploading input data for offloading tasks increases, thereby reducing the transmission delay. In addition, the change of B has a more significant impact on the performance of the AllCloud algorithm. This is because the AllCloud strategy offloads all tasks to the server for execution, while in some periods, the DRL-based algorithm may choose to execute tasks locally, depending on the observed environmental state, and the local execution delay is independent of B.

[0189] like Fig. 9 As shown in Figure 1, as the probability of vehicle task generation P in each time slot increases, the network load gradually increases, resulting in additional task execution delays. Therefore, this prompts the agent to bear higher costs to offload tasks to resource-rich cloud servers, resulting in a reduction in the system's reward. Since the AllCloud algorithm adopts a fixed offloading strategy, its performance is only affected by the size of P during the data upload phase, thus maintaining an almost constant reward.

[0190] Fig.10 The relationship between the reward and the number of vehicles in the network is illustrated. When the number of vehicles is small - that is, the network load is low - the three DRL-based algorithms perform similarly, and each algorithm outperforms the AllCloud algorithm. As the number of vehicles increases, the reward gradually decreases, which is attributed to the limited computing resources of the MEC server; however, the performance of the P-DQN-based method decreases the least. Specifically, when N=15, the proposed method achieves rewards that are 0.25 and 0.18 higher than DDPG and DDQN, respectively; when N increases to 40, this gap widens to 0.62 and 1.19. This effect stems from the network structure of the P-DQN algorithm, which can effectively manage the discrete-continuous mixed space and achieve better performance under the condition of scarce computing resources through reasonable offloading decisions and resource management.

[0191] like Fig.11 As shown in the figure, when the service coefficient λ changes from 60Kilocycle / b to 100Kilocycle / b, the reward value obtained by the algorithm decreases. This is because the increase in λ leads to an increase in task execution delay. Compared with other algorithms, the proposed algorithm obtains higher rewards by optimizing resource allocation by utilizing parameterized action space. In addition, as λ increases, the rate at which the rewards of P-DQN, DDPG and DDQN algorithms decrease is lower than that of the AllCloud algorithm, which shows that the intelligent algorithm can adaptively adjust the offloading strategy to cope with dynamic environmental changes in the vehicle edge computing network.

[0192] Fig.12(a), 12(b) and 12(c) verify the performance of the algorithm and the changes in the allocation strategy under different weights w1, w2, w3, where w2 is set to 0 and w1+w3=1. Specifically, Fig.12 (a) shows the episode rewards under different parameters, and Fig.12 (b) and Fig.12 (c) The average delay and cost of each time slot are analyzed separately. Fig.12 (a) It can be seen that as w1 increases and w3 decreases, the system reward gradually increases. This is because the DRL-based algorithm, under the influence of higher delay weights, may choose a higher cost to minimize the task execution delay. This trend can be seen from Fig.12 (b) and Fig.12 (c). However, the AllCloud algorithm keeps the average delay and cost constant by allocating fixed resources to all vehicles, and the change in its reward comes from the change in the reward calculation method. In addition, although the P-DQN-based algorithm always obtains the highest reward under different parameters, its average delay is slightly higher than DDPG and DDQN. In contrast, the P-DQN algorithm has a lower average cost. This shows that the P-DQN algorithm can more reasonably balance multiple performance indicators, thereby achieving better overall performance.

Claims

1. A resource management method for vehicle edge computing system based on P-DQN, It is characterized in that Build a three-layer task offloading system including local, edge server and cloud layers; construct an optimization problem that jointly considers delay, energy consumption and cost, and frame the problem as a Markov decision process MDP, design the state space and reward function, and introduce a parameterized behavior space to effectively manage the mixed behavior space; In addition, a method based on parameterized Q network P-DQN is proposed to solve the optimization problem.

2. According to the P-DQN-based vehicle edge computing system resource management method of claim 1, It is characterized in that A three-layer task computing model including local processing, edge offloading and cloud offloading is constructed. The system consists of N vehicles, M evenly distributed roadside units (RSUs) and a cloud server. The vehicle set is represented as Each RSU has the same coverage $R$, and the RSU set is expressed as These RSUs are connected to the remote cloud server through wired connections and core networks. The model divides the road into M segments, and vehicles are randomly distributed on these segments. The total length of the road is represented as L. Each RSU is equipped with a MEC server with a computing power of F. The collection of these MEC servers is represented as RSUs equipped with MEC servers can be regarded as edge nodes, providing reliable and rapid task offloading services to vehicles within their coverage area. These edge nodes are interconnected through wired connections. When the access node is overloaded and cannot process computing tasks, the vehicle can offload tasks to adjacent nodes or cloud servers, thereby alleviating the imbalance of edge computing resources. Cloud servers have huge computing resources and can handle highly complex tasks or serve as supplementary resources during periods of insufficient network computing resources. In addition, time is divided into time periods of equal length, represented as t = {1, 2, ..., T}. It is assumed that each vehicle generates at most one task in each time period, and the duration of each time period is Δt.

3. According to the P-DQN-based vehicle edge computing system resource management method of claim 1, It is characterized in that The three-layer task calculation model also takes into account the mobility of the vehicle, assuming that the vehicle speed follows a truncated Gaussian distribution, reflecting the randomness of the speed while excluding negative values. The probability density function of the truncated Gaussian distribution is as follows: in represents the probability density function of Gaussian distribution, μ and σ represent the average speed and standard deviation of the vehicle, respectively, V max =μ+3σ and V min =μ-3σ represent the maximum speed and minimum speed of the vehicle respectively, is the error function.

4. According to the P-DQN-based vehicle edge computing system resource management method of claim 1, It is characterized in that In order to describe the computing tasks of the vehicle, the following task model is adopted: For vehicle i, the computing task generated at time slot t is usually represented as a triple, which includes the data size of the task, the required computing resources, and the deadline of the task, which is expressed as follows: Among them, d i (t) represents the computational task The input data size; ω i (t) = λ i (t)·d i (t) represents the task The computational intensity of the task, specifically, the number of CPU cycles required to complete the task, where λ i (t) is a coefficient that describes d i (t) and ω i (t), called the service coefficient, is determined by the computational complexity of the task; Indicates the task Maximum tolerable delay; Based on the described framework, vehicles can choose to process computing tasks locally or offload them to MEC servers or cloud servers. For each computing task, a vehicle can offload to at most one server. Although vehicles can only communicate directly with the RSU in their area, they can offload tasks to the MEC servers of neighboring RSUs through this RSU. Therefore, for vehicle i, the task in time period t The possible unloading decision set includes M+2 options, denoted as a i (t) = {0, 1, 2, ..., M + 1}. i (t) = 0 means local execution, a i (t) = M + 1 represents cloud offloading, a i (t) = i (1≤i≤M) represents offloading to MEC server i. In addition, for vehicle i, the offloading symbol is defined as Represents a set of uninstallation policies. Specifically, when a i When (t) = 0, Indicates local processing, otherwise when a i When (t) = i (1≤i≤M), Indicates offloading to MEC server i, otherwise when a i When (t) = M + 1, Indicates cloud uninstallation, otherwise 5. According to the P-DQN-based vehicle edge computing system resource management method of claim 1, It is characterized in that When a vehicle decides to unload a task, it first needs to upload the input data of the task to the corresponding RSU, and receive the output data from the RSU after the task is completed. The vehicle-RSU communication model based on vehicle-to-infrastructure V2I communication is as follows: Assuming that the uplink channel from vehicle i to RSU is a Rayleigh fading channel, inter-channel interference is not considered, and the channel gain is expressed as follows: where g im represents the small-scale fast fading power component, which follows an exponential distribution with a mean of one, A represents the path loss constant, and L im is the distance between vehicle i and RSU m, γ represents the attenuation index, β im is a log-normal shadow fading random variable with standard deviation ξ. Assume that the total uplink bandwidth of RSU is B, and at a given time slot t, there are N(t) tasks that need to be offloaded to RSUm. Under these conditions, the uplink transmission rate from vehicle i to RSU m can be calculated as follows: where p i (t) is the transmission power of vehicle i in time slot t, and the channel gain from vehicle i to RSU m is h im (t), the background noise power is σ 2 .

6. A vehicle edge computing system resource management method based on P-DQN according to claim 1, It is characterized in that The task processing models under various task execution strategies selected by the vehicle include: (1) Local execution: The vehicle uses the onboard computing resources to process computing tasks. Assume that the computing capacity of vehicle i is Indicates that the time and energy consumption of completing task Γ locally are calculated as follows: Among them, κ is an energy consumption parameter that depends on the chip architecture. When the task is processed locally, the vehicle does not need to pay for the computing resources of the server operator, so the cost of vehicle i is expressed as (2) MEC server execution: When a vehicle offloads a computing task to a MEC server, it must first upload its input data through the uplink of the RSU. The task is then executed by the MEC server connected to the RSU. Considering the close distance between the RSU and the MEC server, the transmission time of the input data between them can be ignored. The time required for vehicle i to upload input data to MEC server m can be given by the following formula: Assuming that computing tasks are executed in parallel on MEC servers, the computing resources requested by vehicle i from MEC server m are expressed as Therefore, the task The execution time and cost of offloading to MEC server m is: where ρ E Represents the unit price of the computing resources of the MEC server; after the task is executed on the MEC server, the calculation result is transmitted back to the vehicle through the downlink of the RSU. Given that the output data volume of the task is relatively small, the delay and energy consumption of transmitting the result back to the vehicle can be ignored. However, considering the dynamic nature of high-speed vehicles, if vehicle i leaves the coverage area of ​​the corresponding RSU, the calculation result must first be transmitted to the RSU of the current road segment through edge node communication, and then the result is downloaded. Therefore, the total transmission time of the calculation result back to the vehicle can be expressed as: Where Δx represents the number of road segments between vehicle i and edge node m, t inter represents the transmission delay of the calculation result data between adjacent edge nodes. Let x i Let k denote the position of vehicle i on the road, and let k denote the vehicle is located in the coverage area of ​​the kth RSU, then Δx = |km|, where Indicates that x is rounded down. Finally, the energy consumption and cost of the MEC server execution solution can be calculated as: (3) Cloud server execution: When offloading computing tasks to the cloud server, the input data is first uploaded from the vehicle to the corresponding roadside unit (RSU), and then uploaded from the RSU to the cloud server. However, since the data transmission between the cloud server and the RSU involves long-distance communication, the upload time from the RSU to the cloud server and the data transmission time from the cloud server to the RSU must be considered. Therefore, the total delay of the cloud server processing decision is as follows: in represents the computing resources allocated by the cloud server to vehicle i, t c represents the constant time of data transmission between RSU and cloud server, assuming ρ c represents the unit price of cloud server computing resources. The energy consumption and cost of the cloud server processing model are expressed as:

7. A vehicle edge computing system resource management method based on P-DQN according to claim 1, It is characterized in that The optimization problem in the vehicle computing offloading scenario is constructed, and the task The execution delay, energy consumption and cost of are expressed as follows: The utility function of vehicle i is defined as: where w 1 ,w 2 ,w 3 are the weight parameters of delay, energy consumption and cost respectively, and w 1 +w 2 +w 3 =1, despite the task The maximum tolerable delay is However, if the vehicle drives out of the entire road section, the result of the unloading task will not be transmitted back through the RSU. is the remaining time of vehicle i on the road; therefore, the task The actual delay tolerance is Based on the above system model analysis, the unloading of all vehicles and their decisions is expressed as and Furthermore, services are defined as the computational resources and transmission power allocated to vehicles, respectively and In order to maximize the long-term utility of the system, the optimization problem is formulated as follows: a i (t)∈{0,1,…,M+1} (23), T i (t)≤T max ,i∈N (24), P min ≤p i (t)≤P max (27), The above optimization problem aims to determine the optimal unloading decision and resource allocation and Minimize the long-term delay, energy consumption and cost of the system. Constraints (21) and (22) ensure that each vehicle chooses only one offloading option: local, MEC server or cloud server. (23) Decision variables are defined, indicating that the task may be offloaded to any server or executed locally. (24) Time constraints for task execution are specified. (25) The resources allocated to the vehicle by the MEC server must not exceed its total computing capacity. (26) An upper limit is set on the resources that the cloud server can allocate to each vehicle without limiting the total capacity of the cloud server. (27) The transmission power of each vehicle is limited to not exceed the maximum allowed value.

8. A vehicle edge computing system resource management method based on P-DQN according to claim 7, It is characterized in that An offloading algorithm based on deep reinforcement learning is proposed to solve the above optimization problems, including: First, the above optimization problem is transformed into a Markov decision process, MDP can be represented by a four-tuple Characterization, among which, Represents the set of system states, i.e., the state space; represents the action space; r(s|a) is the reward function that depends on the state and action, and is simplified to r(s) when the reward is only affected by the state. P(s′|s,a) is the state transition function, which represents the probability of transferring to state s' after executing action a in state s. However, due to the lack of prior environmental knowledge, the state transition function is usually unknown, which makes it complicated to obtain the optimal solution through dynamic programming of MDP. Therefore, deep reinforcement learning is applied to solve the proposed optimization problem. The goal of the agent is to maximize the long-term cumulative reward. Initially, the agent observes state s from the environment. t And select action a according to the current strategy. After executing the action, the system transfers to state s t+1 , and returns a reward to the agent, which then updates its strategy based on this feedback and eventually converges to the optimal strategy.

9. A vehicle edge computing system resource management method based on P-DQN according to claim 8, It is characterized in that Construct the state space, action space and reward of the deep reinforcement learning algorithm according to the proposed model: (1) State space: For task offloading decisions and resource allocation, we must first determine the server’s computing resource distribution, the variable sizes of all computing tasks, and the vehicle’s position change information. Therefore, in time period t, the state It consists of four components: s t ={F m (t),X i (t),V i (t),D i (t)} (28), for s t The detailed explanation is as follows: F m (t) = [f 1 (t),f 2 (t),…,f M (t)] represents the current computing resource distribution of edge nodes. This array contains the computing resource configuration of each edge node at time t, where f i (t) represents the amount of available computing resources of the i-th edge node. This information is crucial for understanding the load and resource usage of each edge node at a specific point in time, and then more efficient resource allocation and task offloading strategies can be formulated based on this data; X i (t) and V i (t) describes the movement of all vehicles, where X i (t) records the vehicle position at time slot t in detail, while V i (t) represents the speed of the vehicle. This information affects the urgency of the computing task and the choice of offloading to the edge node. The state vector of the vehicle position change is defined as: X i (t)=[x 1 (t),x 2 (t),…,x N (t)] (29), V i (t)=[v 1 (t),v 2 (t),…,v N (t)] (30), Vector D i (t) = [d 1 (t),d 2 (t),…,d N (t)] records the input data size of each task waiting to be executed. The input data size directly affects the decision of task offloading, including the location and method of offloading, which also affects the data transmission time and the computing resources required for computing tasks; (2) Action space: When the state s is observed t When , vehicle i needs to make an unloading decision. The dimension of the action vector depends on the unloading decision and can be expressed as: Among them, a i (t) represents the unloading decision of vehicle i. As mentioned above, a i (t) = 0 means local execution; a i (t)>0 means offloading to the server, defined in space It covers M+2 dimensions. and represent the resources of MEC server m and cloud server allocated to vehicle i, respectively, and p i (t) represents the transmission power of the vehicle, In order to handle heterogeneous action vectors, this paper introduces a parameterized action space with a hierarchical structure. Vehicles first determine their discrete actions a i (t), and then select the associated continuous parameter y ai (t), where y ai (t) is a vector of variable dimension, defined as For all Continuous set of discrete-continuous mixed action spaces It can be expressed as: Reward: The agent adjusts its strategy based on the reward received. According to the objective function (20), the reward at time t is defined as follows: Where R p represents the penalty introduced to ensure that the agent's actions comply with constraint (24). In addition, the calculation formula for the long-term cumulative discounted reward is:

10. A vehicle edge computing system resource management method based on P-DQN according to claim 9, It is characterized in that The vehicle network unloading decision and resource allocation algorithm using P-DQN includes: first, the deterministic policy network μ(θ) is based on the current state s t Generate all continuous parameters, then the state s t The values ​​of the continuous parameters are input into the value network Q(ω), and the algorithm then selects the discrete action corresponding to the highest Q value. Finally, the algorithm combines the selected discrete actions and their corresponding continuous parameters into an action vector and outputs it. In the P-DQN algorithm, the Q network is used to approximate the action-value function based on the Bellman equation. In the mixed action space, the equation can be expressed as follows: Furthermore, when the Q function is fixed, the process of determining continuous parameters can be expressed as a function y(s), and y(s) can be approximated by a deterministic policy network μ(s; θ), whose weight is represented by θ. Therefore, the approximation of the Bellman equation can be expressed as follows: Among them, ω * represents the optimal value of the network weight ω. Similar to DDPG and DQN, P-DQN adopts the experience replay mechanism and uses the target network to calculate the temporal difference TD target. The network weight ω is optimized by minimizing the TD loss. The loss functions for ω and θ are defined by equations (37) and (38), respectively: where y t is the TD target, which is defined as shown in formula (39):

Citation Information

Cited By

  • Multi-agent visual data unloading method based on information perception and reinforcement learning

    CN119917183A

  • Internet of vehicles task unloading method, device, equipment and medium

    CN120523607A