Internet-of-vehicles computing unloading method based on Rainbow DQN

Abstract

Along with rapid development of an artificial intelligence technology, calculation-intensive applications generally need a large number of calculation resources, but the calculation resources of mobile terminal equipment (such as vehicles) are often limited, so that the requirements of the applications are difficult to meet. In order to solve the challenge, the invention provides a deep reinforcement learning method based on a Rainbow DQN algorithm, and the deep reinforcement learning method is specially used for solving the problem of calculation unloading in the V2X (Vehicle-to-Everything, V2X). Under the condition of considering task characteristics and system resources, according to the capability differences of various computing units of cloud, edge and end in the aspects of data operation, analysis, decision making and the like, the method integrates multiple improved deep reinforcement learning methods, aims to solve the problem of insufficient computing capability of end vehicles, optimizes the time delay and energy consumption of the whole system, and improves the system performance. The problem of calculation unloading in an Internet of Vehicles scene is solved.

Description

Technical Field

[0001] The present invention relates to a vehicle networking technology, and in particular to an optimization method in vehicle networking computing offloading, and specifically to a vehicle networking computing offloading method based on a Rainbow DQN architecture. Background Art

[0002] With the rapid development of artificial intelligence technology, computationally intensive applications such as augmented reality, autonomous driving, speech recognition, and natural language processing have gradually become mainstream. These applications usually require a lot of computing resources, while the computing resources of mobile devices (such as vehicles) are often limited and difficult to meet the needs of these applications. To address this challenge, this study proposes a method based on the Rainbow DQN algorithm, which specifically targets the computational offloading problem in the vehicle-to-everything (V2X) network. The algorithm not only significantly improves the performance and stability of DQN, but also can flexibly adapt to environmental changes.

[0003] Rainbow DQN architecture: Based on the Rainbow DQN algorithm, the present invention constructs the Rainbow DQN algorithm by introducing the Dueling mechanism, Prioritized Experience Replay, and Noisy Nets. This architecture divides the Q network into two parts: a state value network and an action advantage network. The state value network estimates the inherent quality of the state without relying on any specific action; the action advantage network evaluates the advantage of each action over other actions in a given state. This separation enables the network to more accurately evaluate the quality of the state and the advantage of the action, thereby improving the accuracy of decision-making. In the Internet of Vehicles environment, by setting different parameters to simulate the computing power, memory size, and bandwidth size of the MEC server, the Rainbow DQN algorithm can be used to dynamically select the optimal strategy for the vehicle and select the most suitable MEC server. This can not only improve task performance, but also effectively reduce energy consumption and extend equipment life.

[0004] In the Internet of Vehicles (IoV), computing task offloading refers to the process of transferring computing tasks on mobile devices (such as smartphones, vehicle systems, etc.) to more powerful computing nodes (such as edge servers, cloud servers) for processing. This is mainly to reduce the computing burden of mobile devices and use more powerful computing resources at the edge or cloud to process computing-intensive or storage-intensive tasks. In real-life scenarios, computing offloading is affected by many factors such as the external environment, user personality, hardware devices or software applications. Therefore, it is extremely important to formulate reasonable and dynamically changing task offloading strategies. Due to the mobility of vehicles and the limited computing resources, traditional cloud computing architectures are difficult to meet the requirements of the IoV for low latency and high reliability. Therefore, edge computing (MEC) is introduced into the IoV to provide lower latency and higher computing power by sinking computing resources to the edge of the network. In the IoV environment, a large amount of data generated by vehicles and roadside devices needs to be processed in real time, and MEC servers provide computing services and data storage services for mobile users (such as vehicles and other intelligent transportation devices), which is very suitable for task migration. Task migration technology can migrate complex computing tasks on vehicle-mounted devices to remote MEC servers through wireless networks for execution, relying on the server's rich computing resources to complete computing tasks, solving the limited computing power and power capacity of vehicle-mounted devices. With the rapid development of the Internet of Vehicles and the continuous improvement of application needs, how to further optimize MEC and task migration technology to meet the high performance, low latency, high reliability and other requirements of the Internet of Vehicles is still an important research direction. In summary, the Internet of Vehicles computing offloading method based on the Rainbow DQN architecture proposed in the present invention provides new ideas and methods for optimizing computing offloading and resource allocation, which helps to improve the overall performance and efficiency of the system. Summary of the invention

[0005] Purpose of the invention: In view of the above-mentioned problems existing in the prior art, a method for offloading computing in the Internet of Vehicles based on Rainbow DQN is proposed to determine the vehicle's location planning, the terminal's offloading decision and the transmission power. Taking into account the characteristics of the task and system resources, this method integrates a variety of improved deep reinforcement learning methods based on the differences in the capabilities of various computing units in the cloud, edge and end in data calculation, analysis, decision-making, etc., aiming to solve the problem of insufficient computing power of the end vehicle, optimize the latency and energy consumption of the entire system, and solve the problem of computing offloading in the Internet of Vehicles scenario.

[0006] Technical solution: In the URLLC (Ultra-Reliable and Low-Latency Communications) scenario, the cloud-edge-end collaborative framework uses the Dueling network architecture to introduce a priority experience playback area to optimize computing task offloading decisions. The main goal is to minimize the delay in task execution and maximize the reliability and efficiency of data transmission. A distributed resource allocation method is adopted, and no centralized processing in the cloud is required. Each V2V link is regarded as an intelligent agent. Each vehicle can intelligently select the optimal offloading strategy for the task based on the current environmental state. The optimization problem is converted into a Markov decision process, and the Markov decision process is solved based on Rainbow DQN to obtain an optimization solution. According to the optimized deep reinforcement learning model, the computing offloading is optimized. The above invention is achieved through the following technical solution: A method for vehicle network computing offloading based on Rainbow DQN, comprising the following steps:

[0007] (1) The IoV computing offloading system based on the cloud-edge computing network architecture consists of cloud servers, edge MEC nodes, and terminal devices. In the IoV communication scenario, there is a base station (BS) and multiple vehicle users (VUs). These vehicle users can be further divided into two categories: vehicle-to-infrastructure (V2I) and vehicle-to-vehicle (V2V);

[0008] (2) Construct a vehicle network resource allocation system model consisting of V2I users and V2V users, where the V2I user set and the V2V user set are represented as M = {1, 2, ..., m} and K = {1, 2, ..., k} respectively;

[0009] (3), the joint problem of computation offloading and resource allocation in vehicular networks is formulated as a Markov decision process;

[0010] (4) Focusing on efficiency and privacy protection in a distributed environment, adopting a multi-agent distributed resource allocation method, taking into account the reliability and latency of V2V and V2I users, and building a network building module of the Rainbow DQN algorithm using a deep reinforcement learning algorithm with the goal of minimizing system latency and energy consumption, including the construction of the Current Q Network and the Target Q Network, and establishing a Prioritized ReplayBuffer;

[0011] (5) In the execution phase, the V2V user obtains the current state S based on local observations t (k) Use the trained model to obtain the optimal V2V user transmission power and channel allocation strategy. In this stage, the V2V user will use the trained deep reinforcement learning model to determine the best action in a given state. Through the Rainbow DQN algorithm, the optimal strategy is dynamically determined for the vehicle, and the most suitable MEC server is selected to improve task performance and reduce energy consumption.

[0012] Furthermore, the step (2) includes the following specific steps:

[0013] (2a) Establish a V2V user resource allocation system model. The system includes M pairs of V2I links and K pairs of V2V links. The uplink frequency band is divided into multiple sub-channels or resource blocks (RBs). Each RB corresponds to a sub-band. Orthogonal spectrum resources are allocated to V2I and V2V communications to reduce interference between them. The sub-channel is represented by the set S = {1, 2, ..., s};

[0014] (2b) In time slot t, the channel power gain of V2V user k in channel s is

[0015] g k,s,t =a k，s,t h k，s，t Expression 1

[0016] where represents the shadow effect and path loss, represents the small-scale fading channel gain. Similarly, we can define the interference channel gain of V2V user k to other V2V users, the interference channel gain of V2V user k to BS, the channel gain of V2I user m to BS, and the interference channel gain of V2I user m to V2V user k;

[0017] (2c) Using a binary indicator variable α m，s and β k，s To describe whether the V2I user and the V2V user occupy the subchannel s (assuming that each user can only occupy one channel at a time, and a channel is allocated to at most one V2I user, and multiple V2V users can share a channel):

[0018]

[0019] (2d), the SINR of the mth V2I link on the sth subchannel can be expressed as:

[0020]

[0021] Similarly, the SINR of the kth V2V link on the sth subchannel can be expressed as

[0022]

[0023] Among them, P m and P k are the transmission powers of V2I user m and V2V user k, σ 2 represents the noise power;

[0024] (2e), interference power I k，s，t for:

[0025]

[0026] (2f), the data rates of V2I user m and V2V user k in channel s and time slot t are:

[0027] R m，s，t =W log2(1+γ m，s，t ) Expression 6

[0028] R k，s,t =W log2(1+γ k，s，t ) Expression 7

[0029] Where W represents the bandwidth of each sub-band;

[0030] (2g), introducing the matrix A = {a k，s，t |k∈K} to indicate whether the V2V user link is reliable. k，s，t is the reliability index of V2V user k in data transmission, defined as:

[0031]

[0032] in, represents the SINR threshold of V2V user k.

[0033] (2e), V2V users are mainly used for periodic safety information transmission, which can reduce the risk of traffic accidents. Therefore, it is necessary to reasonably optimize the parameters of V2V itself, and the objective function is to maximize the total transmission rate of all V2V users, thereby improving the communication performance of V2I users and V2V users. Based on this, the Internet of Vehicles resource allocation problem is established as follows:

[0034]

[0035] Constraints:

[0036]

[0037] Among them, constraints C1, C2 and C3 indicate that each V2I user can only occupy one channel and one channel is allocated to at most one V2I user, while multiple V2V users can share one channel. C4, C5 and C6 are reliability constraints, power constraints and binary variable constraints respectively;

[0038] Furthermore, the step (3) includes the following specific steps:

[0039] (3a) Considering vehicle users (VUs) as agents, for each agent n, at each time step t, it first obtains the current state from the environment through local observation. The state contains information related to resource allocation, including the characteristics of the computing task, the usage of system resources, etc., namely, S t ={D f , C f , T d , R free , B, L, L cloud , L edge} and A t ={a1, a2, a3}

[0040] Among them, the data size of the task is D f , the computational complexity of the task is C f (such as the number of CPU cycles required), the priority or deadline Td of the task; system resource usage: the currently available computing resources R free (e.g., CPU, memory) Current network status, including bandwidth B and latency L Load on the cloud and edge servers L cloud and L edge ;

[0041] Furthermore, the step (4) includes the following specific steps:

[0042] (4a), Network construction: Based on the DuelingNetwork architecture, two different branches are introduced after the convolutional network, one for predicting the value of the state and the other for predicting the advantage of each action. It includes a shared input layer, a value function flow (V), an advantage flow (A), and an output layer that merges the results of the two branches. First, define the value function:

[0043]

[0044] where π represents the action probability distribution and γ∈(0, 1) is the discount factor.

[0045] A(s,a;θ)=Q(s,a;θ)-V(s) Expression 12

[0046] The target value is calculated as follows:

[0047] Q π (s, a; θ) = r + γ (1-D) (V π (s′)+max a′ (Aπ(s′, a′; θ))-(V π (s′)+max a′ (Aπ(s′, a′; θ′)))) Expression 13

[0048] Where r is the immediate reward received after executing action a, s′ is the next state, and D is an indicator variable. If the current state s is the terminal state, then D = 1, otherwise D = 0, V π (s′) is the value function branch of the target network’s estimate of the next state, A π (s′, a′; θ′) is the estimate of the next state and action by the advantage function branch of the target network;

[0049] (4b) Establish a priority experience playback area: set the maximum storage capacity (max_size) and batch size (batch_size) to store the state, action, reward, next state and termination information during the training process, and introduce a priority parameter. Usually, TD-error (temporal difference error) is used to indicate the importance of experience.

[0050] (4c), adding experience: when the agent interacts with the environment, the experience (state S, action a, reward R, next state S′, whether to terminate D) is added to the priority experience playback area, and the priority of the experience is stored at the same time. The initial priority is dynamically calculated according to the TD-error. The selected action is executed in the environment. This action can be to offload the computing task to the cloud or execute it locally, and observe the results, and receive the reward R, next state S′ and whether to terminate D information returned by the environment;

[0051] (4d) Update the agent's experience replay buffer: store the current state S, action a, reward R, next state S' and whether to terminate in the priority experience replay area, and calculate TD-error as the priority of the experience. This area is used to store the state-action-reward-next state sequence during training, allowing the agent to learn from historical data;

[0052] (4e), randomly select a batch of experience samples from the priority experience playback area. For each sample, use the target network to calculate the optimal action value of the next state, and then calculate the target Q value based on the action value. Use the mean square error loss function to calculate the difference between the predicted Q value and the target Q value, and update the parameters of the evaluation network through the back propagation algorithm. When updating the network, each agent needs to minimize the following loss function to achieve gradient descent:

[0053]

[0054] (4f) Update the exploration rate: After a certain number of training steps, the Noisy DQN is used to replace the ε-greedy strategy. The noise parameter σ of the Noisy DQN will gradually decrease as the training progresses, thus achieving a smooth transition from exploration to exploitation.

[0055] (4g) Loop training until convergence, repeating steps (4c) and (4f): continuously selecting actions, executing actions, observing results, storing experience, and training the network. As training proceeds, the impact of noise will gradually decrease, and the agent will rely more on the learned knowledge to make decisions. Beneficial effects: In view of the problem that the limited computing resources of the mobile terminal (such as the vehicle) are difficult to meet the computing power requirements, the present invention proposes a method for vehicle network computing offloading based on Rainbow DQN, which effectively addresses the problem of task offloading in the vehicle network and realizes efficient resource allocation in the vehicle network. This algorithm can dynamically adjust the resource allocation strategy according to the real-time environment and task requirements, thereby improving resource utilization efficiency. In addition, by optimizing computing offloading and edge computing resource allocation, the utilization efficiency of sub-channels is improved. This helps to more effectively utilize limited communication resources and reduce channel congestion and conflicts. In real life scenarios, computing offloading will be affected by multiple factors such as the external environment, user personality, hardware equipment or software applications. The present invention can formulate a reasonable offloading strategy that adapts to these complex factors.

[0056] In summary, the present invention integrates a variety of improved deep reinforcement learning methods through this method, aiming to solve the problem of insufficient computing power of end vehicles and optimize the latency and energy consumption of the entire system, providing an efficient, low-latency, and low-energy solution to solve the problem of computing offloading in the Internet of Vehicles scenario. At the same time, it pays attention to efficiency and privacy protection in distributed environments, and has significant practical value and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of a flow chart based on the Rainbow DQN algorithm provided in an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of a 5G vehicle networking computing offloading communication scenario provided by an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of a deep reinforcement learning model for a 5G vehicle networking computing offloading method provided in an embodiment of the present invention; DETAILED DESCRIPTION

[0060] The core idea of ​​the present invention is: in the joint optimization problem of vehicle networking computing offloading and resource allocation, a distributed resource allocation method is adopted, each V2V link is regarded as an intelligent agent, and a deep reinforcement learning model is established by integrating Double Q-Learning, Dueling Networks, Noisy Nets, and Prioritized Experience Replay mechanisms to obtain the optimal vehicle networking computing offloading strategy.

[0061] The present invention is described in further detail below.

[0062] Step (1): The IoV computing offloading system based on the cloud-edge computing network architecture consists of cloud servers, edge MEC nodes, and terminal devices. In the IoV communication scenario, there is a base station (BS) and multiple vehicle users (VUs). These vehicle users can be further divided into two categories: vehicle-to-infrastructure (V2I) and vehicle-to-vehicle (V2V);

[0063] Step (2) constructs a joint system model for vehicle networking computing offloading and resource allocation consisting of V2I users and V2V users, including a channel model, and the V2I user and V2V user sets are represented as M = {1, 2, ..., m} and K = {1, 2, ..., k}, respectively, including the following steps:

[0064] (2a) Establish a V2V user resource allocation system model. The system includes M pairs of V2I links and K pairs of V2V links. The uplink frequency band is divided into multiple sub-channels or resource blocks (RBs). Each RB corresponds to a sub-band. Orthogonal spectrum resources are allocated to V2I and V2V communications to reduce interference between them. The sub-channel is represented by the set S = {1, 2, ..., s};

[0065] (2b) In time slot t, the channel power gain of V2V user k in channel s is

[0066] g k,s，t =a k,s，t h k，s，t Expression 1

[0067] where represents the shadow effect and path loss, represents the small-scale fading channel gain. Similarly, we can define the interference channel gain of V2V user k to other V2V users, the interference channel gain of V2V user k to BS, the channel gain of V2I user m to BS, and the interference channel gain of V2I user m to V2V user k;

[0068] (2c) Using a binary indicator variable α m,s and β k，s To describe whether the V2I user and the V2V user occupy the subchannel s (assuming that each user can only occupy one channel at a time, and a channel is allocated to at most one V2I user, and multiple V2V users can share a channel):

[0069]

[0070] (2d), the SINR of the mth V2I link on the sth subchannel can be expressed as:

[0071]

[0072] Similarly, the SINR of the kth V2V link on the sth subchannel can be expressed as

[0073]

[0074] Among them, P m and P k are the transmission powers of V2I user m and V2V user k, σ 2 represents the noise power;

[0075] (2e), interference power I k，s，t for:

[0076]

[0077] (2f), the data rates of V2I user m and V2V user k in channel s and time slot t are:

[0078] R m，s，t =W log2(1+γ m,s,t ) Expression 6

[0079] R k，s，t =W log2(1+γ k，s,t ) Expression 7

[0080] Where w represents the bandwidth of each sub-band;

[0081] (2g), introducing the matrix A = {a k,s,t |k∈K} to indicate whether the V2V user link is reliable. k,s，tis the reliability index of V2V user k in data transmission, defined as:

[0082]

[0083] in, represents the SINR threshold of V2V user k.

[0084] (2e), V2V users are mainly used for periodic safety information transmission, which can reduce the risk of traffic accidents. Therefore, it is necessary to reasonably optimize the parameters of V2V itself, and the objective function is to maximize the total transmission rate of all V2V users, thereby improving the communication performance of V2I users and V2V users. Based on this, the Internet of Vehicles resource allocation problem is established as follows:

[0085]

[0086] Constraints:

[0087]

[0088] Among them, constraints C1, C2 and C3 indicate that each V2I user can only occupy one channel and one channel is allocated to at most one V2I user, while multiple V2V users can share one channel. C4, C5 and C6 are reliability constraints, power constraints and binary variable constraints respectively;

[0089] Step (3), constructing the joint optimization problem of computation offloading and resource allocation in the vehicle network as a Markov decision process, including the following steps:

[0090] (3a) Considering vehicle users (VUs) as intelligent agents, for each intelligent agent n, at each time step t, the current state is first obtained from the environment through local observation. This state contains information related to resource allocation, including the characteristics of the computing task, system resource usage, etc., namely

[0091] S t ={D f , C f , T d , R free , B, L, L cloud , L edge}

[0092] A t ={a1, a2, a3}

[0093] Among them, the data size of the task is D f , the computational complexity of the task is C f (such as the number of CPU cycles required), the task priority or deadline T d; System resource usage: currently available computing resources R free (e.g., CPU, memory) Current network status, including bandwidth B and latency L Load on the cloud and edge servers L cloud and L edge ;

[0094] Step (4) focuses on efficiency and privacy protection in a distributed environment, adopts a multi-agent distributed resource allocation method, considers the reliability and latency of V2V and V2I users, and builds a deep reinforcement learning algorithm with the goal of minimizing system latency and energy consumption. It sets the key parameters of the algorithm based on the Dueling network architecture, combines the Double DQN algorithm, and designs the Rainbow DQN training algorithm.

[0095] (4a), Network construction: Based on the Dueling Network architecture, two different branches are introduced after the convolutional network, one for predicting the value of the state and the other for predicting the advantage of each action. It includes a shared input layer, a value function flow (V), an advantage flow (A), and an output layer that merges the results of the two branches. First, define the value function:

[0096]

[0097] Where π represents the action probability distribution and y∈(0, 1) is the discount factor.

[0098] A(s,a;θ)=Q(s,a;θ)-V(s) Expression 12

[0099] The target value is calculated as follows:

[0100] Q π (s, a; θ) = r + γ (1-D) (V π (s′)+max a′ (A π (s′, a′; θ))-(V π (s′)+max a′ (A π (s′, a′; θ′)))) Expression 13

[0101] Where r is the immediate reward received after executing action a, s′ is the next state, and D is an indicator variable. If the current state s is the terminal state, then D = 1, otherwise D = 0, V π (s′) is the value function branch of the target network’s estimate of the next state, A π (s′, a′; θ′) is the estimate of the next state and action by the advantage function branch of the target network;

[0102] (4b) Establish a priority experience playback area: set the maximum storage capacity (max_size) and batch size (batch_size) to store the state, action, reward, next state and termination information during the training process, and introduce a priority parameter. Usually, TD-error (temporal difference error) is used to indicate the importance of experience.

[0103] (4c), adding experience: when the agent interacts with the environment, the experience (state S, action a, reward R, next state S′, whether to terminate D) is added to the priority experience playback area, and the priority of the experience is stored at the same time. The initial priority is dynamically calculated according to the TD-error. The selected action is executed in the environment. This action can be to offload the computing task to the cloud or execute it locally, and observe the results, and receive the reward R, next state S′ and whether to terminate D information returned by the environment;

[0104] (4d) Update the agent's experience replay buffer: store the current state S, action a, reward R, next state S' and whether to terminate in the priority experience replay area, and calculate TD-error as the priority of the experience. This area is used to store the state-action-reward-next state sequence during training, allowing the agent to learn from historical data;

[0105] (4e), randomly select a batch of experience samples from the priority experience playback area. For each sample, use the target network to calculate the optimal action value of the next state, and then calculate the target Q value based on the action value. Use the mean square error loss function to calculate the difference between the predicted Q value and the target Q value, and update the parameters of the evaluation network through the back propagation algorithm. When updating the network, each agent needs to minimize the following loss function to achieve gradient descent:

[0106]

[0107] (4f) Update the exploration rate: After a certain number of training steps, the Noisy DQN is used to replace the ε-greedy strategy. The noise parameter σ of the Noisy DQN will gradually decrease as the training progresses, thus achieving a smooth transition from exploration to exploitation.

[0108] (4g) The training cycle continues until convergence, and steps (4c) and (4f) are repeated: continuously selecting actions, executing actions, observing results, storing experience, and training the network. As the training progresses, the impact of noise will gradually decrease, and the agent will rely more on the learned knowledge to make decisions.

[0109] Step (5), in the execution phase, the V2V user obtains the current state S based on local observations t(k), use the trained model to obtain the optimal V2V user transmit power and channel allocation strategy. In this stage, the V2V user will use the trained Rainbow DQN model to determine the best action in a given state and optimize the latency and energy consumption of the entire system.

[0110] exist Figure 1 In the paper, the offloading flow chart of the Internet of Vehicles based on the Rainbow DQN algorithm is described.

[0111] exist Figure 2 In the 5G vehicle network resource allocation communication scenario, it can be seen that in the vehicle network communication scenario, there is a base station (BaseStation, BS) and multiple vehicle users (Vehicle Users, VUs), vehicle users facing infrastructure (Vehicle-to-Infrastructure, V2I) and vehicle users facing between vehicles (Vehicle-to-Vehicle, V2V).

[0112] exist Figure 3 In the paper, a deep reinforcement learning model for 5G vehicle networking computation offloading and resource allocation is described. It can be seen that the V2V link, as an intelligent agent, selects the channel and transmission power according to the strategy based on the current state and obtains rewards from the environment.

[0113] Based on the description of the present invention, it should not be difficult for those skilled in the art to see that the RainbowDQN-based 5G vehicle networking computing offloading and resource allocation joint optimization method of the present invention can maintain parameter stability and improve system performance.

[0114] The contents not described in detail in the present application belong to the prior art known to the professional and technical personnel in this field.

Claims

1. A method for offloading vehicle network computing based on Rainbow DQN for vehicle network communication scenarios, characterized in that: The steps include: (1) The IoV computing offloading system based on the cloud-edge computing network architecture consists of cloud servers, edge MEC nodes, and terminal devices. In the IoV communication scenario, there is a base station (BS) and multiple vehicle users (VUs). These vehicle users can be further divided into two categories: vehicle-to-infrastructure (V2I) and vehicle-to-vehicle (V2V); (2) Constructing a joint system model of vehicle-to-vehicle computing offloading and resource allocation consisting of V2I users and V2V users, including a channel model; (3) The joint problem of computation offloading and resource allocation in vehicular networks is formulated as a Markov decision process; (4) Construct the network building module of the Rainbow DQN algorithm, including the construction of the Current Q Network and the Target Q Network, and establish the Prioritized Replay Buffer; (5) In the execution phase, the V2V user obtains the current state S based on local observations. t (k) Through the Rainbow DQN algorithm, the optimal strategy is dynamically determined for the vehicle, the most suitable MEC server is selected, and the task performance is improved; Further, the step (3) comprises the following specific steps: (3a) Considering vehicle users (VUs) as intelligent agents, for each intelligent agent n, at each time step t, the current state is first obtained from the environment through local observation. This state contains information related to resource allocation, including the characteristics of the computing task, system resource usage, etc., namely S t ={D f ,C f ,T d ,R free ,B,L,L cloud ,L edge } <h2 style=";text-align:left;direction:ltr">A<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> (a1, a2, a3) Among them, the data size of the task is D f , the computational complexity of the task is C f (such as the number of CPU cycles required), the task priority or deadline T d ; System resource usage: currently available computing resources R free (e.g., CPU, memory) Current network status, including bandwidth B and latency L Load on the cloud and edge servers L cloud and L edge ; Further, the step (4) comprises the following specific steps: (4a) Network construction: Rainbow DQN combines the network architecture of Double DQN and Dueling DQN. Two different branches are introduced after the convolutional network. One branch is used to predict the value of the state, and the other is used to predict the advantage of each action. It includes a shared input layer, a value function flow (V), an advantage flow (A), and an output layer that merges the results of the two branches. First, define the value function: where π represents the action probability distribution and γ∈(0, 1) is the discount factor. A(s,a;θ)=Q(s,a;θ)-V(s) The target value is calculated as follows: Q π (s,a;θ)=r+γ(1-D)(V π (s)+max a′ (A π (s′, a′;θ))-(V π (s′)+max a′ (A π (s′, a′;θ′)))) Where r is the immediate reward received after executing action a, s′ is the next state, and D is an indicator variable. If the current state s is the terminal state, then D = 1, otherwise D = 0, V π (s′) is the value function branch of the target network’s estimate of the next state, A π (s′, a′; θ′) is the estimate of the next state and action by the advantage function branch of the target network, and introduces noise in the output layer of the network. The last layer of the network needs to generate noise parameters and apply them to the calculation of the action value (Q value); (4b) Establish a priority experience playback area: Set the maximum storage capacity (max_size) and batch size (batch_size) to store the state, action, reward, next state and termination information during the training process, and introduce a priority parameter. Usually, TD-error (temporal difference error) is used to indicate the importance of experience. (4c) Adding experience: When the agent interacts with the environment, the experience (state S, action a, reward R, next state S′, whether to terminate D) is added to the priority experience playback area, and the priority of the experience is stored. The initial priority is dynamically calculated according to TD-erTor. The selected action is executed in the environment. This action can be to offload the computing task to the cloud or execute it locally, and observe the results. The reward R, next state S′ and whether to terminate D are received from the environment. (4d) Update the agent's experience replay buffer: store the current state S, action a, reward R, next state S' and whether to terminate in the priority experience replay area, and calculate TD-error as the priority of the experience. This area is used to store the state-action-reward-next state sequence during training, allowing the agent to learn from historical data; (4e) Randomly extract a batch of experience samples from the priority experience playback area. For each sample, use the target network to calculate the optimal action value of the next state, and then calculate the target Q value based on the action value. Use the mean square error loss function to calculate the difference between the predicted Q value and the target Q value, and update the parameters of the evaluation network through the back propagation algorithm. When updating the network, each agent needs to minimize the following loss function to achieve gradient descent: (4f) Update the exploration rate: After a certain number of training steps, the Noisy DQN is used to replace the ε-greedy strategy. The noise parameter σ of the Noisy DQN will gradually decrease as the training progresses, thus achieving a smooth transition from exploration to exploitation. (4g) The training cycle continues until convergence, and steps (4c) and (4f) are repeated: continuously selecting actions, executing actions, observing results, storing experience, and training the network. As the training progresses, the impact of noise will gradually decrease, and the agent will rely more on the learned knowledge to make decisions.

Images