A joint computation offloading and resource allocation method based on multi-agent DDQN

Abstract

The present invention discloses a joint computing offloading and resource allocation method based on multi-agent DDQN. When the local computing resources of the vehicle cannot meet the requirements, the service vehicles and other MEC servers on the road are used to perform the calculations. The method takes into account the dynamic changes of resources during vehicle driving, adopts a centralized training-distributed execution multi-agent reinforcement learning architecture based on DDQN in the 5G cellular vehicle network, takes the computing overhead and communication overhead as the variation in the reward function, realizes the joint computing offloading decision and the optimal allocation of computing and power resources, so as to meet the needs of computing-intensive services in the vehicle network.

Description

Technical Field

[0001] The present invention relates to a vehicle networking technology, and in particular to a computing resource scheduling method in the vehicle networking, and more specifically to a joint computing offloading and resource allocation method based on a multi-agent DDQN. Background Art

[0002] With the increasing number of vehicles and users' higher requirements for product experience, the traditional local computing model can no longer meet the needs of computing-intensive services. The emergence of cloud servers allows vehicles to transfer some complex calculations to remote cloud computing servers to assist in the calculation, and then return the results to the user, thereby making up for the lack of terminal computing resources. However, in the vehicle-to-everything (V2X) communication of the Internet of Vehicles, low latency requirements are of paramount importance and a prerequisite for ensuring traffic safety. The long distance between the cloud server and the terminal leads to problems such as non-negligible communication latency and energy consumption. Some literature uses the mathematical tool of game theory to study the multi-user computing offloading problem of mobile edge computing in a multi-channel wireless interference environment, and realizes efficient computing offloading in a distributed manner. However, the vehicles in the Internet of Vehicles have a fast moving speed, which leads to rapid changes in the communication environment. It is difficult for us to obtain the exact environmental state, and the environmental state also changes rapidly in an unpredictable way. Therefore, it is difficult for traditional model-based methods to make reasonable and accurate choices.

[0003] Deep reinforcement learning is a machine learning algorithm that does not need to make assumptions about the optimization target or perform sub-optimization to reduce complexity like traditional optimization methods. Instead, it utilizes the perception ability of deep learning and the decision-making ability of reinforcement learning. It uses neural networks as function approximators and can solve decision-making problems in the high-dimensional state space of the Internet of Vehicles. Among them, the Double Deep Q Network (DDQN) is a new method for implementing high-dimensional data learning control strategies in deep reinforcement learning. DDQN uses two networks, an online network and a target network, which are used for the selection of agent actions and the estimation of action value functions, respectively. It can effectively solve the problem of over-estimation in the Deep Q Network (DQN), making the training process faster and more reliable. However, the traditional single-agent reinforcement learning method cannot meet the needs of computationally intensive services in the Internet of Vehicles, and the rapidly changing environment also greatly increases the computing cost.

[0004] The present invention proposes a joint computation offloading and resource allocation method based on multi-agent DDQN, which adopts a reinforcement learning architecture of offline training and online execution, and regards the entire Internet of Vehicles as a multi-agent environment. It takes minimizing communication overhead and computational overhead as the optimization goal, and can meet the low-latency, high-dynamic, and computationally intensive service requirements of cellular-vehicle-to-everything (C-V2X). Summary of the invention

[0005] The purpose of the present invention is to propose a joint computing offloading and resource allocation method for Internet of Vehicles based on a multi-agent DDQN architecture, which can reduce the energy consumption of the user end as much as possible on the basis of minimizing the delay. In order to achieve this purpose, the steps adopted by the present invention are:

[0006] Step 1: To minimize application latency and user-side energy consumption, the state space is defined as task information, channel state information, and auxiliary computing device state information; the action space is defined as offloading decisions, allocation of computing resources, and control of transmission power; the reward value includes rewards in the communication stage and the computing stage;

[0007] Step 2: Set the maximum number of training rounds E and the maximum number of steps per round T, update the vehicle position and large-scale fading parameters, and reset the remaining computational tasks of each agent;

[0008] Step 3: Take the state of agent n As the input of the online network, the action of agent n is executed based on the ε greedy policy And store the training data into the memory replay pool;

[0009] Step 4: Select a small batch of samples from the memory replay pool to calculate the Q estimate, train the Q network by minimizing the sum of squared errors between the Q network and the learning target, and use the stochastic gradient descent method to calculate the updated network parameters. If the number of training rounds reaches the maximum number of training rounds E, go to step 5, otherwise jump to step 2 to start a new round of training;

[0010] Step 5: Based on the trained DDQN network, obtain the optimal joint computing offloading and resource allocation strategy.

[0011] Furthermore, the step 1 includes the following specific steps:

[0012] Step 1-1: The vehicle networking system model under the C-V2X network consists of a cloud computing center, a core network, and a network access edge. The network access edge consists of a mobile edge computing (MEC) server and vehicle connected to a base station and a road side computing unit (RSU), and is connected to the cloud computing center through the core network;

[0013] Step 1-2: The state space S includes the agent n’s own task information D n , n∈N C , where N C is the number of client vehicles (Vehicle User Equipment Client, VUEC), channel state information H n , n∈N C And calculate the device status information F, that is,

[0014]

[0015] Among them, the task information D of agent n itself n Including the amount of task data to be processed d n 、The number of CPU revolutions required to complete the task c n , the total amount of module data that can be uninstalled q n , The number of CPU revolutions required for offloaded tasks And task delay threshold Right now

[0016]

[0017] Channel state information H n Includes the channel gain h between VUEC n and the vehicle user equipment server (VUES) VUES v n，v , and the channel gain h between MEC m n，m , denoted as H n ={h n，v ,h n，m The computing device status information F includes the remaining computing resources of VUES v and MEC m Expressed as

[0018] Step 1-3: The action space A includes offloading decisions, allocation of computing resources, and control of transmission power, expressed as

[0019] A n ={o n (t), p n (t), xn (t)} (Expression 3)

[0020] Among them n (t) = {0, 1, 2}, is the offloading decision of agent n at step t, 0, 1, 2 represent local computing, service vehicle computing, and other MEC computing, respectively, p n (t) = {23, 10, 5, 0} dBm, which are four different levels of transmission power, x n (t) = {0, 0.25, 0.5, 1}, which is the computing resource allocation ratio of 4 different levels;

[0021] Step 1-4: The reward includes the rewards of the communication phase and the calculation phase, expressed as

[0022]

[0023] in Reward for the communication phase, benefit comm is a fixed value that can be adjusted during training. is the total cost of the communication phase. Similarly, The purpose of reinforcement learning is to find a strategy that can maximize the expected return starting from any state. The converted expected return is expressed as

[0024]

[0025] Where γ∈[0,1] represents the discount rate, R t+k+1 Represents the reward value at step t+k+1.

[0026] Furthermore, the step 3 includes the following specific steps:

[0027] Step 3-1: Set the state of each agent n Input the online network and output actions according to the ε greedy strategy And apply the actions of all agents to the environment to get an immediate reward R t+1 , go to the next state

[0028] Step 3-2: Update the small-scale fading parameters and convert the training data of each agent Stored in the memory playback pool.

[0029] Furthermore, the step 4 includes the following specific steps:

[0030] Step 4-1: Select a small batch of training sample data from the memory playback pool Constructing a data set;

[0031] Step 4-2: Calculate the Q estimate as

[0032]

[0033] Where γ is the discount rate, θ t and are the parameters of the online network and the target network, respectively. The structures of the two networks are the same, but the online network is used to evaluate the greedy strategy and continuously update θ t , while the target network is used to estimate the value and is updated at regular intervals

[0034] Step 4-3: Define the loss function as

[0035] L t (θ)=∑ D (y t -Q(S t , A t ;θ)) (Expression 7)

[0036] Where D is the sampling set, that is, the data set selected in step 4-1, y t The Q estimate defined by Expression 6;

[0037] Step 4-4: Calculate θ using stochastic gradient descent t The updated value is

[0038]

[0039] Where α is the iteration step size, L t (θ t ) is the loss function defined in Expression 7, and updates θ t So that Q(S t , A t θ t ) is as close as possible to the Q estimate y t ;

[0040] Step 4-5: If t<T, T is the total time step of each round, and t is the current time step, then t=t+1, and jump to step 3; otherwise, set t=0 and jump to step 4-6;

[0041] Step 4-6: If e<E, E is the total number of training rounds, and e is the current round number, then e=e+1, and jump to step 2; otherwise, jump to step 5. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is an algorithm framework diagram of a joint computation offloading and resource allocation method based on multi-agent DDQN proposed by the present invention;

[0043] Figure 2 It is a schematic diagram of the system model adopted by the present invention;

[0044] Figure 3 It is a schematic diagram of the relationship between the state space, action space and reward function in the reinforcement learning model adopted by the present invention;

[0045] Figure 4 This is a diagram of the offloadable task volume-cost simulation result of the present invention;

[0046] Figure 5 is a task vehicle number-cost simulation result diagram of the present invention; DETAILED DESCRIPTION

[0047] The present invention is further described in detail below with reference to the accompanying drawings and examples.

[0048] The algorithm framework diagram of the joint computing offloading and resource allocation method based on multi-agent DDQN proposed in this invention is shown in the attached figure. Figure 1 As shown. Specifically including the following steps:

[0049] Step 1: To minimize application latency and user-side energy consumption, the state space is defined as task information, channel state information, and auxiliary computing device state information; the action space is defined as offloading decisions, allocation of computing resources, and control of transmission power; the reward value includes rewards in the communication stage and the computing stage;

[0050] Step 2: Set the maximum number of training rounds E and the maximum number of steps per round T, update the vehicle position and large-scale fading parameters, and reset the remaining computational tasks of each agent;

[0051] Step 3: Take the state of agent n As the input of the online network, the action of agent n is executed based on the ε greedy policy And store the training data into the memory replay pool;

[0052] Step 4: Select a small batch of samples from the memory replay pool to calculate the Q estimate, train the Q network by minimizing the sum of squared errors between the Q network and the learning target, and use the stochastic gradient descent method to calculate the updated network parameters. If the number of training rounds reaches the maximum number of training rounds E, go to step 5, otherwise jump to step 2 to start a new round of training;

[0053] Step 5: Based on the trained DDQN network, obtain the optimal joint computing offloading and resource allocation strategy.

[0054] Furthermore, the step 1 includes the following specific steps:

[0055] Step 1-1: The vehicle networking system model under the C-V2X network consists of a cloud computing center, a core network, and a network access edge. The network access edge consists of a mobile edge computing (MEC) server and vehicle connected to a base station and a road side computing unit (RSU), and is connected to the cloud computing center through the core network. The system model diagram is shown in the attached figure. Figure 2 As shown in the figure, BS is the base station, RSU is the roadside computing unit, and VUEC is the vehicle computing offloading client, that is, the vehicle that currently has computing tasks, a total of N C VUES is the vehicle computing offloading server, that is, vehicles that currently have no computing tasks and can provide computing assistance to VUEC, a total of N S Vehicle. V2V, V2R and V2B links are vehicle-to-vehicle, vehicle-to-roadside MEC, and vehicle-to-base station MEC links, respectively, using their own dedicated resource pools, from which vehicles can select channel resources for data transmission;

[0056] Step 1-2: The state space S includes the agent n’s own task information D n , n∈N C , where N C is the number of client vehicles (Vehicle User Equipment Client, VUEC), channel state information H n , n∈N C And calculate the device status information F, that is,

[0057]

[0058] Among them, the task information D of agent n itself n Including the amount of task data to be processed d n 、The number of CPU revolutions required to complete the task c n , the total amount of module data that can be uninstalled q n , The number of CPU revolutions required for offloaded tasks And task delay threshold Right now

[0059]

[0060] Channel state information H n Includes the channel gain h between VUEC n and the vehicle user equipment server (VUES) VUES v n，v , and the channel gain h between MEC m n，m , denoted as H n ={h n，v ,hn，m The computing device status information F includes the remaining computing resources of VUES v and MEC m Expressed as

[0061] Step 1-3: The action space A includes offloading decisions, allocation of computing resources, and control of transmission power, expressed as

[0062] A n ={o n (t), p n (t), x n (t)} (Expression 3)

[0063] Among them n (t) = {0, 1, 2}, is the offloading decision of agent n at step t, 0, 1, 2 represent local computing, service vehicle computing, and other MEC computing, respectively, p n (t) = {23, 10, 5, 0} dBm, which are four different levels of transmission power, x n (t) = {0, 0.25, 0.5, 1}, which is the computing resource allocation ratio of 4 different levels;

[0064] Step 1-4: The reward includes the rewards of the communication phase and the calculation phase, expressed as

[0065]

[0066] in Reward for the communication phase, benefit comm is a fixed value that can be adjusted during training. is the total cost of the communication phase. Similarly, The purpose of reinforcement learning is to find a strategy that can maximize the expected return starting from any state. The converted expected return is expressed as

[0067]

[0068] Where γ∈[0,1] represents the discount rate, R t+k+1 Represents the reward value at step t+k+1.

[0069] Furthermore, the step 3 includes the following specific steps:

[0070] Step 3-1: Set the state of each agent n Input the online network and output actions according to the ε greedy strategy And apply the actions of all agents to the environment to get an immediate reward R t+1 , go to the next state

[0071] Step 3-2: Update the small-scale fading parameters and convert the training data of each agent Stored in the memory playback pool.

[0072] Furthermore, the step 4 includes the following specific steps:

[0073] Step 4-1: Select a small batch of training sample data from the memory playback pool Constructing a data set;

[0074] Step 4-2: Calculate the Q estimate as

[0075]

[0076] where θ t and are the parameters of the online network and the target network, respectively. The structures of the two networks are the same, but the online network is used to evaluate the greedy strategy and continuously update θ t , while the target network is used to estimate the value and is updated at regular intervals

[0077] Step 4-3: Define the loss function as

[0078] L t (θ)=∑ D (y t -Q(S t , A t ;θ)) (Expression 7)

[0079] Where D is the sampling set, that is, the data set selected in step 4-1;

[0080] Step 4-4: Calculate the updated θ using stochastic gradient descent t for

[0081]

[0082] Where α is the iteration step size, L t (θ t ) is the loss function defined in (7), and updates θ t So that Q(S t , A t θ t ) is as close as possible to the Q estimate y t ;

[0083] Step 4-5: If t<T, T is the total time step of each round, and t is the current time step, then t=t+1, and jump to step 3; otherwise, set t=0 and jump to step 4-6;

[0084] Step 4-6: If e<E, E is the total number of training rounds, and e is the current round number, then e=e+1, and jump to step 2; otherwise, jump to step 5.

[0085] Furthermore, the step 5 includes the following specific steps:

[0086] Step 5-1: Using the trained DDQN deep reinforcement learning model, input the state S of the system at a certain time t t ;

[0087] Step 5-2: The online network is based on the input state S t Output the optimal action A t , and obtain the optimal offloading decision, computing resource allocation decision, and transmission power.

[0088] In order to verify the effectiveness of the optimization algorithm, this paper conducts simulation based on the Keras library in Python. The simulation results are shown in the attached Figure 4 and Figure 5 The simulation results are described in detail below.

[0089] Figure 4The performance of different methods under different numbers of task vehicles is compared and analyzed, among which MJCORA is a multi-agent reinforcement learning based Joint Computing Offloading and Resource Allocation method (MJCORA) proposed by the present invention, SJCORA is a single-agent reinforcement learning based Joint Computing Offloading and Resource Allocation method (SJCORA), RJCORA is a random selection based Joint Computing Offloading and Resource Allocation method (RJCORA), Local is a local calculation method, Cost is a cost value, which is defined as the weighted sum of latency and energy consumption. The lower the cost value, the better the performance of the method. The results show that as the number of vehicles increases, the cost values ​​obtained by all methods increase with the increase of task vehicles, but the MJCORA method proposed by the present invention always has the lowest cost value among the four methods. When the number of task vehicles reaches 10, the cost value is about 30% lower than that of local calculation. It can be seen that when latency and power consumption are comprehensively considered, the computation offloading solution we proposed has more advantages than local computing and makes up for the defects of local computing. At the same time, it performs better than the SJCORA algorithm and the RJCORA algorithm.

[0090] Figure 5 The performance of different methods under different offloadable task sizes is compared and analyzed. It can be seen that when the amount of computing tasks is small, the local computing performance is optimal and the cost value is the lowest. However, as the amount of computing tasks increases, the transmission delay and power consumption spent on computing offloading are insignificant compared to the time spent on local computing, so computing offloading is more appropriate. In this case, the performance of the MJCORA method proposed in the present invention is lower in cost value and better in performance than the other two methods.

[0091] Therefore, the comprehensive simulation results Figure 4 and Figure 5 It can be seen that the method proposed in the present invention has obvious performance advantages over local computing and the other two methods in computing-intensive services, and reduces the cost value by 100%-200% compared with the RJCORA method.

[0092] Finally, the attached Figure 1 -Attached Figure 3 Provide detailed explanation.

[0093] exist Figure 1 In the figure, the framework diagram of the method proposed in the present invention is described. Two Q networks are used. The online Q network is used to generate a greedy strategy and put the training data into the experience reuse pool. Then, small batch samples are extracted from it and the target Q network is used to calculate the value, thereby updating the network parameter θ t .

[0094] exist Figure 2 In the figure, a system model diagram adopted by the present invention is described, in which the network access edge is connected to the cloud computing center through the core network.

[0095] exist Figure 3 In the paper, the relationship between the state space S, action space A and reward function R in reinforcement learning is described. The agent obtains the current environment state Z t After that, combine the observed state with the current state S t , select action A according to a specific strategy π t , after applying the action to the environment, we get the next observed state and reward R t+1 .

Claims

1. A joint computation offloading and resource allocation method based on multi-agent DDQN, specifically comprising the following steps: Step 1: To minimize application delay and user-side energy consumption, define the state space as task information, channel state information, and auxiliary computing device state information; The action space includes offloading decisions, allocation of computing resources, and control of transmission power; the reward value includes the rewards of the communication phase and the computing phase. The reward value R of the t+1 step is t+1 It can be expressed as in Reward for the communication phase, benefit comm is a fixed value that can be adjusted during training. is the total cost of the communication phase. Similarly, To calculate the reward in the stage, the goal of reinforcement learning is to find an optimal strategy that can maximize the expected return starting from any state. The expected return at the tth step is expressed as Where γ∈[0,1] represents the discount rate, R t+k+1 Represents the reward value at step t+k+1; Step 2: Set the maximum number of training rounds E and the maximum number of steps per round T, update the vehicle position and large-scale fading parameters, and reset the remaining computational tasks of each agent; Step 3: Take the state of agent n at step t As the input of the online network, the action of agent n is executed based on the ε greedy policy And store the training data into the memory replay pool. The specific steps are: Step 3-1: Set the state of each agent n As the input of the online network, D n is the task information of agent n, H n is the channel state information, and outputs actions according to the ε greedy strategy And apply the actions of all agents to the environment to get an immediate reward R t+1 , go to the next state Step 3-2: Update the small-scale fading parameters and convert the training data of each agent Stored in the memory playback pool; Step 4: Select a small batch of samples from the memory replay pool to calculate the Q estimate, train the Q network by minimizing the sum of squared errors between the Q network and the learning target, and use the stochastic gradient descent method to calculate the updated network parameters. If the number of training rounds reaches the maximum number of training rounds E, go to step 5, otherwise jump to step 2 to start a new round of training; Step 5: Based on the trained DDQN network, obtain the optimal joint computing offloading and resource allocation strategy.

2. According to claim 1, a method for joint computation offloading and resource allocation based on multi-agent DDQN is characterized in that Use 2 different Q networks to calculate updated parameter values ​​and select greedy strategies. The specific steps are as follows: Step 4-1: Select a small batch of training sample data from the memory playback pool Constructing a data set; Step 4-2: Calculate the Q estimate as Where γ is the discount rate, θ t and are the parameters of the online network and the target network, respectively. The structures of the two networks are the same, but the online network is used to evaluate the greedy strategy and continuously update θ t , while the target network is used to estimate the value and is updated at regular intervals Step 4-3: Define the loss function as L t (θ)=∑ D (y t -Q(S t ,A t ;i)) Where D is the sampling set, that is, the data set selected in step 4-1, y t is the Q estimate defined in step 4-2; Step 4-4: Calculate θ using stochastic gradient descent t The updated value is Where α is the iteration step size, L t (θ t ) is the loss function defined in step 4-3, update θ t So that Q(S t , A t θ t ) is as close as possible to the Q estimate y t ; Step 4-5: If t<T, T is the total time step of each round, and t is the current time step, then t=t+1, and jump to step 3; otherwise, set t=0 and jump to step 4-6; Step 4-6: If e<E, E is the total number of training rounds, and e is the current round number, then e=e+1, and jump to step 2; otherwise, jump to step 5.

Images