A method for offloading delay-sensitive applications in Internet of Vehicles based on federated learning

Abstract

The present invention discloses a method for unloading delay-sensitive applications in an Internet of Vehicles based on federated learning, the method comprising the following steps: step 1, determining an Internet of Vehicles network architecture based on MEC; step 2, establishing an application model; step 3: determining an MEC node collaboration scheme; step 4, establishing a system model and obtaining an optimization problem; step 5, designing an algorithm combining DDQN and FL and solving the optimization problem. The method divides delay-sensitive applications into mutually dependent subtasks, provides an MEC node collaboration scheme, and models the problem as an optimization problem with computing resources as constraints and minimizing total computing delay as the goal. The problem is solved by combining federated learning with deep reinforcement learning. On the basis of obtaining an unloading method that can minimize the total computing delay, the privacy security of users can also be guaranteed to avoid leakage of user private data.

Description

Technical Field

[0001] The present invention relates to a method for offloading tasks in an Internet of Vehicles (IoV), and in particular to a method for offloading delay-sensitive applications in an IoV based on a combination of deep reinforcement learning and federated learning (FL). Background Art

[0002] The Internet of Vehicles (IoV) is an intelligent integrated network system that takes vehicles as basic information units and aims to establish an improved traffic condition, improved travel efficiency, and expanded forms of information interaction. The rise of IoV can not only promote economic development, but also has huge social potential value, such as reducing traffic congestion and greatly saving traffic costs. At present, many latency-sensitive applications with low latency requirements have emerged in the IoV environment, such as augmented reality and video stream analysis. Although deploying different applications in the core cloud can provide users with convenient and rich services, a large number of devices accessing the core cloud will greatly increase the network load and data transmission delay, which does not meet the requirements of low latency and high reliability of applications. To solve this problem, Mobile Edge Computing (MEC) came into being. Mobile Edge Computing transfers the computing task load from the remote cloud to the edge nodes of the core network closer to the user, such as the Road Side Unit (RSU). By offloading computing-intensive tasks to surrounding vehicles or edge servers, it can meet the processing requirements of huge amounts of data in scenarios such as autonomous driving, and meet the latency sensitivity of some entertainment services. However, the computing and network resources of edge nodes are limited. Therefore, mobile devices should carefully plan whether computing tasks are executed locally or offloaded to edge nodes or the cloud to achieve the goal of minimizing task completion latency.

[0003] In response to this type of task offloading problem in mobile edge computing, Wu et al. proposed a fast offloading algorithm based on support vector machines to meet the fast offloading requirements in vehicle networks; Sun et al. proposed a new computing migration scheme based on policy reinforcement learning, which can dynamically learn the optimal strategy in a dynamic environment; Ning et al. developed an intent-based traffic control system based on deep reinforcement learning, which can dynamically coordinate edge computing and content caching. However, the decision-making of task offloading using traditional machine learning methods will face various security and privacy threats. As an extension of cloud computing, edge computing still has some security issues in cloud computing. Due to the complexity and real-time nature of edge computing service models, the multi-source heterogeneity and perception of data, and the resource-constrained characteristics of terminals, the data security and privacy protection mechanisms in cloud computing environments are no longer suitable for the protection of massive data generated by edge devices. In response to the problem that data security and privacy cannot be guaranteed, Lu et al. used homomorphic encryption to develop a lightweight data aggregation scheme for heterogeneous IoT devices in edge computing, which ensures data confidentiality and data integrity. However, this scheme brings a huge communication burden and does not consider the mobility of edge devices, so it cannot be applied to vehicle networking scenarios. Lyu et al. proposed a privacy-preserving data aggregation scheme based on differential privacy and secret sharing technology. However, this method does not have the characteristics of distribution, but relies on the central controller to train the model and generate and issue decisions, which will increase the communication cost and delay between the vehicle and the central controller. In summary, most of the current research on task offloading in the Internet of Vehicles focuses on centralized machine learning. However, traditional machine learning methods cannot guarantee the privacy and security of users' data, and most of the research on data privacy and security issues does not fit the actual environment of the Internet of Vehicles well and cannot adapt to the dynamically changing network. Summary of the invention

[0004] In order to solve the problem of unloading decision-making for delay-sensitive tasks generated by vehicles during driving and protect the privacy and security of users, the present invention provides a method for unloading delay-sensitive applications in Internet of Vehicles based on federated learning. This method divides delay-sensitive applications into subtasks that are dependent on each other, provides a MEC node collaboration solution, and models the problem as an optimization problem with computing resources as constraints and minimizing the total computing delay as the goal. The problem is solved by combining federated learning with deep reinforcement learning. On the basis of obtaining an unloading method that can minimize the total computing delay, the privacy and security of users can also be protected to avoid leakage of user private data.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A method for unloading delay-sensitive applications in an Internet of Vehicles based on federated learning comprises the following steps:

[0007] Step 1: Determine the network architecture of the Internet of Vehicles based on MEC

[0008] The MEC-based Internet of Vehicles network includes three entities: RSU, MEC edge server and vehicle, among which: RSU is a collection Indicates, M represents the total number of RSUs, RSU number m∈[1,M]; all vehicles are represented by the set Indicates, N represents the total number of vehicles, and the vehicle number n∈[1,N]; all RSUs are evenly distributed along the one-way straight road, and each RSU is configured with a MEC edge server. The computing resource set of the MEC edge server is f m The computing resources of the mth RSU provide comprehensive coverage of communication and computing services for vehicles on the road. Vehicles are connected to the RSUs through wireless communications, so that the delay-sensitive applications generated by the vehicles are handed over to the MEC edge servers on the RSUs for computing;

[0009] Step 2: Build an application model

[0010] Step 21: Divide the delay-sensitive applications generated by each vehicle at random time into subtasks that have dependencies on each other. The input-output dependencies of the tasks generated by vehicle n are represented by a directed acyclic graph The total number of subtasks is represented by T, then The task set is used express; represents a set of vertices, Each vertex in represents a subtask i, and the related parameters C n,i represents the required computing resources of the i-th subtask after the application generated by vehicle n is split; represents a set of edges. If the edge If it exists, it means that there is a dependency relationship between subtask i and subtask j, that is, subtask i is the predecessor subtask of subtask j, and O n,i,j Indicates the size of data transferred between subtasks;

[0011] Step 22: Set the uninstall decision variable λ n,m,i ∈{0,1} represents whether the i-th subtask of vehicle n is chosen to be calculated on the MEC edge server on the m-th RSU. If it is chosen to be calculated on the MEC edge server on the m-th RSU, then λ n,m,i =1, otherwise 0;

[0012] Step 23: Let the resource allocation decision variable f n,m,irepresents the computing resources allocated by the MEC edge server on the mth RSU to the i-th subtask after the application generated by vehicle n is split;

[0013] Step 24: Define the decision variables of all subtasks of vehicle n as λ n =(λ n,1,1 ,...,λ n,1,T ,...,λ n,m,i ,...,λ n,M,1 ,...,λ n,T,T ), the resource allocation decision variables of all subtasks are f n =(f n,1,1 ,...,f n,1,T ,...,f n,m,i ,...,f n,M,1 ,...,f n,M,T );

[0014] Step 25: define the set of all direct predecessor subtasks of subtask i as pred(i), and the set of all direct successor subtasks of subtask i as succ(i);

[0015] Step 3: Determine the MEC node collaboration plan

[0016] Step 31: The vehicle generates a delay-sensitive application at a random time. At this time, the vehicle receives the current computing resource information of all RSUs broadcast by the RSU where it is located. And according to Make decisions;

[0017] Step 32: The vehicle uploads all input data and decision information to the RSU where the vehicle is currently located. The RSU uses wired transmission between RSUs to distribute each subtask to each RSU where the vehicle decision is offloaded based on the vehicle's decision information, and requests the MEC edge server connected to them to collaborate on computing;

[0018] Step 3: When all subtasks are calculated, that is, the final calculation results are obtained, the RSU transmits the calculation results to the RSU where the vehicle is located at this time, and transmits the results back to the vehicle;

[0019] Step 4: Establish a system model and obtain the optimization problem

[0020] Step 41: Establish a communication model

[0021] The communication model includes two types: communication between RSUs and communication between vehicles and RSUs, where:

[0022] RSUs use wired communication, and the transmission rate of wired communication is constant r. RSUIndicates that data O is transmitted between RSU and RSU n,i,j Required transmission delay for:

[0023]

[0024] Wireless communication is adopted between vehicles and RSUs. It is assumed that vehicles can be allocated non-overlapping frequency bands to avoid mutual interference during data transmission, and each vehicle is allocated the same bandwidth, denoted by B. The channel gain of the uplink between all vehicles and all RSUs is represented by the vector h = (h 1,1 ,...,h 1,M ,...,h n,m ,...,h N,1 ,...h N,M ) represents the channel gain h between vehicle n and the mth RSU n,m It is expressed as:

[0025]

[0026] Among them, A d represents the antenna gain, f c represents the carrier frequency, PL represents the path loss index, and d represents the distance between the vehicle and the RSU;

[0027] According to Shannon's theorem, the transmission rate r between vehicle n and the mth RSU is obtained n,m :

[0028]

[0029] Among them, y n represents the transmission power of vehicle n, N0 represents the power of Gaussian additive white noise;

[0030] Data O is transmitted between vehicle n and the mth RSU n,0,j Required transmission delay for:

[0031]

[0032] Step 42: Establish a calculation model

[0033] Define the start time RT n,m,j and the termination time FT n,m,i , start time RT n,m,i Refers to the earliest time when the i-th subtask generated by vehicle n can start computing on the m-th RSU after the application is split, and the termination time FT n,m,i It refers to the time when the i-th subtask after the application generated by vehicle n is split and the calculation is completed on the m-th RSU;

[0034] The computational delay of the i-th subtask after application segmentation generated by vehicle n on the m-th RSU is:

[0035]

[0036] Therefore, the starting time RT of the i-th subtask generated by vehicle n after application segmentation can be obtained: n,m,i and the termination time FT n,m,i , if subtask i is the direct successor subtask of the virtual starting subtask 0, then the starting time of subtask i is the sum of the communication delay of the application input data from the vehicle to the RSU and the communication delay between RSUs. Conversely, the starting time of subtask i is the sum of the maximum end time of all its direct predecessor subtasks and the communication delay between RSUs, that is:

[0037]

[0038] The termination time FT of the i-th subtask generated by vehicle n after application segmentation n,m,i is equal to its starting time RT n,m,i And the sum of the computation delay, that is:

[0039]

[0040] The total computational delay of the application generated by vehicle n is:

[0041] t n =RT n,m,T+1 ;

[0042] Step 43: Establish the optimization problem

[0043] The computational delay t of the application generated by vehicle n is derived through the communication model and computational model established in step 41 and step 42. n , we get the following optimization problem:

[0044]

[0045]

[0046]

[0047]

[0048] t n ≤t max

[0049] Among them, t n represents the computational latency of completing the application generated by vehicle n, tmax represents the maximum deadline for completing this task; Constraints 1 and 2 represent that each subtask can only select one RSU for calculation and must select one RSU; Constraint 3 represents that the computing resources allocated by the MEC edge server on the mth RSU for the i-th subtask of vehicle n need to be greater than 0 and less than or equal to the remaining computing resources of the MEC edge server on the RSU; Constraint 4 represents that the total computing delay required to complete the task must be less than or equal to the maximum deadline for completing this task;

[0050] Step 5: Design an algorithm that combines DDQN and FL and solve the optimization problem

[0051] Step 51: Establish a Markov decision process

[0052] The state space includes the amount of data transmitted by vehicles for delay-sensitive applications. and the required computing resources Variable channel gain Real-time changing vehicle speed and location The remaining computing resources of the MEC edge server in the RSU Among them, v n is the speed of vehicle n, x n is the current location of vehicle n;

[0053] The action space includes and resource decision variables λ n,i The index number of the RSU that is unloaded for the i-th subtask decision of vehicle n;

[0054] The reward function is the negative of the total computational delay, i.e. -t n ;

[0055] Step 52: Design an algorithm combining DDQN and FL

[0056] Step 521: Establish a Q network and a TargetQ network with the same structure at the client (i.e., the vehicle end) and the center end, and initialize all parameters θ of the Q network and all parameters θ' of the Target Q network, where θ'=θ;

[0057] Step 522, initialize state s and experience replay pool;

[0058] Step 523: Input state s into the client's Q network, obtain the Q values ​​corresponding to all actions, select action a using the ε greedy algorithm, obtain the reward r and the state s' at the next moment through the current state s and action a, store [s, a, r, s'] into the experience revisit pool, and then replace the current state s with the state s' at the next moment;

[0059] Step 524: If the number of storage times at this time meets the learning interval step length, start training the neural network. The specific steps are as follows:

[0060] Sample b samples from the experience replay pool to calculate the current target Q value, use the mean-variance loss function, and update the parameters of the Q network through gradient back propagation. If the number of training times meets the target network update step, the parameters of the Q network are copied to the Target Q network, that is, θ'=θ; if the number of training times meets the aggregation step, the Q network and Target network parameters of the client selected by the central server are uploaded to the central server for aggregation, and then the central server sends the aggregated parameters to all clients, and the clients update the local model parameters;

[0061] Step 525: Repeat step 524 until the model reaches convergence;

[0062] Step 53: Use the algorithm combining DDQN and FL to solve the optimization problem

[0063] The current state s is input into the trained network, and the above optimization problem is solved by combining DDQN with FL to obtain the offloading decision that can minimize the total computing delay, including the offloading location of each subtask i and the computing resources allocated by the MEC edge server on the mth RSU to the i-th subtask of vehicle n.

[0064] Compared with the prior art, the present invention has the following advantages:

[0065] 1. The present invention solves the task offloading decision problem of delay-sensitive applications that can be divided into interdependent subtasks generated during vehicle driving in the Internet of Vehicles environment. It uses the MEC server of the edge device for calculation, which makes up for the shortcomings of local computing and cloud computing, such as high latency and unstable communication. The stability of communication between RSU and vehicle is guaranteed through stable wired transmission between RSUs and reselection of backhaul RSU and other mechanisms.

[0066] 2. The present invention uses an algorithm that combines DDQN and FL to effectively solve the problem of data privacy information leakage caused by traditional machine learning, and can ensure the convergence speed and model performance. FL uses the method of transferring model intermediate parameters rather than the traditional method of transferring data used in machine learning to protect the user's data privacy security. The user's private data is retained locally and the data will not be leaked to the outside, meeting the needs of user privacy protection and data security. Moreover, the status of each participant is equal, and fair cooperation can be achieved. It can also ensure that each vehicle can perform encrypted exchange of information and model parameters while maintaining independence, and grow at the same time.

[0067] 3. The algorithm combining DDQN and FL designed by the present invention can effectively reduce the total computing delay required to complete the task compared to the algorithm that relies entirely on local computing; compared to the greedy algorithm, the resource allocation strategy in this algorithm can prevent the uneven distribution of MEC servers on some RSUs while the MEC servers on some RSUs are busy, and can effectively improve the task completion rate; compared to the traditional centralized machine learning algorithm, this algorithm can improve the task completion rate while protecting the privacy and security of users, and the vehicle can make autonomous decisions based on the current environment; compared to some distributed machine learning algorithms, each vehicle has complete autonomy over local data and can decide when and how to join federated learning.

[0068] 4. The present invention realizes the function of jointly training a model by many vehicles without leaking privacy data. After the model training is completed, each vehicle can download all or part of the global model according to its own needs, and then use the local model to make independent decisions. Compared with most task offloading methods and privacy protection methods, the present invention can well adapt to the dynamic and high-speed vehicle networking environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flowchart of a method for offloading delay-sensitive applications in Internet of Vehicles based on federated learning;

[0070] Figure 2 For the MEC vehicle networking environment;

[0071] Figure 3 A directed acyclic graph of an application divided into interdependent subtasks;

[0072] Figure 4 It is an experience replay mechanism;

[0073] Figure 5 It is the update and iteration mechanism of the DDQN algorithm;

[0074] Figure 6 It is the basic architecture based on DDQN and FL algorithms.

[0075] Figure 7 It is the convergence curve of the reward function of Q-learning, DQN, and DDQN algorithms;

[0076] Figure 8 It is the convergence curve of the reward function of the algorithm combining DDQN and FL;

[0077] Fig. 9 is the effect of parameter P on convergence;

[0078] Fig.10 is the effect of parameter E on convergence;

[0079] Fig.11 Analyze the environmental adaptability of the algorithm;

[0080] Fig.12 Comparison of resource utilization between algorithms;

[0081] Fig.13 The impact of the number of vehicles on the task completion rate of each algorithm;

[0082] Fig.14 The influence of vehicle speed on the task completion rate of each algorithm. DETAILED DESCRIPTION

[0083] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0084] The present invention provides a method for unloading delay-sensitive applications in an Internet of Vehicles based on federated learning, which mainly solves the task unloading decision problem of delay-sensitive applications that can be divided into mutually dependent subtasks generated during vehicle driving in an Internet of Vehicles environment, and at the same time solves the data privacy leakage problem caused by vehicles using deep reinforcement learning training models. Specifically, the present invention solves how to train a model that can obtain an unloading decision that minimizes the total computing delay of delay-sensitive application tasks generated by vehicles in an environment where vehicles are driving at high speeds, while ensuring data privacy security.

[0085] like Figure 1 As shown, the method comprises the following steps:

[0086] Step 1: Determine the network architecture of the Internet of Vehicles based on MEC.

[0087] The MEC-based Internet of Vehicles network of the present invention includes three entities: RSU, MEC edge server and vehicle. Figure 2 The present invention considers a straight one-way road, and RSU uses a set Represented by and evenly distributed along the one-way straight road, M represents the total number of RSUs, and RSU number m∈[1,M]. All vehicles are represented by the set Represents, N represents the total number of vehicles, and the vehicle number n∈[1,N]. Each RSU is configured with a MEC edge server, and these RSUs provide comprehensive coverage of communication and computing services for vehicles on the road. Vehicles can connect to the RSU through wireless communication, so that the delay-sensitive applications generated by the vehicle are handed over to the MEC edge server on the RSU for calculation. The RSU and MEC server are connected by optical fiber, and the bandwidth is large enough. The RSU and MEC edge server are set to a one-to-one relationship, so RSU can also be used to refer to the MEC server. The input data and calculation results are transmitted between RSUs by wired transmission. The present invention takes into account the reality that the computing resources of the MEC edge server are limited, and uses the collection To represent the computing resources of the MEC edge server on each RSU.

[0088] Step 2: Create an application model.

[0089] Each vehicle will generate a delay-sensitive application at a random time. The application can be divided into subtasks that have dependencies on each other. The input-output dependency of the task generated by vehicle n can be represented by a directed acyclic graph To express, if Figure 3 That is, if you want to calculate subtask 6, you must wait for subtask 1, subtask 2, and subtask 3 to complete their calculations and pass their results before you can continue to calculate subtask 6. Directed Acyclic Graph Each vertex in the directed acyclic graph represents a subtask i. A collection of tasks is available To express, the relevant parameter C n,i represents the required computing resources of the ith subtask after the application segmentation generated by vehicle n. For each edge Represents the dependency relationship between subtask i and subtask j, with O n,i,j In particular, the present invention introduces two virtual subtasks 0 and T+1 as the start task and the end task, and ensures that the application starts and ends at the vehicle end by forcing the execution of these two virtual subtasks locally, that is, C n,0 =C n,T+1 = 0. Let the unloading decision variable λ n,m,i∈{0,1} represents whether the i-th subtask of vehicle n is chosen to be calculated on the MEC edge server on the m-th RSU. If it is chosen to be calculated on the MEC edge server on the m-th RSU, then λ n,m,i =1, otherwise 0; let the resource allocation decision variable f n,m,i represents the computing resources allocated by the MEC edge server on the mth RSU to the i-th subtask of vehicle n. The decision variables of all subtasks of vehicle n are defined as n =(λ n,1,1 ,...,λ n,1,T ,...,λ n,m,i ,...,λ n,M,1 ,...,λ n,M,T ), the resource allocation decision variable for all subtasks is f n =(f n,1,1 ,...,f n,1,T ,...,f n,m,i ,...,f n,M,1 ,...,f n,M,T ).

[0090] In order to better describe the dependency relationship and calculation process of subtasks, the set of direct predecessor subtasks and the set of direct successor subtasks of subtasks are defined. The set of all direct predecessor subtasks of subtask i is represented by pred(i). The set of all direct successor subtasks of subtask i is represented by succ(i). Figure 3 As shown, the direct predecessor subtask pred(9) of subtask 9 includes subtasks 5, 6, and 7, and the direct successor subtask succ(1) of subtask 1 includes subtasks 6 and 7.

[0091] Step 3: Determine the MEC node collaboration plan.

[0092] First, the vehicle will generate a delay-sensitive application at a random time. At this time, the vehicle will receive the current computing resource information of all RSUs broadcast by the RSU where it is located. And according to Make a decision. Then, the vehicle uploads all input data and decision information to the RSU where the vehicle is currently located. The RSU uses the wired transmission between RSUs to distribute each subtask to the RSU where each vehicle decision is offloaded based on the vehicle's decision information, and requests the MEC edge server connected to them to collaborate in computing to reduce computing latency, thereby reducing the unreliability of the vehicle due to its high-speed mobility to a certain extent. Finally, when all subtasks are calculated, that is, when the final calculation results are obtained, due to the high-speed movement of the vehicle, the vehicle is likely to be out of the coverage of the initial RSU. Therefore, the RSU will transmit the calculation results to the RSU where the vehicle is located at this time, and transmit the results back to the vehicle. This strategy of reselecting the RSU for returning the results will also improve the reliability of offloading. For virtual subtasks 0 and T+1, since they are forced to be executed locally, that is, on the vehicle, it is guaranteed that the start and end of the application are both performed on the vehicle side.

[0093] Step 4: Establish system model.

[0094] Communication model:

[0095] The communication model includes two types: communication between RSUs and communication between vehicles and RSUs, where:

[0096] RSUs use wired communication. Since wired communication has a very high transmission rate, the transmission rate is constant r. RSU To represent. Then the data transmitted between RSU and RSU is O n,i,j Required transmission delay for:

[0097]

[0098] Wireless communication is used between vehicles and RSUs. The present invention assumes that vehicles can be allocated non-overlapping frequency bands to avoid mutual interference between vehicles during data transmission, and each vehicle is allocated the same bandwidth, represented by B. The channel gain of the uplink between all vehicles and all RSUs is represented by the vector h = (h 1,1 ,...,h 1,M ,...,h n,m ,...,h N,1 ,...h N,M ). The channel gain between vehicle n and the mth RSU is directly related to the distance, so the channel gain is expressed as:

[0099]

[0100] Among them, A drepresents antenna gain, fc represents carrier frequency, PL represents path loss exponent, and d represents the distance between vehicle and RSU.

[0101] According to Shannon's theorem, the transmission rate r between vehicle n and the mth RSU can be obtained as n,m :

[0102]

[0103] Among them, y n represents the transmission power of vehicle n, and N0 represents the power of Gaussian additive white noise.

[0104] Since only the direct successor subtask of the virtual starting subtask 0 needs to use wired transmission between the vehicle and the RSU, the data O transmitted between vehicle n and the mth RSU n,0,j Required transmission delay for:

[0105]

[0106] Computational model:

[0107] First define two moments: start time RT n,m,i and the termination time FT n,m,i . Start time RT n,m,i Refers to the earliest time when the i-th subtask generated by vehicle n can start computing on the m-th RSU after the application is split. For subtask i, it needs to wait for all its direct predecessor subtasks to complete computing and transfer data before it can start computing, so RT n,m,i It is equal to the moment when subtask i receives the output data of all direct predecessor subtasks. Termination time FT n,m,i Refers to the time when the i-th subtask after the application generated by vehicle n is split and the calculation is completed on the m-th RSU, that is, the starting time RT n,m,i Add calculation time.

[0108] The computational delay of the i-th subtask after application segmentation generated by vehicle n on the m-th RSU is:

[0109]

[0110] Among them, C n,i represents the required computing resources of the ith subtask after the application generated by vehicle n is split, f n,m,i represents the computing resources allocated by the mth RSU to the ith subtask after the application is split for vehicle n. In particular, for the virtual start subtask and end subtask, the computing demand is 0, that is, C n,0 =C n,T+1 =0.

[0111] Therefore, the starting time RT of the i-th subtask generated by vehicle n after application segmentation can be obtained: n,m,i and the termination time FT n,m,i If subtask i is the direct successor subtask of the virtual starting subtask 0, the starting time of subtask i is the sum of the communication delay of the application input data from the vehicle to the RSU and the communication delay between RSUs. Otherwise, the starting time of subtask i is the sum of the maximum end time of all its direct predecessor subtasks and the communication delay between RSUs, that is:

[0112]

[0113] The termination time FT of the i-th subtask generated by vehicle n after application segmentation n,m,i is equal to its starting time RT n,m,i And the sum of the computation delay, that is:

[0114]

[0115] According to the above formula, the total computing delay of the application generated by vehicle n can be obtained. Since the amount of data of the calculation result is very small, the delay required for returning the calculation result can be ignored. Therefore, the total computing delay of the application generated by vehicle n is the termination time of the virtual end subtask. Since the virtual end subtask has no computing requirements, it is also equivalent to the start time of the virtual end subtask, that is:

[0116] t n =RT n,m,T+1 .

[0117] Step 5: Establish the optimization problem.

[0118] The present invention mainly studies the offloading method that can minimize the total computing delay, so the optimization goal is to calculate the total computing delay of the application generated by vehicle n. Therefore, the optimization problem can be written as:

[0119]

[0120]

[0121]

[0122]

[0123] t n ≤t max

[0124] Among them, t nrepresents the computational latency of completing the application generated by vehicle n, t max Represents the maximum deadline for completing this task. Constraints 1 and 2 mean that each subtask can only select one RSU for calculation and must select one RSU; Constraint 3 means that the computing resources allocated by the MEC edge server on the mth RSU for the i-th subtask of vehicle n need to be greater than 0 and less than or equal to the remaining computing resources of the MEC edge server on the RSU; Constraint 4 means that the total computing delay required to complete the task must be less than or equal to the maximum deadline for completing the task.

[0125] It can be clearly seen that since this problem is an integer programming problem and is an NP-hard problem, it is a non-convex problem and it is difficult to solve it in a short time using general mathematical optimization methods.

[0126] Step 6: Establish the Markov decision process.

[0127] State Space:

[0128] The state space includes the amount of data transmitted by vehicles for delay-sensitive applications. and the required computing resources Variable channel gain Real-time changing vehicle speed and location The remaining computing resources of the MEC edge server in the RSU

[0129] Action Space:

[0130] From the optimization problem, we know that each subtask generated by each vehicle can only select one RSU for offloading calculation, so we define λ n,i is the index number of the RSU that is unloaded by the i-th subtask decision of vehicle n. Its relationship with the unloading decision variable is:

[0131]

[0132] Therefore, the action space consists of and resource decision variables

[0133] Reward function:

[0134] The present invention considers the unloading method that minimizes the total computing delay of the application generated by the vehicle, so the reward function is set to the negative value of the total computing delay, that is, -t n .

[0135] Step 7: Solve using the algorithm combining DDQN and FL.

[0136] The present invention adopts an algorithm combining DDQN and FL. Compared with the Q-learning algorithm, the DDQN algorithm is more suitable for situations where the state space and action space are larger, and the convergence speed is much faster than the Q-learning algorithm. Compared with the DQN algorithm, both DDQN and DQN use the mechanism of experience replay, such as Figure 4 As shown, where s t and t+1 represents the state at time t and time t+1, {a1,...,a n} represents all actions, a t represents the action taken at time t, r t Indicates taking action a at time t t The reward obtained after t ,a t ) means taking action a at time t t The Q value of the network output. t After that, the network will output the Q value obtained by taking each action, select the action according to the ε greedy strategy, and get the state s at the next moment after interacting with the environment. t+1 and the reward r t . The experience e of each step length t t ={s t ,a t ,r t ,s t+1}Stored in the data set In the training process, experience samples are randomly selected from the sample pool to update the Q network. However, during the iterative update, the DQN algorithm uses a maximization function, which will make the value obtained by the network greater than the actual action value, that is, the overestimation problem. The DDQN algorithm uses different neural networks to implement action selection and action evaluation respectively to solve the overestimation problem.

[0137] The update iterative process of the DDQN algorithm is as follows Figure 5 As shown in the figure. Both the DQN algorithm and the DDQN algorithm use two neural networks, namely the Q network and the Target Q network. The DQN algorithm always selects the maximum output value of the Target Q network. The DDQN algorithm first finds the action with the maximum output value from the Q network, and then finds the output value of the Target Q network corresponding to this action. The DDQN algorithm uses Q(s t ,a1) as the predicted value of the network, r t +γQ(s t+1,a1) as the actual value of the network, and perform error back propagation. Where γ is the loss factor. In the DDQN algorithm, there are two value functions, one for selecting actions (the strategy of the current state) and one for evaluating the value of the current state. The parameters of these two value functions are denoted as θ and θ', respectively. The error of the DDQN algorithm is expressed as:

[0138]

[0139] By introducing FL on the basis of DDQN algorithm, a model with the same performance can be trained on the basis of protecting user privacy and security. Figure 6 As shown. The algorithm based on DDQN and FL is mainly divided into two parts: server and client. The server and each local client (i.e., vehicle) have the same Q network and Target Q network. Through communication between the server and the client, a model is trained together without leaking the data of the local client. The specific steps are as follows:

[0140] Server side: First, initialize the parameters. Then, for the tth round of training, randomly select m clients, all of which are trained in parallel. After all clients have finished updating, the local model parameters are transmitted to the server. The server will integrate the local model parameters of the m randomly selected clients to obtain the latest global parameters. Finally, the server distributes the latest global parameters to all clients for the next round of updates.

[0141] Local client: First, each local client must perform a fixed number of updates based on its own data, represented by E here, calculate the gradient of these data losses, perform gradient descent updates, and obtain new local model parameters. Then, the updated local model parameters are transmitted to the central server, and the server integrates the calculated parameters to obtain the latest global model parameters. Finally, the client receives the latest global model parameters sent by the server and performs the next round of updates.

[0142] The algorithm selected by the present invention for aggregating parameters on the server side is the Federated Averaging Algorithm (FedAvg), that is, the central server initializes the model parameters and executes several rounds. Assuming that there are K clients in total, the clients selected in each round simultaneously use the model parameters w of the tth round issued by the server. t Train the model locally and get the local model parameters And upload it back to the server. The server aggregates the model parameters of each client in a weighted average manner according to the number of samples from each party to obtain the next round of model w t+1 :

[0143]

[0144] The specific pseudo code of the algorithm based on the combination of DDQN and FL is shown in Table 1.

[0145] Table 1: Algorithms based on the combination of DDQN and FL

[0146]

[0147] The definitions of all the above parameters are shown in Table 2:

[0148] Table 2 Parameter meaning

[0149]

[0150]

[0151] Step 8: Experimental results and analysis

[0152] The present invention considers a 5km one-way straight road with 10 evenly distributed RSUs on the road, and each RSU is equipped with a MEC edge server. The initial computing resource of each MEC edge server is 50GHz. The number of vehicles is 40, the distance from the roadside ranges from [25,35]m, and the vehicle speed is [30,90]km / h. All vehicle applications follow the general task call graph, where the number of nodes ranges from 9 to 13 and the number of directed edges is 13 to 17. In addition, the arrival of applications follows a Poisson process. The data size transmitted between adjacent subtasks is [6,8]Mbit, and the computational amount of the subtask is [2,5]Gcycles. Other simulation parameters are shown in Table 3.

[0153] Table 3 Simulation parameters

[0154] parameter value <![CDATA[A d ]]> 4 <![CDATA[f c ]]> 915MHz PL 3 <![CDATA[y n ]]> 0.1W B 2MHz <![CDATA[N0]]> <![CDATA[10 -13 In]]> <![CDATA[r RSU ]]> 1Gbps

[0155] The convergence curves of the reward functions of the Q-learning, DQN, and DDQN algorithms are as follows: Figure 7 As shown, the algorithm combining DDQN and FL is as follows Figure 8 As shown in the figure, it can be seen that the Q-learning algorithm cannot converge due to the large state space and action space. The convergence speed of the algorithm combined with DDQN and FL is faster than that of the DQN algorithm, but slower than that of the DDQN algorithm. This is because the system architecture of FL causes the convergence speed of the algorithm to be slightly slower, but it can ensure the privacy security of users, and the convergence speed is still within the allowable range.

[0156] Then analyze the effect of parameter m and parameter E on the convergence speed. To clearly observe the trend of the curve, take every 200 values ​​as a group and make an average. Parameter m is the number of clients that perform calculations in each round, parameter P is the proportion of clients that perform calculations in each round, and parameter E is the number of times each client updates its local parameters in each round of updates. First, fix parameter E and change parameter P. The results are as follows: Fig. 9 As shown in the figure, it can be seen that increasing the number of local clients selected can significantly speed up the convergence speed. Secondly, the parameter P is fixed and the parameter E is changed. The results are as follows Fig.10 As shown in the figure, it can be found that increasing E can also speed up the convergence. However, when the parameter E is over-optimized, that is, when the parameter E is too large, the convergence speed is not obvious. However, when the parameter E is increased, although the convergence speed can be accelerated, the number of communications between the client and the server can be reduced, thereby reducing the communication cost, but the amount of data on the client must be large enough to support it to train a model, and the client will also have a large amount of calculation. Therefore, the value of parameter E needs to balance the communication cost and the local client calculation amount, and be selected according to the actual situation.

[0157] Next, we analyze whether the algorithm based on the combination of DDQN and FL can meet the changes in the environment, that is, after a specific model is trained, if the environment changes, can it still adapt to the changes in the environment, and can it be trained again into a new model based on the old model and reach convergence. The present invention changes the environment twice based on the trained model. The changes in the environment are mainly reflected in the computing resources required for each subtask, the amount of data transmitted, and the structure of the directed acyclic graph of dependent tasks. The results are as follows Fig.11 As shown, it can be found that after each change of the environment, the model can still reach new convergence, and the convergence speed will gradually accelerate.

[0158] The algorithm of the present invention is compared with other algorithms, among which: the LOC algorithm means that all computing tasks are calculated on the vehicle without relying on the computing power of the MEC server; the greedy algorithm means that all subtasks are offloaded to the RSU closest to the vehicle, and occupy the maximum computing resources that can be occupied; the AHP algorithm is a computing task distribution and offloading algorithm based on DQN, and computing resources are allocated based on the priority of the task; the algorithm of the present invention is the FL+DDQN algorithm.

[0159] First, the RSU resource utilization of these algorithms is compared. Since the LOC algorithm does not involve the occupation of computing resources of the MEC edge server, only the other three algorithms are considered. The results are as follows: Fig.12As shown in the figure. Since the greedy algorithm selects the nearest RSU, it is easy to cause the computing resources of the middle RSU to be occupied a lot, and even cause subsequent tasks to be in the waiting stage, while the computing resources of the outer RSUs have not been used. The AHP algorithm and the FL+DDQN algorithm will not ignore the outer RSUs because they take resource allocation into consideration.

[0160] Then we compared the effect of vehicle speed on task completion rate. The results are as follows: Fig.13 As shown. For the LOC algorithm, the task success rate has been fluctuating around 30%, mainly because the computing resources of the vehicle itself are limited and can only handle a certain number of tasks. When there are many vehicles, the greedy algorithm can easily cause a waiting situation, resulting in the inability to calculate subsequent tasks and exceeding the response time. The AHP algorithm takes into account the computing resources of the vehicle itself and the edge server. The task completion rate will decrease with the change of vehicles. The main reason is that some vehicles cannot use the computing resources in the edge server because they are preempted by high-priority vehicles, resulting in the inability to complete the task within the deadline. When the number of vehicles is too large, the FL+DDQN algorithm will cause all RSUs to have no computing resources or few remaining computing resources, which will cause the task to fail. However, compared with the above algorithms, the algorithm of the present invention is less affected by the number of vehicles.

[0161] Finally, the impact of vehicle speed on task completion rate was compared. The results are as follows: Fig.14 As shown. There is no obvious change in the LOC algorithm either, because although the vehicle maintains a high speed, the vehicle speed has no effect on the completion method that relies on its own computing power. The task completion rate of the greedy algorithm suddenly decreases with the increase of vehicle speed. The main reason is that as the vehicle speed increases, the time required for actual communication between the vehicle and each MEC edge server decreases sharply. Therefore, the set of edge servers that can be actually allocated to each vehicle may decrease sharply, and the vehicle may drive out of the section, resulting in the inability to receive the result. The vehicle speed has little effect on the algorithm of the present invention and the AHP algorithm, but it will also cause a decrease in the task completion rate, mainly because the vehicle may drive out of the section, resulting in the inability to receive the result.

[0162] In addition to being ahead of the compared algorithms in performance, the task offloading method of the present invention also has advantages that other algorithms do not have, namely, the task offloading method of the present invention can effectively protect the user's data privacy and security and prevent user data leakage by retaining the user's local data locally.

Claims

1. A method for unloading delay-sensitive applications in Internet of Vehicles based on federated learning, characterized in that The method comprises the following steps: Step 1: Determine the network architecture of the Internet of Vehicles based on MEC The MEC-based Internet of Vehicles network includes three entities: RSU, MEC edge server and vehicle, among which: RSU is a collection Indicates, M represents the total number of RSUs, RSU number m∈[1,M]; all vehicles are represented by the set Indicates, N represents the total number of vehicles, and the vehicle number n∈[1,N]; all RSUs are evenly distributed along the one-way straight road, and each RSU is configured with a MEC edge server. The computing resource set of the MEC edge server is f m The computing resources of the mth RSU provide comprehensive coverage of communication and computing services for vehicles on the road. Vehicles are connected to the RSUs through wireless communications, so that the delay-sensitive applications generated by the vehicles are handed over to the MEC edge servers on the RSUs for computing; Step 2: Build an application model Step 21: Divide the delay-sensitive applications generated by each vehicle at random time into subtasks that have dependencies on each other. The input-output dependencies of the tasks generated by vehicle n are represented by a directed acyclic graph The total number of subtasks is represented by T, then The task set is used express; represents a set of vertices, Each vertex in represents a subtask i, and the related parameters C n,i represents the required computing resources of the i-th subtask after the application generated by vehicle n is divided; represents a set of edges. If the edge If it exists, it means that there is a dependency relationship between subtask i and subtask j, that is, subtask i is the predecessor subtask of subtask j, and O n,i,j Indicates the size of data transferred between subtasks; Step 22: Set the uninstall decision variable λ n,m,i ∈{0,1} represents whether the i-th subtask of vehicle n is chosen to be calculated on the MEC edge server on the m-th RSU. If it is chosen to be calculated on the MEC edge server on the m-th RSU, then λ n,m,i =1, otherwise 0; Step 23: Let the resource allocation decision variable f n,m,i represents the computing resources allocated by the MEC edge server on the mth RSU to the i-th subtask after the application generated by vehicle n is split; Step 24: Define the decision variables of all subtasks of vehicle n as λ n =λ n,1,1 ,...,λ n,1,T ,...,λ n,m,i ,...,λ n,M,1 ,...,λ n,M,T ), the resource allocation decision variable for all subtasks is f n =(f n,1,1 ,...,(f n,1,T ,...,f n,m,i ,...,f n,M,1 ,...,f n,M,T ); Step 25: define the set of all direct predecessor subtasks of subtask i as pred(i), and the set of all direct successor subtasks of subtask i as succ(i); Step 3: Determine the MEC node collaboration plan Step 31: The vehicle generates a delay-sensitive application at a random time. At this time, the vehicle receives the current computing resource information of all RSUs broadcast by the RSU where it is located. And according to Make decisions; Step 32: The vehicle uploads all input data and decision information to the RSU where the vehicle is currently located. The RSU uses wired transmission between RSUs to distribute each subtask to each RSU where the vehicle decision is offloaded based on the vehicle's decision information, and requests the MEC edge server connected to them to collaborate on computing; Step 3: When all subtasks are calculated, that is, the final calculation results are obtained, the RSU transmits the calculation results to the RSU where the vehicle is located at this time, and transmits the results back to the vehicle; Step 4: Establish a system model and obtain the optimization problem The computational delay t of the application generated by vehicle n is derived through the communication model and the computational model. n , we get the following optimization problem: t n ≤t max Among them, t n represents the computational latency of completing the application generated by vehicle n, t max represents the maximum deadline for completing this task; Constraints 1 and 2 represent that each subtask can only select one RSU for calculation and must select one RSU; Constraint 3 represents that the computing resources allocated by the MEC edge server on the mth RSU for the i-th subtask of vehicle n need to be greater than 0 and less than or equal to the remaining computing resources of the MEC edge server on the RSU; Constraint 4 represents that the total computing delay required to complete the task must be less than or equal to the maximum deadline for completing this task; Step 5: Design an algorithm that combines DDQN and FL and solve the optimization problem Step 51: Establish a Markov decision process The state space includes the amount of data transmitted by vehicles for delay-sensitive applications. and the required computing resources Variable channel gain Real-time changing vehicle speed and location The remaining computing resources of the MEC edge server in the RSU v n is the speed of vehicle n, x n is the current location of vehicle n; The action space includes and resource decision variables λ n,i The index number of the RSU that is unloaded for the i-th subtask decision of vehicle n; The reward function is the negative value of the total computation delay, i.e. -t n ; Step 52: Design an algorithm combining DDQN and FL Step 521: Establish a Q network and a Target Q network with the same structure at the client end, i.e., the vehicle end and the center end, and initialize all parameters θ of the Q network and all parameters θ' of the Target Q network, where θ'=θ; Step 522, initialize state s and experience replay pool; Step 523: Input state s into the client's Q network, obtain the Q values ​​corresponding to all actions, select action a using the ε greedy algorithm, obtain the reward r and the state s' at the next moment through the current state s and action a, store [s, a, r, s'] into the experience revisit pool, and then replace the current state s with the state s' at the next moment; Step 524: If the number of storage times at this time meets the learning interval step length, start training the neural network. The specific steps are as follows: Sample b samples from the experience replay pool to calculate the current target Q value, use the mean-variance loss function, and update the parameters of the Q network through gradient back propagation. If the number of training times meets the target network update step, the parameters of the Q network are copied to the Target Q network, that is, θ'=θ; if the number of training times meets the aggregation step, the Q network and Target network parameters of the client selected by the central server are uploaded to the central server for aggregation, and then the central server sends the aggregated parameters to all clients, and the clients update the local model parameters; Step 525: Repeat step 524 until the model reaches convergence; Step 53: Use the algorithm combining DDQN and FL to solve the optimization problem The current state s is input into the trained network, and the above optimization problem is solved by combining DDQN with FL to obtain the offloading decision that can minimize the total computing delay, including the offloading location of each subtask i and the computing resources allocated by the MEC edge server on the mth RSU to the i-th subtask of vehicle n.

2. The method for unloading delay-sensitive applications in Internet of Vehicles based on federated learning according to claim 1 is characterized in that The RSU and MEC server are connected via optical fiber, and there is a one-to-one relationship between the RSU and the MEC edge server.

3. The method for unloading delay-sensitive applications in Internet of Vehicles based on federated learning according to claim 1 is characterized in that The communication model is established as follows: The communication model includes two types: communication between RSUs and communication between vehicles and RSUs, where: RSUs use wired communication with each other. The transmission rate of wired communication is expressed by a constant r. RSU Indicates that data O is transmitted between RSU and RSU n,i,j Required transmission delay for: Wireless communication is adopted between vehicles and RSUs. It is assumed that vehicles can be allocated non-overlapping frequency bands to avoid mutual interference during data transmission, and each vehicle is allocated the same bandwidth, denoted by B. The channel gain of the uplink between all vehicles and all RSUs is represented by the vector h = (h 1,1 ,...,h 1, M,...,h n,m ,...,h N,1 ,...h N,M ) represents the channel gain h between vehicle n and the mth RSU n,m It is expressed as: Among them, A d represents the antenna gain, f c represents the carrier frequency, PL represents the path loss index, and d represents the distance between the vehicle and the RSU; According to Shannon's theorem, the transmission rate r between vehicle n and the mth RSU is obtained n,m : Among them, y n represents the transmission power of vehicle n, N0 represents the power of Gaussian additive white noise; Data O is transmitted between vehicle n and the mth RSU n,0,j Required transmission delay for:

4. The method for unloading delay-sensitive applications in Internet of Vehicles based on federated learning according to claim 1 is characterized in that The method for establishing the calculation model is as follows: Define the start time RT n,m,i and the termination time FT n,m,i , start time RT n,m,i Refers to the earliest time when the i-th subtask generated by vehicle n can start computing on the m-th RSU after the application is split, and the termination time FT n,m,i It refers to the time when the i-th subtask after the application generated by vehicle n is split and the calculation is completed on the m-th RSU; The computational delay of the i-th subtask after application segmentation generated by vehicle n on the m-th RSU is: Therefore, the starting time RT of the i-th subtask generated by vehicle n after application segmentation can be obtained: n,m,i and the termination time FT n,m,i , if subtask i is the direct successor subtask of the virtual starting subtask 0, then the starting time of subtask i is the sum of the communication delay of the application input data from the vehicle to the RSU and the communication delay between RSUs. Conversely, the starting time of subtask i is the sum of the maximum end time of all its direct predecessor subtasks and the communication delay between RSUs, that is: In the formula, Transmit data between RSU and RSU n,i,j The required transmission delay, Transmit data O between vehicle n and the mth RSU n,0,j Required transmission delay; The termination time FT of the i-th subtask generated by vehicle n after application segmentation n,m,i is equal to its starting time RT n,m,i And the sum of the computation delay, that is: The total computational delay of the application generated by vehicle n is: t n =RT n,m,T+1 。

Images