Task offloading method based on federated deep reinforcement learning in vehicle edge computing

Abstract

The present invention relates to a task offloading method based on federated deep reinforcement learning in vehicle edge computing, including: 1. Modeling the task offloading problem as an optimization problem: finding a set of optimal task offloading strategies to minimize the average response time of all vehicles in the system to reach the task; 2. Constructing an offloading action Q value prediction model; treating each vehicle as an intelligent agent, and training the offloading action Q value prediction model in a distributed manner using a DQN method in a runtime environment; each intelligent agent trains its own model through its own observable data; in the training process, the models of each intelligent agent are aggregated in combination with a federated learning framework, and finally a general offloading action Q value prediction model suitable for each intelligent agent is obtained; 3. In the runtime environment, the obtained general model is used to select the offloading action by comparing the Q values ​​of each offloading action; repeating the above process, and determining the offloading location for each subtask. The method is conducive to optimizing the average response time of the entire system.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle edge computing, and specifically relates to a task offloading method based on federated deep reinforcement learning in vehicle edge computing. Background Art

[0002] With the development of my country's Internet of Vehicles industry, smart cars are becoming more and more popular, and the Internet of Vehicles application service system is becoming increasingly rich to meet people's needs. These applications are usually computationally intensive and latency-sensitive, while the computing resources on the vehicle side are usually limited, making it difficult to complete these computationally intensive tasks under latency constraints. Mobile edge computing deploys computing resources at the edge closer to the user to provide users with high-performance and low-latency services. Based on mobile edge computing, computational offloading can make full use of the computing resources in the system when processing tasks, and further optimize the response time of computationally intensive and latency-sensitive tasks.

[0003] The existing technology widely uses heuristic methods to search for offloading solutions, which usually takes tens of seconds or even hundreds of seconds to solve the approximate optimal result, which does not meet the delay constraints of Internet of Vehicles applications. In reality, the computing resources owned by each vehicle and edge server are usually different, and the computing resources required for the tasks of each vehicle are also different. The mobility of the vehicle causes the position between the vehicle and the edge server to change as the vehicle moves, resulting in the transmission rate and transmission delay between the vehicle and the edge server to change dynamically. Traditional heuristic methods are usually unable to adapt to dynamically changing environments. When the environment changes dynamically, the offloading strategy needs to be re-solved. In order to meet the delay constraints, an offline method is used to solve a better offloading strategy in a runtime environment with very little solution time. Deep Reinforcement Learning (DRL) is an offline method that trains the Q value prediction model of the offloading action online and then uses it for offloading decisions in the runtime environment. It has been applied to task offloading in mobile edge computing.

[0004] Existing research is usually limited to the task unloading of a single vehicle. In order to conform to the actual situation, the scenario of multiple vehicles unloading at the same time is considered. Existing methods usually make unloading decisions and schedule the tasks of all vehicles centrally at the edge. The disadvantage is that the information of all vehicles needs to be collected when making decisions, which not only has a large amount of data but is also time-consuming. Considering the decentralized nature of vehicles in the system, a distributed method is used for task unloading. Specifically, each vehicle is regarded as an intelligent agent, and each vehicle decides its own unloading strategy. In addition, a federated learning framework is used when training deep reinforcement learning models. By leveraging the learning progress of other vehicles to reduce training time, a universal model applicable to all vehicles can be obtained. Summary of the invention

[0005] The purpose of the present invention is to provide a task offloading method based on federated deep reinforcement learning in vehicle edge computing, which is conducive to optimizing the average response time of the entire system.

[0006] To achieve the above object, the technical solution adopted by the present invention is: a task offloading method based on federated deep reinforcement learning in vehicle edge computing, comprising the following steps:

[0007] Step 1: Model the task offloading problem as an optimization problem: find a set of optimal task offloading strategies to minimize the average response time of all vehicles in the system to reach the task;

[0008] Step 2: Construct a Q-value prediction model for unloading actions; treat each vehicle as an intelligent agent, and use the DQN method to train the Q-value prediction model for unloading actions in a distributed manner in the runtime environment; the training data includes the vehicle's task arrival rate λ, the computing node's service rate μ, the unit task transmission delay D between computing nodes, the task unloading scheme F, the computing node's load rate ρ, and the system's average response time T; each intelligent agent trains its own model with its own observable data, and the model is used to predict the Q-value of different unloading actions in different runtime environments; during the training process, the models of each intelligent agent are aggregated in combination with the federated learning framework, and finally a universal unloading action Q-value prediction model applicable to each intelligent agent is obtained;

[0009] Step 3: In the runtime environment, use the general unloading action Q value prediction model obtained in step 2 to predict the Q values ​​of different unloading actions based on the vehicle's task arrival rate λ, the computing node's service rate μ, the unit task volume transmission delay D between computing nodes, the task unloading scheme F and the computing node's load rate ρ. Select the unloading action by comparing the Q values ​​of each unloading action; repeat the above process to gradually determine the unloading location for each subtask.

[0010] Furthermore, the problem model of the task offloading problem is constructed as follows:

[0011] Assume that N roadside units are deployed on the roads in a certain area, and each roadside unit is equipped with an edge server with computing power to process tasks; the set of roadside units is denoted as E = {e1, e2, ..., e N},e j represents the jth roadside unit; there are M vehicles in this area, denoted as V = {v1, v2, ..., v M},v irepresents the i-th vehicle; each vehicle has computing resources to process its own tasks; vehicles and roadside units with computing resources are collectively referred to as computing nodes; each vehicle accesses the nearest K roadside units through wireless networks and multi-access edge computing technology for communication and data transmission, and uses the computing resources of edge servers for data processing;

[0012] When a vehicle generates a task, it is assumed that the task is independent of each other and is arbitrarily divided into multiple subtasks during operation. The overhead of task division is negligible. Each subtask is offloaded to any one of the K roadside units for execution, or executed locally.

[0013] Assume that the vehicle's task arrival rate per unit time follows a Poisson distribution; use the vector λ = [λ1, λ2, ..., λ M ] represents the task arrival rate of M vehicles, that is, the number of tasks generated by vehicles per unit time; using vector represents the service rate of the computing node, that is, the amount of tasks processed by the computing node per unit time, μ vi Indicates vehicle v i The service rate, μ ej Represents roadside unit e j The service rate of M vehicles and N roadside units is expressed by the matrix D:

[0014]

[0015] Among them, d i，0 =0 means vehicle v i The transmission delay of the unit task to itself, d i，j Indicates vehicle v i and roadside unit e j The transmission delay of unit task between them;

[0016] The unloading strategy of M vehicles per unit time is expressed by matrix F:

[0017]

[0018] Among them, f i，0 Indicates vehicle v i The amount of tasks executed locally, f i，j Indicates vehicle v i Unloading to roadside unit j The amount of tasks;

[0019] Using Vectors Represents the amount of tasks required to calculate node load per unit time, where Indicates vehicle v i The amount of workload, Represents roadside unit e j The amount of tasks loaded; Assume that the vehicle can only unload tasks to the K roadside units it can access or execute locally, but cannot unload tasks to other vehicles for execution; Therefore, vehicle v i The amount of tasks loaded is only related to the amount of tasks executed locally, that is, Roadside Unit j The load task is related to the unloading strategy of M vehicles, that is,

[0020] Assuming that each computing node has only one server for computing tasks, the M / M / 1 queuing model is used to model each computing node separately; for each computing node, the unit task amount stay time, that is, the sum of the waiting time of the unit task amount in the queue and the execution time required for the unit task amount is calculated as:

[0021]

[0022] Because each computing node processes tasks in parallel, the vehicle v i The arrival task response time is the latest execution completion time of all computing nodes:

[0023] T i =max(f i，0 ·t i，0 , f i，1 ·t i，1 , ..., f i，N ·t i，N )

[0024] Among them, t i，j Indicates that the vehicle v i The arriving tasks are offloaded to computing node j, The average response time per unit task is composed of the residence time and transmission time per unit task:

[0025] t i，j =T a (w j , μ j )+d i，j

[0026] When j = 0, it means that the vehicle processes the corresponding subtask locally, and the vehicle's local load and service rate are used for calculation, that is, Otherwise, it means that the vehicle unloads the corresponding subtask to the roadside unit e j Execute, using roadside unit e j The load task volume and service rate are calculated, that is,

[0027] For the entire system, the average response time for processing all vehicle arrival tasks is:

[0028]

[0029] The optimization goal of the proposed real-time task offloading method is to find a set of offloading strategies F to minimize the response time of all vehicle arrival tasks in the entire system:

[0030]

[0031] stC1:

[0032] C2:

[0033] C3:

[0034] C4:

[0035] C5:

[0036] C6:

[0037] Among them, constraint C1 means that each task must be completely divided and unloaded to each available computing node; C2 means that the task load of each roadside unit cannot exceed its maximum load capacity, ρ max Indicates the maximum load rate that the roadside unit can bear; C3 indicates that the task arrival rate of each vehicle is less than its service rate; C4 indicates that the service rate of each roadside unit is greater than the service rate of all vehicles; C5 indicates that the unit task transmission delay and transmission task volume between the vehicle and the roadside unit must be non-negative; C6 indicates that the task volume that the vehicle needs to handle is not greater than its task arrival rate.

[0038] Furthermore, the offloading action Q value prediction model is constructed as follows:

[0039] Each vehicle is considered as an intelligent agent, which interacts with the environment to learn and make decisions; let the vehicle v i The K accessible roadside units are set E i ={e i，1 , e i，2 , ..., e i，K}, whose indexes in the roadside unit set E are j1, j2, ..., j K ,Right now Remember vehicle v i The service rate of the K roadside units that it can access is Vehicle iThe transmission delay of the unit task volume of the K accessible roadside units is t time step vehicle v i The unloading plan for the arrival task is t time step v i The load rate of the K accessible roadside units is

[0040] Based on the constructed problem model, define the Markov decision process quadruple of each agent's state, action, state transfer function and reward function:

[0041] State space: vehicle v i The state at time step t is defined as the vector The observable runtime environment information μ i , d i and performance indicators composition;

[0042] A fingerprint is introduced into the state of each vehicle to perform task unloading without knowing the unloading strategies of other vehicles; the fingerprint is a low-dimensional vector containing information about other agents, consisting of [e, ∈], where e is the number of rounds of current training and ∈ is the exploration rate in ∈-greedy; vehicle v i The state at time step t is defined as the vector

[0043] Action Space: The action space is defined as a i = [0, 1, 2, ..., K], vehicle v i The action at time step t is recorded as The task volume is divided into two parts according to a fixed ratio δ i Divided into several subtasks, each action determines the execution location of each subtask; action Indicates vehicle v i Execute the corresponding subtask locally; action Indicates vehicle v i Unload the corresponding subtask to the roadside unit Execute on;

[0044] State transfer function: In state Next action Can transfer to next state The runtime environment parameter μ i and d i remains unchanged; fingerprint [e,∈] is transferred to [e′,∈′] during the training process; and Limit Action Transfer to and The details are as follows:

[0045]

[0046]

[0047] Reward function: In order to guide the agent's unloading decision to minimize the average response time of the system, the reward function is defined as a negative number related to the average response time of the system:

[0048]

[0049] Among them, T t represents the average response delay accumulated by the system at time step t, T t+1 Indicates that all agents have completed their actions After that, the average response delay accumulated by the system at the t+1 time step.

[0050] Furthermore, the training method of the unloading action Q value prediction model is:

[0051] First, initialize the exploration rate, experience replay pool, federated model parameters, online networks of each agent, and target network parameters;

[0052] For each training cycle, initialize the current system average response time, offloading strategy, and the workload of each computing node;

[0053] During the training process, the network parameters are updated through federated averaging every C iterations, and the target network parameters of each agent are updated;

[0054] The arrival task of each vehicle is divided into several subtasks using a fixed ratio; for each subtask, each vehicle first observes the current state, and then selects the unloading action of the current subtask according to the ∈-greedy method;

[0055] Then, all vehicles execute actions uniformly, and the environment updates information based on the actions of each agent, including the current system average response time, unloading strategy, and the amount of tasks loaded on each computing node;

[0056] Each car receives a global reward function and moves to the next state;

[0057] Afterwards, each vehicle stores the current unloaded experience in the experience replay pool, and randomly extracts B experiences from the experience replay pool. After calculating the target Q value and loss function, the RMSprop optimizer is used to update the online network parameters.

[0058] In order to allow the agent to explore better experiences without letting too many poor experiences affect the training process, the exploration rate is updated through linear descent;

[0059] After a set number of iterations, the final network parameters are obtained using federated averaging.

[0060] Furthermore, in the runtime environment, the trained offloading action Q value prediction model is used to make offloading action decisions, including:

[0061] First, the trained model is distributed to each vehicle, and the current system average response time, offloading strategy, and the workload of each computing node are initialized;

[0062] Next, each vehicle observes the current state, and uses the unloading action Q value prediction model to evaluate the Q value of each unloading action in the current state, selects the unloading action with the largest Q value, and updates the current system average response time, unloading strategy, and the task volume of each computing node load;

[0063] The offloading location of each subtask is determined in turn to obtain the final average system response time.

[0064] Compared with the prior art, the present invention has the following beneficial effects: it provides a task offloading method based on federated deep reinforcement learning in vehicle edge computing. The method first models the task offloading problem as an optimization problem, and its optimization goal is to enable each vehicle to minimize the average response time of the system by selecting the optimal task offloading strategy; then each vehicle is regarded as an intelligent agent, and the task model and offloading decision of other vehicles are unknown. DQN is used to train the Q value prediction model of each offloading action in a distributed manner, and a federated learning framework is used to combine the training information of all vehicles to obtain a general model applicable to all vehicles; finally, the unloading location is determined for each subtask through the obtained general model. This method can obtain a system average response time close to the ideal solution within an acceptable algorithm execution time, so that the average response time of the entire system is approximately optimal. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is an implementation framework diagram of an embodiment of the present invention;

[0066] Figure 2 is a structural block diagram of an online network and a target network in an embodiment of the present invention;

[0067] Figure 3 is the system average response time (Average Response Time) and algorithm execution time (Execution Time) of the method and other methods in the embodiment of the present invention in the scenario where K=2, N=3;

[0068] Figure 4 is the system average response time (Average Response Time) and algorithm execution time (Execution Time) of the method and other methods in the embodiment of the present invention in the scenario where K=2, N=15;

[0069] Figure 5 is the system average response time (Average Response Time) and algorithm execution time (Execution Time) of the method and other methods in the embodiment of the present invention in the scenario of K={0,1,2},N=3;

[0070] Figure 6 It is the system average response time (Average Response Time) and algorithm execution time (Execution Time) of the method in the embodiment of the present invention and other methods in the scenario of K={0,1,2},N=15. DETAILED DESCRIPTION

[0071] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0072] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0073] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0074] This embodiment provides a task offloading method based on federal deep reinforcement learning (TOFDRL) in vehicle edge computing, including the following steps:

[0075] Step 1: Model the task offloading problem as an optimization problem: find a set of optimal task offloading strategies to minimize the average response time of all vehicles in the system to reach the task.

[0076] Step 2: Construct a Q-value prediction model for unloading actions; regard each vehicle as an intelligent agent, and use the DQN method to train the Q-value prediction model for unloading actions in a distributed manner in the runtime environment; the training data includes the vehicle's task arrival rate λ, the service rate μ of the computing node, the unit task transmission delay D between the computing nodes, the task unloading scheme F, the load rate ρ of the computing node, and the average response time T of the system; each intelligent agent trains its own model through its own observable data, and the model is used to predict the Q-value of different unloading actions in different runtime environments; during the training process, the models of each intelligent agent are aggregated in combination with the federated learning framework, and finally a universal unloading action Q-value prediction model suitable for each intelligent agent is obtained.

[0077] Step 3: In the runtime environment, use the general unloading action Q value prediction model obtained in step 2 to predict the Q values ​​of different unloading actions based on the vehicle's task arrival rate λ, the computing node's service rate μ, the unit task volume transmission delay D between computing nodes, the task unloading scheme F and the computing node's load rate ρ. Select the unloading action by comparing the Q values ​​of each unloading action; repeat the above process to gradually determine the unloading location for each subtask.

[0078] 1 Problem Model

[0079] 1.1 System Model

[0080] Assume that N roadside units (RSUs) are deployed on the roads in a certain area. Each RSU is equipped with an edge server with computing power to process tasks. The set of RSUs is denoted as E = {e1, e2, ..., e N},e j represents the jth roadside unit; there are M vehicles in this area, denoted as V = {v1, v2, ..., v M},v i represents the i-th vehicle; each vehicle has computing resources to process its own tasks; vehicles and roadside units with computing resources are collectively referred to as computing nodes; each vehicle accesses the K nearest roadside units through wireless networks and multi-access edge computing technology for communication and data transmission, and uses the computing resources of edge servers for data processing.

[0081] 1.2 Task Model

[0082] The vehicle will generate a task. Assuming that the task is independent of each other, it can be arbitrarily divided into multiple subtasks at runtime, and the overhead of task division is negligible; each subtask is offloaded to any one of the K roadside units for execution, or executed locally.

[0083] Assume that the vehicle's task arrival rate per unit time follows a Poisson distribution; use the vector λ = [λ1, λ2, ..., λ M ] represents the task arrival rate of M vehicles, that is, the number of tasks generated by vehicles per unit time; using vector It represents the service rate of the computing node, that is, the amount of tasks processed by the computing node per unit time. Indicates vehicle v i The service rate, Represents roadside unit e j The service rate of M vehicles and N roadside units is expressed by the matrix D:

[0084]

[0085] Among them, d i，0 =0 means vehicle v i The transmission delay of the unit task to itself, d i，j Indicates vehicle v i and roadside unit e j The transmission delay of unit task volume between them.

[0086] The unloading strategy of M vehicles per unit time is expressed by matrix F:

[0087]

[0088] Among them, f i，0 Indicates vehicle v i The amount of tasks executed locally, f i，j Indicates vehicle v i Unloading to roadside unit j the amount of tasks.

[0089] Using Vectors Represents the amount of tasks required to calculate node load per unit time, where Indicates vehicle v i The amount of workload, Represents roadside unit e j The amount of tasks loaded; Assume that the vehicle can only unload tasks to the K roadside units it can access or execute locally, but cannot unload tasks to other vehicles for execution; Therefore, vehicle v i The amount of tasks loaded is only related to the amount of tasks executed locally, that is, Roadside Unit j The load task is related to the unloading strategy of M vehicles, that is,

[0090] 1.3 Computational Model

[0091] Assuming that each computing node has only one server for computing tasks, the M / M / 1 queuing model is used to model each computing node separately; for each computing node, the unit task amount stay time, that is, the sum of the waiting time of the unit task amount in the queue and the execution time required for the unit task amount is calculated as:

[0092]

[0093] Because each computing node processes tasks in parallel, the vehicle v i The arrival task response time is the latest execution completion time of all computing nodes:

[0094] T i =max(f i，0 ·t i，0 , f i，1 ·t i，1 , ..., f i，N ·t i，N )

[0095] Among them, t i，j Indicates that the vehicle v i The arriving tasks are offloaded to computing node j, The average response time per unit task is composed of the residence time and transmission time per unit task:

[0096] t i，j =T a (w j , μ j )+d i，j

[0097] When j = 0, it means that the vehicle processes the corresponding subtask locally, and the vehicle's local load and service rate are used for calculation, that is, Otherwise, it means that the vehicle unloads the corresponding subtask to the roadside unit e j Execute, using roadside unit e j The load task volume and service rate are calculated, that is,

[0098] For the entire system, the average response time for processing all vehicle arrival tasks is:

[0099]

[0100] 1.4 Problem Definition

[0101] The optimization goal of the proposed real-time task offloading method is to find a set of offloading strategies F to minimize the response time of all vehicle arrival tasks in the entire system:

[0102]

[0103] stC1:

[0104] C2:

[0105] C3:

[0106] C4:

[0107] C5:

[0108] C6:

[0109] Among them, constraint C1 means that each task must be completely divided and unloaded to each available computing node; C2 means that the task load of each roadside unit cannot exceed its maximum load capacity, ρ max Indicates the maximum load rate that the roadside unit can bear; C3 indicates that the task arrival rate of each vehicle is less than its service rate; C4 indicates that the service rate of each roadside unit is greater than the service rate of all vehicles; C5 indicates that the unit task transmission delay and transmission task volume between the vehicle and the roadside unit must be non-negative; C6 indicates that the task volume that the vehicle needs to handle is not greater than its task arrival rate.

[0110] 2. Unloading Action Q Value Prediction Model

[0111] 2.1 Construction of the Q-value prediction model for offloading actions

[0112] Each vehicle is considered as an intelligent agent, which interacts with the environment to learn and make decisions; let the vehicle v i The K accessible roadside units are set E i ={e i，1 , e i，2 , ..., e i，K}, whose indexes in the roadside unit set E are j1, j2, ..., j K ,Right now Remember vehicle v i The service rate of the K roadside units that it can access is Vehicle i The transmission delay of the unit task volume of the K accessible roadside units is t time step vehicle v i The unloading plan for the arrival task is t time step v i The load rate of the K accessible roadside units is

[0113] Based on the constructed problem model, define the Markov decision process quadruple of each agent's state, action, state transfer function and reward function:

[0114] State space: vehicle v i The state at time step t is defined as the vector The observable runtime environment information μ i , d i and performance indicators In addition, in a multi-vehicle environment, the unloading decision of each vehicle will be affected by other vehicles. However, when the current vehicle makes an unloading decision, it does not know the unloading strategies of other vehicles. In order to perform task unloading without knowing the unloading strategies of other vehicles, a fingerprint is introduced in the state of each vehicle. The fingerprint is a low-dimensional vector containing information about other agents, consisting of [e, ∈], where e is the number of rounds of current training and ∈ is the exploration rate in ε-greedy. In summary, vehicle v i The state at time step t is defined as the vector

[0115] Action Space: The action space is defined as a i = [0, 1, 2, ..., K], vehicle v i The action at time step t is recorded as The task volume is divided into two parts according to a fixed ratio δ i Divided into several subtasks, each action determines the execution location of each subtask; action Indicates vehicle v i Execute the corresponding subtask locally; action Indicates vehicle v i Unload the corresponding subtask to the roadside unit Execute on.

[0116] State transfer function: In state Next action Can transfer to next state The runtime environment parameter μ i and d i remains unchanged; fingerprint [e,∈] is transferred to [e′,∈′] during the training process; and Limit Action Transfer to and The details are as follows:

[0117]

[0118]

[0119] Reward function: In order to guide the agent's unloading decision to minimize the average response time of the system, the reward function is defined as a negative number related to the average response time of the system:

[0120]

[0121] Among them, T t represents the average response delay accumulated by the system at time step t, T t+1 Indicates that all agents have completed their actions After that, the average response delay accumulated by the system at the t+1 time step.

[0122] Figure 1 This is the implementation framework diagram of the method of the present invention. Each vehicle uses the classic deep reinforcement learning algorithm DQN for training and decision-making. Each DQN agent observes the vehicle edge computing environment to obtain the state, and then selects the unloading action through the ε-greedy method. After the unloading action is executed, the environment changes, and each agent will obtain the reward value of the action evaluated by the environment and observe the new state. In order to avoid the overestimation of the Q value of the next state, each DQN agent has a target network with the same architecture as the online network in addition to an online network for online training and updating, which is used to obtain the Q value of the next state. The structures of the online network and the target network are shown in the figure. Figure 2 shown.

[0123] 2.2 Training of the Q-value prediction model for offloading actions

[0124] The training process of the offloading action Q-value prediction model is shown in Algorithm 1. First, the exploration rate, experience replay pool, federated model parameters, online network of each agent, and target network parameters are initialized (line 1). For each training cycle, the current system average response time, offloading strategy, and the task volume of each computing node load are initialized (line 3). During the training process, the network parameters need to be updated by federated average for each C round of iterations, and the target network parameters of each agent are updated (line 4). The arrival task of each vehicle is divided into several subtasks using a fixed ratio. For each subtask, each vehicle first observes the current state, and then selects the offloading action of the current subtask according to the ∈-greedy method (lines 6-8). Then, all vehicles perform actions uniformly, and the environment updates information according to the actions of each agent, including the current system average response time, offloading strategy, and the task volume of each computing node load (line 9). Each vehicle receives a global reward function and moves to the next state (line 11). After that, each vehicle stores the currently unloaded experience in the experience replay pool, and randomly extracts B experiences from the experience replay pool. After calculating the target Q value and loss function, the online network parameters are updated using the RMSprop optimizer (lines 12-15). In order to allow the agent to explore better experiences but not allow too many poor experiences to affect the training process, the exploration rate needs to be updated through linear descent (line 18). After a certain number of iterations, the final network parameters are obtained using federated averaging (lines 21-22).

[0125]

[0126]

[0127] 2.3 Runtime decision of uninstall action

[0128] In the runtime environment, the trained offloading action Q value prediction model is used to make offloading action decisions. The main steps are shown in Algorithm 2. First, the trained model is distributed to each vehicle, and the current system average response time, offloading strategy, and the task load of each computing node are initialized (lines 1-2). Next, each vehicle observes the current state (e and ∈ in the state are set to the values ​​of the last training step), and the offloading action Q value prediction model is used to evaluate the Q value of each offloading action executed in the current state. The offloading action with the largest Q value is selected, and the current system average response time, offloading strategy, and the task load of each computing node are updated (lines 4-7). The offloading location of each subtask is determined in turn, and the final system average response time is obtained (line 9).

[0129]

[0130]

[0131] 3 Experimental simulation and results

[0132] 3.1 Experimental Setup

[0133] In order to simulate the diversity of scenarios, consider the number of roadside units in the system N = {3, 15}, the number of vehicles M = {5, 15, 25}, and the number of roadside units that each vehicle can access K = {0, 1, 2}. The task arrival rate λ of the vehicles in the system follows the normal distribution of N(10, 4), the fixed ratio of the divided tasks δ = 4%; the service rate μ of the vehicle v Obeys the normal distribution of N(12,6); the service rate of the roadside unit μ e It follows the normal distribution of N(15, 6); the transmission delay d of the unit task between the vehicle and the roadside unit follows the uniform distribution of U(0.1, 0.2). In order to prevent the vehicle from being unable to handle its own tasks, the vehicle's task arrival rate must be less than the vehicle's service rate. In order to prevent the roadside unit from receiving too many tasks unloaded by vehicles and being unable to handle these tasks, the upper limit of the load rate of the roadside unit is set to ρ max = 85%. When the load rate of the roadside unit exceeds this upper limit, the vehicle's unloading request will be rejected, and the remaining task volume will be executed locally on the vehicle. The specific simulation parameter settings are shown in Table 1.

[0134] Table 1 Simulation parameter settings of the system model

[0135]

[0136] This method is implemented based on Python 3.8 and TensorFlow 2.8.0, using a neural network consisting of an input layer, two hidden layers consisting of fully connected layers, and an output layer. Each fully connected layer in the hidden layer has 128 hidden neurons, and the activation function uses ReLU. The number of training rounds e and the synchronization frequency C of the federated average network parameters and the target network parameters are set to 3000 and 100 respectively; the experience replay pool capacity c, the training batch size B, the learning rate α, and the discount rate γ are set to 65535, 32, 0.001, and 0.99 respectively; the initial value of the exploration rate ∈ is 1, and the final value is 0.1. It is linearly decayed in the first 600 rounds and remains unchanged after decaying to the final value.

[0137] 3.2 Experimental Results and Analysis

[0138] In different scenarios, this method (TOFDRL) is compared with the following methods in terms of system response time and algorithm execution time:

[0139] 1) PSO-GA (Particle Swarm Optimization-Genetic Algorithm): A heuristic method with fast convergence and high precision. The problem solved by this invention is difficult to traverse the entire solution space to find the optimal solution using brute force enumeration. Therefore, PSO-GA is considered an ideal solution.

[0140] 2) Rule-based method: Use fixed percentage rules to divide and offload tasks.

[0141] 3) Multi-agent deep reinforcement learning (MADRL): Instead of using the federated learning framework for training, each agent performs model training and unloading decisions based only on the local information it observes.

[0142] Figures 3 to 6 The average response time (Average Response Time) and algorithm execution time (Execution Time) of this method and other classic methods are compared in various scenarios with different numbers of vehicles, different numbers of roadside units, and fixed or random numbers of accessible roadside units. Figure 3 The performance of this method is compared with other classic methods in the scenario of K = 2, N = 3. The average system response time obtained by this method is 36.94% and 30.03% better than that of MADRL and Rule-based methods, respectively, and only 10.63% different from the ideal solution. Figure 4 The performance of this method is compared with other classic methods in the scenario of K = 2, N = 15. The average system response time obtained by this method is 32.38% and 41.23% better than that of MADRL and Rule-based methods, respectively, and only 19.48% different from the ideal solution. Figure 5 The performance of this method is compared with other classic methods in the scenario of K = {0, 1, 2}, N = 3. The average system response time obtained by this method is 22.78% and 17.01% better than that of MADRL and Rule-based methods, respectively, and only 7.69% different from the ideal solution. Figure 6 The performance of this method is compared with other classic methods in the scenario of K = {0, 1, 2}, N = 15. The average system response time obtained by this method is 20.28% and 18.62% better than that of MADRL and Rule-based methods, respectively, and only 5.75% different from the ideal solution. The above results of the average system response time verify that this method can achieve near-optimal performance in different scenarios.

[0143] Comparing the algorithm execution time, the ideal solution solves the offloading solution, and the system average response time is shorter than that of this method, but it requires a longer execution time, about the order of hundreds of seconds, which does not meet the delay constraints of delay-sensitive tasks in the Internet of Vehicles. The Rule-based method basically does not involve mathematical operations, and only requires an algorithm execution time of one hundredth to one thousandth of a second, but it is too general, resulting in the system average response time of the offloading solution obtained by it being higher than that of this method. The MADRL method differs from this method only in the training phase, and the offloading decision process in the runtime environment is basically the same as this method. Therefore, the execution time of the two methods is in the same order of magnitude, both in the millisecond level, which meets the delay constraints of delay-sensitive tasks in the Internet of Vehicles, but the system average response time obtained by this method is better than that of the MADRL method. The above results verify the superiority of this method in solving the system average response time close to the ideal solution within an acceptable algorithm execution time.

[0144] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A task offloading method based on federated deep reinforcement learning in vehicle edge computing, characterized in that: The following steps are involved: Step 1: Model the task offloading problem as an optimization problem: find a set of optimal task offloading strategies to minimize the average response time of all vehicles in the system to reach the task; Step 2: Construct a Q-value prediction model for unloading actions; treat each vehicle as an intelligent agent, and use the DQN method to train the Q-value prediction model for unloading actions in a distributed manner in the runtime environment; the training data includes the vehicle's task arrival rate λ, the computing node's service rate μ, the unit task transmission delay D between computing nodes, the task unloading scheme F, the computing node's load rate ρ, and the system's average response time T; each intelligent agent trains its own model with its own observable data, and the model is used to predict the Q-value of different unloading actions in different runtime environments; during the training process, the models of each intelligent agent are aggregated in combination with the federated learning framework, and finally a universal unloading action Q-value prediction model applicable to each intelligent agent is obtained; Step 3: In the runtime environment, the general unloading action Q value prediction model obtained in step 2 is used to predict the Q values ​​of different unloading actions according to the vehicle's task arrival rate λ, the computing node's service rate μ, the unit task transmission delay D between computing nodes, the task unloading scheme F and the computing node's load rate ρ. The unloading action is selected by comparing the Q values ​​of each unloading action. The above process is repeated to gradually determine the unloading location for each subtask. The problem model of the task offloading problem is constructed as follows: Assume that N roadside units are deployed on the roads in a certain area, and each roadside unit is equipped with an edge server with computing power to process tasks; the set of roadside units is denoted as E = {e1, e2, ..., e N },e j represents the jth roadside unit; there are M vehicles in this area, denoted as V = {v1, v2, ..., v M },v i represents the i-th vehicle; each vehicle has computing resources to process its own tasks; vehicles and roadside units with computing resources are collectively referred to as computing nodes; each vehicle accesses the nearest K roadside units through wireless networks and multi-access edge computing technology for communication and data transmission, and uses the computing resources of edge servers for data processing; When a vehicle generates a task, it is assumed that the task is independent of each other and is arbitrarily divided into multiple subtasks during operation. The overhead of task division is negligible. Each subtask is offloaded to any one of the K roadside units for execution, or executed locally. Assume that the vehicle's task arrival rate per unit time follows a Poisson distribution; use the vector λ = [λ1, λ2, ..., λ M ] represents the task arrival rate of M vehicles, that is, the number of tasks generated by vehicles per unit time; using vector It represents the service rate of the computing node, that is, the amount of tasks processed by the computing node per unit time. Indicates vehicle v i The service rate, Represents roadside unit e j The service rate of M vehicles and N roadside units is expressed by the matrix D: Among them, d i，0 =0 means vehicle v i The transmission delay of the unit task to itself, d i，j Indicates vehicle v i and roadside unit e j The transmission delay of unit task between them; The unloading strategy of M vehicles per unit time is expressed by matrix F: Among them, f i，0 Indicates vehicle v i The amount of tasks executed locally, f i，j Indicates vehicle v i Unloading to roadside unit j The amount of tasks; Using Vectors Represents the amount of tasks required to calculate node load per unit time, where Indicates vehicle v i The amount of workload, Represents roadside unit e j The amount of tasks loaded; Assume that the vehicle can only unload tasks to the K roadside units it can access or execute locally, but cannot unload tasks to other vehicles for execution; Therefore, vehicle v i The amount of tasks loaded is only related to the amount of tasks executed locally, that is, Roadside Unit j The load task is related to the unloading strategy of M vehicles, that is, Assuming that each computing node has only one server for computing tasks, the M / M / 1 queuing model is used to model each computing node separately; for each computing node, the unit task amount stay time, that is, the sum of the waiting time of the unit task amount in the queue and the execution time required for the unit task amount is calculated as: Because each computing node processes tasks in parallel, the vehicle v i The arrival task response time is the latest execution completion time of all computing nodes: T i =max(f i，0 ·t i，0 ,f i，1 ·t i，1 ,...,f i，N ·t i，N ) Among them, t i，j Indicates that the vehicle v i The arrival tasks are offloaded to the computing nodes The average response time per unit task is composed of the residence time and transmission time per unit task: t i，j =T a (w j ,μ j )+d i，j When j = 0, it means that the vehicle processes the corresponding subtask locally, and the vehicle's local load and service rate are used for calculation, that is, Otherwise, it means that the vehicle unloads the corresponding subtask to the roadside unit e j Execution, using roadside unit e j The load task volume and service rate are calculated, that is, For the entire system, the average response time for processing all vehicle arrival tasks is: The optimization goal of the real-time task offloading method is to find a set of offloading strategies F to minimize the response time of all vehicle arrival tasks in the entire system: Among them, constraint C1 means that each task must be completely divided and unloaded to each available computing node; C2 means that the task load of each roadside unit cannot exceed its maximum load capacity, ρ max Indicates the maximum load rate that the roadside unit can bear; C3 indicates that the task arrival rate of each vehicle is less than its service rate; C4 indicates that the service rate of each roadside unit is greater than the service rate of all vehicles; C5 indicates that the unit task transmission delay and transmission task volume between the vehicle and the roadside unit must be non-negative; C6 indicates that the task volume that the vehicle needs to handle is not greater than its task arrival rate; The Q value prediction model of the offloading action is constructed as follows: Each vehicle is considered as an intelligent agent, which interacts with the environment to learn and make decisions; let the vehicle v i The K accessible roadside units are set E i ={e i，1 , e i，2 , ..., e i，K }, whose indexes in the roadside unit set E are j1, j2, ..., j K ,Right now Remember vehicle v i The service rate of the K roadside units that it can access is Vehicle i The transmission delay of the unit task volume of the K accessible roadside units is t time step vehicle v i The unloading plan for the arrival task is t time step v i The load rate of the K accessible roadside units is Based on the constructed problem model, define the Markov decision process quadruple of each agent's state, action, state transfer function and reward function: State space: vehicle v i The state at time step t is defined as the vector The observable runtime environment information μ i , d i and performance indicators composition; A fingerprint is introduced into the state of each vehicle to perform task unloading without knowing the unloading strategies of other vehicles; the fingerprint is a low-dimensional vector containing information about other agents, consisting of [e, ∈], where e is the number of rounds of current training and ∈ is the exploration rate in ∈-greedy; vehicle v i The state at time step t is defined as the vector Action Space: The action space is defined as a i = [0, 1, 2, ..., K], vehicle v i The action at time step t is recorded as The task volume is divided into two parts according to a fixed ratio δ i Divided into several subtasks, each action determines the execution location of each subtask; action Indicates that vehicle vi executes the corresponding subtask locally; action Indicates vehicle v i Unload the corresponding subtask to the roadside unit Execute on; State transfer function: In state Next action Can transfer to next state The runtime environment parameter μ i and d i remains unchanged; fingerprint [e,∈] is transferred to [e′,∈′] during the training process; and According to the action Transfer to and The details are as follows: Reward function: In order to guide the agent's unloading decision to minimize the average response time of the system, the reward function is defined as a negative number related to the average response time of the system: Among them, T t represents the average response delay accumulated by the system at time step t, T t+1 Indicates that all agents have completed their actions After that, the average response delay accumulated by the system at the t+1 time step.

2. The task offloading method based on federated deep reinforcement learning in vehicle edge computing according to claim 1 is characterized in that: The training method of the unloading action Q value prediction model is: First, initialize the exploration rate, experience replay pool, federated model parameters, online networks of each agent, and target network parameters; For each training cycle, initialize the current system average response time, offloading strategy, and the workload of each computing node; During the training process, the network parameters are updated through federated averaging every C iterations, and the target network parameters of each agent are updated; The arrival task of each vehicle is divided into several subtasks using a fixed ratio; for each subtask, each vehicle first observes the current state, and then selects the unloading action of the current subtask according to the ∈-greedy method; Then, all vehicles execute actions uniformly, and the environment updates information based on the actions of each agent, including the current system average response time, unloading strategy, and the amount of tasks loaded on each computing node; Each car receives a global reward function and moves to the next state; Afterwards, each vehicle stores the current unloaded experience in the experience replay pool, and randomly extracts B experiences from the experience replay pool. After calculating the target Q value and loss function, the RMSprop optimizer is used to update the online network parameters. In order to allow the agent to explore better experiences without letting too many poor experiences affect the training process, the exploration rate is updated through linear descent; After iterating for a set number of rounds, the final network parameters are obtained using federated averaging.

3. The task offloading method based on federated deep reinforcement learning in vehicle edge computing according to claim 2 is characterized in that: In the runtime environment, the trained offloading action Q value prediction model is used to make offloading action decisions, including: First, the trained model is distributed to each vehicle, and the current system average response time, offloading strategy, and the workload of each computing node are initialized; Next, each vehicle observes the current state, and uses the unloading action Q value prediction model to evaluate the Q value of each unloading action in the current state, selects the unloading action with the largest Q value, and updates the current system average response time, unloading strategy, and the task volume of each computing node load; The offloading location of each subtask is determined in turn to obtain the final average system response time.

Images