Task offloading and cache update method based on federated reinforcement learning in internet of vehicles

Abstract

The application discloses a task offloading and cache updating method based on federal reinforcement learning in Internet of Vehicles, and steps are as follows: an Internet of Vehicles scene of intelligent vehicle and infrastructure communication is constructed, including a base station and a central cloud server with computing and cache capabilities; an optimization model is established with the sum of vehicle task computing delay and energy consumption benefit minimization as an optimization target; a vehicle obtains an input state by sensing the Internet of Vehicles environment, and takes offloading strategy and resource request as output actions; a base station obtains an input state by vehicle historical request information, and takes cache updating strategy as an output action; task offloading, resource allocation and cache updating strategies are obtained through online output actions of the vehicle and the base station; and a network is trained through offline gradient descent and federal aggregation. The application introduces federal learning into the offline learning link of deep reinforcement learning, realizes distributed task offloading and cache updating, and has better scalability and feasibility in a high-dynamic Internet of Vehicles environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wireless communication and Internet of Vehicles, and particularly relates to a task offloading and cache updating method based on federated reinforcement learning in Internet of Vehicles. BACKGROUND

[0002] As one of the application scenarios of 5G, Internet of Vehicles supports many vehicle-mounted application services, and these services often need to meet low latency, high reliability and high stability. Mobile edge computing realizes lower latency and lower energy consumption by sinking computing resources and storage resources to the network edge, and has become a technology that attracts much attention.

[0003] Task offloading is one of the important application scenarios of mobile edge computing, which provides computing services to mobile applications to improve the service quality of Internet of Vehicles application groups. Efficient operation of Internet of Vehicles relies on efficient offloading decisions and reasonable resource allocation in Internet of Vehicles systems.

[0004] As one of the key technologies of information centralization, cache can effectively reduce the transmission of repeated content in the network. By caching content (such as video, computing framework, etc.) at network edge nodes (such as base stations, roadside units, etc.), traffic load, access delay and network cost can be greatly reduced.

[0005] Deep reinforcement learning has been widely used in the field of edge computing. In most existing researches, task offloading in Internet of Vehicles environment is usually a traditional single-agent centralized algorithm (Single DQN, etc.), which has the disadvantages of dimension explosion and poor scalability, or a semi-distributed multi-agent reinforcement learning (such as MADDPG, etc.) with distributed decision-making and centralized evaluation, which has the risk of privacy leakage and is difficult to provide good service quality in the high dynamic Internet of Vehicles environment.

[0006] The prior art of a multi-intelligent device task offloading decision method based on deep reinforcement learning (CN202210362289.6) does not consider the timeliness of edge computing tasks. In most existing scenarios, offloading tasks have a certain delay tolerance. The existing invention uses a centralized training and distributed decision-making algorithm, which requires a large amount of communication resources for centralized training in high-density user scenarios, has the risk of privacy leakage, and does not consider the cacheability of tasks. SUMMARY

[0007] In order to overcome the shortcomings of centralized single-agent reinforcement learning and semi-distributed multi-agent reinforcement learning algorithm in the prior art, that is, easy leakage of user data and poor scalability, the present application proposes a task offloading and cache updating method based on federated reinforcement learning in Internet of Vehicles, introduces federated learning into the offline learning link of deep reinforcement learning, adopts a fully distributed algorithm, realizes distributed task offloading and cache updating, avoids leakage of user data, protects the privacy of users, has better scalability and feasibility in a high-dynamic Internet of Vehicles environment, realizes user-side distributed task offloading, resource allocation, power control and edge-side cache updating, and solves the problem of minimizing delay and energy consumption in a distributed manner.

[0008] The object of the present application is achieved at least by one of the following technical solutions.

[0009] The task offloading and cache updating method based on federated reinforcement learning in Internet of Vehicles comprises the following steps:

[0010] S1, constructing an Internet of Vehicles scene of intelligent vehicle and infrastructure communication, including an edge server and a center cloud server with computing and caching capabilities;

[0011] S2, establishing an optimization model with the optimization objective of minimizing the sum of vehicle task computing delay and energy consumption benefit;

[0012] S3, the intelligent vehicle end taking the task demand of the intelligent vehicle and the state of the edge server as the input state of the federated deep reinforcement learning network, and taking the connection behavior, transmission power and requested computing resource of the intelligent vehicle and the edge end as the output action;

[0013] S4, the base station taking the historical cache request of the current edge server as the input state of the federated deep reinforcement learning network, and taking the cache replacement strategy as the output action;

[0014] S5, the vehicle agent outputs a task offloading and resource allocation scheme according to the state, and the edge agent outputs a cache update according to the input state, so as to maximize the quality of service of Internet of Vehicles;

[0015] S6, training the network through offline gradient descent and federated aggregation, updating the target network weight, aggregating the target network of each agent in the multi-agent environment, and distributing it to each agent.

[0016] Further, in step S1, a dynamic Internet of Vehicles scene needs to be established:

[0017] The Internet of Vehicles scene is a highway and a road section with continuously moving vehicles, wherein the number of vehicles obeys a Poisson distribution based on vehicle speed, and there are two base stations in the scene, each of which is equipped with an edge server and a cache server;

[0018] where the set of vehicles is U = {1,...,u,...,U}, the set of base stations is K = {1,...,k,...K}, the set of task types is N = {1,...,n,...,N}, U is the total number of vehicles, K is the total number of base stations; the computational task of vehicle u is defined as Task u = (D u , C u , T max , n) in each time slot, where D u is the data volume of the task, C u is the required computing power, T max is the delay tolerance of the current task, and n is the current task number, and the popularity of the vehicle task is modeled as a Zipf distribution.

[0019] Further, a communication model between vehicles and base stations needs to be established, in which the wireless channel model is based on 3GPP TR36.885, as follows:

[0020] The channel gain of vehicle u and the base station within its range considers large-scale and small-scale fading, and is expressed as:

[0021]

[0022] A is the path loss parameter; μ u is the fading coefficient of vehicle u, which is subject to an independent complex exponential distribution with a mean of 1; β u is the shadow fading of vehicle u, which is subject to a lognormal distribution with a standard deviation; L u，k represents the distance between vehicle u and base station k, and γ is the attenuation exponent;

[0023] The wireless communication between vehicles and base stations is based on orthogonal frequency division multiple access (OFDMA), so the bandwidth of vehicle u and base station k is divided by all vehicles connected to the current base station that need to upload, the uplink signal-to-noise ratio is proportional to the vehicle transmission power and the channel gain, and the uplink rate of vehicle u and base station k is R u，k , according to the Shannon formula:

[0024]

[0025] In the above formula, p u is the uplink transmission power of vehicle u, B is the total bandwidth of the base station, h u，k is the channel gain of vehicle u and base station k, σ 2 is the Gaussian white noise power, a u，k is a binary vehicle offloading indicator variable, a u，k = 1 means that vehicle u and base station k establish a connection behavior, in particular, a ｕ，0 is a vehicle local computing indicator variable, a u，0= 1, i.e., the computing task of vehicle u in the current time slot is local computing; y k，n is a base station cache indicator variable with 0-1 binary, when y u，n = 1, i.e., the base station k corresponds to the cache server that caches the task type n, otherwise y k，n = 0. Where the uplink signal-to-noise ratio of vehicle u and base station k needs to meet:

[0026] Where the uplink signal-to-noise ratio of vehicle u and base station k needs to meet:

[0027]

[0028] In the above formula, γ th is the noise threshold, when the uplink signal-to-noise ratio is lower than the noise threshold, the task offloading fails;

[0029] The uplink delay of vehicle u and base station k is:

[0030]

[0031] In the above formula, D u is the task data volume, R u，k is the uplink rate of vehicle u and base station k;

[0032] The uplink energy consumption of vehicle u and base station k is:

[0033]

[0034] In the above formula, p u is the transmit power of vehicle u, is the uplink delay of vehicle u and base station k.

[0035] Further, a computing model of vehicle and base station needs to be established, according to the space-time distribution of vehicle and the connection behavior of vehicle and base station, vehicle u needs to request computing resource to edge server k, wherein f u，k is the computing resource applied by vehicle u to base station k, needs to meet the following constraints:

[0036]

[0037] f u，k ∈ (0, F k )

[0038] In the above formula, F k is the total computing power of edge server equipped by base station k;

[0039] When vehicle u selects local processing of computing task, the computing delay is as shown in the following formula:

[0040]

[0041] In the above formula, C u is the required computing power of the current task, f u is the local computing resource of vehicle u; ε is the vehicle chip energy consumption coefficient, which is determined by the vehicle-mounted chip technology of the vehicle;

[0042] The energy consumption of vehicle u local computing is shown in the following formula:

[0043]

[0044] In the above formula, C u is the required computing power of the current task, f u is the local computing resource of vehicle u; ε is the vehicle chip energy consumption coefficient, which is determined by the vehicle-mounted chip technology of the vehicle;

[0045] When the base station k does not cache the data of the current task, the computing delay of vehicle u offloading to base station k is shown in the following formula:

[0046]

[0047] The computing energy consumption of vehicle u offloading to base station k is shown in the following formula:

[0048]

[0049] Assuming that the cache server corresponding to base station k has cached the data of task n, it can perform task computing without uplink, which can effectively save uplink resources;

[0050] The cache server of base station k needs to satisfy the following constraints:

[0051]

[0052] In the above formula, G k is the total cache size of the cache server corresponding to base station k, y k，n is the cache variable, if base station k caches the data of task n, y k，n = 1, otherwise y k，n = 0, D n is the data volume corresponding to task n;

[0053] Considering the limited cache space of the base station and the time-varying characteristics of content popularity, the cache server corresponding to base station k updates its cache list according to the current vehicle task request characteristics; x n，k represents the cache update strategy of the current cache server, x n，k = 1, which means that base station k adds content n to the cache server, otherwise it does not cache, x n，k needs to satisfy the following constraints:

[0054]

[0055] The above formula represents replacing the cache content at most once every T time slot.

[0056] Further, in step S2, an optimization model is established with the optimization objective of minimizing the sum of the computation time delay and the energy consumption benefit of the vehicle task, and the specific process is as follows:

[0057] The total time delay of the computation task requested by the vehicle u at time slot t is as follows:

[0058]

[0059] In the above formula, is the computation time delay of the vehicle when local computation is selected, a u，0 is a local computation indication variable, is the local computation time delay; is the total time delay of the task when the edge is unloaded and the cache is hit, composed of the computation time delay y k，n , a u，k is a corresponding indication variable; is the total time delay of the task when the edge is unloaded but the cache is not hit, composed of the computation time delay and the transmission time delay ;

[0060] The energy consumption of the computation task requested by the vehicle u is as follows:

[0061]

[0062] In the above formula, is the computation energy consumption of the vehicle when local computation is selected, a u，0 is a local computation indication variable, is the local computation energy consumption; is the total energy consumption of the task when the edge is unloaded and the cache is hit, composed of the computation energy consumption y k，n , a u，k is a corresponding indication variable; is the total energy consumption of the task when the edge is unloaded but the cache is not hit, composed of the computation energy consumption and the transmission time delay.

[0063] Further, the optimization problem is represented as minimizing the weighted time delay energy consumption sum of the Internet of Vehicles edge computing system:

[0064]

[0065]

[0066] ​

[0067]

[0068]

[0069]

[0070]

[0071] C7: sinr>γ th

[0072]

[0073]

[0074] wherein the optimization formula: x n，k is a cache replacement variable; U is a vehicle set, is a base station set, T max is the delay tolerance of the task, β1 and β2 are weighting coefficients, τ u and E u are the delay and energy consumption of the vehicle u for this task offloading; in the constraint condition, sinr is the uplink signal-to-noise ratio, γ th is the noise threshold, G k is the upper limit of the cache capacity of the base station k corresponding cache server;

[0075] The first constraint condition C1 indicates that the vehicle is binary offloading when offloading the task, without task segmentation;

[0076] The second constraint condition C2 indicates that the vehicle must make an offloading decision at the current time slot;

[0077] The third constraint condition C3 indicates that the computing resource allocation needs to be less than the maximum computing power of the edge server corresponding to the base station

[0078] The fourth constraint condition C4 indicates that the sum of the computing resources allocated by the edge server corresponding to the base station cannot exceed the total computing resources;

[0079] The fifth constraint condition C5 indicates that the vehicle uplink transmission power cannot exceed the maximum transmission power P max ;

[0080] The sixth constraint condition C6 indicates that the execution time of the vehicle task needs to be less than the delay tolerance;

[0081] The seventh constraint condition C7 indicates that the vehicle uplink signal-to-noise ratio needs to be greater than the noise threshold;

[0082] The eighth constraint condition C8 indicates that the cache content of the cache server corresponding to the base station cannot exceed the total cache resources;

[0083] The ninth constraint condition C9 indicates that only one cache content is replaced each time.

[0084] Further, in step S3, the input state of the vehicle agent based on the federated deep reinforcement learning network is constructed, and the state of each vehicle agent is represented as follows:

[0085] S u (t) = [Task u (t), H u，1 (t),..., H u，K (t), y 1，n (t),..., y K，n (t)]

[0086] wherein Task u (t) is represented as a task model, including the current task data volume, the required computing capability, the time delay tolerance, and the task number, H u，1 (t),..., H u，K (t) is represented as the channel gain of the vehicle u perceiving and establishing a wireless connection with the base station set K, y 1，n (t),..., y K，n (t) is represented as whether the base station set corresponding to the cache server caches the computing task of the vehicle u (wherein n is obtained in the vehicle task model);

[0087] The action of each vehicle agent is represented as follows:

[0088]

[0089] wherein a u，0 (t),..., a u，K (t) is represented as the connection behavior of the vehicle u with the base station k at the current time slot, in particular, a u，0 (t) is represented as whether the vehicle u decides to locally compute, p u (t) is represented as the uplink transmission power of the vehicle u, is the computing resource applied for by the vehicle u to the base station;

[0090] The reward function of each vehicle agent is designed as follows:

[0091]

[0092] wherein β1 and β2 are weighting coefficients; τ u (t), E u (t) are respectively the time delay and the energy consumption of the vehicle u this time for task offloading; T max is the time delay tolerance of the task; sinr(t) is the uplink signal-to-noise ratio, γ th is the uplink noise tolerance; G(·) is represented as follows:

[0093]

[0094] When the parameter p1 is less than the threshold value, the function G(·) outputs p1, and when higher than the threshold value, the function G(·) outputs a positive value μ1, -(β1τ u +β2E u ) represents the quality of service of the current offloading decision, the lower the weighted delay and energy consumption, the higher the reward, G(·) is a decision function, if the offloading decision violates the constraint condition, a corresponding penalty is given, and if the condition is met, a reward is given; G(sinr-γ th , μ1) is a decision whether the uplink signal-to-noise ratio is higher than the signal-to-noise threshold; G(T max -τ u , μ2) judges whether the task execution delay meets the delay tolerance, is a decision whether the computing resources requested by the vehicle to the base station are overloaded;

[0095] The cooperation between vehicles is completed by combining the federated learning distributed learning framework with the offline learning link of the agent, and a plurality of vehicles are periodically selected to obtain a global model according to the following formula, and the Q-target Net parameter set of the agent corresponding to the vehicle u is The aggregation process is shown in the following formula:

[0096]

[0097] In the above formula, let be the model parameter of the target network of the agent corresponding to the vehicle u, W u be the experience replay buffer capacity of the agent corresponding to the vehicle u, be the sum of the experience replay buffer capacities of all participant agents.

[0098] Further, in step S4, the input state of the edge node agent based on the federated deep reinforcement learning network is constructed, and the state of each edge node agent is represented as follows:

[0099] S k (t)=[ψ s (t),ψ m (t),ψ l (t)]

[0100] Wherein ψ s (t), ψ m (t), ψ l (t) are the total number of requests for each content in a period of short-term, medium-term and long-term;

[0101] The action of each edge node agent is represented as follows:

[0102] αk (t) = [x k，0 (t),..., x k，N (t)]

[0103] x k，0 (t),..., x k，N (t) respectively represent whether the cache server corresponding to base station k replaces task type n;

[0104] The reward function of each edge node agent is designed as follows:

[0105]

[0106] The reward of the above formula serves as an important indicator for guiding the gradient descent of the decision network, characterizes the sum of the weighted cache hit rates of the base stations k in the T up time slots after executing the cache replacement strategy, where H k (t+τ) is the cache hit rate of the cache server corresponding to base station k in the t+τ time slot; The numerator is the sum of the cached task data amounts in the entire simulation scenario, the denominator is the global task data amount, μ4 is an adjustment parameter, and the division represents the global content coverage rate; the function l(·) represents 1 if the content in the parentheses is true, and 0 otherwise; the higher the global content coverage rate, the greater the reward;

[0107] The federated learning distributed learning framework is combined with the offline learning link of the agent to complete the cooperation between the edge nodes, and the Q-target Net parameter set of the agent corresponding to base station k is set as The aggregation process is shown in the following formula:

[0108]

[0109] In the above formula, Θ G is the global model after aggregation, W k is the size of the experience replay cache pool of each participating aggregation base station; is the sum of the experience replay cache pool capacities of the base stations participating in aggregation.

[0110] Further, in step S6, the network of each agent is trained based on offline learning and federated learning, and the offline learning steps are as follows:

[0111] Initialize the experience replay pool of each agent, the target network weight and the policy network weight; initialize the vehicle networking scene environment to obtain the input state of each agent; the policy network reasons according to the input state, and selects an action according to the action selection policy; the action is applied to the environment to obtain a reward, and is transferred to the next state; the current state transition pair, including the current state, the executed action, the reward and the next state, is stored in the experience replay pool; samples are extracted from the experience pool in small batches to train the policy network, and the target network weight is updated.

[0112] Further, in step S6, each agent network is trained based on offline learning and federated learning, and the federated learning steps are as follows:

[0113] Each agent updates its own network through online learning and offline learning; the vehicle side clusters based on its own offloading decision; each agent uploads its own Target Net model parameters to the aggregation center according to the clustering result, the aggregation center obtains a global model through weighted averaging, and the global model is distributed to each agent, and each agent updates its own network model parameters through the global model.

[0114] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0115] (1) The task offloading and cache updating method based on federated reinforcement learning in the vehicle networking disclosed in the present application obtains a suboptimal solution to the task offloading problem and the cache updating problem under the condition of low time complexity, aiming at the optimization problem of maximizing the service quality of task offloading in the vehicle networking environment.

[0116] (2) The task offloading and cache updating method based on federated reinforcement learning in the vehicle networking disclosed in the present application realizes distributed task offloading and cache updating, avoids user data leakage compared with centralized and semi-distributed algorithms, protects the privacy of users, and has good scalability; the present application optimizes the cache replacement strategy according to the dynamic popularity of vehicles, which can greatly reduce traffic load, access delay and network cost.

[0117] (3) The task offloading and cache updating method based on federated reinforcement learning in the vehicle networking disclosed in the present application has higher service quality compared with other algorithms. BRIEF DESCRIPTION OF DRAWINGS

[0118] Figure 1 It is a vehicle networking highway scene schematic diagram disclosed in the present application;

[0119] Figure 2 It is a flow chart of the task offloading and cache updating method based on federated reinforcement learning in the vehicle networking disclosed in the embodiment of the present application;

[0120] Figure 3Figure for comparison of cache hit rate of different algorithms in Example 1 when cache server capacity is different;

[0121] Figure 4 Figure for comparison of offloading success rate of different algorithms in Example 2 when the number of vehicles is different;

[0122] Figure 5 Figure for comparison of latency of different algorithms in Example 2 when the number of vehicles is different;

[0123] Figure 6 Figure for comparison of energy consumption of different algorithms in Example 2 when the number of vehicles is different;

[0124] Figure 7 Figure for comparison of offloading success rate of different algorithms in Example 3 when the average task request content is different;

[0125] Figure 8 Figure for comparison of latency of different algorithms in Example 3 when the average task request content is different;

[0126] Figure 9 Figure for comparison of energy consumption of different algorithms in Example 3 when the average task request content is different. DETAILED DESCRIPTION

[0127] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0128] Example 1

[0129] As shown in Figure 2 , it is a task offloading and cache updating method based on federated reinforcement learning in a vehicle Internet, which maximizes the service quality of a vehicle Internet user group based on federated reinforcement learning, and the specific steps include:

[0130] The vehicle Internet scene of the present embodiment is a six-lane highway with a length of two kilometers as shown in Figure 1 , and there are vehicles continuously driving on the road section, wherein the number of vehicles obeys a Poisson distribution based on vehicle speed, and there are two base stations in the scene, wherein each base station is equipped with an edge server and a cache server.

[0131] Wherein the vehicle set is U = {1,..., u,..., U}, the base station set is K = {1,..., k,..., K}, the task type set is N = {1,..., n,..., N}, U is the total number of vehicles, and K is the total number of base stations; in each time slot, the computing task of vehicle u is defined as Task u = (D u , C u , Tmax , n), where D u is the task data volume, C u is the required computing power, T max is the time delay tolerance of the current task, and n is the current task number. The popularity of vehicle tasks is modeled as a Zipf distribution.

[0132] The channel gain of vehicle u communicating with the base station in its range considers large-scale and small-scale fading, and is expressed as:

[0133]

[0134] A is the path loss parameter; μ u is the small-scale fading coefficient of vehicle u, which is subject to an independent complex exponential distribution with a mean of 1; β u is the shadow fading of vehicle u, which is subject to a lognormal distribution with a standard deviation; L u，k denotes the distance between vehicle u and base station k, and γ is the attenuation exponent.

[0135] Wireless communication between vehicles and base stations is based on orthogonal frequency division multiple access (OFDMA), so the bandwidth of vehicle u and base station k is divided by all vehicles connected to the current base station that need to upload. The uplink signal-to-noise ratio is proportional to the vehicle transmission power and the channel gain. The uplink rate of vehicle u and base station k is R u，k , according to the Shannon formula:

[0136]

[0137] In the above formula, p u is the uplink transmission power of vehicle u, B is the total bandwidth of the base station, h u，k is the channel gain of vehicle u and base station k, σ 2 is the Gaussian white noise power, a u，k is a 0-1 binary vehicle offloading indicator variable, a u，k = 1 if vehicle u offloads tasks to base station k, and a u，k = 0 otherwise. In particular, a u，0 is a vehicle local computing indicator variable, a u，0 = 1 if the computing task of vehicle u in the current time slot is local computing, and vice versa; when the vehicle chooses to offload to the base station, y k，n is a 0-1 binary base station cache indicator variable, y k，n = 1 if the cache server corresponding to base station k caches the task type n, and y k，n = 0 otherwise.

[0138] where the uplink signal-to-noise ratio of vehicle u and base station k needs to satisfy:

[0139]

[0140] The uplink time delay of vehicle u and base station k is:

[0141]

[0142] The uplink energy consumption of vehicle u and base station k is:

[0143]

[0144] S1.3, a computing model of vehicle and base station is established, according to the space-time distribution of vehicle and the connection behavior of vehicle and base station, vehicle u needs to request computing resource from base station k, wherein f u，k The computing resource applied by vehicle u to base station k needs to meet the following constraints:

[0145]

[0146] f u，k ∈(0,F k )

[0147] In the above formula, F k is the total computing power of the edge server equipped by base station k.

[0148] When vehicle u selects local processing of the computing task, the computing time delay is as follows:

[0149]

[0150] In the above formula, C u is the required computing power of the current task, and f u is the local computing resource of vehicle u.

[0151] The energy consumption of local computing of vehicle u is as follows:

[0152]

[0153] In the above formula, C u is the required computing power of the current task, and f u is the local computing resource of vehicle u. ε is the energy consumption coefficient of vehicle chip, which is determined by the vehicle chip technology equipped by vehicle.

[0154] When base station k does not cache the data of the current task, the computing time delay of vehicle u offloading to base station k is as follows:

[0155]

[0156] The computing energy consumption of vehicle u offloading to base station k is as follows:

[0157]

[0158] Assuming that the base station k corresponds to the cache server has cached the data of task n, no uplink is needed to perform task calculation, which can effectively save uplink resources.

[0159] Where the cache server of base station k needs to satisfy the following constraints:

[0160]

[0161] In the above formula, G k is the total cache size of the cache server corresponding to base station k, y k，n is a cache variable, if base station k caches the data of task n, y k，n =1, otherwise y k，n =0, D n is the data volume corresponding to task n;

[0162] Considering the limited cache space of the base station and the time-varying characteristics of content popularity, the cache server corresponding to base station k updates its cache list according to the current vehicle task request characteristics; x n，k represents the cache update strategy of the current cache server, x n，k =1, which means that base station k adds content n to the cache server, otherwise it is not cached, x n，k needs to satisfy the following constraints:

[0163]

[0164] The above formula means replacing one cache content at most every T time slot.

[0165] Further, in step S2, an optimal model is recommended with the optimization objective of minimizing the sum of vehicle task calculation delay and energy consumption benefit, as follows:

[0166] At time slot t, the calculation task delay requested by vehicle u is as follows:

[0167]

[0168] At time slot t, the calculation task energy consumption requested by vehicle u is as follows:

[0169]

[0170] The optimization problem is represented as minimizing the weighted delay energy consumption sum of the Internet of Vehicles edge computing system:

[0171]

[0172]

[0173]

[0174]

[0175]

[0176]

[0177]

[0178] C7: sinr>γ th

[0179]

[0180]

[0181] wherein:

[0182] Constraint C1 means that the vehicle is binary offloading when task offloading, without task splitting;

[0183] Constraint C2 means that the vehicle must make offloading decisions in the current time slot;

[0184] Constraint C3 means that the computation resource allocation needs to be less than the maximum computation power of the edge server corresponding to the base station

[0185] Constraint C4 means that the sum of the computation resources allocated by the edge server corresponding to the base station cannot exceed the total computation resource;

[0186] Constraint C5 means that the uplink transmission power of the vehicle cannot exceed the maximum transmission power P max ;

[0187] Constraint C6 means that the execution time of the vehicle task needs to be less than the delay tolerance;

[0188] Constraint C7 means that the uplink signal-to-noise ratio of the vehicle needs to be less than the noise threshold;

[0189] Constraint C8 means that the cache content of the cache server corresponding to the base station cannot exceed the total cache resource;

[0190] Constraint C9 means that only one cache content is replaced each time.

[0191] Further in step S3, the input state of the vehicle intelligent agent based on federated reinforcement learning is constructed, and the state of each vehicle intelligent agent is represented as follows:

[0192] S u (t) = [Task u (t), H u，1 (t),..., H u，K (t), y 1，n (t),..., yK，n (t)]

[0193] where Task u (t) denotes the task model, including the current task data volume, required computing capability, latency tolerance, and task number, H u，1 (t),..., H u，K (t) denotes the channel gain of vehicle u perceiving the wireless connection with base station set K, y 1，n (t),..., y K，n (t) denotes whether the computing task of vehicle u is cached in the corresponding cache server of base station set (where n is obtained in the vehicle task model);

[0194] The action of each vehicle agent is denoted as follows:

[0195]

[0196] where a u，0 (t),..., a u，K (t) denotes the connection behavior of vehicle u with base station k in the current time slot, in particular, a u，0 (t) denotes whether vehicle u decides to perform local computing, p u (t) denotes the uplink transmission power of vehicle u, is the computing resource applied for by vehicle u to the base station;

[0197] The reward function of each vehicle agent is designed as follows:

[0198]

[0199] where β1 and β2 are weighting coefficients and the sum is 1; τ u , E u are the latency and energy consumption of vehicle u for this task offloading; T max is the latency tolerance of the task; sinr is the uplink signal-to-noise ratio, γ th is the uplink noise tolerance; G(·) is denoted as follows:

[0200]

[0201] The general meaning of the G(·) function is that when the parameter p1 is less than the threshold value, the function G(·) outputs p1, and when it is higher than the threshold value, the function G(·) outputs a positive value μ1, -(β1τ u (t) + β2E u (t)) represents the quality of service of the current offloading decision, and the lower the weighted latency and energy consumption, the higher the reward, G(·) is a decision function, and if the offloading decision violates the constraint condition, a corresponding penalty is given, and if the condition is met, a reward is given; G(sinr(t)-γth , μ1) is to judge whether the uplink signal-to-noise ratio is higher than the signal-to-noise threshold; G(T max -τ u (t), μ2) judges whether the task execution delay meets the delay tolerance, is to judge whether the computing resources requested by the vehicle to the base station are overloaded;

[0202] Since the reward maximization in the above formula requires the cooperation of all vehicle agents, in order to realize the cooperation between agents, the federated learning distributed learning framework is used in combination with the offline learning link of the agent to complete the cooperation between vehicles. A number of vehicles are periodically selected to obtain the global model according to the following formula, and the Q-target Net parameter set of the agent corresponding to the vehicle u is The aggregation process is shown in the following formula:

[0203]

[0204] In the above formula, let be the model parameters of the agent target network corresponding to the vehicle u, W u be the experience replay buffer capacity of the agent corresponding to the vehicle u, be the sum of the experience replay buffer capacities of all participant agents.

[0205] Further, in step S4, the input state of the edge node agent based on federated reinforcement learning is constructed, and the state of each edge node agent is represented as follows:

[0206] S k (t) = [ψ s (t), ψ m (t), ψ l (t)]

[0207] Where ψ s (t), ψ m (t), ψ l (t) are the total number of requests for each content in a period of short, medium and long term;

[0208] The action of each edge node agent is represented as follows:

[0209] α k (t) = [x k，0 (t),..., x k，N (t)]

[0210] In the above formula, x k，0 (t),..., x k，N (t) respectively represent whether the cache server corresponding to the base station k replaces the task type n;

[0211] The reward function of each edge node agent is designed as follows:

[0212]

[0213] The reward of the above formula serves as an important indicator for guiding the gradient descent of the decision network, The weighted cache hit rate of the band attenuation coefficient χ in the time slot after representing the execution of the cache replacement strategy, up The sum of the weighted cache hit rate of the band attenuation coefficient χ in the time slot, k (t+τ) is the cache hit rate of the base station k corresponding to the cache server t+τ time slot; The numerator is the sum of the cached task data amount in the entire simulation scenario, the denominator is the global task data amount, μ4 is an adjustment parameter, and the division represents the global content coverage rate. The l(·) function represents 1 if the bracket is true, and 0 otherwise. The higher the global content coverage rate, the greater the reward;

[0214] Since reward maximization requires the cooperation of all edge agents, to achieve cooperation between agents, a federated learning distributed learning framework is used in combination with the offline learning of the agents to complete the cooperation between edge nodes. Let the Q-target Net parameter set of the agent corresponding to the edge node k be The aggregation process is shown in the following formula:

[0215]

[0216] In the above formula, Θ G is the global model after aggregation, which is then distributed to the network model of each agent. In the above formula, Θ G is the global model after aggregation, and W u is the size of the experience replay cache pool of each participating aggregation base station; is the sum of the cache pool capacities of the set of participating aggregation base stations. The aggregation process aims to break the local optimum through cooperation between vehicles. After aggregation, it is then distributed to the network model of each agent.

[0217] Further, in step S6, each agent network is trained based on offline learning and federated learning, and the steps are as follows:

[0218] Initialize the experience replay pool, target network weights, and policy network weights; initialize the Internet of Vehicles scenario environment to obtain the input state; the policy network makes inferences according to the input state and selects actions according to the action selection policy; the action is applied to the environment to obtain the reward and is transferred to the next state; the current state transition pair, including the current state, the executed action, the reward, and the next state, is stored in the experience replay pool; samples are extracted from the experience pool to train the policy network and update the target network weights; every T period, the target networks of each agent in the multi-agent environment are aggregated, and then distributed to each agent.

[0219] Table 1 is a simulation parameter table of the present example

[0220] Table 1. Simulation parameter setting table

[0221]

[0222] Figure 3 Fig. 1 is a cache hit rate comparison chart of different algorithms in the present embodiment when the cache server capacity is different, wherein the number of vehicles is set to 20, the scheme LRU is that when the cache space is full, the new content replaces the content in the original cache that has not been requested for the longest time until the cache space is not full. The scheme LFU is that when the cache space is full, the new content replaces the content in the original cache that has the lowest request frequency until the cache space is not full. The scheme FIFO is that when the cache space is full, the new content replaces the content in the original cache that is cached first until the cache space is not full. When the cache capacity increases, more cache content can be cached at the edge, so the cache hit rate increases as a whole. At the same time, when the cache capacity is limited, the method can effectively adapt to the high dynamics of cache content in the Internet of Vehicles, and obtain a higher cache hit rate in the same cache space.

[0223] Example 2

[0224] In the present embodiment, the mean value of the average requested content of the fixed vehicle is 400 kb, and the number of vehicles is between 20 and 55.

[0225] Figure 4 Fig. 2 is a successful offloading rate comparison chart of different algorithms in the present embodiment when the number of vehicles is different. The offloading method of the comparison algorithm MADQN uses a multi-agent reinforcement learning algorithm to make a decision, and the cache update uses a dynamic programming algorithm. The offloading method of the comparison algorithm ACCO uses a greedy and heuristic algorithm to make a decision, and the cache update uses a dynamic programming algorithm. Compared with the comparison algorithm, the present method has a great advantage in the index of successful offloading rate. At the same time, when the number of vehicles increases, the resource contention is fierce, and the traditional multi-agent reinforcement learning algorithm has a rapid decline in the successful offloading rate due to the collision of resource contention in the distributed environment, while the present method can still maintain a good successful offloading rate.

[0226] Figure 5 Fig. 3 is a time delay comparison chart of different algorithms in the present embodiment when the number of vehicles is different. Compared with the comparison algorithm, the present method has a great advantage in the index of time delay. When the resources are sufficient, all algorithms can basically maintain a good service quality. When the number of vehicles increases, although the overall offloading time delay of all algorithms increases, the present method can effectively reduce the average time delay of the vehicle-mounted application in a distributed manner.

[0227] Figure 6For the energy consumption comparison chart of different algorithms in the embodiment when the number of vehicles is different, compared with the comparison algorithm, the method has great advantages in the energy consumption index, and the method can reduce the average energy consumption of the vehicle-mounted application by selecting the effective transmission power and cooperation between vehicles.

[0228] Embodiment 3

[0229] In the embodiment, the number of fixed vehicles is 25, and the average request content size is between 400-1000 kb.

[0230] Figure 7 For the offloading success rate comparison chart of different algorithms in the embodiment when the average request content size is different, the comparison algorithm MADQN offloading method uses a distributed multi-agent reinforcement learning algorithm to make a decision, and the cache update uses a dynamic programming algorithm, the comparison algorithm ACCO offloading method is based on a greedy and heuristic algorithm to make a decision, and the cache update uses a dynamic programming algorithm, compared with the comparison algorithm, the method has great advantages in the offloading success rate index, when the average request content size increases, the resource contention is fierce, the traditional multi-agent reinforcement learning in the distributed environment is collided due to the resource contention, and the offloading success rate decreases rapidly, and the method can maintain a good offloading success rate.

[0231] Figure 8 For the time delay comparison chart of different algorithms in the embodiment when the average request content size is different, compared with the comparison algorithm, the method has great advantages in the time delay index, when the resources are sufficient, all algorithms can basically maintain a good service quality, when the average request content size increases, although the overall offloading time delay of all algorithms increases, the method can complete the cooperation between agents in a distributed manner and effectively reduce the average time delay of the vehicle-mounted application offloading.

[0232] Figure 9 For the energy consumption comparison chart of different algorithms in the embodiment when the average request content size is different, compared with the comparison algorithm, the method has great advantages in the energy consumption index, and the method can reduce the average energy consumption of the vehicle-mounted application by selecting the effective transmission power and cooperation between vehicles.

[0233] The above embodiment is a preferred embodiment of the present application, but the embodiment of the present application is not limited by the above embodiment, and any change, modification, substitution, combination, simplification made without departing from the spirit and principles of the present application should be an equivalent replacement method, and all are included in the protection scope of the present application.

Claims

1. A method for task offloading and cache updating based on federated reinforcement learning in the Internet of Vehicles, characterized in that, Includes the following steps: S1. Constructing a vehicle-to-everything (V2X) scenario for communication between intelligent vehicles and infrastructure, including edge servers and central cloud servers with computing and caching capabilities; A communication model between the vehicle and the base station needs to be established, where the wireless channel model is based on 3GPP TR 36.885, as detailed below: vehicle u The channel gain for communication with base stations within its range, taking into account both large-scale and small-scale fading, is expressed as: ； A represents the path fading parameter; For vehicles u The fading coefficients of follow an independent complex exponential distribution with a mean of 1; For vehicles u The shadow fading follows a log-normal distribution with standard deviation; Indicates vehicle u With base station k The distance, where γ is the decay exponent; Wireless communication between vehicles and base stations is based on Orthogonal Frequency Division Multiple Access (OFDMA), therefore vehicles u With base station k The bandwidth is equally shared by all vehicles connected to the current base station that require uplink access. The uplink signal-to-noise ratio is proportional to the vehicle's uplink transmit power and channel gain. u With base station k The uplink speed is According to Shannon's formula, we get: In the above formula, For vehicles u uplink transmit power, The total bandwidth of the base station, For vehicles u With base station k Channel gain, The power of Gaussian white noise. A vehicle unloading indicator variable in binary format (0-1). Vehicles u With base station k Establishing a connection involves offloading tasks to the base station. k The corresponding edge server, Calculate indicator variables locally for the vehicle. Vehicles in the current time slot u The computation task is performed locally; A base station cache indicator variable in binary format (0-1), when At that time, i.e., base station k The corresponding caching server caches the task type. n， Conversely ;vehicle u With base station k The uplink signal-to-noise ratio must meet the following requirements: = > In the above formula The noise threshold is used to determine when the uplink signal-to-noise ratio is lower than the noise threshold, at which point the task offloading fails. vehicle u With base station k The uplink latency is: In the above formula For the amount of task data, For vehicles u With base station k Uplink speed; vehicle u With base station k The uplink energy consumption is: * In the above formula For vehicles u The transmission power, For vehicles u Uplink delay with base station k; A computational model for vehicles and base stations needs to be established, based on the spatiotemporal distribution of vehicles and their connection behavior with base stations. u Need to go to the edge server k Request computing resources, where For vehicles u to base station k The requested computing resources must meet the following constraints: <= ∈ (0, ) In the above formula, For base stations k The total computing power of the equipped edge servers; When the vehicle u When selecting local processing of computation tasks, computation latency As shown in the following formula: In the above formula, The computing power required for the current task. For vehicles u Local computing resources; vehicle u Locally calculated energy consumption As shown in the following formula: * In the above formula, The computing power required for the current task. For vehicles u Local computing resources; The vehicle chip energy consumption coefficient is determined by the manufacturing process of the onboard chip equipped in the vehicle. vehicle u Offloaded to base station k The calculation delay is shown in the following formula: vehicle u Offloaded to base station k Computational energy consumption As shown in the following formula: * Assuming base station k The corresponding caching server has cached the task. n Data can be processed without uplinking, effectively saving uplink resources; Among them, base stations k The cache server must meet the following constraints: In the above formula For base stations k The total cache size of the corresponding cache server A base station cache indicator variable in binary format (0-1), when At that time, i.e., base station k The corresponding caching server caches the task type. n， Conversely ; Considering the limited cache space of base stations and the time-varying nature of content popularity, base stations k The corresponding cache server updates its cache list based on the characteristics of the current vehicle task request; This represents the current cache update policy of the cache server. , representing a base station k Add content n To the cache server, The following constraints must be met: The above formula represents every A time slot can replace at most one cached item. S2. Establish an optimization model with the optimization objective of minimizing the sum of vehicle task computation latency and energy consumption efficiency; in step S2, the optimization model is established with the optimization objective of minimizing the sum of vehicle task computation latency and energy consumption efficiency, as follows: In the time slot t ,vehicle u The total latency of the requested computation task is as follows: In the above formula, Calculation latency when selecting local computation for the vehicle For local calculation of indicator variables, For local calculation latency; The total latency of the task when it is unloaded at the edge and a cache hit occurs is calculated from the latency. composition, , For the corresponding indicator variable; The total latency of a task that is unloaded at the edge but misses the cache is calculated from the latency. With transmission delay composition; vehicle u The energy consumption of the requested computation task is as follows: In the above formula, Select the computing power consumption when calculating locally for the vehicle. For local calculation of indicator variables, Calculate energy consumption locally; The total energy consumption of the task when it is unloaded at the edge and the cache is hit is calculated from the energy consumption. composition, , For the corresponding indicator variable; The total energy consumption of a task that is unloaded at the edge but misses the cache is calculated from the energy consumption. With transmission delay composition; The optimization problem can be expressed as minimizing the weighted latency and energy consumption of the vehicle-to-everything (V2X) edge computing system: s.t.C1: ∈ C2: = 1 C3: ∈ C4: <= C5: ∈ C6: < C7: > C8: <= C9: <= 1 In the optimization formula: Replace variables for cache; For vehicle assembly, For base station set, Tolerance for task delays, and The weighted coefficients sum to 1. and These represent the time delay and energy consumption for vehicle u during this unloading task; among the constraints, Uplink signal-to-noise ratio. As the noise threshold, For base stations k The corresponding cache server cache capacity limit; The first constraint C1 indicates that the vehicle is unloaded in binary mode during task unloading, and no task splitting is performed. The second constraint C2 means that the vehicle in the current time slot must make an unloading decision; The third constraint, C3, indicates that the allocated computing resources must be less than the maximum computing power of the edge server corresponding to the base station. The fourth restriction, C4, means that the total computing resources allocated to the edge servers corresponding to the base station must not exceed the upper limit of computing resources. The fifth restriction, C5, means that the vehicle's uplink transmission power cannot exceed the maximum transmission power. ; The sixth constraint, C6, means that the execution time of the vehicle task must be less than the delay tolerance. The seventh constraint, C7, indicates that the vehicle's uplink signal-to-noise ratio must be greater than the noise threshold. ; The eighth restriction, C8, means that the cached content of the cache server corresponding to the base station must not exceed the total cache resources; The ninth constraint, C9, means that only one cached item is replaced at a time; S3. The intelligent vehicle uses its own task requirements and the state of the edge server as the input state of the federated deep reinforcement learning network, and the connection behavior between the intelligent vehicle and the edge, the transmission power, and the requested computing resources as the output actions. In step S3, the input state of the vehicle agent based on the federated deep reinforcement learning network is constructed, and the state representation of each vehicle agent is as follows: in, This is represented as a task model, including the current task data volume, required computing power, latency tolerance, and task number. Indicates vehicle u Sensing and base station set Channel gain for establishing wireless connections This indicates whether the cache server corresponding to the base station set caches vehicles. u The computational task (of which) n (obtained from the vehicle task model); The actions of each vehicle agent are represented as follows: in, Represented as vehicles u Current time slot and base station k The connection behavior, in particular, Represented as vehicles u Whether to decide on local computation Represented as vehicles u uplink transmit power, For vehicles u The computing resources requested from the base station; The reward function for each vehicle agent is designed as follows: in and These are weighted coefficients, and their sum is 1; , vehicles u The latency and energy consumption of this task's unloading process; Tolerance for task delays; For uplink signal-to-noise ratio, For uplink noise margin; It is represented as follows: G ( , ) = When parameter When less than the threshold, the function Output When the function is above the threshold Output positive value , The service quality characterized by the current offloading decision is determined by the weighted average latency and energy consumption; the lower the weighted average latency and energy consumption, the higher the reward. This is a decision function; if the unloading decision violates the constraints, a corresponding penalty is imposed, and if the conditions are met, a reward is given. To determine whether the uplink signal-to-noise ratio is higher than the signal-to-noise threshold; Does the delay in the execution of the judgment task meet the delay tolerance? To determine whether the computing resources requested by the vehicle from the base station are overloaded; By combining a federated learning distributed learning framework with the offline learning stage of the intelligent agent, cooperation between vehicles is achieved. A number of vehicles are periodically selected to obtain a global model according to the following formula, enabling the vehicles to... u The Q-target Net parameter set corresponding to the agent is The polymerization process is shown in the following formula: In the above formula, let For vehicles u The corresponding model parameters of the agent target network, For vehicles u The corresponding capacity of the agent experience replay cache pool. , which is the sum of the capacity of the experience replay cache pool for all participating agents; S4. The base station uses the historical cache requests of the current edge server as the input state of the federated deep reinforcement learning network and uses the cache replacement strategy as the output action. In step S4, the input states of edge node agents based on the federated deep reinforcement learning network are constructed. The state representation of each edge node agent is as follows: in This refers to the total number of requests for each piece of content over a short, medium, and long period of time. The action of each edge node agent is represented as follows: In the above formula Representing base stations k Should the corresponding cache server replace the task type? n ; The reward function for each edge node agent is designed as follows: = + The reward in the above formula serves as an important indicator for guiding gradient descent in decision-making networks. After the cache replacement policy is executed Band attenuation coefficient within time slot The sum of weighted cache hit rates, where For base stations k Corresponding caching server Cache hit rate within the time slot; The numerator is the sum of the cached task data in the entire simulation scenario, and the denominator is the global task data volume. To adjust the parameters, the division represents the global content coverage. The function is represented as 1 if the expression inside the parentheses is true, and 0 otherwise; the higher the global content coverage, the greater the reward. By combining a federated learning distributed learning framework with the offline learning phase of the intelligent agent, cooperation between edge nodes is achieved, and base stations are set up. k The Q-target Net parameter set corresponding to the agent is The polymerization process is shown in the following formula: In the above formula For the aggregated global model, The size of the experience replay buffer pool for each participating aggregation base station; The sum of the cache pool capacities of the base stations participating in the aggregation; S5. The vehicle intelligent agent outputs a task offloading and resource allocation scheme based on the status, and the edge intelligent agent outputs a cache update based on the input status, thereby maximizing the quality of vehicle network services. S6. Train the network through offline gradient descent and federated aggregation, update the target network weights, aggregate the target networks of each agent in the multi-agent environment to obtain the global model, and then distribute the global model to each agent.

2. The task offloading and cache update method based on federated reinforcement learning in the Internet of Vehicles according to claim 1, characterized in that, In step S1, a dynamic vehicle-to-everything (V2X) scenario needs to be established: The vehicle-to-everything (V2X) scenario involves vehicles constantly moving on highways and road sections, with the number of vehicles following a Poisson distribution based on vehicle speed. The scenario includes two base stations, each equipped with an edge server and a cache server. The vehicle collection is as follows The base station set is The task type set is , The total number of vehicles. The total number of base stations; in each time slot, the number of vehicles u The computational task is defined as = ,in For the amount of task data, For the required computing power, Tolerance for latency in the current task. Given the current task number, the popularity of vehicle tasks is modeled as following a Zipf distribution.

3. The task offloading and cache update method based on federated reinforcement learning in the Internet of Vehicles according to claim 1, characterized in that, In step S6, the networks of each agent are trained based on offline learning and federated learning. The offline learning steps are as follows: Initialize the experience replay pool, target network weights, and policy network weights for each agent; initialize the vehicle network scenario environment and obtain the input states of each agent; the policy network infers based on the input states and selects actions based on the actions; the actions are applied to the environment, a reward is obtained, and the agent is transferred to the next state; the current state transition pair, including the current state, the executed action, the reward, and the next state, is stored in the experience replay pool. The policy network is trained by drawing small batches of samples from the experience pool, and the weights of the target network are updated.

4. The task offloading and cache update method based on federated reinforcement learning in the Internet of Vehicles according to claim 1, characterized in that, In step S6, the networks of each agent are trained based on offline learning and federated learning. The federated learning steps are as follows: Each agent updates its own network through online and offline learning; the vehicle performs clustering based on its own unloading decisions; each agent uploads its own Target Net model parameters to the aggregation center based on the clustering results, the aggregation center obtains the global model through weighted averaging and distributes it to each agent, and each agent updates its own network model parameters through the global model.

Images