An asynchronous federated optimization method for selecting vehicles based on DDPG algorithm

Abstract

The present invention provides an asynchronous federated optimization method for selecting vehicles based on a DDPG algorithm, including setting the system state, action and reward of time slot t according to the vehicle's own transmission rate, the size of available computing resources and the vehicle position; selecting vehicles participating in training according to the system action of time slot t to obtain a selected vehicle; the selected vehicle performs local training using local data to obtain a corresponding local model; considering the hysteresis effect of the training delay on the local model trained by the vehicle, the local model is weighted and optimized to obtain a weight-optimized local model; the trained vehicle asynchronously uploads the weight-optimized local model to a roadside unit for asynchronous federated aggregation, and finally obtains a global model through multiple rounds of repeated training. The method of the present invention is simple in calculation, and the system model is reasonable. Simulation experiments verify that the method can obtain a higher global model accuracy in a vehicle environment.

Description

Technical Field

[0001] The present invention relates to the field of vehicle network technology, and in particular to an asynchronous federated optimization method for selecting vehicles based on a DDPG (Deep Deterministic Policy Gradient) algorithm. Background Art

[0002] With the development of science and technology, Internet of Vehicles technology has gradually emerged, making our travel more convenient. At the same time, intelligent services on various vehicles are also gradually emerging. Therefore, vehicles will generate some computing tasks according to demand while driving on the road. However, in traditional cloud computing services, since the cloud is far away from the vehicle, uploading will produce a large delay, which is not suitable for high-speed moving vehicle scenarios, so vehicle-mounted edge computing has emerged. Among them, vehicles can upload computing tasks to roadside units (RSUs) with certain computing power and close to the vehicle for task processing, thereby greatly reducing the delay of task processing. However, the task calculation of the vehicle requires the vehicle to upload local data to the roadside unit for processing by the roadside unit. This creates privacy and security issues. Vehicle users may be worried about privacy leakage and are unwilling to upload local data. So federated learning technology was born. Specifically, federated learning performs a certain number of global aggregations at the roadside unit. In the first round of training, the vehicle first downloads the initialized global model from the roadside unit, then performs local training using local data. After the training is completed, the local model is uploaded instead of the local data. After receiving the local models of all vehicles at the roadside unit, the local model is aggregated, and then the second round is repeated until the specified number of times is reached. This greatly protects the privacy of vehicle users.

[0003] However, in traditional federated learning, the roadside unit needs to wait until all vehicles have uploaded their local models before updating the global model. If a certain vehicle is being trained and the upload delay is too long, other vehicles will be out of the coverage of the roadside unit and will be unable to participate in global training. This is how asynchronous federated training came about. Specifically, the vehicle uses local data to complete local training and then uploads the local model. At the roadside unit, it aggregates the global model every time it receives a local model uploaded by a vehicle. This can update the global model at the roadside unit faster without waiting for other vehicles to upload.

[0004] Since the vehicles themselves have a certain degree of mobility, time-varying channel conditions are caused, resulting in time-varying transmission rates, which causes vehicles to have different transmission delays. At the same time, since different vehicles have different time-varying computing resources and the amount of local data they have is different, different local training delays are caused. In the asynchronous federated training process, since the vehicles upload local models asynchronously, when a vehicle has not yet uploaded its local model to the roadside unit, the roadside unit has already updated the global model based on other uploaded local models, so the local model of the vehicle has a certain degree of obsolescence. Obsolescence is related to local training delay and transmission delay. Therefore, it is crucial to consider the impact of the above factors on the accuracy of the global model at the roadside unit.

[0005] Since there may be bad nodes in the vehicle, that is, the vehicle itself has few available computing resources, the amount of local data is small, and the local model after local training will be disturbed to a certain extent. The existence of bad nodes will greatly affect the accuracy of the global model at the roadside unit. Therefore, it is very necessary to select the vehicle nodes participating in the global aggregation.

[0006] Therefore, taking into account the mobility of vehicles, time-varying channel conditions, the time-varying available computing resources of the vehicles themselves, the different amounts of local data of vehicles, and the existence of bad nodes in vehicles, the present invention proposes an asynchronous federated optimization method for selecting vehicles based on the DDPG algorithm. Summary of the invention

[0007] To this end, an embodiment of the present invention provides an asynchronous federated optimization method for selecting vehicles based on the DDPG algorithm, which is used to solve the problem of low accuracy of the generated global model in the prior art due to the mobility of vehicles, time-varying channel conditions, the time-varying available computing resources of the vehicles themselves, the different local data amounts of vehicles, and the existence of bad vehicle nodes.

[0008] In order to solve the above problems, an embodiment of the present invention provides an asynchronous federated optimization method for selecting vehicles based on a DDPG algorithm, the method comprising:

[0009] S1: Set the system state, action and reward for time slot t according to the vehicle's own transmission rate, available computing resources and vehicle location;

[0010] S2: Select the vehicles participating in the training according to the system action at time slot t, and obtain the selected vehicles;

[0011] S3: The selected vehicle performs local training using local data to obtain a corresponding local model;

[0012] S4: Considering the hysteresis effect of training delay and transmission delay on the local model trained by the vehicle, the weight of the local model is optimized to obtain a local model after weight optimization;

[0013] S5: After training, the vehicle asynchronously uploads the weight-optimized local model to the roadside unit for asynchronous federated aggregation. After multiple rounds of repeated training, the roadside unit finally obtains the global model.

[0014] Preferably, in step S1, setting the system state, action and reward of time slot t according to the vehicle's own transmission rate, available computing resource size and vehicle position includes:

[0015] The system state of the set time slot t is:

[0016] s(t)=(Tr(t),μ(t),d x (t),a(t-1))

[0017] Where s(t) is the system state at time slot t, Tr(t) represents the set of transmission rates of all vehicles at time slot t, μ(t) is the set of available computing resources of all vehicles at time slot t, and d x (t) is the set of position coordinates of all vehicles along the x-axis at time slot t, and a(t-1) is the system action at time slot t-1;

[0018] The system action for setting the time slot t is:

[0019] a(t)=(λ 1 (t),λ 2 (t),…,λ K (t))

[0020] Where a(t) is the system action at time slot t, λ i (t), i∈[1,K] represents the probability of selecting vehicle i, let λ 1 (0) = λ 2 (0) = ... = λ K (0) = 1;

[0021] The system reward for setting time slot t is:

[0022]

[0023] Where r(t) is the system reward at time slot t, ω 1 and ω 2 is a non-negative weight factor, a di (t) is the system action in time slot t, λ i (t),i∈[1,K] represents the probability of selecting vehicle i, Loss(t) is the loss value calculated in asynchronous federated training, is the delay caused by local training of vehicle i, is the transmission delay of vehicle i uploading the local model in time slot t.

[0024] Preferably, in step S2, selecting the vehicle to participate in the training according to the system action of the time slot t, and obtaining the selected vehicle comprises the following steps:

[0025] S21: Set a d (t)=(a d1 (t),a d2 (t),…,a dK (t));

[0026] S22: λ i (t) is normalized and λ is set i (t) values ​​greater than or equal to 0.5 correspond to a di (t) is recorded as 1, otherwise it is 0, and the final set a d (t) is composed of 0 and 1, 1 means selecting a vehicle, and 0 means not selecting a vehicle.

[0027] Preferably, based on the system reward at time slot t, the expected long-term discounted reward of the system can be expressed as:

[0028]

[0029] Among them, γ∈(0,1) is the discount factor, N is the total number of time slots, μ is the system strategy, and J(μ) is the expected long-term discounted reward of the system.

[0030] Preferably, in step S3, the selected vehicle is locally trained using local data to obtain a corresponding local model, including the following steps:

[0031] S31: At time slot t, vehicle V k Download the global model w from the roadside unit t-1 , where, at time slot 1, the global model at the roadside unit is initialized to w using a convolutional neural network 0 ;

[0032] S32: Vehicle V k The local data is trained based on the convolutional neural network. The local training consists of l rounds. In the mth (m∈[1,l]) round of local training, the vehicle V k First, the label probability of each local data a, i.e., y a Input to the local model w k,m The convolutional neural network is then used to obtain the predicted probability of the convolutional neural network for each data label. The cross entropy loss function is used to calculate w k,mThe loss value is calculated as follows:

[0033]

[0034] S33: Use the stochastic gradient descent algorithm to update the local model. The formula is as follows:

[0035]

[0036] in, f k (w k,m ) is the gradient, η is the learning rate;

[0037] S34: Vehicle V k Use the updated local model to perform m+1 rounds of local training. When the number of local training rounds reaches l, the local training stops and the vehicle obtains the updated local model w k .

[0038] Preferably, the training delay is:

[0039]

[0040] in, is the delay caused by local training of vehicle i, C 0 The number of CPU cycles required to train a data set, μ i is the computing resource of vehicle i, measured by the CPU cycle frequency. Each vehicle i (1≤i≤K) carries a different amount of data D i .

[0041] Preferably, the transmission delay is:

[0042]

[0043]

[0044] d i (t)=||P i (t)-P r ||

[0045] in, is the transmission delay of vehicle i uploading the local model in time slot t, |w| is the size of the local model obtained by local training for each vehicle, tr i (t) is the transmission rate of vehicle i in time slot t, B is the transmission bandwidth, p 0 is the transmission power of each vehicle, a fixed value, h i (t) is the channel gain at time slot t, α is the path loss exponent, σ 2 is the noise power, the position P of vehicle i at time slot t i(t) is set to (d ix (t),d y ,0), where d ix (t) and d y are the distances of vehicle i from the antenna of the roadside unit along the x-axis and y-axis at time slot t, respectively. y is a fixed value, d ix (t) = d i0 +vt,d i0 is the coordinate of the initial position of vehicle i along the x-axis, v is the vehicle speed, t is the time slot, and the antenna height of the roadside unit is set to H r , then the antenna position of the roadside unit is expressed as P r =(0,0,H r ).

[0046] Preferably, an autoregressive model is used to construct h i (t) and h i (t-1), namely:

[0047]

[0048] Among them, ρ i is the normalized channel correlation coefficient between consecutive time slots, e(t) is the error vector that follows a complex Gaussian distribution and is related to h i (t) related, according to Jack's fading spectrum, Among them J 0 (·) is a zero-order Bessel function of the first kind and is the Doppler frequency of vehicle i, Λ is the wavelength, θ is the moving direction, that is, x 0 =(1,0,0) and the uplink communication direction is P r -P i (t), so

[0049] Preferably, in step S4, the method for weight optimization of the local model is:

[0050] The local model is weighted and optimized, wherein the weight includes training weight and transmission weight, and the training weight is:

[0051]

[0052] Among them, β 1,k is the training weight, m 1 ∈(0,1) is a parameter that makes β 1,k As the local training latency increases, it decreases. For vehicle V k Local computing latency;

[0053] The transmission weight is:

[0054]

[0055] Among them, β 2,k (t) is the transmission weight, m 2 ∈(0,1) is a parameter that makes β 2,k (t) decreases as the transmission delay increases, For vehicle V k transmission delay;

[0056] According to the formula w kw =w k *β 1,k *β 2,k , get the local model after weight optimization;

[0057] Among them, w k is the local model, w kw is the local model after weight optimization, β 1,k is the training weight, β 2,k (t) is the transmission weight.

[0058] Preferably, in step S5, the trained vehicle asynchronously uploads the weight-optimized local model to the roadside unit for asynchronous federated aggregation. After multiple rounds of repeated training, the roadside unit finally obtains the global model, which specifically includes:

[0059] Preferably, when the vehicle V k After uploading the weight-optimized local model to the roadside unit, the roadside unit performs a global aggregation, and the formula is as follows:

[0060] w new =βw old +(1-β)w kw

[0061] Among them, w old is the current global model at the roadside unit, w new is the updated global model, w kw is the local model after weight optimization, β∈(0,1) is the aggregation ratio;

[0062] At the beginning of each time slot, when the roadside unit receives the first uploaded local model, w old =w t-1 , when the roadside unit receives the local models of all selected vehicles and gets the updated K 1 The global model w t After that, the global model update of this time slot ends.

[0063] It can be seen from the above technical solutions that the present invention has the following advantages:

[0064] An embodiment of the present invention provides an asynchronous federated optimization method for selecting vehicles based on a DDPG algorithm. The present invention uses a deep reinforcement learning algorithm to select vehicles participating in training according to the vehicle's own transmission rate, the size of available computing resources, and the vehicle's position, and removes bad nodes that may exist in the vehicle; the vehicle adopts asynchronous federated training, and the roadside unit aggregates the global model once each time it receives a local model uploaded from the vehicle, which can update the global model at the roadside unit faster without waiting for other vehicles to upload; when the vehicle performs local modeling, the hysteresis effect of training delay and transmission delay on the local model trained by the vehicle is considered, and the weight of the local model is optimized, thereby improving the accuracy of the global model at the roadside unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0066] Figure 1 A flowchart of an asynchronous federated optimization method for selecting a vehicle based on a DDPG algorithm provided in an embodiment;

[0067] Figure 2 Schematic diagram of the scene framework of the method of the present invention;

[0068] Figure 3 A diagram showing two quantities in the reward for the test phase: the loss value in asynchronous federated learning and the sum of vehicle delays, i.e., the sum of local delay and transmission delay;

[0069] Figure 4 This is a schematic diagram showing the accuracy comparison between the method of the present invention and asynchronous federated learning and federated learning in the test phase when bad nodes exist;

[0070] Figure 5 This is a schematic diagram showing the comparison of the loss of the method of the present invention with asynchronous federated learning and federated learning in the test phase when bad nodes exist;

[0071] Figure 6 A schematic diagram showing the comparison of the accuracy of the method of the present invention with asynchronous federated learning without local weight processing and federated learning without local weight processing in the case of selecting nodes;

[0072] Figure 7 A schematic diagram showing the comparison of the loss of the method of the present invention with asynchronous federated learning without local weight processing and federated learning without local weight processing in the case of selecting nodes;

[0073] Figure 8 Schematic diagram showing the comparison of the training delay between the method of the present invention and federated learning as the number of global rounds increases;

[0074] Fig. 9 It is a schematic diagram showing the accuracy comparison between the method of the present invention and the global model of traditional asynchronous federated learning under selected nodes under different beta values. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] like Figure 1 As shown, an embodiment of the present invention proposes an asynchronous federated optimization method for selecting a vehicle based on a DDPG algorithm, the method comprising:

[0077] S1: Set the system state, action and reward for time slot t according to the vehicle's own transmission rate, available computing resources and vehicle location;

[0078] S2: Select the vehicles participating in the training according to the system action in time slot t, and obtain the selected vehicles;

[0079] S3: The selected vehicle performs local training using local data to obtain a corresponding local model;

[0080] S4: Considering the hysteresis effect of training delay and transmission delay on the local model trained by the vehicle, the weight of the local model is optimized to obtain a local model after weight optimization;

[0081] S5: After training, the vehicle asynchronously uploads the weight-optimized local model to the roadside unit for asynchronous federated aggregation. After multiple rounds of repeated training, the roadside unit finally obtains the global model.

[0082] The present invention provides an asynchronous federated optimization method for selecting vehicles based on a DDPG algorithm. The present invention sets time slots according to the vehicle's own transmission rate, the size of available computing resources, and the vehicle's position, selects vehicles participating in training, and removes bad nodes that may exist in the vehicle; the selected vehicle uses local data for local training to obtain a corresponding local model. When the vehicle performs local modeling, the hysteresis effect of training delay and transmission delay on the local model trained by the vehicle is considered, and the weight of the local model is optimized, thereby improving the accuracy of the global model at the roadside unit; the trained vehicle asynchronously uploads the weight-optimized local model to the roadside unit for asynchronous federated aggregation, and through multiple rounds of repeated training, the roadside unit finally obtains a global model. The vehicle of the present invention adopts asynchronous federated training, and the roadside unit aggregates the global model once each time it receives a local model uploaded from the vehicle, which can update the global model at the roadside unit faster without waiting for the upload of other vehicles. The method of the present invention is simple to calculate, and the system model is reasonable. Simulation experiments verify that the method can obtain a higher global model accuracy in a vehicle environment.

[0083] Figure 2 This is a schematic diagram of the scenario framework of the method of the present invention. A deep reinforcement learning algorithm is used to select vehicles participating in the training based on the vehicle's own transmission rate, the size of available computing resources, and the vehicle's location. The selected vehicles then use asynchronous federation technology to train the local model and then upload it to the roadside unit to ultimately obtain a more accurate global model.

[0084] Furthermore, step S1 includes:

[0085] The system state, action and reward of time slot t are set according to the vehicle's own transmission rate, available computing resources and vehicle location, including:

[0086] Since the mobility of a vehicle can be reflected by its position change, the training time and upload time of the vehicle's local model are related to the vehicle's own time-varying available computing resources and the current channel conditions. Therefore, the system state s(t) at time slot t is defined as:

[0087] s(t)=(Tr(t),μ(t),d x (t),a(t-1))

[0088] Where s(t) is the system state at time slot t, Tr(t) represents the set of transmission rates of all vehicles at time slot t, μ(t) is the set of available computing resources of all vehicles at time slot t, and d x (t) is the set of position coordinates of all vehicles along the x-axis at time slot t, and a(t-1) is the system action at time slot t-1.

[0089] Since the purpose of the present invention is to select a better vehicle for asynchronous federated learning training according to the current state, the system action a(t) in time slot t is defined as:

[0090]

[0091] Where a(t) is the system action at time slot t, λ i (t), i∈[1,K] represents the probability of selecting vehicle i, let λ 1 (0) = λ 2 (0) = ... = λ K (0)=1.

[0092] The present invention aims to select vehicles with better performance for asynchronous federated training to obtain a more accurate global model at the roadside unit, while considering the delay and the accuracy of the global model. Therefore, the system reward r(t) at time slot t is defined as:

[0093]

[0094] Where r(t) is the system reward at time slot t, ω 1 and ω 2 is a non-negative weight factor, a di (t) is the system action in time slot t, λ i (t),i∈[1,K] represents the probability of selecting vehicle i, Loss(t) is the loss value calculated in asynchronous federated training, is the delay caused by local training of vehicle i, is the transmission delay of vehicle i uploading the local model in time slot t.

[0095] Then the expected long-term discounted reward of the system can be expressed as:

[0096]

[0097] Among them, γ∈(0,1) is the discount factor, N is the total number of time slots, μ is the system strategy, and J(μ) is the expected long-term discounted reward of the system.

[0098] Furthermore, step S2 includes:

[0099] In order to select a specific vehicle, let the set a d (t)=(a d1 (t),a d2 (t),…,a dK (t)), i (t) is normalized and λ is set i (t) values ​​greater than or equal to 0.5 correspond to a di (t) is recorded as 1, otherwise it is 0. The final set ad (t) is composed of 0 and 1, 1 means selecting a vehicle, and 0 means not selecting a vehicle.

[0100] Furthermore, step S3 includes:

[0101] The selected vehicle uses local data for local training to obtain the corresponding local model, including the following steps:

[0102] S31: At time slot t, vehicle V k Download the global model w from the roadside unit t-1 , where, at time slot 1, the global model at the roadside unit is initialized to w using a convolutional neural network 0 ;

[0103] S32: Vehicle V k The local data is trained based on the convolutional neural network. The local training consists of l rounds. In the mth (m∈[1,l]) round of local training, the vehicle Vk first converts the label probability of each local data a, i.e., y a Input to the local model w k,m The convolutional neural network is then used to obtain the predicted probability of the convolutional neural network for each data label. The cross entropy loss function is used to calculate w k,m The loss value is calculated as follows:

[0104]

[0105] S33: Use the stochastic gradient descent algorithm to update the local model. The formula is as follows:

[0106]

[0107] in, f k (w k,m ) is the gradient, η is the learning rate;

[0108] S34: Vehicle V k Use the updated local model to perform m+1 rounds of local training. When the number of local training rounds reaches l, the local training stops and the vehicle obtains the updated local model w k .

[0109] Furthermore, step S4 includes:

[0110] When the vehicle is performing local training, training delay and transmission delay will occur. The training delay is:

[0111]

[0112] in, is the delay caused by local training of vehicle i, C 0 The number of CPU cycles required to train a data set, μ i is the computing resource of vehicle i, measured by the CPU cycle frequency. Each vehicle i (1≤i≤K) carries a different amount of data D i ;

[0113] The transmission delay is:

[0114]

[0115]

[0116] d i (t)=||P i (t)-P r ||

[0117] in, is the transmission delay of vehicle i uploading the local model in time slot t, |w| is the size of the local model obtained by local training for each vehicle, tr i (t) is the transmission rate of vehicle i in time slot t, B is the transmission bandwidth, p 0 is the transmission power of each vehicle, a fixed value, h i (t) is the channel gain at time slot t, α is the path loss exponent, σ 2 is the noise power, the position P of vehicle i at time slot t i (t) is set to (d ix (t),d y ,0), where d ix (t) and d y are the distances of vehicle i from the antenna of the roadside unit along the x-axis and y-axis at time slot t, respectively. y is a fixed value, d ix (t) = d i0 +vt,d i0 is the coordinate of the initial position of vehicle i along the x-axis, v is the vehicle speed, t is the time slot, and the antenna height of the roadside unit is set to H r , then the antenna position of the roadside unit is expressed as P r =(0,0,H r ).

[0118] Among them, the autoregressive model is used to construct h i (t) and h i (t-1), namely:

[0119]

[0120] Among them, ρ iis the normalized channel correlation coefficient between consecutive time slots, e(t) is the error vector that follows a complex Gaussian distribution and is related to h i (t) related, according to Jack's fading spectrum, Among them J 0 (·) is a zero-order Bessel function of the first kind and is the Doppler frequency of vehicle i, Λ is the wavelength, θ is the moving direction, that is, x 0 =(1,0,0) and the uplink communication direction is P r -P i (t), so

[0121] Different from traditional asynchronous federated learning, the present invention takes into account the hysteresis effect of training delay and transmission delay on the local model trained by the vehicle. Specifically, since there will be a certain delay in the local training of the vehicle and uploading the local model to the roadside unit, when a vehicle is in the process of local training and uploading to the roadside unit, it is possible that the roadside unit has received the local model uploaded by other vehicles and updated the global model. In this case, the local model trained by this vehicle has a certain hysteresis. Therefore, the present invention has a certain delay in the local training of the vehicle V k The local model is weighted, that is, the training weight and transmission weight are set. The specific calculation method is as follows:

[0122] The local model is weighted and optimized, wherein the weight includes training weight and transmission weight, and the training weight is:

[0123]

[0124] Among them, β 1,k is the training weight, m 1 ∈(0,1) is a parameter that makes β 1,k As the local training latency increases, it decreases. For vehicle V k Local computing latency;

[0125] The transmission weight is:

[0126]

[0127] Among them, β 2,k (t) is the transmission weight, m 2 ∈(0,1) is a parameter that makes β 2,k (t) decreases as the transmission delay increases, For vehicle V k transmission delay;

[0128] According to the formula wkw =w k *β 1,k *β 2,k , get the local model after weight optimization;

[0129] Among them, w k is the local model, w kw is the local model after weight optimization, β 1,k is the training weight, β 2,k (t) is the transmission weight.

[0130] Furthermore, step S5 includes:

[0131] After training, the vehicle asynchronously uploads the weight-optimized local model to the roadside unit for asynchronous federated aggregation. After multiple rounds of repeated training, the roadside unit finally obtains the global model, which includes:

[0132] When the vehicle V k After uploading the weight-optimized local model to the roadside unit, the roadside unit performs a global aggregation, and the formula is as follows:

[0133] w new =β wold +(1-β)w kw

[0134] Among them, w old is the current global model at the roadside unit, w new is the updated global model, w kw is the local model after weight optimization, β∈(0,1) is the aggregation ratio;

[0135] At the beginning of each time slot, when the roadside unit receives the first uploaded local model, w old =w t-1 , when the roadside unit receives the local models of all selected vehicles and gets the updated K 1 The global model w t After that, the global model update of this time slot ends.

[0136] At the same time, the average loss Loss(t) of the vehicles participating in the training can be obtained, which can be expressed as:

[0137]

[0138] Among them, f k (w k ) is the local model w k The loss value.

[0139] In order to further illustrate the principles and beneficial effects of the present invention, a description is given below in conjunction with specific experiments.

[0140] The present invention aims to find an optimal strategy μ * to maximize the expected long-term discounted reward of the system.

[0141] The overall algorithm specifically adopted in the present invention includes two parts, an algorithm of a training phase based on a DAFL (Data-Free Learning) framework and an algorithm of a testing phase based on the DAFL framework.

[0142] The algorithm steps of the training phase based on the DAFL framework are shown in Table 1.

[0143] Table 1

[0144]

[0145]

[0146] The present invention provides an asynchronous federated optimization method for selecting vehicles based on the DDPG algorithm, wherein the DDPG algorithm is based on an actor-critic network architecture. The actor network is used for policy improvement, and the critic network is used for policy evaluation. Specifically, the actor network is used to approximate the policy μ, and its approximate policy is represented as μ δ The actor network is based on the policy μ δ And observe the state and output actions.

[0147] The present invention improves and evaluates the strategy through iteration to finally obtain the optimal strategy. In order to ensure the stability of the algorithm, the DDPG algorithm also uses a target network composed of a target actor network and a target critic network, and its architecture is the same as that of the actor network and the critic network.

[0148] Set δ as the actor network parameter, ξ as the critic network parameter, δ * is the optimized actor network parameter, ξ * is the optimized critic network parameter, δ 1 is the target actor network parameter, ξ 1 is the target critic network parameter. τ is the update parameter of the target network, Δ t is the noise of action exploration in time slot t. I is the mini-batch size. Next, the algorithm of the training phase will be introduced in detail.

[0149] First, randomly initialize δ and ξ, and at the same time replace δ in the target network with 1 and 1Initialize to δ and ξ respectively. At the same time, the experience playback buffer R b Initialize.

[0150] Next, the algorithm will execute E max In the first round, the positions of all vehicles, channel status, and available computing resources of the vehicles are reset. And set λ 1 (0) = λ 2 (0) = ... = λ K (0) = 1, then in the first time slot, the system can obtain the initial state s(1) = (Tr(1), μ(1), d x (1), a(0)). At the same time, CNN (Convolutional Neural Networks) is used to initialize the global model w at the roadside unit. 0 .

[0151] After that, the algorithm will be executed continuously from time slot 1 to the maximum number of time slots N. In the first time slot, the actor network gets the output μ according to the state δ (s|δ), where a random noise Δ is added to the action t , so the system gets action a(1) = μ δ (s(1)|δ)+Δ t . Then calculate a according to the action d (1), determine the vehicle selected for this time slot. The selected vehicle performs asynchronous federated training, that is, the vehicle trains a local model based on local data, and then asynchronously uploads it to the roadside unit to update the global model, and then calculates the loss value Loss(1). At the same time, the local training delay and transmission delay of the vehicle are calculated, so that the system reward under time slot 1 can be obtained. Then, the vehicle position is updated, the channel conditions and the vehicle's own available computing resources are recalculated, and the vehicle's transmission rate is updated so that the system can observe the next state s(2). The tuple (s(1), a(1), r(1), s(2)) is then stored in R b middle.

[0152] When R b When the number of tuples in is less than or equal to I, the system directly inputs the next state into the actor network and proceeds to the next iteration.

[0153] When R b When the number of tuples in is greater than I, the parameters δ, ξ, δ in the actor network, critic network, and target network are 1 and 1 Start updating to maximize J(μ δ ). The parameters δ of the actor network are moving towards J(μ δ ) is the gradient direction of Update. Will follow the strategy μ under s(t) and a(t) δ The action value function is set to Its expression is:

[0154]

[0155] It represents the long-term expected discounted reward of the system in time slot t.

[0156] Solution It can be solved by Gradient Instead, the critic network uses the parameter ξ to Approximately Q ξ (s(t),a(t)).

[0157] Next, we will introduce the parameters δ, ξ, δ under time slot t. 1 and 1 The update method. When R b When the number of tuples in is greater than I, the system starts from R b Randomly select I tuples from to form a mini-batch. Let (s x ,a x ,r x ,s′ x ), x∈[1,2,…,I] is the xth tuple in the mini-batch. Then the system first converts s′ x Input the target actor network to get the output action Then s′ x and a′ x Input the target critic network and get the output action value function Then the target value can be calculated as:

[0158]

[0159] Then, according to s x and a x , the critic network will have an output Q ξ (s x ,a x ), then the loss of tuple x can be calculated as:

[0160] L x =[y x -Q ξ (s x ,a x )] 2

[0161] When all tuples are input into the critic network and the target network, the loss function is obtained:

[0162]

[0163] The critic network is constructed by Use the gradient descent method to minimize the loss function L(ξ) and update the parameter ξ.

[0164] Similarly, actor networks are implemented by Use the gradient ascent method to maximize J(μ δ ) to update the parameter δ. The action value function is calculated by approximating the critic network, and the formula is as follows:

[0165]

[0166] Where Q ξ The input is

[0167] At the end of time slot t, update the parameters of the target network, and the update formula is:

[0168] ξ 1 ←τξ+(1-τ)ξ 1

[0169] δ 1 ←τδ+(1-τ)δ 1

[0170] Where τ is a constant and satisfies τ<<1.

[0171] Finally, the system inputs s′ into the actor network and starts the iterative calculation of the next time slot. When the time slot t reaches the maximum value N, the round ends. Then the system re-initializes the state value s(1) = (Tr(1), μ(1), d x (1), a(0)), and proceed to the next round of training. When the number of rounds reaches the maximum value E max The training ends when , and the parameters of the optimized actor network, critic network, target actor network and target critic network are obtained, namely δ * , * , and

[0172] The test phase simulates the critic network, target actor network, and target critic network in the training phase. And uses the optimal parameter δ * The optimal strategy.

[0173] The algorithm steps of the testing phase based on the DAFL framework are shown in Table 2.

[0174] Table 2

[0175] <![CDATA[1. For each episode 1 ≤ epi ≤ E′ max Execute:]]> 2. Reset the simulation parameters of the system model and initialize the global model at the roadside unit 3. Get the initial state s(1) 4. For each time slot 1≤t≤N, execute: <![CDATA[5. Generate action a = μ δ (s|δ)]]> <![CDATA[6. Calculate a d , and determine the selected vehicle]]> 7. The selected vehicle undergoes weight-based AFL update training 8. Obtain reward r and next state s′ from the current system

[0176] Through the above experiments, the method of the present invention has the following conclusions:

[0177] 1. During the test phase, as the step size increases, the loss value in the asynchronous federation, i.e., the local delay, gradually decreases, and the vehicle delay, i.e., the transmission delay, remains within a certain range, e.g. Figure 3 shown.

[0178] 2. In the test phase, in the presence of bad nodes, the method of the present invention has higher accuracy and lower loss than asynchronous federated learning and federated learning, such as Figure 4 , Figure 5 shown.

[0179] 3. In the case of selecting nodes, the method of the present invention has higher accuracy and lower loss than asynchronous federated learning without local weight processing and federated learning without local weight processing, such as Figure 6 , Figure 7 shown.

[0180] 4. As the number of global rounds increases, the training delay of the method of the present invention is smaller than that of federated learning, such as Figure 8 shown.

[0181] 5. Under different beta values, the accuracy of the method of the present invention is higher than that of the global model of the traditional asynchronous federated learning under the selected node, such as Fig. 9 shown.

[0182] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

[0183] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0184] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0185] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0186] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. An asynchronous federated optimization method for selecting vehicles based on the DDPG algorithm, characterized in that: include: S1: Set the system state, action and reward for time slot t according to the vehicle's own transmission rate, available computing resources and vehicle location; S2: Select the vehicles participating in the training according to the system action at time slot t, and obtain the selected vehicles, specifically including: S21: Set a d (t)=(a d1 (t),a d2 (t),…,a dK (t)); S22: λ i (t) is normalized and λ is set i (t) values ​​greater than or equal to 0.5 correspond to a di (t) is recorded as 1, otherwise it is 0. The final set a d (t) is composed of 0 and 1, 1 means selecting a vehicle, 0 means not selecting a vehicle; S3: The selected vehicle performs local training using local data to obtain a corresponding local model; S4: Considering the hysteresis effect of training delay and transmission delay on the local model trained by the vehicle, the weight of the local model is optimized to obtain a local model after weight optimization; The method for optimizing the weight of the local model is: The local model is weighted, and the weights include training weights and transmission weights. The training weights are: Among them, β 1,k is the training weight, m1∈(0,1) is the parameter, which makes β 1,k As the local training latency increases, it decreases. For vehicle V k Local computing latency; The transmission weight is: Among them, β 2,k (t) is the transmission weight, m2∈(0,1) is the parameter, which makes β 2,k (t) decreases as the transmission delay increases, For vehicle V k transmission delay; According to the formula w kw =w k *β 1,k *β 2,k , get the local model after weight optimization; Among them, w k is the local model, w kw is the local model after weight optimization, β 1,k is the training weight, β 2,k (t) is the transmission weight; S5: After training, the vehicle asynchronously uploads the weight-optimized local model to the roadside unit for asynchronous federated aggregation. After multiple rounds of repeated training, the roadside unit finally obtains the global model.

2. The asynchronous federated optimization method for selecting vehicles based on the DDPG algorithm according to claim 1, characterized in that: In step S1, the system state, action and reward of time slot t are set according to the vehicle's own transmission rate, available computing resource size and vehicle position, including: The system state of the set time slot t is: s(t)=(Tr(t),μ(t),d x (t),a(t-1)) Where s(t) is the system state at time slot t, Tr(t) represents the set of transmission rates of all vehicles at time slot t, μ(t) is the set of available computing resources of all vehicles at time slot t, and d x (t) is the set of position coordinates of all vehicles along the x-axis at time slot t, and a(t-1) is the system action at time slot t-1; The system action for setting the time slot t is: a(t)=(λ1(t),λ2(t),…,λ K (t)) Where a(t) is the system action at time slot t, λ i (t), i∈[1,K] represents the probability of selecting vehicle i, let λ1(0)=λ2(0)=…=λ K (0) = 1; The system reward for setting time slot t is: Where r(t) is the system reward for time slot t, ω1 and ω2 are non-negative weight factors, and a di (t) is the system action in time slot t, λ i (t),i∈[1,K] represents the probability of selecting vehicle i, Loss(t) is the loss value calculated in asynchronous federated training, is the delay caused by local training of vehicle i, is the transmission delay of vehicle i uploading the local model in time slot t.

3. The asynchronous federated optimization method for selecting vehicles based on the DDPG algorithm according to claim 2, characterized in that: Based on the system reward at time slot t, the expected long-term discounted reward of the system can be expressed as: Among them, γ∈(0,1) is the discount factor, N is the total number of time slots, μ is the system strategy, and J(μ) is the expected long-term discounted reward of the system.

4. The asynchronous federated optimization method for selecting vehicles based on the DDPG algorithm according to claim 1, characterized in that: In step S3, the selected vehicle uses local data to perform local training to obtain a corresponding local model, including the following steps: S31: At time slot t, vehicle V k Download the global model w from the roadside unit t-1 , where, at time slot 1, the global model at the roadside unit is initialized to w0 using a convolutional neural network; S32: Vehicle V k The local data is trained based on the convolutional neural network. The local training consists of l rounds. In the mth (m∈[1,l]) round of local training, the vehicle V k First, the label probability of each local data a, i.e., y a Input to the local model w k,m The convolutional neural network is then used to obtain the predicted probability of the convolutional neural network for each data label. The cross entropy loss function is used to calculate w k,m The loss value is calculated as follows: S33: Use the stochastic gradient descent algorithm to update the local model. The formula is as follows: in, f k (w k,m ) is the gradient, η is the learning rate; S34: Vehicle V k Use the updated local model to perform m+1 rounds of local training. When the number of local training rounds reaches l, the local training stops and the vehicle obtains the updated local model w k .

5. The asynchronous federated optimization method for selecting vehicles based on the DDPG algorithm according to claim 1, characterized in that: In step S4, the training delay is: in, is the delay caused by local training of vehicle i, C0 is the number of CPU cycles required to train one data, μ i is the computing resource of vehicle i, measured by the CPU cycle frequency. Each vehicle i (1≤i≤K) carries a different amount of data D i .

6. The asynchronous federated optimization method for selecting vehicles based on the DDPG algorithm according to claim 1, characterized in that: In step S4, the transmission delay is: d i (t)=‖P i (t)-P r ‖ in, is the transmission delay of vehicle i uploading the local model in time slot t, |w| is the size of the local model obtained by local training for each vehicle, tr i (t) is the transmission rate of vehicle i in time slot t, B is the transmission bandwidth, p0 is the transmission power of each vehicle, which is a constant value, and h i (t) is the channel gain at time slot t, α is the path loss exponent, σ 2 is the noise power, the position P of vehicle i at time slot t i (t) is set to (d ix (t),d y ,0), where d ix (t) and d y are the distances of vehicle i from the antenna of the roadside unit along the x-axis and y-axis at time slot t, respectively. y is a fixed value, d ix (t) = d i0 +vt,d i0 is the coordinate of the initial position of vehicle i along the x-axis, v is the vehicle speed, t is the time slot, and the antenna height of the roadside unit is set to H r , then the antenna position of the roadside unit is expressed as P r =(0,0,H r ).

7. The asynchronous federated optimization method for selecting vehicles based on the DDPG algorithm according to claim 6, characterized in that: An autoregressive model is used to construct h i (t) and h i (t-1), namely: Among them, ρ i is the normalized channel correlation coefficient between consecutive time slots, e(t) is the error vector that follows a complex Gaussian distribution and is related to h i (t) related, according to Jack's fading spectrum, where J0(·) is the zero-order Bessel function of the first kind and is the Doppler frequency of vehicle i, Λ is the wavelength, θ is the moving direction, that is, x0 = (1,0,0) and the uplink communication direction, that is, P r -P i (t), so 8. The asynchronous federated optimization method for selecting vehicles based on the DDPG algorithm according to claim 1, characterized in that: In step S5, the trained vehicle asynchronously uploads the weight-optimized local model to the roadside unit for asynchronous federated aggregation. After multiple rounds of repeated training, the roadside unit finally obtains the global model, which specifically includes: When the vehicle V k After uploading the weight-optimized local model to the roadside unit, the roadside unit performs a global aggregation, and the formula is as follows: In new =βw old +(1-β)in kw Among them, w old is the current global model at the roadside unit, w new is the updated global model, w kw is the local model after weight optimization, β∈(0,1) is the aggregation ratio; At the beginning of each time slot, when the roadside unit receives the first uploaded local model, w old =w t-1 , when the roadside unit receives the local models of all selected vehicles and obtains the global model w after updating K1 times t After that, the global model update of this time slot ends.

Images