Edge resource allocation method for Internet of Vehicles based on federated multi-agent reinforcement learning

Abstract

The present invention discloses a method for allocating edge resources of an Internet of Vehicles based on federated multi-agent reinforcement learning, specifically: inputting an Internet of Vehicles environment, initializing the local Q network and federated network parameters of the agent, and modeling the optimization problem; dividing the agent into two categories, α and β, according to whether the agent can obtain a reward, and observing the local state and inputting the Q network in the current time slot respectively; encrypting the output of the Q network, and outputting the joint action decision of the two types of vehicle agents through the federated network; then the α vehicle agent obtains the global reward fed back by the system, and the cache pool stores the sample data of the current time slot; when the number of samples is sufficient, the α-type and β-type vehicle agents update the parameters of the local Q network and the federated network respectively; after the current training round is over, the Internet of Vehicles environment is reset, and the next round of training is started. The present invention improves the connectivity of the Internet of Vehicles under the premise of privacy protection, while reducing switching overhead and energy loss.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technologies, and in particular to a method for allocating edge resources in an Internet of Vehicles based on federated multi-agent reinforcement learning. Background Art

[0002] In recent years, vehicles, as an indispensable mode of transportation for human travel, have become an important carrier for driving rapid economic development and improving people's quality of life. As a product of the rapid development of modern industry and mobile communications, vehicle networking technology can effectively handle delay-sensitive services such as road safety, traffic efficiency, autonomous driving, and real-time information interaction in intelligent transportation systems. However, with the explosive growth of the number of vehicle users, traditional single access methods and communication resource allocation schemes have been difficult to meet the needs of various types of transportation at the same time. Since most of the vehicle networking services occurring around vehicles only need to provide short-range communication in the surrounding area, and most tasks are time-sensitive, dedicated short-range communication technologies with vehicle-to-vehicle communication and vehicle-to-infrastructure communication as the main communication methods have been widely used to support high-speed transmission (Jameel F, Wyne S, Javed MA, et al. Interference-aided vehicular networks: Future research opportunities and challenges [J]. IEEE Communications Magazine, 2018, 56 (10): 36-42.). However, as a typical heterogeneous wireless network with high mobility, the nodes with fixed coverage in the Internet of Vehicles are difficult to provide services continuously and stably, which brings expensive switching overhead, may cause transmission interruption, and even lead to the inability to effectively complete traffic safety business processing. Therefore, efficient and fast user access solutions are crucial to achieve high efficiency and high performance of the Internet of Vehicles.

[0003] With the development of artificial intelligence, deep reinforcement learning (DRL) is considered to be an effective solution to complex sequential decision-making problems. It can learn the best strategy by directly interacting with the environment without any prior knowledge of system dynamics (e.g., wireless channels, vehicle positions, etc.), avoiding the need for state transition matrices required when using traditional solutions to solve Markov decision problems. In recent years, scholars have tried to introduce DRL algorithms to solve edge resource allocation problems. Khan et al. decomposed the tasks centrally processed in the cloud into each roadside unit for local training, and used a distributed reinforcement learning framework to design a user access solution. Ultimately, they were able to meet the transmission rate requirements of each vehicle while minimizing the network coordination overhead (Khan H, Elgabli A, Samarakoon S, et al. Reinforcement learning-based vehicle-cell association algorithm for highly mobile millimeter wave communication [J]. IEEE Transactions on Cognitive Communications and Networking, 2019, 5 (4): 1073-1085.). However, although existing research has designed a variety of user access solutions for the Internet of Vehicles scenario and provided collaborative management strategies for communication and energy resources based on different business needs, the training of the DRL algorithm relies on a large amount of information interaction between vehicle users, which brings serious privacy leakage problems. Summary of the invention

[0004] The purpose of the present invention is to provide a method for allocating edge resources in an Internet of Vehicles based on federated multi-agent reinforcement learning, thereby improving the connectivity of the Internet of Vehicles and reducing switching overhead and energy loss under the premise of privacy protection.

[0005] The technical solution to achieve the purpose of the present invention is: a method for allocating edge resources in an Internet of Vehicles based on federated multi-agent reinforcement learning, the specific steps of which include:

[0006] Step 1: Input the Internet of Vehicles environment, initialize the agent local Q network and federated network parameters, and model the optimization problem;

[0007] Step 2: According to whether the agent can obtain rewards, it is divided into two categories: α-type vehicle agent and β-type vehicle agent. In the current time slot, the α-type and β-type vehicle agents respectively obtain the channel state between themselves and the roadside unit, the observable roadside unit position, and the roadside unit position associated with the previous time slot, and cascade them as the input of the Q network;

[0008] Step 3: Use the Gaussian difference method to encrypt the Q network output, and output the joint edge access decision of the α-type and β-type vehicle agents and the power allocation decision of the downlink through the federated network;

[0009] Step 4: The α-type vehicle agent obtains the connection gain, switching overhead, and energy loss trade-off reward fed back by the system, and the system inputs the sample data of the current time slot into the cache pool;

[0010] Step 5: Determine whether the sample size is sufficient. If yes, proceed to step 6; otherwise, proceed directly to step 7.

[0011] Step 6: When the number of samples is sufficient, the α-type and β-type vehicle agents update the parameters of the local Q network and the federated network respectively, and clear the cache pool after the update is completed;

[0012] Step 7: Determine whether the current training round is over. If not, return to step 2 to start the next round of training. If yes, proceed to step 8.

[0013] Step 8: Determine whether convergence has occurred. If not, reset the Internet of Vehicles environment and return to step 1. If yes, the training ends and the Internet of Vehicles edge resource allocation is completed.

[0014] Compared with the prior art, the present invention has the following significant advantages: (1) In order to solve the problem of large-scale information interaction caused by centralized processing of single-agent reinforcement learning algorithms, the multi-agent scenario in the Internet of Vehicles is considered, and vehicle users make collaborative decisions to improve the connectivity of the Internet of Vehicles while reducing switching overhead and energy loss; (2) By adopting the Gaussian difference method to protect the privacy of the output value of the local Q network of the agent, it is impossible for the agents to share the observed environmental state information, which greatly improves the security of local data; (3) The proposed scheme based on federated multi-agent reinforcement learning promotes model training by sharing encrypted training data, thereby improving the performance indicators of the system.

[0015] The present invention will be further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the vehicle network edge resource allocation method based on federated multi-agent reinforcement learning of the present invention.

[0017] Figure 2 Schematic diagram of a vehicle networking system model according to an embodiment of the present invention.

[0018] Figure 3 This is a graph showing how the convergence performance of the Internet of Vehicles system changes with period in an embodiment of the present invention.

[0019] Figure 4This is a graph showing how the average performance of vehicle users in the Internet of Vehicles system varies with the number of roadside units in an embodiment of the present invention.

[0020] Figure 5 This is a graph showing how the average performance of vehicle users in the Internet of Vehicles system varies with the maximum threshold of downlink transmission power in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention proposes a method for allocating edge resources in the Internet of Vehicles based on federated multi-agent reinforcement learning. Specifically, within a unit time slot t, a single vehicle user acts as an agent, observes the channel state between the vehicle and the roadside unit and the location information of the roadside unit, and uses a neural network to train a joint action strategy, thereby improving the transmission rate, reducing the switching overhead and energy loss, and achieving a trade-off between the three. Figure 1-2 , including the following steps:

[0022] Step 1: Input the Internet of Vehicles environment, initialize the agent local Q network and federated network parameters, and model the optimization problem;

[0023] Step 2: According to whether the agent can obtain rewards, the α-type and β-type vehicle agents respectively obtain the channel state between themselves and the roadside unit, the observable roadside unit position, and the roadside unit position associated with the previous time slot in the current time slot, and cascade them as the input of the Q network;

[0024] Step 3: Use the Gaussian difference method to encrypt the Q network output, and output the joint edge access decision of the α-type and β-type vehicle agents and the power allocation decision of the downlink through the federated network;

[0025] Step 4: The α-type vehicle agent obtains the connection gain, switching overhead, and energy loss trade-off reward fed back by the system, and the system inputs the sample data of the current time slot into the cache pool;

[0026] Step 5: Determine whether the sample size is sufficient. If yes, proceed to step 6; otherwise, proceed to step 7.

[0027] Step 6: When the number of samples is sufficient, the α-type and β-type vehicle agents update the parameters of the local Q network and the federated network respectively, and clear the cache pool after the update is completed;

[0028] Step 7: Determine whether the current training round is over. If not, return to step 2 to start the next round of training. If yes, proceed to step 8.

[0029] Step 8: Determine whether convergence has occurred. If not, reset the Internet of Vehicles environment and return to step 1. If yes, the training ends and the Internet of Vehicles edge resource allocation is completed.

[0030] As a specific implementation, the inputting of the Internet of Vehicles environment in step 1 specifically includes:

[0031] (1) Time slot model: The continuous training time is discretized into multiple time slots, expressed as The channel state information and system parameters remain unchanged within the duration of a single time slot, but may change randomly between different time slots.

[0032] (2) Network model: A multi-lane urban highway model is established, where roadside units supporting edge communication are evenly distributed on both sides of the highway. Indicates that vehicles travel towards each other from both ends of the highway, and by establishing a link between the vehicle and the infrastructure, the transmission rate of their own data is improved. express.

[0033] (3) Vehicle movement model: The speed change of the vehicle follows the following Gauss-Markov random process:

[0034]

[0035] Where v(t) represents the speed of the vehicle at time slot t, v(t-1) represents the speed of the vehicle at time slot t-1, represents the approximate mean of velocity, ζ represents the approximate variance of velocity, ξ represents the degree of memory, and z represents an uncorrelated random Gaussian process with zero mean and unit variance.

[0036] (4) Switching model: Assume that the vehicle can observe nearby O max The association variable between vehicle k and all roadside units is defined as

[0037]

[0038] in It means that the roadside unit r is associated with vehicle k in time slot t, and vice versa.

[0039] Considering that the coverage of the roadside unit is limited and a single time slot can only serve one vehicle, a switch occurs when the roadside unit associated with the vehicle in the adjacent time slot changes, that is:

[0040]

[0041] Among them, H k (t) represents the number of switches generated by vehicle k between adjacent time slots, 1 {·} It means that the value is set to 1 if the constraint is satisfied, otherwise it is set to 0.

[0042] (5) Power allocation model: The transmission power of the downlink roadside unit adopts a discrete level distribution, expressed as [P min ,P max ] range. Let represents the downlink transmission power configured by the roadside unit r associated with vehicle k in time slot t, then the power allocation variable of the downlink roadside unit configured by vehicle k is

[0043]

[0044] in It indicates that vehicle k selects p as the power of the downlink roadside unit in time slot t, that is, on the contrary,

[0045] (6) Wireless communication model: Consider that the mutual interference between the vehicle and the infrastructure link has been eliminated by the interference elimination method. Assume that the channel power gain consists of small-scale fading (Rayleigh fading) and path loss. Let represents the channel gain between vehicle k and roadside unit r. According to Shannon's formula, given the roadside unit r associated with vehicle user k in time slot t and its configured downlink power, the transmission rate that vehicle k can obtain is expressed as:

[0046]

[0047] In addition, it is assumed that the minimum data rate required by all vehicles in each time slot is the same and fixed, and R min express.

[0048] As a specific implementation method, the optimization problem modeling described in step 1 is specifically as follows:

[0049] Considering the optimization of joint edge access and power allocation, an optimization problem is constructed, and the optimization goal is to maximize the trade-off between connection gain, switching overhead, and energy loss:

[0050]

[0051]

[0052]

[0053] Where ω1, ω2, ω3 are weight coefficients of connection gain, switching overhead and energy loss respectively, and ω1+ω2+ω3=1; Formula C1 ensures seamless connection of vehicle users, and Formula C2 reflects the minimum data rate constraint required by vehicle users.

[0054] As a specific implementation method, according to whether the agent can obtain rewards, step 2 is divided into two categories: α-type vehicle agent and β-type vehicle agent, specifically;

[0055] Considering privacy protection, the vehicle agent can only observe its own status information and cannot obtain rewards from the system in real time or accurately. According to whether the agent can obtain rewards, it can be divided into two categories:

[0056] α-type vehicle agent: can observe its own local state and obtain corresponding system rewards in a timely and accurate manner;

[0057] β-type vehicle agent: can observe its own local state, but cannot obtain system rewards due to privacy protection.

[0058] As a specific implementation, in step 2, the α-type and β-type vehicle agents respectively obtain the channel state between themselves and the roadside unit, the observable roadside unit position, and the roadside unit position associated with the previous time slot in the current time slot, specifically:

[0059] At time slot t, each vehicle user acts as an intelligent agent and obtains its own observation state by interacting with the environment. The state of vehicle k at time slot t is expressed as:

[0060]

[0061] in, represents the channel state information between vehicle k and its observable roadside unit in time slot t; represents the position of the roadside unit observable by vehicle k in time slot t; Represents the position of the roadside unit associated with vehicle k at time slot t-1.

[0062] As a specific implementation, step 3 outputs the joint edge access decision of the α-type and β-type vehicle agents and the power allocation decision of the downlink through the federated network, specifically:

[0063] Determine the actions of each vehicle agent in interacting with the environment, including the selection of the associated roadside unit and the selection of downlink transmission power. At time slot t, the action of vehicle k is expressed as:

[0064]

[0065] As a specific implementation, the α-type vehicle agent in step 4 obtains a trade-off reward of connection gain, switching overhead, and energy loss fed back by the system, specifically:

[0066] When all vehicle agents have completed their actions, the environment will give back a global reward. Define the average trade-off of each user as the global reward, expressed as:

[0067]

[0068] As a specific implementation method, the agent local Q network and the federated network are initialized in step 1, and the Gaussian difference method is used to encrypt the Q network output in step 3, specifically:

[0069] (1) Local Q-network: α-type and β-type vehicle agents and local Q-networks are constructed to estimate the action-value function

[0070]

[0071] in represents the future cumulative reward with a discount factor, and γ represents the discount factor. Therefore, the local Q network output value of the α-type and β-type vehicle agents is defined as Q α (·;θ α ) and Q β (·;θ β ), where θ α and θ β are the weight parameters of the deep neural network.

[0072] (2) Gaussian difference: Based on privacy protection, the Gaussian difference method is used to add a random variable that follows a Gaussian distribution to the output of the local Q network of the α-type and β-type vehicle agents, respectively, which is defined as:

[0073]

[0074]

[0075] (3) Federation network: A multi-layer perceptron network is used as the federation network, with the encrypted Q network output as input, and the output joint decision is used to predict the joint action, which is expressed as:

[0076]

[0077] where MLP(·;θ MLP ) represents a multi-layer perceptron network, and [·|·] represents a cascade operation.

[0078] When the α-type and β-type vehicle agents update their network models, they regard the encrypted Q network output of the other party as a constant, namely:

[0079]

[0080]

[0081] As a specific implementation, the step 4 of inputting the sample data of the current time slot into the buffer pool is specifically as follows:

[0082] In order to improve the stability of the reinforcement learning algorithm, the experience replay method is used; the experience buffer pool is used to store the parameters of each step in the previous learning process, and randomly extract some samples from the previous samples for learning in the subsequent learning process. The experience pool is represented as The N groups of samples extracted by the α-type vehicle agent are expressed as The samples extracted by the β-type vehicle agent are expressed as When the number of stored samples reaches the upper limit, old samples will be removed to reserve space for new samples.

[0083] As a specific implementation, when the number of samples is sufficient, the α-type and β-type vehicle agents update the parameters of the local Q network and the federated network respectively, specifically:

[0084] In the actual model training process, there are online networks and target networks to prevent frequent updates and reduce divergence and oscillation in training. The online network continuously updates parameters to train the neural network and calculate the Q estimate; the target network temporarily fixes parameters and updates them once in a while to calculate the Q target value. Considering that the β-type vehicle agent cannot obtain the global reward of the system feedback, the Q estimate of the federated network can only be calculated by the α-type vehicle agent and shared with β, expressed as:

[0085]

[0086] Among them, r j is the global reward obtained by the α-type vehicle agent, and γ represents the discount factor.

[0087] Therefore, the loss function of the network is expressed as:

[0088]

[0089]

[0090] In actual calculations, the stochastic gradient descent method is generally used to optimize the loss function. During the training process, first, the α-type vehicle agent calculates y j And pass it to the β-type vehicle agent; then, the β-type vehicle agent updates its own Q network and multi-layer perceptron network and changes the parameter θ β ,θ MLP as well as Pass to the α-type vehicle agent:

[0091]

[0092]

[0093] Finally, the α-type vehicle agent updates its own Q network and federated network parameters based on the received parameters:

[0094]

[0095]

[0096] After updating all networks, if the vehicle round ends, the training process ends and the optimal strategy π is output * , otherwise start the next training.

[0097] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0098] Example

[0099] This embodiment provides a method for allocating edge resources in an Internet of Vehicles based on federated multi-agent reinforcement learning, which is described in detail below:

[0100] 1. Establish the Internet of Vehicles system model:

[0101] The present invention uses Python software to simulate the scenario of a multi-lane urban highway. The total length of the road is set to 1 km, and 12 roadside units are evenly distributed on both sides. The radius of its service range is 200 meters. Consider a pair of vehicles driving towards each other from both ends of the road. The vehicle can observe the information of the four nearest roadside units at most and select one to access. The vehicle movement model follows a Gaussian random process, where the initial value of the vehicle is ξ=0.1,ζ=0.1. The channel model between the vehicle and the roadside unit is as follows, path loss model: Small-scale fading follows Rayleigh distribution. The downlink transmission power is [23,35] dBm, and the minimum transmission rate is constrained to 8 bit / s / Hz.

[0102] 2. Establish a federated multi-agent reinforcement learning algorithm framework:

[0103] The federated multi-agent reinforcement learning algorithm framework is derived from the Deep Q network (DQN) algorithm, and the action strategy is determined by the value function estimation algorithm. The algorithm framework is mainly composed of the Q network and the federated network, in which a three-layer fully connected neural network is used as the local Q network of the agent, which is the input layer, the output layer and a hidden layer, and the number of neurons in the hidden layer is 80; a multi-layer perceptron network is used as the federated network, and the output value of the Q network processed by Gaussian difference is used as the federated network input, and the action value function is output to predict the joint action.

[0104] 3. Training phase of the algorithm:

[0105] First, input the current state sα (t), s β (t), defined as That is, the channel state observable by the vehicle agent, the roadside unit position, and the roadside unit position associated with the previous time slot. Secondly, the action output by the federated network is defined as the joint action of the vehicle agent. That is, the edge access decision of the vehicle agent and the selection of downlink power.

[0106] After the algorithm inputs the state, according to the joint action predicted by the federated network, after the current action interacts with the environment, the α-type vehicle agent obtains a global reward And transfer to the next state s α (t+1),s β (t+1). The network is then updated by minimizing the loss function that changes at each iteration i. During training, a decaying learning rate η (decaying from 0.01 to 0.001) and a discount factor γ = 0.9 are selected. 2000 training sets and 100 test sets are used, the number of samples for one training is set to 32, and the weights of the reward function are ω1 = 0.5, ω2 = 0.25, and ω3 = 0.25. First, the α-type vehicle agent calculates y j And pass it to the β-type vehicle agent; then, the β-type vehicle agent updates its own Q network and multi-layer perceptron network and changes the parameter θ β ,θ MLP as well as Pass it to the α-type vehicle agent; finally, the α-type vehicle agent updates its own Q network and federated network parameters based on the received parameters. After updating all networks, if the vehicle round ends, the training process ends and the optimal strategy π is output * , otherwise start the next training.

[0107] Consider the following benchmark scenario: Given that the α-type vehicle agent can train its model independently and the β-type vehicle agent shares a similar system environment, we can only train the model of the α-type vehicle agent to obtain the joint action strategy of the α-type and β-type vehicle agents, namely the DQN_α algorithm (whose input is s α (t), output the joint action Q value of the vehicle agent).

[0108] like Figure 3 As shown in the figure, first, the proposed federated multi-agent reinforcement learning algorithm has better convergence performance than the comparison algorithm. This shows that even if the training data is protected from noise, the β-type vehicle agent can still provide unobservable training strategies to help the α-type vehicle agent learn the joint strategy. Secondly, since the state information of both agents is effectively utilized, the proposed algorithm greatly reduces the number of switching while ensuring a satisfactory transmission rate.

[0109] like Figure 4 As shown in the figure, as the number of roadside units increases, the average vehicle-user trade-off of both schemes increases, and the transmission rate follows the same trend. The reason is that as the number of roadside units increases, vehicles have more opportunities to connect to closer roadside units, thereby increasing the data rate. In addition, thanks to the federated strategy learning between α-type and β-type vehicle agents, the federated multi-agent reinforcement learning algorithm proposed in this description can improve the connection gain on the basis of privacy protection and significantly reduce the switching overhead.

[0110] like Figure 5 As shown in Figure 2, this paper further studies the impact of the downlink transmit power threshold on the average performance of vehicle users. First, since increased transmit power helps to increase the data rate, the total trade-off and connection benefit increase with the increase of the power threshold. However, since the power allocation is not directly related to the switching overhead, the number of switching remains almost unchanged. Second, the federated multi-agent reinforcement learning framework proposed in this paper shares the encrypted Q-values ​​of α-type and β-type vehicle agents, which can promote the training of the model under the premise of privacy protection, and then prompt the vehicle agents to make more effective joint decisions under different downlink transmit power thresholds.

[0111] In summary, the present invention targets the unknown high-dynamic topology and channel state characteristics of the Internet of Vehicles, studies the problem of joint edge access and power allocation of the Internet of Vehicles based on privacy protection, and proposes a method for allocating edge resources in the Internet of Vehicles based on federated multi-agent reinforcement learning. This scheme adopts the Gaussian difference method to retain the privacy of interactions between vehicle agents during decision-making training; at the same time, it introduces a multi-layer perceptron model to share encrypted training data, which enables the agents to better perform model training. The method for allocating edge resources in the Internet of Vehicles based on federated multi-agent reinforcement learning proposed in the present invention can improve the connectivity of the Internet of Vehicles while protecting the privacy information of the local state of vehicle users, while reducing switching overhead and energy loss, thereby supporting various ultra-high reliability and low-latency communication applications in the Internet of Vehicles.

Claims

1. A method for allocating edge resources in an Internet of Vehicles based on federated multi-agent reinforcement learning, characterized in that: The specific steps include: Step 1: Input the Internet of Vehicles environment, initialize the agent local Q network and federated network parameters, and model the optimization problem; Step 2: According to whether the agent can obtain rewards, it is divided into two categories: α-type vehicle agent and β-type vehicle agent. In the current time slot, the α-type and β-type vehicle agents respectively obtain the channel state between themselves and the roadside unit, the observable roadside unit position, and the roadside unit position associated with the previous time slot, and cascade them as the input of the Q network; Step 3: Use the Gaussian difference method to encrypt the Q network output, and output the joint edge access decision of the α-type and β-type vehicle agents and the power allocation decision of the downlink through the federated network; Step 4: The α-type vehicle agent obtains the connection gain, switching overhead, and energy loss trade-off reward fed back by the system, and the system inputs the sample data of the current time slot into the cache pool; Step 5: Determine whether the sample size is sufficient. If yes, proceed to step 6; otherwise, proceed directly to step 7. Step 6: When the number of samples is sufficient, the α-type and β-type vehicle agents update the parameters of the local Q network and the federated network respectively, and clear the cache pool after the update is completed; Step 7: Determine whether the current training round is over. If not, return to step 2 to start the next round of training. If yes, proceed to step 8. Step 8: Determine whether convergence has occurred. If not, reset the Internet of Vehicles environment and return to step 1. If yes, the training ends and the Internet of Vehicles edge resource allocation is completed. The input vehicle networking environment described in step 1 specifically includes: (1) Time slot model: The continuous training time is discretized into multiple time slots, expressed as The channel state information and system parameters remain constant within the duration of a single time slot, but may vary randomly between different time slots; (2) Network model: A multi-lane urban highway model is established, where roadside units supporting edge communication are evenly distributed on both sides of the highway. Indicates that vehicles travel towards each other from both ends of the highway, and by establishing a link between the vehicle and the infrastructure, the transmission rate of their own data is improved. express; (3) Vehicle movement model: The speed change of the vehicle follows the following Gauss-Markov random process: Among them, υ(t) represents the speed of the vehicle at time slot t, υ(t-1) represents the speed of the vehicle at time slot t-1, represents the approximate mean of velocity, ζ represents the approximate variance of velocity, ξ represents the degree of memory, and z represents an uncorrelated random Gaussian process with zero mean and unit variance; (4) Switching model: Assume that the vehicle can observe nearby O max The information of the roadside units is collected and the associated roadside units are adaptively selected; the association variable between vehicle k and all roadside units is defined as in It means that the roadside unit r is associated with vehicle k in time slot t, and vice versa. Considering that the coverage of the roadside unit is limited and a single time slot can only serve one vehicle, when the roadside unit associated with the vehicle in the adjacent time slot changes, switching occurs, that is: Among them, H k (t) represents the number of switches generated by vehicle k between adjacent time slots, 1 {·} Indicates that it is set to 1 if the constraint is met, otherwise it is set to 0; (5) Power allocation model: The transmission power of the downlink roadside unit adopts a discrete level distribution, expressed as [P min ,P max ] within the range of P levels; let represents the downlink transmission power configured by the roadside unit r associated with vehicle k in time slot t, then the power allocation variable of the downlink roadside unit configured by vehicle k is for in It indicates that vehicle k selects p as the power of the downlink roadside unit in time slot t, that is, on the contrary, (6) Wireless communication model: Consider that the mutual interference between the vehicle and the infrastructure link has been eliminated by the interference elimination method in the system; assume that the channel power gain is composed of small-scale fading, namely Rayleigh fading, and path loss; let represents the channel gain between vehicle k and roadside unit r. According to Shannon's formula, given the roadside unit r associated with vehicle user k in time slot t and its configured downlink power, the transmission rate R that vehicle k can obtain is k (t) is expressed as: In addition, it is assumed that the minimum data rate required by all vehicles in each time slot is the same and fixed, and R min express; Step 1 describes modeling the optimization problem, specifically: Considering the optimization of joint edge access and power allocation, an optimization problem is constructed, and the optimization goal is to maximize the trade-off between connection gain, switching overhead, and energy loss: Where ω1, ω2, ω3 are weight coefficients of connection gain, switching overhead and energy loss respectively, and ω1+ω2+ω3=1; Formula C1 ensures seamless connection of vehicle users, and Formula C2 reflects the minimum data rate constraint required by vehicle users.

2. The method for allocating edge resources in an Internet of Vehicles based on federated multi-agent reinforcement learning according to claim 1 is characterized in that: According to whether the agent can obtain rewards, it is divided into two categories: α-type vehicle agent and β-type vehicle agent, specifically; Considering privacy protection, the vehicle agent can only observe its own status information and cannot obtain rewards from the system in real time or accurately. According to whether the agent can obtain rewards, it is divided into two categories: α-type vehicle agent: can observe its own local state and obtain corresponding system rewards in a timely and accurate manner; β-type vehicle agent: can observe its own local state, but cannot obtain system rewards due to privacy protection.

3. The method for allocating edge resources in an Internet of Vehicles based on federated multi-agent reinforcement learning according to claim 2 is characterized in that: In step 2, the α-type and β-type vehicle agents obtain the channel state between themselves and the roadside unit, the observable roadside unit position, and the roadside unit position associated with the previous time slot in the current time slot, specifically: At time slot t, each vehicle user acts as an intelligent agent and obtains its own observation state by interacting with the environment. The state of vehicle k at time slot t is expressed as: in, represents the channel state information between vehicle k and its observable roadside unit in time slot t; represents the position of the roadside unit observable by vehicle k in time slot t; Represents the position of the roadside unit associated with vehicle k at time slot t-1.

4. The method for allocating edge resources in an Internet of Vehicles based on federated multi-agent reinforcement learning according to claim 3 is characterized in that: Step 3 outputs the joint edge access decision of the α-type and β-type vehicle agents and the power allocation decision of the downlink through the federated network, specifically: Determine the actions of each vehicle agent in interacting with the environment, including the selection of the associated roadside unit and the selection of the downlink transmission power; at time slot t, the action of vehicle k is expressed as:

5. The method for allocating edge resources in an Internet of Vehicles based on federated multi-agent reinforcement learning according to claim 4 is characterized in that: In step 4, the α-type vehicle agent obtains the trade-off reward of connection gain, switching overhead and energy loss fed back by the system, which is specifically: When all vehicle agents have completed their actions, the environment will give back a system global reward. The average trade-off of each user is defined as the global reward, expressed as:

6. The method for allocating edge resources in an Internet of Vehicles based on federated multi-agent reinforcement learning according to claim 5, characterized in that: In step 1, the local Q network and the federated network of the intelligent agent are initialized, and in step 3, the Gaussian difference method is used to encrypt the output of the Q network, specifically: (1) Local Q-network: Local Q-networks are constructed for the α-type and β-type vehicle agents respectively to estimate the action-value function in represents the future cumulative reward with a discount factor γ; defines the local Q network output value of the α-type and β-type vehicle agents as Q α (·;θ α ) and Q β (·;θ β ), where θ α and θ β They are the weight parameters of the deep neural network; (2) Gaussian difference: Based on privacy protection, the Gaussian difference method is used to add a random variable that follows a Gaussian distribution to the output of the local Q network of the α-type and β-type vehicle agents, respectively, which is expressed as: (3) Federated network: A multi-layer perceptron network is used as the federated network, with the encrypted Q network output as input, and the output is a joint decision to predict the action, expressed as: where MLP(·;θ MLP ) represents a multilayer perceptron network, [·|·] represents a cascade operation; When the α-type and β-type vehicle agents update the network model, the two types of vehicle agents regard the encrypted Q network output of the other party as a constant, that is:

7. The method for allocating edge resources in an Internet of Vehicles based on federated multi-agent reinforcement learning according to claim 6 is characterized in that: Step 4 of inputting the sample data of the current time slot into the buffer pool is as follows: The experience replay method is used to improve the stability of the reinforcement learning algorithm; the experience buffer pool is used to store the parameters of each step in the previous learning process, and randomly extract some samples from the previous samples for learning in the subsequent learning process; the experience pool is represented as The N groups of samples extracted by the α-type vehicle agent are expressed as The samples extracted by the β-type vehicle agent are expressed as When the number of stored samples reaches the upper limit, old samples will be removed to reserve space for new samples.

8. The method for allocating edge resources in an Internet of Vehicles based on federated multi-agent reinforcement learning according to claim 7, characterized in that: When the number of samples is sufficient as described in step 6, the α-type and β-type vehicle agents update the parameters of the local Q network and the federated network respectively, specifically: In the actual model training process, there are online networks and target networks. The online network continuously updates parameters to train the neural network and calculate the Q estimation value. The target network temporarily fixes parameters and updates them once in a while to calculate the Q target value. Considering that the β-type vehicle agent cannot obtain the global reward of the system feedback, the Q estimate of the federated network can only be calculated by the α-type vehicle agent and shared with β, expressed as: Among them, r j is the global reward obtained by the α-type vehicle agent, γ represents the discount factor; Therefore, the loss function of the network is expressed as: In actual calculation, the stochastic gradient descent method is used to optimize the loss function. During the training process, first, the α-type vehicle agent calculates y j And pass it to the β-type vehicle agent; then, the β-type vehicle agent updates its own Q network and multi-layer perceptron network and changes the parameter θ β ,θ MLP as well as Pass to the α-type vehicle agent: Finally, the α-type vehicle agent updates its own Q network and federated network parameters based on the received parameters: After updating all networks, if the vehicle round ends, the training process ends and the optimal strategy π is output * , otherwise start the next training.

Images