Internet of vehicles spectrum resource sharing method oriented to queue system
Through deep reinforcement learning algorithms, the vehicle queue system model is trained, and spectrum resource allocation is optimized, which solves the problem of unstable transmission of vehicle network queues in complex environments, and improves link performance and sharing stability.
Patent Information
- Application Number
- CN202411199802.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-22
AI Technical Summary
The existing Internet of Vehicle queue processing methods perform well in discrete low-dimensional environments, but poorly perform well in continuous and complex environments, which cannot ensure the reliability and efficiency of link transmission, resulting in errors in message sharing.
The vehicle queue system model is trained using deep reinforcement learning algorithms, and spectrum resource allocation is performed through real-time network states, optimizing the vehicle's transmission rate to infrastructure links and the vehicle's payload transmission success rate to vehicle links.
The link performance and sharing stability of the Internet of Vehicles queue system are improved, and through intelligent resource allocation and management of interference, the total transmission rate of the link and the success rate of the payload are maximized.
Smart Images

Figure CN119012210A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of vehicle network spectrum sharing, and in particular relates to a vehicle network spectrum resource sharing method for a queuing system. Background Art
[0002] With the continuous development of the new generation of mobile communication technology, Vehicle-to-Everything (V2X) has attracted widespread attention from all over the world due to its potential value in promoting the development of intelligent transportation systems (ITS) and smart cities. Technically speaking, the vehicle-to-everything connects vehicles to mobile networks to achieve full connectivity of vehicle-to-infrastructure (V2I), vehicle-to-person (V2P), vehicle-to-network (V2N) and vehicle-to-vehicle (V2V). In the V2X network, different types of links support various in-vehicle applications with heterogeneous quality of service (QoS) requirements. Specifically, entertainment, traffic efficiency information and other applications not related to safety are usually A large amount of data needs to be transmitted between the base station and the vehicle, and a high-capacity V2I link is an ideal solution. At the same time, safety-related applications such as cooperative driving and danger warning often involve the transmission of safety-related information between nearby vehicles in a periodic or triggered manner, which requires the support of a reliable V2V link. Based on the application of artificial algorithms, interconnected self-driving cars have become an important means to ensure traffic safety and reduce environmental pollution, and are considered to be a basic component of the future intelligent transportation system. Platoon driving is the first step to achieve autonomous driving and is considered to be one of the most representative potentials for overcoming transportation costs. A platoon is led by a platoon leader (Platoon In the platoon system, PMs regularly share information (including vehicle speed, acceleration, location and application data) with PL, so that PMs can travel closely together at a fixed speed and head-on interval. This method helps to reduce vehicle energy consumption and improve driving safety and comfort. In addition, each platoon must obtain enough information in the surrounding environment to understand other platoons and vehicles in the environment, which requires vehicles to have high-performance communication capabilities. However, with the continuous expansion of Internet of Vehicles applications and the continuous improvement of communication needs, the existing spectrum resources are obviously not enough to meet all communication needs. Therefore, in order to ensure low-latency and high-reliability communication services in the Internet of Vehicles platoon system, a new spectrum resource allocation scheme is needed.
[0003] The existing Internet of Vehicles queue processing methods can only handle discrete low-dimensional environments, or perform poorly in continuous complex environments. Therefore, they have certain limitations and cannot ensure the transmission reliability of the link's effective load, resulting in errors in periodic message sharing. There is room for improvement. Summary of the invention
[0004] The purpose of the present invention is to propose a spectrum resource sharing method for Internet of Vehicles (IoV) for a queue system in order to solve the problem that the existing IoV queue processing method can only process discrete low-dimensional environments or performs poorly in continuous complex environments.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] A spectrum resource sharing method for an Internet of Vehicles (IoV) for a queuing system, comprising:
[0007] The spectrum allocation decision is obtained by using a deep reinforcement learning algorithm to train the vehicle platoon system model based on the real-time network status;
[0008] Allocate spectrum resources to maximize the total transmission rate of the vehicle-to-infrastructure link and ensure that the payload of the vehicle-to-vehicle link achieves an optimized delivery success rate under the desired delay constraints.
[0009] As a further description of the above technical solution:
[0010] The reinforcement learning method for training the vehicle platoon system model using a deep reinforcement learning algorithm is:
[0011] The cellular-based multi-queue vehicle communication network model consists of one RSU, M vehicle queues, M V2I links and N V2V links, where each vehicle queue has O vehicles;
[0012] The sets of V2I links and V2V links are represented by M = {1, ..., M} and N = {1, ..., N} respectively;
[0013] V2V communication provides periodic vehicle information transmission between PM and PL;
[0014] V2I communication supports applications that require large amounts of data to be transmitted between PL and RSU;
[0015] Orthogonal frequency division multiplexing (OFDM) is used in the model. It is assumed that the V2I link for uploading data has been pre-allocated with orthogonal spectrum sub-bands. There are I sub-bands in total, of which the mth V2I link occupies the ith sub-band. The transmission power of the V2I link is a fixed value, and the V2V link can share the spectrum resources allocated to the V2I link.
[0016] set up and are the transmission power of the transmitting end of the m-th V2I link and the transmission power of the n-th V2V link on the i-th subband, respectively, which are subject to the following constraints:
[0017]
[0018] A Boolean spectrum selection variable ρ is defined n [i], which indicates whether the nth V2V link is transmitted on the i-th subband. If so, then ρ n [i] = 1, otherwise 0, and each V2V link can only access one orthogonal subband at most, that is: ∑ n ρ n [i]≤1;
[0019] Then for the V2I link, the calculation formula for the signal-to-noise ratio of the mth V2I link on the i-th subband is:
[0020]
[0021] Where g n,m [i] is the interference channel gain from V2V link n to V2I link m on subband i, g m,B [i] is the channel gain of V2I link m transmitted on the i-th subband, σ 2 is the noise power;
[0022] For the V2V link, the calculation expression of the SINR received by the nth V2V link on the i-th subband is:
[0023]
[0024]
[0025] Where g n [i] represents the channel power gain of V2V link n on the i-th subband, I n [i] is the interference power, g m,n [i] represents the interference channel gain of V2I link m to V2V link n on the i-th subband, g n',n [i] represents the interference channel gain of V2V link n' to V2V link n on the i-th subband, represents the transmission power of the V2V link n';
[0026] According to Shannon's formula, the transmission rate of V2I link m on subband i can be expressed as
[0027]
[0028] Where W is the sub-band spectrum bandwidth;
[0029] Similarly, the channel capacity calculation formula for the nth V2V link on the ith subband is:
[0030]
[0031] Among them, the high data transmission rate and reliable message delivery requirements of V2I and V2V link applications are the optimization goals;
[0032] That is, the capacity of the V2I link in represents the minimum tolerable throughput. The V2V link is mainly used to ensure the reliable transmission of vehicle data, where vehicles generate data periodically at different frequencies. Let B represent the size of the periodically generated V2V payload in bits. Δt is the channel coherence time, and t is the index of the coherence time. Within the constraint time T, the probability of successful transmission of the payload of a data packet of size B is:
[0033]
[0034] For all n∈N, m∈M, the transmission power of the V2V link is a continuously adjustable variable At the same time, the total transmission rate of all V2I links and the probability of successful transmission of V2V link payload are maximized. The optimization objectives and constraints are:
[0035]
[0036] st:∑ n ρ n [i]≤1
[0037]
[0038] 0≤t≤T
[0039]
[0040] As a further description of the above technical solution:
[0041] It also includes establishing a Markov decision process (MDP), which is mainly composed of a state space (S), an action space (A), and a reward function (R);
[0042] At time slot t, the environment state is The queue members in the Internet of Vehicles queue are intelligent agents, which can Adopt a strategy The environment changes to the next state Feedback rewards
[0043] As a further description of the above technical solution:
[0044] The state space is: the observed local information state of agent n has its own channel gain g n [i], the interference channel power gain g of V2V link n' to V2V link n n`,n [i], interference channel gain g of V2V link n to V2I link m on subband i n,m [i], channel gain g of the V2I link m to the base station in the i-th subband m,B [i], and the interference noise power I from all other links n [i];
[0045] In addition to this channel information, because the optimization objective also takes into account the transmission success rate of the V2V payload, the agent also needs to consider its own remaining transmission load D n and the remaining transmission time T n , which is expressed as follows:
[0046]
[0047] Among them G n [i] = {g n [i],g n`,n [i],g n,B [i],g m,B [i]}.
[0048] As a further description of the above technical solution:
[0049] The action space is:
[0050] Based on the observed state, each agent will make a decision on the continuous control of the transmission power. The transmission power of the transmitter of the V2V link is a continuous variable with a value range of The action decision of agent k is represented as:
[0051] As a further description of the above technical solution:
[0052] The reward function is:
[0053] Optimize the total transmission rate of all V2I links and on the other hand improve the success probability of V2V payload delivery within time T, where the optimal target of the total transmission rate of all V2I links is at time slot t, maximizing the total transmission rate of all V2I links. The reward function is defined as
[0054]
[0055] For each agent n, the reward L nSet to the transmission rate of the V2V link until the payload is fully delivered, after which let the reward be a constant β, which is greater than the maximum V2V link transmission rate;
[0056] Therefore, the reward function for successful transmission of the V2V link payload at time step t is defined as:
[0057]
[0058] The goal of agent learning is to maximize the total transmission rate of the V2I link while improving the transmission success rate of the V2V link payload. Therefore, at time slot t, the total optimization objective function set according to the formula is as follows:
[0059]
[0060] Where λ and β are the weights of the V2I link capacity and the V2V link transmission rate, respectively, and λ+β=1. The values of λ and β correspond to different service requirements. If λ approaches zero and β approaches 1, the agent will pay more attention to the total transmission rate of the V2I link, and the platoon vehicles will need higher bandwidth. On the contrary, the platoon vehicles have higher requirements for driving safety, and the agent will focus on improving the rate at which the V2V link transmits information. If the user has no special requirements, λ and β can be set to 0.5.
[0061] As a further description of the above technical solution:
[0062] It also includes the use of a multi-agent reinforcement learning algorithm based on proximal policy optimization (PPO) to adopt a centralized training and distributed execution framework to perform reinforcement learning on the overall optimization objective function, including:
[0063] The distributed execution framework consists of an actor network with parameter θ and a critic network with parameter φ. The Actor network and the Critic network have strategy functions π(a k |s k ,θ) and the value estimation function V(s k ,φ), where θ and φ are the weights of the Actor network and the Critic network;
[0064] According to the observed state s t , input to the Actor network, thereby obtaining action a t ;
[0065] Execute action a t After that, the agent will receive an instantaneous reward r t and the state s at the next moment t+1 , and store the experience in the experience replay pool;
[0066] During the training process, the agent interacts with the environment many times, generating and collecting a batch of experience {s t ,a t ,r t ,s t+1};
[0067] Use the collected experience to optimize the Actor network and Critic network;
[0068] During the optimization process, the Actor network and the Critic network calculate the loss values of the strategy and the global state value function respectively based on the empirical data to update the parameters θ and φ;
[0069] The loss value of the Actor network is calculated by the following formula:
[0070]
[0071] in and π θ denote the old strategy and the current strategy respectively, S(s t ) indicates the state is s t The action entropy at that time increases exploration during the training phase to avoid falling into the local optimum;
[0072] The meaning of the clip function is that if the first term is less than the second term, 1-ε is output, and if the first term is greater than the third term, 1+ε is output. The purpose is to limit the probability ratio to a reasonable range;
[0073] ε is a hyperparameter, usually 0.1;
[0074] ψ is a hyperparameter that balances exploration and exploitation;
[0075] To balance the bias and variance of the policy gradient, The state value V(s) is used in the form of Generalized Advantage Estimation (GAE). t ) is estimated, and the estimated deviation and variance are weighted and balanced by introducing the parameter λ. The specific formula is:
[0076]
[0077]
[0078] Define V φ (s t ) is the state value fitted by the Critic network of the agent, and the Critic network loss function is calculated by the following formula:
[0079]
[0080] The Actor network is updated by maximizing the objective function, and the Critic network is updated by minimizing the loss function.
[0081] As a further description of the above technical solution:
[0082] The multi-agent reinforcement learning algorithm adopts the centralized training algorithm process steps as follows:
[0083] S1. Initialize the actor network and critic network parameters;
[0084] S2, initialize the environment and experience pool;
[0085] S3, For e = 1 to episode
[0086] S4, reset the Internet of Vehicles environment;
[0087] S5, update vehicle position and large-scale fading;
[0088] S6, For step = 1 to 100;
[0089] S7, action sampling;
[0090] S8, observe the initial environment state s of the vehicle agent t ;
[0091] S9. Input the state into the Actor network to get action a t ;
[0092] S10, the agent performs action a t Then get the state s at the next moment t+1 And get the reward value r t ;
[0093] S11, will (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool;
[0094] S12, calculating the discounted return and advantage function according to the formula;
[0095] S13, extracting training samples from the experience pool for training;
[0096] S14, calculate the Actor network and Critic network losses and update the network parameters;
[0097] S15, End.
[0098] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0099] In the present invention, the vehicle network spectrum resource sharing method designed for the queue system can optimize the link performance, and manage the interference between vehicle network pairs through intelligent resource allocation design. Each intelligent agent can make different power choices to share spectrum resources to maximize the total transmission rate of the link and the transmission success rate of the link payload, thereby improving the sharing stability of the queue system. BRIEF DESCRIPTION OF THE DRAWINGS
[0100] Figure 1 A schematic diagram of the system model structure of a vehicle network spectrum resource sharing method for a queuing system proposed by the present invention;
[0101] Figure 2 A schematic diagram of the interaction between an intelligent agent and the environment in an Internet of Vehicles (IoV) of a method for sharing spectrum resources in an IoV system for a queuing system proposed by the present invention;
[0102] Figure 3 This is a result diagram showing how the reward values of different algorithms of a spectrum resource sharing method for a queuing system of an Internet of Vehicles proposed by the present invention change with the number of iterations;
[0103] Figure 4 This is a graph showing the probability of successful transmission of V2V link loads under different load sizes for a vehicle network spectrum resource sharing method for a queuing system proposed by the present invention;
[0104] Figure 5 This is a result diagram of the total transmission rate of the V2I link under different load sizes of a vehicle network spectrum resource sharing method for a queuing system proposed by the present invention;
[0105] Figure 6 This is a graph showing the variation of the residual load of the V2V link over time using the MADDPG algorithm when the load size of the vehicle network spectrum resource sharing method for a queuing system proposed by the present invention is 8*1060 bytes;
[0106] Figure 7 This is a graph showing the variation of the residual load of the V2V link of the MAPPO algorithm over time when the load size of the vehicle network spectrum resource sharing method for a queuing system proposed in the present invention is 8*1060 bytes. DETAILED DESCRIPTION
[0107] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0108] See also Figure 1-Figure 2 The present invention provides a technical solution: a vehicle network spectrum resource sharing method for a queuing system, comprising:
[0109] The spectrum allocation decision is obtained by using a deep reinforcement learning algorithm to train the vehicle platoon system model based on the real-time network status;
[0110] Allocate spectrum resources to maximize the total transmission rate of the vehicle-to-infrastructure link and ensure that the payload of the vehicle-to-vehicle link achieves an optimized delivery success rate under the desired delay constraints.
[0111] Specifically: a V2X system consisting of multiple vehicle queues and an RSU, wherein the PL and the RSU perform periodic communication through a V2I link, and the PMs and the PL of the queue in which they are located perform periodic message transmission through a V2V link. Multiple V2V links in the queue share the spectrum occupied by the V2I link. The optimization goal of this embodiment is to maximize the capacity of the V2I link while ensuring the payload transmission reliability of the V2V link to achieve periodic message sharing.
[0112] In the scenario of vehicle platooning, this application proposes a spectrum resource allocation scheme based on multi-agent reinforcement learning. The purpose is to optimize spectrum resource allocation and ensure high probability delivery of V2V payload while maximizing V2I link capacity;
[0113] The spectrum resource sharing of multiple V2V links is modeled as a multi-agent problem, and the decision-making process of each agent is modeled as a Markov process. A PPO-based distributed power allocation algorithm is proposed in a continuous action environment.
[0114] like Figure 1As shown in the figure, the cellular-based multi-queue vehicle communication network model consists of one RSU, M vehicle queues, M V2I links and N V2V links, where each vehicle queue has O vehicles. The sets of V2I links and V2V links are represented by M = {1, ..., M} and N = {1, ..., N} respectively. V2V communication provides periodic mutual transmission of vehicle information between PM and PL, ensuring driving safety. V2I communication provides support for applications that require large amounts of data to be transmitted between PL and RSU. Orthogonal frequency division multiplexing multiple access technology is used in the model. It is assumed that the V2I link for uploading data has been pre-allocated with orthogonal spectrum sub-bands, with a total of I sub-bands, of which the mth V2I link occupies the i-th sub-band, the V2I link transmission power is a fixed value, and the V2V link can share the spectrum resources allocated to the V2I link. Assume and are the transmission power of the transmitting end of the m-th V2I link and the transmission power of the n-th V2V link on the i-th subband, respectively, which are subject to the following constraints:
[0115]
[0116] Define a Boolean spectrum selection variable ρ n [i], which indicates whether the nth V2V link is transmitted on the i-th subband. If so, then ρ n [i] = 1, otherwise 0, and each V2V link can only access one orthogonal subband at most, that is, Σ n ρ n [i]≤1. Then for the V2I link, the calculation formula for the signal-to-noise ratio of the mth V2I link on the i-th subband is
[0117]
[0118] Where g n,m [i] is the interference channel gain from V2V link n to V2I link m on subband i, g m,B [i] is the channel gain of V2I link m transmitted on the i-th subband, σ 2 is the noise power.
[0119] For the V2V link, the calculation expression of the SINR received by the nth V2V link on the i-th subband is:
[0120]
[0121]
[0122] Where g n [i] represents the channel power gain of V2V link n on the i-th subband. n[i] is the interference power, g m,n [i] represents the interference channel gain of V2I link m to V2V link n on the i-th subband, g n',n [i] represents the interference channel gain of V2V link n' to V2V link n on the i-th subband, represents the transmit power of the V2V link n'.
[0123] According to Shannon's formula, the transmission rate of V2I link m on subband i can be expressed as
[0124]
[0125] Where W is the sub-band spectrum bandwidth.
[0126] Similarly, the channel capacity calculation formula for the nth V2V link on the ith subband is:
[0127]
[0128] This embodiment considers a multi-objective optimization problem, maximizing the total transmission rate of the V2I link and the reliability of the V2V link at the same time. The optimization objectives are selected to reflect the high data transmission rate and reliable message delivery requirements of the applications of the V2I and V2V links, respectively. V2I links are often used to support high-transmission-rate entertainment services, so an appropriate design goal is to maximize their and capacity in represents the minimum tolerable throughput. The V2V link is mainly used to ensure the reliable transmission of vehicle data, where vehicles generate data periodically at different frequencies. Let B represent the size of the periodically generated V2V payload in bits. Δt is the channel coherence time, and t is the index of the coherence time. Within the constraint time T, the probability of successful transmission of the payload of a data packet of size B is:
[0129]
[0130] In summary, the resource allocation problem can be described as follows: for all n∈N, m∈M, the transmission power of the V2V link is a continuously adjustable variable At the same time, maximize the total transmission rate of all V2I links and the probability of successful transmission of V2V link payload. That is, the optimization objectives and constraints are:
[0131]
[0132]
[0133] 0≤t≤T
[0134]
[0135] Furthermore, in the spectrum resource allocation scenario of the Internet of Vehicles, multiple V2V links attempt to access the limited spectrum occupied by the V2I link, which can be modeled as a multi-agent deep reinforcement problem. Each PM is an agent that interacts with the unknown communication environment to gain experience, make decisions based on experience, and feedback to the environment to get rewards.
[0136] Markov decision model, in order to use multi-agent deep reinforcement learning methods to solve problems. First, we need to establish a Markov decision process (MDP), which mainly consists of a state space (StateSpace, S), an action space (Action Space, A) and a reward function (Reward Function, R). At time slot t, the environment state is The agent is based on the current state of the environment Adopt a strategy The environment changes to the next state Feedback rewards
[0137] State space, for each agent, only its own state can be obtained and actions The global channel conditions and the actions of other agents are unknown, so the local information state observed by agent n has its own channel gain g n [i], the interference channel power gain g of V2V link n' to V2V link n n`,n [i], interference channel gain g of V2V link n to V2I link m on subband i n,m [i], channel gain g of the V2I link m to the base station in the i-th subband m,B [i], and the interference noise power I from all other links n [i] In addition to this channel information, because the optimization objective also takes into account the transmission success rate of the V2V payload, the agent also needs to consider its own remaining transmission load D n and the remaining transmission time T n , which is expressed as follows:
[0138]
[0139] Among them, G n [i] = {g n [i],g n`,n [i],g n,B [i],g m,B[i]}.
[0140] Action space, based on the observed state, each agent will make a decision on the continuous control of the transmission power. In most of the literature, the transmission power of the V2V link transmitter is a continuous variable, and its value range is The action decision of agent k is represented as:
[0141] Reward function,The goal of the reward function is to maximize the total transmission rate of all V2I links on the one hand, and to increase the probability of successful V2V payload delivery within time T on the other hand. The first goal is to maximize the total transmission rate of all V2I links at time slot t. The reward function is defined as
[0142]
[0143] For each agent n, the reward L n Set to the transmission rate of the V2V link until the payload is fully delivered, and then let the reward be a constant β, which is greater than the maximum V2V link transmission rate. Therefore, the reward function for the successful transmission of the V2V link payload at time step t is defined as:
[0144]
[0145] The goal of agent learning is to maximize the total transmission rate of the V2I link while improving the transmission success rate of the V2V link payload. Therefore, at time slot t, the total optimization objective function set according to the formula is as follows:
[0146]
[0147] Where λ and β are the weights of the V2I link capacity and V2V link transmission rate, respectively, and λ+β=1. The values of λ and β correspond to different service requirements. If λ approaches zero and β approaches 1, the agent pays more attention to the total transmission rate of the V2I link, and the platoon vehicles need more high bandwidth required for entertainment applications such as video streaming. On the contrary, the platoon vehicles have higher requirements for driving safety, and the agent will focus on improving the rate at which the V2V link transmits information. Of course, if the user has no special requirements, λ and β can be set to 0.5.
[0148] Furthermore, the multi-agent reinforcement learning algorithm based on proximal policy optimization (PPO) adopts a centralized training and distributed execution framework. The algorithm framework is as follows Figure 2 As shown in the figure, it is mainly composed of an actor network with parameter θ and a critic network with parameter φ. The Actor network and the Critic network have strategy functions π(a k|s k ,θ) and the value estimation function V(s k ,φ), where θ and φ are the weights of the Actor network and the Critic network;
[0149] According to the observed state s t , input to the Actor network, thereby obtaining action a t . Perform action a t After that, the agent will receive an instantaneous reward r t and the state s at the next moment t+1 , and stores the experience in the experience replay pool. During the training process, the agent interacts with the environment many times, generating and collecting a batch of experience {s t ,a t ,r t ,s t+1}. The Actor network and the Critic network are optimized using the collected experience. During the optimization process, the Actor network and the Critic network calculate the loss values of the strategy and the global state value function based on the empirical data to update the parameters θ and φ. The loss value of the Actor network is calculated by the following formula;
[0150]
[0151] in and π θ denote the old strategy and the current strategy respectively, S(s t ) indicates the state is s t The action entropy at the time of training increases exploration during the training phase to avoid falling into the local optimum; the meaning of the clip function is that if the first term is less than the second term, 1-ε is output, and if the first term is greater than the third term, 1+ε is output. The purpose is to limit the probability ratio to a reasonable range. ε is a hyperparameter, generally 0.1. ψ is a hyperparameter that balances exploration and utilization. To weigh the deviation and variance of the policy gradient, The state value V(s) is used in the form of Generalized Advantage Estimation (GAE). t ) is estimated, and the estimated deviation and variance are weighted and balanced by introducing the parameter λ. The specific formula is:
[0152]
[0153]
[0154] Define V φ (s t) is the state value fitted by the Critic network of the agent, and the Critic network loss function is calculated by the following formula:
[0155]
[0156] The Actor network is updated by maximizing the objective function, and the Critic network is updated by minimizing the loss function. The specific algorithm flow is shown in Algorithm 1:
[0157] 1 Initialize the actor network and critic network parameters
[0158] 2 Initialize the environment and experience pool
[0159] 3For e=1to episode
[0160] 4Reset the Internet of Vehicles environment
[0161] 5. Update vehicle position and large-scale fading
[0162] 6For step=1to 100
[0163] 7 Action Sampling
[0164] 8 Observe the initial environment state s of the vehicle agent t
[0165] 9 Input the state into the Actor network to get action a t
[0166] 10 Agent performs action a t Then get the state s at the next moment t+1 And get the reward value r t
[0167] 11 will (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool, End
[0168] 12 Calculate the discounted return and advantage function according to the formula
[0169] 13 Extract training samples from the experience pool for training
[0170] 14 Calculate the Actor network and Critic network losses and update network parameters
[0171] 15End.
[0172] Experimental example
[0173] See also Figure 3-Figure 7, the simulation experiment was carried out using python language and PyTorch through the Anaconda platform. In the urban road simulation, a base station is considered in the middle, and the base station is surrounded by two 375-meter-wide and 649-meter-long two-way dual-lane vehicle network topology scenarios, and the vehicle queue is randomly generated at the center of the lane. The vehicle queue model sets the traffic environment according to the urban case of 3GPP TR36.885. The parameters such as vehicle density, speed, moving direction, vehicle channel, and V2V data traffic are introduced in detail in the model. Table 1 shows the main simulation parameters and the channel model of V2V and V2I links. In the multi-agent reinforcement learning spectrum resource allocation algorithm based on PPO, the deep reinforcement learning parameters are set as shown in Table 1, where the learning rate of the Actor network and the Critic network is 0.001;
[0174] Table 1 Simulation parameter settings
[0175]
[0176]
[0177] Compared algorithms, 1. Multi-agent deep reinforcement learning based on discrete action space (MADQN): In this algorithm, a single agent follows the DQN algorithm, that is, each agent selects discrete actions from the action set {0, 0.003W, 0.006W, 0.012W, 0.025W, 0.5W, 0.1W, 0.2W} based on the current Internet of Vehicles environment. The DQN algorithm has only one neural network, namely the Q network. The DQN algorithm also adopts a dual network structure of experience pool and evaluation network and target network.
[0178] 2. Random optimization algorithm (Random): At each time step, the vehicle selects the transmission power on the V2V link in a uniformly distributed random manner.
[0179] 3. Multi-Agent Deep Reinforcement Learning in Continuous Action Space (MADDPG)
[0180] Simulation results, according to Figure 3 By comparing the results, it can be seen that in the set vehicle simulation environment, the reward values of the four algorithms used perform differently. The MAPPO algorithm performs well in the convergence of reward values, with fast and stable convergence speed, and achieves the highest reward value. The MADDPG algorithm can converge and stabilize after about 1500 iterations, but the effect is not as good as the MAPPO algorithm. However, the MADQN algorithm finds it difficult to converge in this environment, and the reward value fluctuates greatly, showing a relatively poor performance. In comparison, the reward value of the Random algorithm shows a state of slight fluctuations. In summary, the algorithm provided in this embodiment is better than the other three algorithms in terms of stability and effect:
[0181] Figure 4 The figure shows the probability of successful transmission of the V2V link payload as the vehicle load size gradually increases. The figure shows that as the load size increases, the performance of each algorithm gradually decreases. The high initial transmission success rate is because the system has abundant spectrum resources, which can be reasonably allocated to complete the load transmission. In addition, the figure also shows that the MAPPO algorithm has a higher probability of successful load transmission than other algorithms under different load sizes, indicating that the algorithm still has better performance after the vehicle environment changes.
[0182] Figure 5 The figure shows the change of the total transmission rate of the V2I link when the vehicle load gradually increases. As the vehicle load gradually increases, the total capacity of the V2I link of each algorithm will decrease. This is because the increase in load will cause the V2V link to transmit the load for a longer time, which will in turn cause stronger interference to the V2I link, resulting in a decrease in the performance of the V2I link. Under the same load conditions, the total transmission rate of the V2I link of the algorithm proposed in this embodiment is still higher than that of other baseline algorithms.
[0183] In order to better reflect the advantages of MAPPO algorithm training, Figure 6 and Figure 7 In the figure, we compared the changes in the size of the remaining payload transmitted by four agents per unit time in MAPPO and MADDPG. In order to adapt to more situations, we chose the payload size to be 8*1060 bytes. Figure 6 The figure shows the change of the remaining load of MADDPG. It can be seen from the figure that links 1, 3 and 4 have faster transmission rates and all complete the transmission in about 8ms, but link 2 succeeds in transmission after 1ms. Figure 7 In the example of using the MAPPO algorithm proposed in this embodiment, the four links all showed a high transmission speed at the beginning, and then the speed of link 2 slowed down, so that the other three links could complete the transmission quickly. After the transmission of the three links was completed, link 2 completed its load transmission at a high rate. This proves that the MAPPO algorithm is superior to the MADDPG algorithm and other algorithms;
[0184] In summary, this embodiment proposes a queue-oriented multi-agent reinforcement learning algorithm to optimize the performance of V2V and V2I links. V2V and V2I links use the same spectrum, so intelligent resource allocation design is required to manage interference. Each agent makes different power selections to share spectrum resources to maximize the total transmission rate of the V2I link and the transmission success rate of the V2V link payload. The simulation results show that the resource sharing method of this application can achieve good performance in a variety of variable situations, which is better than other baseline algorithms.
[0185] In the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance; the term "plurality" refers to two or more, unless otherwise clearly defined. The terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0186] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A spectrum resource sharing method for Internet of Vehicles for queuing system. It is characterized in that include: The spectrum allocation decision is obtained by using a deep reinforcement learning algorithm to train the vehicle platoon system model based on the real-time network status; Allocate spectrum resources to maximize the total transmission rate of the vehicle-to-infrastructure link and ensure that the payload of the vehicle-to-vehicle link achieves an optimized delivery success rate under the desired delay constraints.
2. According to the method for sharing spectrum resources of Internet of Vehicles for a queuing system in accordance with claim 1, It is characterized in that The reinforcement learning method for training the vehicle platoon system model using a deep reinforcement learning algorithm is: The cellular-based multi-queue vehicle communication network model consists of one RSU, M vehicle queues, M V2I links and N V2V links, where each vehicle queue has O vehicles; The sets of V2I links and V2V links are represented by M = {1, ..., M} and N = {1, ..., N} respectively; V2V communication provides periodic vehicle information transmission between PM and PL; V2I communication supports applications that require large amounts of data to be transmitted between PL and RSU; Orthogonal frequency division multiplexing (OFDM) is used in the model. It is assumed that the V2I link for uploading data has been pre-allocated with orthogonal spectrum sub-bands. There are I sub-bands in total, of which the mth V2I link occupies the ith sub-band. The transmission power of the V2I link is a fixed value, and the V2V link can share the spectrum resources allocated to the V2I link. set up and are the transmission power of the transmitting end of the m-th V2I link and the transmission power of the n-th V2V link on the i-th subband, respectively, which are subject to the following constraints: A Boolean spectrum selection variable ρ is defined n [i], which indicates whether the nth V2V link is transmitted on the i-th subband. If so, then ρ n [i] = 1, otherwise 0, and each V2V link can only access one orthogonal subband at most, that is: ∑ n ρ n [i]≤1; Then for the V2I link, the calculation formula for the signal-to-noise ratio of the mth V2I link on the i-th subband is: Where g n,m [i] is the interference channel gain from V2V link n to V2I link m on subband i, g m,B [i] is the channel gain of V2I link m transmitted on the i-th subband, σ 2 is the noise power; For the V2V link, the calculation expression of the SINR received by the nth V2V link on the i-th subband is: Where g n [i] represents the channel power gain of V2V link n on the i-th subband, I n [i] is the interference power, g m,n [i] represents the interference channel gain of V2I link m to V2V link n on the i-th subband, g n',n [i] represents the interference channel gain of V2V link n' to V2V link n on the i-th subband, represents the transmission power of the V2V link n'; According to Shannon's formula, the transmission rate of V2I link m on subband i can be expressed as Where W is the sub-band spectrum bandwidth; Similarly, the channel capacity calculation formula for the nth V2V link on the ith subband is: Among them, the high data transmission rate and reliable message delivery requirements of V2I and V2V link applications are the optimization goals; That is, the capacity of the V2I link in represents the minimum tolerable throughput. The V2V link is mainly used to ensure the reliable transmission of vehicle data, where vehicles generate data periodically at different frequencies. Let B represent the size of the periodically generated V2V payload in bits. Δt is the channel coherence time, and t is the index of the coherence time. Within the constraint time T, the probability of successful transmission of the payload of a data packet of size B is: For all n∈N, m∈M, the transmission power of the V2V link is a continuously adjustable variable At the same time, the total transmission rate of all V2I links and the probability of successful transmission of V2V link payload are maximized. The optimization objectives and constraints are: st:∑ n ρ n [i]≤1 0≤t≤T 3. According to the method for sharing spectrum resources of Internet of Vehicles for a queuing system in accordance with claim 2, It is characterized in that It also includes establishing a Markov decision process (MDP), which is mainly composed of a state space (S), an action space (A), and a reward function (R); At time slot t, the environment state is The queue members in the Internet of Vehicles queue are intelligent agents, which can Adopt a strategy The environment changes to the next state Feedback rewards 4. According to claim 3, a vehicle network spectrum resource sharing method for a queuing system, It is characterized in that The state space is: the observed local information state of agent n has its own channel gain g n [i], the interference channel power gain g of V2V link n' to V2V link n n`,n [i], interference channel gain g of V2V link n to V2I link m on subband i n,m [i], channel gain g of the V2I link m to the base station in the i-th subband m,B [i], and the interference noise power I from all other links n [i]; In addition to this channel information, because the optimization objective also takes into account the transmission success rate of the V2V payload, the agent also needs to consider its own remaining transmission load D n and the remaining transmission time T n , which is expressed as follows: Among themG n [i]={g n [i],g n`,n [i],g n,B [i],g m,B [and]}.
5. According to the method for sharing spectrum resources of Internet of Vehicles for a queuing system in accordance with claim 3, It is characterized in that The action space is: Based on the observed state, each agent will make a decision on the continuous control of the transmission power. The transmission power of the transmitter of the V2V link is a continuous variable with a value range of The action decision of agent k is represented as:
6. A vehicle network spectrum resource sharing method for a queuing system according to claim 3, It is characterized in that The reward function is: Optimize the total transmission rate of all V2I links and on the other hand improve the success probability of V2V payload delivery within time T, where the optimal target of the total transmission rate of all V2I links is at time slot t, maximizing the total transmission rate of all V2I links. The reward function is defined as For each agent n, the reward L n Set to the transmission rate of the V2V link until the payload is fully delivered, after which let the reward be a constant β, which is greater than the maximum V2V link transmission rate; Therefore, the reward function for successful transmission of the V2V link payload at time step t is defined as: The goal of agent learning is to maximize the total transmission rate of the V2I link while improving the transmission success rate of the V2V link payload. Therefore, at time slot t, the total optimization objective function set according to the formula is as follows: Where λ and β are the weights of the V2I link capacity and the V2V link transmission rate, respectively, and λ+β=1. The values of λ and β correspond to different service requirements. If λ approaches zero and β approaches 1, the agent will pay more attention to the total transmission rate of the V2I link, and the platoon vehicles will need higher bandwidth. On the contrary, the platoon vehicles have higher requirements for driving safety, and the agent will focus on improving the rate at which the V2V link transmits information. If the user has no special requirements, λ and β can be set to 0.
5.
7. The vehicle network spectrum resource sharing method for a queuing system according to claim 3, It is characterized in that It also includes the use of a multi-agent reinforcement learning algorithm based on proximal policy optimization (PPO) to adopt a centralized training and distributed execution framework to perform reinforcement learning on the overall optimization objective function, including: The distributed execution framework consists of an actor network with parameter θ and a critic network with parameter φ. The Actor network and the Critic network have strategy functions π(a k |s k ,θ) and the value estimation function V(s k , φ), where θ and φ are the weights of the Actor network and the Critic network; According to the observed state s t , input to the Actor network, thereby obtaining action a t ; Execute action a t After that, the agent will receive an instantaneous reward r t and the state s at the next moment t+1 , and store the experience in the experience replay pool; During the training process, the agent interacts with the environment many times, generating and collecting a batch of experience {s t , a t , r t ,s t+1 }; Use the collected experience to optimize the Actor network and Critic network; During the optimization process, the Actor network and the Critic network calculate the loss values of the strategy and the global state value function respectively based on the empirical data to update the parameters θ and φ; The loss value of the Actor network is calculated by the following formula: in and π θ denote the old strategy and the current strategy respectively, S(s t ) indicates the state is s t The action entropy at that time increases exploration during the training phase to avoid falling into the local optimum; The meaning of the clip function is that if the first term is less than the second term, 1-ε is output, and if the first term is greater than the third term, 1+ε is output. The purpose is to limit the probability ratio to a reasonable range; ε is a hyperparameter, usually 0.1; ψ is a hyperparameter that balances exploration and exploitation; To balance the bias and variance of the policy gradient, The state value V(s) is used in the form of Generalized Advantage Estimation (GAE). t ) is estimated, and the estimated deviation and variance are weighted and balanced by introducing the parameter λ. The specific formula is: Define V φ (s t ) is the state value fitted by the Critic network of the agent, and the Critic network loss function is calculated by the following formula: The Actor network is updated by maximizing the objective function, and the Critic network is updated by minimizing the loss function.
8. A vehicle network spectrum resource sharing method for a queuing system according to claim 7, It is characterized in that The multi-agent reinforcement learning algorithm adopts the centralized training algorithm process steps as follows: S1. Initialize the actor network and critic network parameters; S2, initialize the environment and experience pool; S3, For e = 1 to episode; S4, reset the Internet of Vehicles environment; S5, update vehicle position and large-scale fading; S6, For step = 1 to 100; S7, action sampling; S8, observe the initial environment state s of the vehicle agent t ; S9. Input the state into the Actor network to get action a t ; S10, the agent performs action a t Then get the state s at the next moment t+1 And get the reward value r t ; S11, will (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool; S12, calculating the discounted return and advantage function according to the formula; S13, extracting training samples from the experience pool for training; S14, calculate the Actor network and Critic network losses and update the network parameters; S15, End.
Citation Information
Cited By
Double-stage collaborative optimization method based on multi-agent deep reinforcement learning
CN119922575A
Vehicle formation communication resource allocation method and device and electronic equipment
CN120857283A
Vehicle platoon communication resource allocation method and device, and electronic equipment
CN120857283B
Internet of vehicles resource scheduling method, device and system, and storage medium
CN120935799A
