Ultra-dense Internet of Vehicles adaptive resource allocation method based on deep reinforcement learning

Abstract

The invention discloses an ultra-dense Internet of Vehicles adaptive resource allocation method based on deep reinforcement learning, and the method comprises the steps: building a communication model of the ultra-dense Internet of Vehicles, the communication model comprises a base station, a transmitting vehicle and a receiving vehicle, the base station exchanges data with the vehicles through a V2I link, and the vehicles communicate with each other through a V2V link; determining a weighted interference model taking the receiving vehicle as the center according to the interference relationship between the vehicles in the communication model; according to the communication model and the weighted interference model, a resource allocation model of the communication model is established, and the resource allocation model is an optimization problem which aims at the maximum resource reuse rate and no conflict of allocated resources under a certain constraint condition; converting the optimization problem into a Markov decision process problem; and solving by using a FedAvg-AC algorithm to obtain a resource allocation strategy of the ultra-dense Internet of Vehicles. The resource allocation method has the beneficial effect that the resource allocation efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle networks, and specifically, relates to an ultra-dense vehicle network adaptive resource allocation method based on deep reinforcement learning. Background Art

[0002] With the rapid development of intelligent transportation, Internet of Vehicles (IoV) technology has become an increasingly important component of modern urban traffic management. As more and more vehicles are equipped with advanced communication and sensing capabilities, these intelligent vehicles can exchange and collaborate in real time through vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications, thus forming an ultra-dense Internet of Vehicles (IoV). In this ultra-dense IoV network, overlapping communication ranges inevitably lead to interference between vehicles. In addition, the different interference capabilities between different vehicles produce a phenomenon called differential interference (or differential interference) in dense IoV networks. This differential interference leads to data packet loss, reduced transmission rate, and increased transmission delay, which significantly complicates channel allocation and power allocation in IoV networks. Therefore, it is an urgent and critical challenge to determine an effective solution to mitigate the differential interference caused by overlapping vehicles and ensure reliable V2V communication in ultra-dense IoV deployments.

[0003] In recent years, the contradiction between communication demand and communication resources in the Internet of Vehicles has become increasingly intensified. Many researchers have proposed a variety of resource allocation optimization schemes to solve the network resource allocation problem in the Internet of Vehicles. Someone proposed a spectrum access scheme based on game theory for opportunistic spectrum access in cognitive radio vehicle-mounted self-organizing networks, which enables vehicles to access licensed frequency bands in a distributed manner. In order to cope with the challenge of large-scale data transmission in the current network architecture, someone used the Long Short-Term Memory (LSTM) network to predict dynamic throughput and developed a resource allocation algorithm based on genetic algorithm, which greatly improved the resource utilization efficiency. For high-speed fleets, a joint optimization problem is proposed in the prior art, which aims to simultaneously optimize the task offloading decision, communication and computing resource allocation scheme of the fleet and base station to maximize utility and network throughput. A multi-agent neighboring strategy optimization algorithm Ly-MAPPO is used to efficiently manage cooperation overhead. The algorithm relies on local observation and uses Lyapunov optimization technology for resource management in the Internet of Vehicles. In addition, other communication infrastructures can also provide assistance for resource management in vehicle networks. For example, an optimization algorithm based on a dynamic digital twin model is based on a two-stage incentive mechanism derived from the Stackelberg game and the Alternating Direction Method of Multipliers (ADMM) distributed algorithm. The algorithm aims to optimize resource allocation in drone-assisted Internet of Vehicles to improve vehicle user satisfaction and overall energy efficiency. However, these schemes have high computational overhead, low efficiency, and slow response, and cannot meet the real-time dynamic resource requirements of the Internet of Vehicles. Therefore, there is an urgent need to implement a new robust computing method to solve the resource allocation problem in the Internet of Vehicles.

[0004] With the development of machine learning, reinforcement learning (RL) technology has become an effective means to manage and control vehicle network resources by making intelligent decisions. There is a hybrid task offloading scheme using deep reinforcement learning in the prior art, focusing on vehicle-to-edge (V2E) and vehicle-to-vehicle (V2V) offloading to optimize resource utilization and management under strict delay constraints and dynamic task characteristics. The prior art integrates double deep Q-network (DDQN) and deep deterministic policy gradient to recommend actions with discrete-continuous hybrid variables, aiming to reduce computational costs. In order to cope with the challenges brought by dynamic environments, the technology of applying reinforcement learning techniques to handle high-dimensional and continuous state-action spaces has emerged. However, the existing reinforcement learning (RL) models in the above studies are centralized and do not consider the unreliable communication connections typical of ultra-dense and dynamic vehicle networks. A distributed RL framework based on MultiIntelligent Deep Deterministic Policy Gradient (MADDPG) is used to manage wireless resources in the Internet of Vehicles. In order to improve the performance of RL in the Internet of Things, a semi-synchronous federated learning protocol has emerged. Later, an algorithm based on federated learning emerged, which balanced resource consumption by considering node characteristics and resource load. For resource management of air-ground integrated networks in 6G vehicular networks, a resource allocation method using federated learning and dueling dual-depth Q networks (FedD3QN) has emerged in the prior art, which can reduce the link delay from vehicle to infrastructure. However, these studies did not consider channel and power allocation under differential interference in ultra-dense deployment. In ultra-dense vehicular networks, the overlap of vehicle communication ranges leads to different interference capabilities of the same vehicle to different vehicles, and this differential interference leads to a decrease in data transmission rate, packet loss rate, and network throughput. These challenges seriously affect the reliability of vehicle communications and complicate resource allocation in ultra-dense vehicular networks. Summary of the invention

[0005] In view of this, an object of the present invention is to provide an adaptive resource allocation method for ultra-dense Internet of Vehicles based on deep reinforcement learning.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A method for adaptive resource allocation in ultra-dense Internet of Vehicles based on deep reinforcement learning, comprising:

[0008] A communication model of an ultra-dense vehicle network is established, wherein the communication model includes a base station and K vehicles, wherein the K vehicles include F sending vehicles and R receiving vehicles, wherein the base station exchanges data with the vehicle via a V2I link, and the vehicles communicate with each other via a V2V link, wherein the sending vehicle sends a signal to the receiving vehicle, wherein the number of V2V links is N;

[0009] According to the interference relationship between vehicles in the communication model, a weighted interference model centered on the receiving vehicle is determined;

[0010] According to the communication model and the weighted interference model, a resource allocation model of the communication model is established, wherein the resource allocation model is an optimization problem with the goal of maximizing resource reuse rate and non-conflicting allocated resources under certain constraints;

[0011] Converting the optimization problem into a Markov decision process problem;

[0012] The FedAvg-AC algorithm is used to solve the Markov decision process problem and obtain the resource allocation strategy for ultra-dense Internet of Vehicles.

[0013] Furthermore, a weighted interference model centered on the receiving vehicle is determined, specifically including:

[0014] For each receiving vehicle, determining the interference weight of each sending vehicle to the receiving vehicle and generating a weight matrix;

[0015] According to the plurality of sending vehicles, the plurality of receiving vehicles and the weight matrix, an interference weight graph G={F, R, W} is established, wherein F is a vertex set, representing a set of sending vehicles; R is an edge set, representing a set of receiving vehicles, and W represents a weight matrix;

[0016] The interference weight map is transformed into the correlation matrix H g Represents, where the incidence matrix H g A row of represents the interference weight of a sending vehicle to each receiving vehicle, and the correlation matrix H g One column represents the interference weight of a certain receiving vehicle from each sending vehicle;

[0017] According to the correlation matrix and resource allocation scheme, the interference level v of each receiving vehicle is determined r , where r = 1, 2, ..., R, and R represents the total number of received vehicles;

[0018] Determine the overall interference level I of ultra-dense vehicular networks.

[0019] Furthermore, the interference level of each receiving vehicle is expressed as:

[0020]

[0021] Among them, ν r represents the interference level of the rth receiving vehicle, c represents the resource allocation of the sending vehicle; h r Represents the incidence matrix H g The combination of all elements in the rth column of r Indicates c and h r The result of multiplying and summing the elements at the corresponding positions in c; c[i] represents the resources allocated to the i-th node in c, that is, the resources allocated to the i-th sending vehicle, ν r = 0 means there is no interference in the rth receiving vehicle, otherwise, ν r ≠0.

[0022] Furthermore, the optimization problem of the resource allocation model is expressed as:

[0023]

[0024] Among them, λ1,λ2∈(0,1), M represents the set of channels, M={1,2,…,M}, M represents the total number of channels, η r represents the resource reuse rate, P represents the power set, P={p1,p2,…,p F}, p f represents the signal transmission power selected by the f-th transmitting vehicle when transmitting a signal; N = {1, 2, ..., N} represents the set of V2V links; η0 represents the resource reuse rate of the entire network;

[0025] represents the data transmission rate of the nth V2V link at time t, R min represents the minimum transmission rate of the channel, t∈{1,2,...,T}, T represents the total time;

[0026] represents the signal-to-noise ratio of the nth V2V link at time t; γ0 represents the signal-to-noise ratio threshold;

[0027] l n,m Indicates the selection of the mth channel by the nth V2V link. When the nth V2V link selects the mth channel, l n,m =1; otherwise, when the nth V2V link does not select the mth channel, l n,m =0.

[0028] Furthermore, converting the optimization problem into a Markov decision process problem includes:

[0029] Treat each V2V link as an intelligent agent;

[0030] According to the vehicle network information at time t, the state space at time t is represented as S = {MF ,γ,M,P},M F represents the channel selection of F sending vehicles in the communication model, γ represents the set of signal-to-noise ratios of all channels, M represents the total number of channels, and P represents the set of transmission powers selected by all sending vehicles;

[0031] The set of action spaces is represented as: A = {a1, a2, ..., a T}, a t = {l n,m ,p f,m} represents the action at time t, which includes the channel selection l n,m and power selection p f,m , p f,m represents the signal transmission power when the f-th transmitting vehicle selects the m-th channel, f∈{1,2,...,F}, n∈{1,2,...,N}, m∈{1,2,...,M};

[0032] The reward value obtained by each agent based on the action at time t is:

[0033] The action value function is: Among them, E represents the expectation, the network state space S0 represents the initialization value of the state space; π represents the strategy; ρ represents the attenuation factor; Q π represents the cumulative reward when starting the network state space S0 under strategy π.

[0034] Furthermore, the FedAvg-AC algorithm specifically includes:

[0035] Construct a global model and a local AC model. Each agent corresponds to a local AC model. Each agent uses the global model to construct a local AC model. During the global model training process, each agent replays the local experience from the corresponding local experience pool D n Randomly select a preset number of data sets B n To update the local AC model corresponding to the agent; after a round of learning, the parameters of the global model are obtained by weighted averaging the parameters of each local AC model.

[0036] Furthermore, the method specifically comprises:

[0037] The parameter w of the global model at time t-1 is t-1 As the parameter of each local AC model at time t-1 n∈{1,2,...,N};

[0038] According to the parameters of each local AC model at time t-1 Determine the loss function of each local AC model at time t-1

[0039] Computing Gradients Get the parameters of the local AC model at time t Update local AC model;

[0040] For each local AC model parameter Perform weighted averaging to obtain the global model parameter w at time t t ;

[0041] The global model parameter w at time t is t Sent to each local AC model for the next round of training.

[0042] Furthermore, for each local AC model parameter w t n Perform weighted averaging to obtain the global model parameter w at time t t , expressed as: Among them, |B n | represents data set B n The amount of data in .

[0043] Furthermore, the local AC model includes an actor network π(s, a; θ) and a critic network q(s, a; ψ), where the actor network is used to identify the action strategy in the vehicle network and its parameter is θ, and the critic network is used to evaluate the quality of the strategy and its parameter is ψ;

[0044] The gradient descent method is used to update θ and ψ, thereby updating the parameters of the AC model. Specifically,

[0045]

[0046] Among them, δ t =q t -(r t +ρ·q t+1 ), α and β represent the learning rate, q t =q(s t ,a t ,ψ t ), r t represents the reward value at time t, q t represents the critic network at time t.

[0047] The beneficial effects of the present invention are:

[0048] The present invention first evaluates the communication interference between vehicles in V2V scenarios and constructs a weighted interference model centered on the receiver (i.e., the interference receiving vehicle) to visualize the interference relationship between various transmitters in a dense vehicle network environment;

[0049] The present invention formulates a resource allocation problem in the presence of differential interference in dense IoVs, defines a combinatorial optimization problem, takes network throughput and resource reuse rate as objective functions, and aims to maximize network throughput and resource reuse rate; and, in order to quantify the degree of conflict in ultra-dense Internet of Vehicles, defines the degree of conflict; finally, by establishing the constraints that the degree of conflict must be zero, the signal-to-noise ratio must be greater than a threshold, and the transmission rate must be greater than a threshold, conflict-free resource allocation is achieved;

[0050] The present invention transforms the combinatorial optimization problem of conflict-free resource allocation in a dense vehicle network environment into a Markov decision process (MDP) model, in which the reward function is complexly designed to comply with constraints such as maximizing network throughput and optimizing resource reuse. Under this MDP framework, the present invention uses an actor-critic (AC) algorithm to solve the combinatorial optimization problem of resource allocation in a high-density vehicle network.

[0051] In order to accelerate the convergence speed of the reinforcement learning algorithm and improve the learning efficiency, the present invention proposes a reinforcement learning algorithm based on the FedAvg-AC (Federated Actor Critic) framework.

[0052] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0054] Figure 1 is the vehicle network scene graph;

[0055] Figure 2 is a schematic diagram of the interference model;

[0056] Figure 3 It is a resource management framework based on FedAvg-AC;

[0057] Figure 4 It is a graph of convergence and different learning rates;

[0058] Figure 5 It is a SINR comparison chart;

[0059] Figure 6 This is a comparison chart of network throughput;

[0060] Figure 7 This is a comparison chart of resource reuse rates. DETAILED DESCRIPTION

[0061] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, rather than for limiting the protection scope of the present invention.

[0062] The present invention proposes a method for adaptive resource allocation of ultra-dense vehicle networks based on deep reinforcement learning, the method comprising:

[0063] Step 1: Establish a communication model for ultra-dense vehicle networks;

[0064] Step 2: Determine a weighted interference model centered on the receiving vehicle based on the interference relationship between vehicles in the communication model;

[0065] Step 3: According to the communication model and the weighted interference model, a resource allocation model of the communication model is established, wherein the resource allocation model is an optimization problem with the goal of maximizing resource reuse rate and non-conflicting allocated resources under certain constraints;

[0066] Step 4: Convert the optimization problem into a Markov decision process problem;

[0067] Step 5: Use the FedAvg-AC algorithm to solve the Markov decision process problem and obtain the resource allocation strategy for ultra-dense Internet of Vehicles.

[0068] Figure 1 is the scene graph of the vehicle network. Figure 1 As shown, the communication model includes a base station and K vehicles, and the K vehicles include F sending vehicles and R receiving vehicles. The sending vehicle (also called a sending end, a sender, a sending end or a sending vehicle, etc.) sends a signal to the receiving vehicle (also called a receiving end, a receiving end, a receiving end or a receiving vehicle). In the present invention, only the interference of the sender to the receiver is considered. The base station exchanges data with the vehicle through the V2I link, and the vehicles communicate with each other through the V2V link, where the number of V2V links is N; N = {1,2,...,N} represents the set of all V2V links; it should be noted that the present invention considers a time slot communication system, and each V2V link can only select one channel to transmit a signal in a time slot, and for the power of the interfering sender (i.e., the sending vehicle), each interfering sender can only select one transmission power. Define M = {1,2,…,M} to represent the set of channels, and M represents the total number of channels. P = {p1,p2,…,p F} represents the power set, p frepresents the signal transmission power selected by the f-th transmitting vehicle when sending a signal, f∈{1,2,...,F}.

[0069]

[0070] Among them, l n,m Indicates the selection of the mth channel by the nth V2V link. When the nth V2V link selects the mth channel, l n,m =1; otherwise, when the nth V2V link does not select the mth channel, l n,m =0.

[0071] The V2V wireless channel follows the free space loss mathematical model, so the channel gain of the nth V2V link in the mth channel in time slot t (also called “time t”) is It can be expressed as:

[0072]

[0073] in, represents the small-scale fading power component of the nth V2V link in the mth channel at time t, and assumes that it obeys an exponential unit mean distribution, It represents the large-scale fading power component of the nth V2V link in the mth channel at time t that is not affected by frequency.

[0074] According to the Rayleigh fading channel model, the signal-to-interference-plus-noise ratio (SINR) of the mth channel at time t is Can be defined as:

[0075]

[0076] in, represents the signal transmission power of the nth V2V link in the mth channel at time t, σ 2 represents the power of additive Gaussian white noise in the channel; and They represent the transmit power and channel gain of the nth V2V link at time t respectively.

[0077] SINR is the ratio of the signal power to the combined interference and noise power in the system, as shown in equation (3). A higher SINR indicates that the receiver can more clearly discern the useful signal amidst reduced interference and noise levels.

[0078] The data transmission rate of the nth V2V link is:

[0079]

[0080] In vehicle networks, SINR is used as a key quality of service (QoS) parameter to evaluate the reliability of data transmission. When the instantaneous SINR of the V2V link drops to the threshold γ o When the SINR exceeds the specified threshold, the receiver has difficulty in accurately decoding the information sent. As a result, this leads to a breakdown in communication between vehicles. Therefore, in order to ensure reliable V2V transmission, it is critical that the SINR exceeds the specified threshold, so the following equation needs to be satisfied:

[0081]

[0082] Next, the interference types in dense vehicle networks are analyzed. The present invention establishes an interference model centered on interference receivers to propose resource allocation problems in ultra-dense vehicle networks.

[0083] Figure 2 is a schematic diagram of the interference model. Figure 2 As shown, the interference model includes 5 receivers (i.e., receiving vehicles, such as Figure 2 R1, R2, ..., R5) and 8 transmitters (i.e., sending vehicles, such as Figure 2 S1, S2, ..., S8), in IoV, the interference weight w is defined at the receiver side:

[0084]

[0085] Among them, P S Represents the transmitter's transmission power, and multiple w can form a weight matrix (i.e. Figure 6 The interference matrix in .

[0086] According to the interference weight of the receiver, interference can be defined into two types, strong interference and weak interference.

[0087] Strong interference: When the interference weight of a single transmitter on a receiver is greater than or equal to the threshold w0, that is, the interference of a single transmitter on a receiver will affect the normal communication of the receiver, this interference phenomenon is called strong interference, w≥w0=1. For example, Figure 2 The interference of S1 to R1 is strong interference.

[0088] Weak interference: When the interference weight of a single transmitter on the receiver is less than the threshold, it is called weak interference, w<w0=1, for example Figure 2 The interference from S8 to R5 is weak interference.

[0089] For the receiver, the weak interference caused by a single transmitter often does not affect the normal communication between vehicles. However, when the receiver is interfered by multiple receivers, it will cause the vehicles to be unable to communicate normally. This interference phenomenon is called cumulative interference. For example, Figure 2 The interference of S5 and S6 on R2. Whether it is cumulative interference or strong interference, it will cause the receiving end to be unable to receive data normally, reduce the data transmission rate, and seriously affect the normal communication between vehicles.

[0090] In order to quantify the interference degree in the interference model, an interference matrix is ​​established and the interference degree is set to measure the interference degree in the in-vehicle Internet.

[0091] Determine a weighted interference model centered on the interference receiving vehicle, including:

[0092] For each interfering receiving vehicle, determining the interference weight of each sending vehicle to the receiving vehicle, and generating a weight matrix;

[0093] According to the plurality of sending vehicles, the plurality of receiving vehicles and the weight matrix, an interference weight graph G={F, R, W} is established, wherein F is a vertex set, representing a set of sending vehicles; R is an edge set, representing a set of receiving vehicles, and W represents a weight matrix;

[0094] Next, the interference weight map is transformed into the correlation matrix H g To represent, its matrix element h(a,e) can be expressed as:

[0095]

[0096] Where h(a,e) represents H g The element in the ath row and eth column, h(a,e)=w means that vertex a is on edge e, and w represents the weight of a on edge e, where a represents an element in F, that is, the sending end; e represents an element in R, that is, the receiving end.

[0097] Figure 1 The interference weight map shown can be represented by the matrix H (i.e., the correlation matrix H g )express:

[0098]

[0099] The interference weight graph can directly display the interference relationship in IoV. When allocating resources, transmitters with strong or cumulative interference cannot be allocated the same resource blocks. For example, S1 on R1 in H cannot be divided into the same resource blocks as S2 and S4.

[0100] To measure the interference in the receiver, we can use the correlation matrix H g and resource allocation scheme to determine the interference level v of each interfering receiving vehicle r , where r = 1, 2, ..., R, and R represents the total number of interference receiving vehicles;

[0101] The present invention only considers the interference of the signal sender to the signal receiver. One receiving end may be interfered by multiple sending end vehicles. The interference degree of each receiving vehicle is expressed as:

[0102]

[0103] Among them, ν r represents the interference level of the rth receiving vehicle, c represents the resource allocation of the sending vehicle; h r Represents the incidence matrix H g The combination of all elements in the rth column of r Indicates c and h r The result of multiplying and summing the elements at the corresponding positions in c; c[i] represents the resources allocated to the i-th node in c, that is, the resources allocated to the i-th sending vehicle, The coefficient of the same resource can be extracted, ν r = 0 means there is no interference in the rth interference receiving vehicle, otherwise, ν r ≠0.

[0104] For example, if the resources allocated in the rth column (e.g., column 1) are (A, A, B, B, C, C, D, D), that is, resource A is allocated to S1 and S2, resource B is allocated to S3 and S4, resource C is allocated to S5 and S6, and resource D is allocated to S7 and S8, that is, c = [A, A, B, B, C, C, D, D] in formula (9), hr = [1, 0.3, 0, 0.5, 0, 0, 0, 0]T in formula (9), then c·h r =1A+0.3A+0.5B=1.3A+0.5B, c[1]=A, then In this way, the coefficients of the resource A that is also allocated will be extracted.

[0105] Then, the interference level I of the entire ultra-dense vehicle network can be determined:

[0106]

[0107] The vehicle network resource allocation problem considering differential interference can be transformed into a combinatorial optimization problem. The main goal of the present invention is to optimize the resource allocation in ultra-densely deployed vehicle networks, avoid resource conflicts, and improve the network throughput of the entire vehicle network. Therefore, the conflict-free resource allocation problem in densely deployed vehicle networks can be described as follows:

[0108]

[0109] Among them, λ1,λ2∈(0,1), M represents the set of channels, M={1,2,…,M}, M represents the total number of channels, η rrepresents the resource reuse rate, P represents the power set, P={p1,p2,…,p F}, p f represents the signal transmission power selected by the f-th transmitting vehicle when transmitting a signal; N = {1, 2, ..., N} represents the set of V2V links; η r represents the resource reuse rate of the rth interference receiving vehicle;

[0110] represents the data transmission rate of the nth V2V link at time t, R min represents the minimum transmission rate of the channel, t∈{1,2,...,T}, T represents the total time;

[0111] represents the signal-to-noise ratio of the nth V2V link at time t; γ0 represents the signal-to-noise ratio threshold;

[0112] l n,m Indicates the selection of the mth channel by the nth V2V link. When the nth V2V link selects the mth channel, l n,m =1; otherwise, when the nth V2V link does not select the mth channel, l n,m =0;

[0113] For the above constraints, constraint C1 can ensure that the minimum network throughput is not less than the minimum network throughput in the IoV network, constraint C2 ensures that the V2V links can communicate with each other, that is, the signal-to-noise ratio is greater than the minimum threshold, constraints C3 and C4 represent the constraints on channel and power selection in IoV resource allocation, and constraint C5 ensures that the IoV network resource allocation should satisfy conflict-free resource allocation. The problem represented by formula (11) is a time-averaged combinatorial optimization problem, which is mathematically challenging and computationally difficult.

[0114] To this end, the present invention develops an MDP model to solve the resource management problem, that is, to solve the combinatorial optimization problem, and proposes an action-critic (AC) resource allocation method based on reinforcement learning. In order to speed up the learning speed, a federated reinforcement learning method, namely the FedAvg-AC algorithm, is proposed.

[0115] The above combinatorial optimization problem in the ultra-dense IoV setting can be formulated as an MDP problem, which is Markov and fully observable.

[0116] Converting the optimization problem into a Markov decision process problem involves:

[0117] Treat each V2V link as an intelligent agent;

[0118] According to the vehicle network information at time t, the state space at time t is represented as S = {MF ,γ,M,P},M F represents the channel selection of F sending vehicles in the communication model, γ represents the set of signal-to-noise ratios of all channels, M represents the total number of channels, and P represents the set of transmission powers (or transmission powers) selected by all sending vehicles (or senders);

[0119] The set of action spaces is represented as: A = {a1, a2, ..., a T}, a t = {l n,m ,p k,m} represents the action at time t, which includes the channel selection l n,m and power selection p k,m , p k,m represents the transmission power when the k-th vehicle selects the m-th channel, k∈{1,2,...,K}, n∈{1,2,...,N}, m∈{1,2,...,M};

[0120] The reward value obtained by each agent based on the action at time t is:

[0121]

[0122] That is, the objective function is used as the reward value function;

[0123] The action value function is:

[0124]

[0125] Where E represents the expectation, the network state space S0 represents the initial state space (i.e., the initialization value of the state space); π represents the strategy; ρ represents the attenuation factor; Q π represents the cumulative reward when starting the network state space S0 under strategy π.

[0126] During the training process, if the resource management scheme is optimal, that is, the allocated resources do not conflict and the resource reuse rate is maximized, the reward value is maximized, and vice versa, if the allocated resources conflict and the resource reuse rate is low, the reward value is minimum and negative. In the reinforcement learning network constructed by the present invention, each intelligent agent is constantly looking for a strategy to maximize the reward value.

[0127] MDP models can be solved using Q-learning, policy gradient methods, and deep Q-learning. However, the slow convergence of the Q function or its approximation in Q-learning often leads to suboptimal performance and prevents the optimal policy from being discovered within a reasonable number of iterations. In addition, Q-learning is inefficient in learning stochastic policies, especially in environments with continuous action spaces, such as those found in IoV networks where channel conditions and transmit power vary.

[0128] In contrast, policy gradient methods manipulate policies directly in policy space and can usually converge to local optimal policies faster than Q-learning. To address the challenge of learning optimal policies for intelligent resource management in continuous state and action spaces, actor-critic (AC) learning algorithms integrate policy and value learning and aim to converge efficiently.

[0129] In the AC framework, the actor (i.e., actor) and critic (i.e., critic) components are crucial. The actor determines the strategy for selecting actions by observing the network state, while the critic evaluates the strategy using the rewards obtained from the environment feedback. In a vehicle network environment where V2V connections act as agents, each connection autonomously senses the current network conditions and takes actions based on its learned strategy, operating in a decentralized manner. After these actions, the IoV environment provides new states and immediate feedback to the agents, allowing them to update their strategies for subsequent iterations.

[0130] The AC model (i.e., AC framework) includes an actor network π(s, a; θ) and a critic network q(s, a; ψ). The actor network is used to identify the action strategy in the vehicle network, and its parameter is θ. The critic network is used to evaluate the quality of the strategy, and its parameter is ω.

[0131] The gradient descent method is used to update θ, which can be expressed as:

[0132]

[0133] The temporal difference learning scheme is also used in AC reinforcement learning to calculate the temporal difference (TD) error between the estimated value and the true value, which can be expressed as:

[0134] δ t =q t -(r t +ρ·q t+1 ), (15)

[0135] When using the critic network to estimate the state value function, ψ can be updated by gradient descent, which can be expressed as:

[0136]

[0137] Among them, α and β represent the learning rate, q t =q(s t ,a t ,ψ t ), r t represents the reward value at time t; q t represents the critic network at time t.

[0138] However, the AC algorithm converges slowly. To solve this problem, the present invention uses a federated learning method to improve the convergence speed of the AC algorithm. In the considered ultra-dense IoV network, all agents use the global model of the FedAvg-AC server to build a local AC network. During the global model training, each agent randomly extracts a small amount of data set B from the local experience replay pool D to update its own local AC model. The local update of the nth RL agent minimizes the target Q-network L(w t ). Specifically, each agent replays the local experience pool D corresponding to the agent k Randomly select a preset number of data sets B k To update the local AC model corresponding to the agent; after a round of learning, the parameters of the global model are obtained by weighted averaging the parameters of each local AC model, thereby deriving the FedAvg-AC global network.

[0139] The loss function L(w t ) can be expressed as follows:

[0140]

[0141] Among them, L(w t ) represents the global model loss function, represents the local AC model loss function, |B n | represents data set B n The amount of data in .

[0142] Specifically, the FedAvg-AC algorithm includes:

[0143] Build a global model and a local AC model. Each agent corresponds to a local AC model, and each agent uses the global model to build a local AC model.

[0144] During the global model training process, the following operations are performed for training:

[0145] The parameter w of the global model at time t-1 is t-1 As the parameter of each local AC model at time t-1

[0146] According to the parameters of each local AC model at time t-1 Determine the loss function of each local AC model at time t-1

[0147] Computing Gradients Get the parameters of the local AC model at time t Update the local AC model, that is:

[0148]

[0149] Next, the parameters of each local AC model are Perform weighted averaging to obtain the global model parameter w at time t t ,Right now:

[0150]

[0151] Then, the server sets the global model parameter w at time t t Sent to each local AC model (e.g., by mass email) for the next round of training.

[0152] Algorithm 1 summarizes the training process. The algorithm flow is as follows:

[0153]

[0154] The following simulation experiments are conducted using the urban scenario in 3GPP TR 36.885 to verify the effectiveness of the proposed algorithm. The FedAvg-AC algorithm is implemented in PyTorch. The parameters of the proposed FedAvg-AC are detailed in Table 1.

[0155] Table 1 Simulation experiment parameters

[0156]

[0157] In order to verify the effectiveness of the algorithm in this paper, the simulation performance was compared with other algorithms such as random network resource allocation (RM), network resource allocation based on greedy algorithm (GA), and network resource allocation based on maximum node degree algorithm (MND).

[0158] Figure 4 The convergence behavior of the FedAvgAC-based network resource management method across various learning rates is shown. Throughout the training phase, a fixed number of 20 V2V links are used and the reward value stabilizes as the training progresses. The learning rate plays a crucial role in controlling the learning dynamics of the network model, affecting how quickly the algorithm converges to the global optimum. A learning rate that is too low may cause the algorithm to get stuck in a local optimum. As shown in the figure, setting the learning rate to 0.0001 shows a significant improvement in the convergence performance. Therefore, a learning rate of 0.0001 is selected for subsequent experiments. Figure 4 The effectiveness of the FedAvg-AC network resource management algorithm is emphasized.

[0159] The advantages of the present invention are described below by analyzing three indicators (Signal to Interference and Noise Ratio SINR, network throughput and resource reuse rate).

[0160] SINR is the ratio of the signal power to the combined interference and noise power in the system, as shown in equation (3). A higher SINR indicates that the receiver can more clearly discern the useful signal amidst reduced interference and noise levels.

[0161] Network throughput: This performance indicator evaluates the network throughput of the Internet of Vehicles after the resource allocation algorithm has allocated all communication link resources, which can be expressed as:

[0162]

[0163] in, Represents the average SINR. The higher the network throughput, the higher the maximum rate that the network can accept, and the better the network performance.

[0164] Resource reuse rate: This performance metric indicates the extent to which network resources are effectively utilized within a specific time period. A high network resource reuse rate means that network resources are fully utilized, reducing idle waste and improving overall network performance and efficiency. It can be expressed as:

[0165]

[0166] Where M represents the total number of resources (i.e., the total number of channels), k Indicates the total number of resources utilized.

[0167] Figure 5 The relationship between the number of V2V links and SINR is shown. A higher SINR value means that the network has lower interference. Figure 5 The SINR values ​​of this algorithm and the other three comparison algorithms under different transmission links are compared. Compared with the other three algorithms, this algorithm has a higher signal to interference and noise ratio. This is because the algorithm can effectively avoid interference, improve the signal to interference and noise ratio, and ensure the transmission rate by adaptively selecting the best resource allocation scheme. The simulation results show that the algorithm can effectively avoid interference and improve the signal to interference and noise ratio.

[0168] Figure 6 The network throughput achieved using various algorithms in a connected vehicle environment is shown. It is clear that as the number of V2V connections increases, the network throughput also increases. This can be attributed to the enhanced transmission link, which enables a larger amount of data to be transmitted simultaneously, thereby improving the overall network throughput. In addition, the FedAvg-AC algorithm proposed in the present invention is also significantly better than other algorithms. This superiority can be attributed to the strong adaptability of the algorithm. Therefore, Figure 6 It is verified that the proposed algorithm significantly improves the network throughput of the system.

[0169] Figure 7The relationship between different numbers of transmitters and resource reuse in dense vehicle networking is shown. A higher resource reuse rate indicates that the system effectively utilizes the available communication resources. When the transmitter has a fixed transmission power, Figure 7 The resource reuse rate of the proposed algorithm is compared with that of other three comparison algorithms under different numbers of reuse times. The results show that the proposed algorithm outperforms similar algorithms in terms of resource reuse rate. This improvement is due to the ability of the algorithm to adaptively select the best resource allocation scheme based on the current network status. Therefore, Figure 7 It is verified that the proposed algorithm significantly improves the resource reuse rate.

[0170] In summary, the present invention aims at the resource allocation problem in the scenario of ultra-dense overlapping interference, establishes a weighted interference model, and proposes a resource allocation algorithm based on FedAvg-AC to allocate channel and power resources. By studying the characteristics of the overlapping communication distances of ultra-dense vehicles, the relationship between the overlapping communication distances of ultra-dense vehicles and interference is analyzed. The weighted interference model is used to evaluate the interference level of the entire vehicle network, and the resource allocation problem is redefined as a combinatorial optimization problem. To address this problem, a method for establishing an MDP model of an ultra-dense vehicle network based on a weighted interference model is explored. Then, an AC-based resource allocation algorithm is designed for the MDP model. In order to accelerate the convergence speed of the AC algorithm and improve the computing power of the network, a FedAvg-AC resource allocation algorithm based on federated learning is proposed. Finally, computer simulation is used to verify the effectiveness of the algorithm. This study provides valuable insights into resource management of ultra-dense vehicle networks.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for adaptive resource allocation in ultra-dense Internet of Vehicles based on deep reinforcement learning, characterized in that: include: A communication model of an ultra-dense vehicle network is established, wherein the communication model includes a base station and K vehicles, wherein the K vehicles include F sending vehicles and R receiving vehicles, wherein the base station exchanges data with the vehicle via a V2I link, and the vehicles communicate with each other via a V2V link, wherein the sending vehicle sends a signal to the receiving vehicle, wherein the number of V2V links is N; According to the interference relationship between vehicles in the communication model, a weighted interference model centered on the receiving vehicle is determined; According to the communication model and the weighted interference model, a resource allocation model of the communication model is established, wherein the resource allocation model is an optimization problem with the goal of maximizing resource reuse rate and non-conflicting allocated resources under certain constraints; Converting the optimization problem into a Markov decision process problem; The FedAvg-AC algorithm is used to solve the Markov decision process problem and obtain the resource allocation strategy for ultra-dense Internet of Vehicles.

2. The method for adaptive resource allocation of ultra-dense vehicle networks based on deep reinforcement learning according to claim 1 is characterized in that: Determine a weighted interference model centered on the receiving vehicle, including: For each receiving vehicle, determining the interference weight of each sending vehicle to the receiving vehicle and generating a weight matrix; According to the plurality of sending vehicles, the plurality of receiving vehicles and the weight matrix, an interference weight graph G={F, R, W} is established, wherein F is a vertex set, representing a set of sending vehicles; R is an edge set, representing a set of receiving vehicles, and W represents a weight matrix; The interference weight map is transformed into the correlation matrix H g Represents, where the incidence matrix H g A row of represents the interference weight of a sending vehicle to each receiving vehicle, and the correlation matrix H g One column represents the interference weight of a certain receiving vehicle from each sending vehicle; According to the correlation matrix and resource allocation scheme, the interference level v of each receiving vehicle is determined r , where r = 1, 2, ..., R, and R represents the total number of received vehicles; Determine the overall interference level I of ultra-dense vehicular networks.

3. The method for adaptive resource allocation of ultra-dense vehicle networks based on deep reinforcement learning according to claim 2 is characterized in that: The interference level of each receiving vehicle is expressed as: Among them, ν r represents the interference level of the rth receiving vehicle, c represents the resource allocation of the sending vehicle; h r Represents the incidence matrix H g The combination of all elements in the rth column of r Indicates c and h r The result of multiplying and summing the elements at the corresponding positions in c; c[i] represents the resources allocated to the i-th node in c, that is, the resources allocated to the i-th sending vehicle, ν r = 0 means there is no interference in the rth receiving vehicle, otherwise, ν r ≠0.

4. The method for adaptive resource allocation of ultra-dense vehicle networks based on deep reinforcement learning according to claim 1 is characterized in that: The optimization problem of the resource allocation model is expressed as: C3:l n,m ∈{0,1} C5: I = 0, Among them, λ1,λ2∈(0,1), M represents the set of channels, M={1,2,…,M}, M represents the total number of channels, η r represents the resource reuse rate, P represents the power set, P={p1,p2,…,p F }, p f represents the signal transmission power selected by the f-th transmitting vehicle when transmitting a signal; N = {1, 2, ..., N} represents the set of V2V links; η0 represents the resource reuse rate of the entire network; represents the data transmission rate of the nth V2V link at time t, R min represents the minimum transmission rate of the channel, t∈{1,2,...,T}, T represents the total time; represents the signal-to-noise ratio of the nth V2V link at time t; γ0 represents the signal-to-noise ratio threshold; l n,m Indicates the selection of the mth channel by the nth V2V link. When the nth V2V link selects the mth channel, l n,m =1; otherwise, when the nth V2V link does not select the mth channel, l n,m =0.

5. The method for adaptive resource allocation of ultra-dense vehicle networks based on deep reinforcement learning according to claim 4 is characterized in that: Converting the optimization problem into a Markov decision process problem involves: Treat each V2V link as an intelligent agent; According to the vehicle network information at time t, the state space at time t is represented as S = {M F ,γ,M,P},M F represents the channel selection of F sending vehicles in the communication model, γ represents the set of signal-to-noise ratios of all channels, M represents the total number of channels, and P represents the set of transmission powers selected by all sending vehicles; The set of action spaces is represented as: A = {a1, a2, ..., a T }, a t = {l n,m ,p f,m } represents the action at time t, which includes the channel selection l n,m and power selection p f,m , p f,m represents the signal transmission power when the f-th transmitting vehicle selects the m-th channel, f∈{1,2,...,F}, n∈{1,2,...,N}, m∈{1,2,...,M}; The reward value obtained by each agent based on the action at time t is: The action value function is: Among them, E represents the expectation, the network state space S0 represents the initialization value of the state space; π represents the strategy; ρ represents the attenuation factor; Q π represents the cumulative reward when starting the network state space S0 under strategy π.

6. The method for adaptive resource allocation of ultra-dense vehicle networks based on deep reinforcement learning according to claim 5 is characterized in that: The FedAvg-AC algorithm specifically includes: Construct a global model and a local AC model. Each agent corresponds to a local AC model. Each agent uses the global model to construct a local AC model. During the global model training process, each agent replays the local experience from the corresponding local experience pool D n Randomly select a preset number of data sets B n To update the local AC model corresponding to the agent; after a round of learning, the parameters of the global model are obtained by weighted averaging the parameters of each local AC model.

7. The method for adaptive resource allocation of ultra-dense vehicle networks based on deep reinforcement learning according to claim 6 is characterized in that: The method specifically comprises: The parameter w of the global model at time t-1 is t-1 As the parameter of each local AC model at time t-1 n∈{1,2,...,N}; According to the parameters of each local AC model at time t-1 Determine the loss function of each local AC model at time t-1 Computing Gradients Get the parameters of the local AC model at time t Update local AC model; For each local AC model parameter Perform weighted averaging to obtain the global model parameter w at time t t ; The global model parameter w at time t t Sent to each local AC model for the next round of training.

8. The method for adaptive resource allocation of ultra-dense vehicle networks based on deep reinforcement learning according to claim 6 is characterized in that: For each local AC model parameter Perform weighted averaging to obtain the global model parameter w at time t t , expressed as: Among them, |B n | represents data set B n The amount of data in .

9. The method for adaptive resource allocation of ultra-dense vehicle networks based on deep reinforcement learning according to claim 6 is characterized in that: The local AC model consists of an actor network π(s,a;θ) and a critic network q(s,a;ψ). The actor network is used to identify the action strategy in the vehicle network, and its parameter is θ. The critic network is used to evaluate the quality of the strategy, and its parameter is ψ. The gradient descent method is used to update θ and ψ, thereby updating the parameters of the AC model. Specifically, Among them, δ t =q t -(r t +ρ·q t+1 ), α and β represent the learning rate, q t =q(s t ,a t ,ψ t ), r t represents the reward value at time t, q t represents the critic network at time t.

Images