A joint optimization method for resource allocation in Internet of Vehicles based on dominance strategy

Abstract

The present invention provides a joint optimization method for Internet of Vehicles resource allocation based on a dominance strategy, comprising: firstly, modeling the resource allocation problem as a multi-agent deep reinforcement learning problem based on a continuous action space, constructing a high-speed dynamic Internet of Vehicles environment model, taking maximizing the total throughput of the vehicle-to-infrastructure communication link and ensuring the high reliability transmission of the vehicle-to-vehicle communication link as the optimization goal, and using the improved MADDPG‑A2C algorithm to optimize the system performance. The results show that compared with the traditional resource allocation method, the algorithm shows certain improvements in key performance indicators such as throughput, latency, and spectrum utilization. At the same time, compared with the MADDPG algorithm, the improved algorithm has the characteristics of strong stability, high sample efficiency, and wide applicability.

Description

Technical Field

[0001] The present invention relates to a joint optimization method for resource allocation of Internet of Vehicles based on a dominant strategy. Background Art

[0002] With the rapid development of intelligent transportation systems, vehicular ad-hoc networks (VANETs), as one of its core technologies, are gradually becoming a hot topic in research and application. VANETs achieves interconnection and interoperability by promoting vehicle-to-vehicle communication (V2V), vehicle-to-infrastructure communication (V2I), and vehicle-to-pedestrian communication (V2P), and provides a series of intelligent transportation services, including but not limited to road safety, traffic management, and infotainment. Due to the dynamic changes in the network structure of VANETs, ​​it has encountered major challenges in resource allocation, which has a significant impact on the reliability, operation efficiency, and service quality of the network.

[0003] The current research focuses on how to optimize the allocation strategy of spectrum resources to reduce co-channel interference while enhancing the system's data processing capabilities and service levels, which is crucial for the design of vehicle communication systems. Traditional resource allocation methods often rely on centralized control strategies or pre-defined rules, which are difficult to achieve ideal results when dealing with highly dynamic and uncertain vehicle networking environments. In contrast, reinforcement learning (RL) provides a mechanism for autonomous learning and decision-making, allowing vehicles to dynamically adjust their behavior according to changes in the environment and find the optimal resource allocation strategy.

[0004] As a branch of reinforcement learning, Multi-Agent Reinforcement Learning (MARL) is more suitable for solving problems in naturally distributed, multi-participant systems such as the Internet of Vehicles. Through MARL, each agent (i.e., vehicle) can not only learn how to interact with the environment, but also learn how to achieve common goals in the interaction with other agents, such as resource sharing, conflict avoidance, etc., thereby optimizing the performance of the entire system.

[0005] Whether at home or abroad, the research on resource allocation of Internet of Vehicles based on multi-agent reinforcement learning has shown great potential and broad application prospects. Through these studies, not only can the performance of Internet of Vehicles be improved, but also reference can be provided for other types of network systems. However, current research still faces some common challenges, including algorithm complexity, computing resource requirements, and real-time guarantee. Future research needs to conduct more in-depth exploration in improving algorithm efficiency, reducing resource consumption, and ensuring system stability. Summary of the invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide a joint optimization method for Internet of Vehicles resource allocation based on a dominant strategy in view of the deficiencies of the prior art, aiming to improve the overall transmission efficiency, quality and stability of the Internet of Vehicles communication system, thereby optimizing the performance of the entire network. The method of the present invention comprises the following steps:

[0007] Step 1: Establish the Internet of Vehicles environment;

[0008] Step 2: Establish optimization objectives and constraints;

[0009] Step 3: Establish a joint resource allocation optimization algorithm based on MADDPG (Multi-Agent Deep Deterministic Policy Gradient Algorithm, MADDPG) and A2C to achieve spectrum allocation of orthogonal sub-channels for vehicle-to-infrastructure communication and power selection for inter-vehicle communication in complex environments, and maximize the total throughput of the V2I (Vehicle-to-Infrastructure, V2I) link while ensuring high-reliability transmission of the V2V (Vehicle-to-Vehicle, V2V) link.

[0010] Step 1 includes: setting a vehicle networking environment including one base station and more than two vehicle users, wherein the vehicle users include vehicle-to-infrastructure communication V2I users and vehicle-to-vehicle communication V2V users; the vehicle-to-infrastructure communication link is used for the communication between the vehicle and the base station and the roadside unit, and the vehicle-to-vehicle communication link is used for direct information exchange between vehicles; the vehicle-to-infrastructure communication link set is S V2I , S V2I = {1, 2, ..., M}, the set of inter-vehicle communication links is S V2V , S V2V ={1, 2, ..., N}; M represents the total number of vehicle-to-infrastructure communication links, N represents the total number of vehicle-to-vehicle communication links, and the values ​​of M and N are natural numbers.

[0011] Step 1 also includes: assuming that all vehicles participating in the communication are equipped with a single antenna device, each vehicle-to-infrastructure communication V2I user communicates only through one vehicle-to-infrastructure communication uplink, and the spectrum resources of the vehicle-to-infrastructure communication uplink are evenly divided into M orthogonal sub-channels; allowing more than two vehicle-to-vehicle communication V2V users to share the spectrum resources of each vehicle-to-infrastructure communication link, thereby realizing spectrum sharing.

[0012] Step 1 also includes: defining the signal-to-noise-plus-interference ratio of the mth vehicle-to-infrastructure communication link and the nth vehicle-to-vehicle communication link channel as:

[0013]

[0014] in represents the signal-to-noise-and-interference ratio of the mth vehicle-to-infrastructure communication link, represents the transmission power of the mth vehicle-to-infrastructure communication link, g n [m] represents the channel power gain of the nth inter-vehicle communication link on the mth orthogonal subchannel, g m，B represents the channel power gain of the mth vehicle-to-infrastructure communication link, σ 2 represents the Gaussian white noise power, It represents the interference degree caused by the nth vehicle-to-vehicle communication link to the mth vehicle-to-vehicle-to-infrastructure communication link, and the expression is:

[0015]

[0016] Among them, the symbol It is used to indicate whether the m-th vehicle-to-infrastructure communication link shares the channel with the n-th vehicle-to-vehicle communication link. The value is 1 if they share the channel and 0 if they do not.

[0017] represents the transmission power of the nth inter-vehicle communication link, g n，B [m] represents the interference channel gain caused by the nth vehicle-to-vehicle communication link to the mth vehicle-to-infrastructure communication link;

[0018] I1 represents the interference caused by the mth vehicle-to-infrastructure communication link to the nth vehicle-to-vehicle communication link, which is expressed as:

[0019]

[0020] in, represents the transmission power of the mth vehicle-to-infrastructure communication link, g m，n represents the interference channel gain when the n-th vehicle-to-vehicle communication link is shared with the m-th vehicle-to-infrastructure communication link;

[0021] I2 represents the interference caused by other inter-vehicle communication links except the nth inter-vehicle communication link, and the expression is:

[0022]

[0023] Among them, φ k [m] indicates whether the mth vehicle-to-infrastructure communication link shares a channel with the remaining k vehicle-to-vehicle communication links except the nth one. The value is 1 if it shares a channel, and 0 if it does not. represents the transmission power of the inter-vehicle communication link set k except the nth inter-vehicle communication link on the orthogonal subchannel of the mth vehicle-to-infrastructure communication link, g k，n [m] represents the interference channel gain of the inter-vehicle communication link set k except the n-th inter-vehicle communication link on the n-th inter-vehicle communication link;

[0024] Define the transmit calculation power g as:

[0025] g=ah[m] (6)

[0026] Where a represents the large-scale fading that is independent of frequency, and h[m] represents the small-scale fading channel gain that is dependent on frequency.

[0027] Step 1 also includes: using L V2V Denote the path loss of the V2V user in vehicle-to-vehicle communication, L V2I represents the path loss of the vehicle-to-infrastructure communication V2I user, and the expressions are:

[0028]

[0029] L V2I =128.1+37.6lg(d) (8)

[0030] Among them, d is a distance parameter; f c Indicates the carrier frequency;

[0031] According to Shannon's formula, the total throughput of the mth vehicle-to-infrastructure communication link is obtained and the total throughput of the nth inter-vehicle communication link They are:

[0032]

[0033] Wherein, lb represents the logarithmic expression with base 2, and B is the channel bandwidth;

[0034] Construct the effective transmission probability ψ n for:

[0035]

[0036] Where B1 represents the size of the periodically generated vehicle-to-vehicle communication link payload; △T is the coherence time of the channel; It represents the total throughput of the nth inter-vehicle communication link within unit time t, and T is the maximum value of the parameter per unit time.

[0037] Step 2 includes: establishing the following optimization objectives:

[0038]

[0039] Where ∑n′ represents the total number of active inter-vehicle communication links; η represents the number of communicating vehicles; max is the maximum value function; E represents the function expression of the expected value;

[0040] Construct the following constraints:

[0041]

[0042]

[0043] Among them, p max I represents the maximum transmission power allowed for the vehicle-to-infrastructure communication link and the vehicle-to-vehicle communication link; max It represents the maximum interference allowed between the vehicle-to-infrastructure communication link and the vehicle-to-vehicle communication link, in dBm; R min represents the minimum tolerable transmission rate of the mth vehicle-to-infrastructure communication link; ρ0 represents the minimum setting effective transmission probability; Indicates the minimum tolerable coherence time of the channel;

[0044] Formula (14) represents the maximum transmission power allowed for the vehicle-to-infrastructure communication link and the vehicle-to-vehicle communication link; Formula (15) represents the interference constraint at the link receiving end; Formula (16) represents the minimum tolerable throughput constraint for the vehicle-to-infrastructure communication link; Formula (17) represents the effective transmission probability constraint for the vehicle-to-vehicle communication link; Formula (18) represents the maximum transmission delay allowed for the vehicle-to-vehicle communication link; Formula (19) represents whether the vehicle-to-vehicle communication link reuses the spectrum resources of the vehicle-to-infrastructure communication link, which is a binary function. Formula (20) represents the maximum tolerable throughput constraint for the vehicle-to-vehicle communication link, that is, the maximum transmission rate constraint.

[0045] Step 3 includes: establishing a joint optimization algorithm for resource allocation based on MADDPG (Multi-Agent Deep Deterministic Policy Gradient Algorithm, MADDPG) and A2C, treating the Internet of Vehicles scenario as a multi-agent deep reinforcement learning task, where each communicating vehicle acts as an independent agent, which accumulates experience through interaction with the environment and learns to achieve the optimal strategy to optimize rewards; in a fully collaborative learning mode, the agent selects the transmission power for data transmission according to the real-time environmental state; the algorithm is divided into two stages: centralized learning and distributed execution. In the centralized learning stage, the agent trains the policy network and the value function network; in the distributed execution stage, the agent selects the optimal strategy π according to the state information and rewards * To minimize the loss L(θ), specifically:

[0046] Step 3-1, design state space:

[0047] In time slot t, the environment state currently encountered by agent n is recorded as Agent n outputs corresponding actions according to the current strategy π In the entire connected vehicle system, the joint state space of all agents is defined as s t , after the strategy π output, we get the joint action A t , and get a common reward R after performing the action together t ; In the neural network, the state of agent n By probability Transfer to the next state The probability The expression is:

[0048]

[0049] in represents the action selection of agent n at time step t; p r represents the probability function;

[0050] The local channel information observed by agent n is defined as follows: the channel gain g of its own link n [m], interference channel gain g caused by other inter-vehicle communication links k，n [m], interference channel gain from itself to the base station g n，B [m], interference channel gain g of the mth vehicle-to-infrastructure communication link to the nth vehicle-to-vehicle communication link m，n [m];

[0051] The expression is:

[0052]

[0053] Step 3-2, design action space:

[0054] The spectrum resources of each vehicle-to-infrastructure communication link are evenly divided into M orthogonal sub-channels. N vehicle-to-vehicle communication links can select one of the M spectrum sub-channels to share and adjust the transmit power to perform transmission. The continuous power value The limit is defined as The action selected by agent n at time step t It is expressed as:

[0055]

[0056] Step 3-3, design reward function:

[0057] Design two reward functions Re V2I and Re V2V :

[0058]

[0059] Where Φ is a constant; for a V2V user that has completed transmission, this constant Φ is greater than the payload transmission rate of the V2V link, thereby encouraging the V2V user to increase the transmission rate;

[0060] Through the weight coefficient method, the final reward function Re is obtained total for:

[0061]

[0062] Where λ is the weight coefficient, and λ∈(0,1); is the reward factor; V2V success represents the probability of successful transmission of the inter-vehicle communication link; is a penalty factor, which is 0 when a specific constraint condition is met and is a parameter ζ when the constraint condition is not met; the specific constraint condition is set to the minimum vehicle-to-infrastructure communication link rate tolerance;

[0063] The expression of parameter ζ is:

[0064]

[0065] Among them, χ is a tuning parameter;

[0066] The goal is to find the optimal strategy π * , so that the cumulative expected reward is maximized:

[0067]

[0068] Among them, Re max represents the maximum cumulative expected reward, E π is a function expression for the expected value under the π strategy, Represents the decay factor, which is used to control the discount of future rewards; τ represents the time step; when τ = 0, Re t+τ Represents the current reward return. When τ is not 0, Re t+τ represents the relevant rewards at other time steps; Represents the sum of current rewards and future rewards;

[0069] Step 3-4, implement the deep deterministic policy gradient algorithm DDPG;

[0070] Steps 3-5, implement the multi-agent deep deterministic policy gradient MADDPG algorithm.

[0071] Step 3-6, establish the advantage-based actor-critic A2C algorithm; use TD (Temporal Difference Error, TD) error to calculate the advantage function of the A2C algorithm

[0072] Step 3-7, establish the MADDPG-A2C algorithm.

[0073] Step 3-4 includes: the deep deterministic policy gradient algorithm DDPG includes the following neural network:

[0074] Main Actor Network: The main Actor network is used to map states to actions. The parameters of the main Actor network are recorded as θ. The main Actor network maps the state s of the environment at time step t to t As input, and generate corresponding action μ(s t |θ);

[0075] Target Actor Network: The target Actor network is used to generate target actions. The initial parameters are set to θ′ and generate actions μ′(s t |θ′); The parameters of the target Actor network are gradually updated synchronously from the parameters θ of the main Actor network by adopting a smooth update strategy;

[0076] Main Critic Network: The main Critic network is used to learn the action value function. With ω as the initial parameter, the main Critic network will receive the state s of the environment at time step t t and the corresponding action a t As input, and output the corresponding state value function Q(s t , a t |ω); The parameters of the main critic network are updated by gradient descent to minimize the gap between the estimated action value function and the target action value;

[0077] Target Critic Network: The target critic network is used to evaluate the value of the target action. It is initially parameterized with the parameter ω′ and the output state value function is Q′(s t , a t |ω′); The parameters of the target critic network are gradually updated from the parameters of the main critic network through soft updating;

[0078] The parameters θ of the main Actor network are updated via gradient descent using the following chain rule:

[0079]

[0080] in, represents the gradient of the loss function with respect to the input parameter θ during back propagation, Represents the policy function about action a t The gradient of Q(s t , a t |ω) represents the output of the main Critic network, represents the gradient operator for the parameter θ, μ(s t |θ) represents the output of the main Actor network;

[0081] The update of the main Critic network first uses the output of the target Actor network as the input of the target Critic network to obtain the target value y of the state value function:

[0082] y=r+γQ′(s′ t ,μ′(s t ′|θ′)|ω′) (30)

[0083] Among them, μ′(s′ t |θ′) represents the output of the target Actor network, γ is a hyperparameter, and s′ t Represents the next state;

[0084] Next, the state value function is calculated using the main critic network, and the main critic network is reversely updated according to the method of minimizing the loss function. The loss function L is defined as:

[0085]

[0086] Among them, D n represents the number of samples drawn; Q(s t , a t |ω) represents the output of the main Critic network;

[0087] The target actor network and the target critic network optimize their parameters through a progressive update method, expressed as:

[0088] ω′←τω+(1-τ)ω′; θ′←τθ+(1-τ)θ′ (32)

[0089] Here, τ is a key hyperparameter used to control the smoothness of the update.

[0090] Steps 3-5 include:

[0091] Multi-agent environment modeling: The MADDPG algorithm regards the environment as a game environment containing more than two agents, and the strategy of each agent needs to consider the influence of other agents;

[0092] Shared experience replay buffer: The MADDPG algorithm uses a shared experience replay buffer, from which all agents can sample experience to train their own policy networks;

[0093] Centralized training and distributed execution: The MADDPG algorithm combines the strategies of centralized learning and distributed operation. It optimizes the strategy through centralized learning and uses a distributed approach in the execution phase, that is, a global main critic network is used to evaluate the behavior value of all agents, and the strategy network of each agent is trained independently.

[0094] Adaptive reward: MADDPG algorithm designs an adaptive reward function;

[0095] In steps 3-6, the advantage function The calculation formula is:

[0096]

[0097] in, Is to perform an action The total expected reward after is generated by the main Actor network; is in state The state value function under ;

[0098] Steps 3-7 include: defining each independent agent n, where n∈(1, 2, ..., N), according to the state Output corresponding actions through the Actor network The strategy function of the main Actor network is denoted as π n In the built car networking system, each agent is equipped with an independent main actor network, corresponding to its unique policy function (each agent follows the π policy, which usually refers to the probability distribution of the agent's action selection based on the current state in reinforcement learning), and a main critic network; each main critic network must evaluate the policy outputs generated by all actor networks. The input of each main critic network integrates the policy outputs of all N main actor networks and finally gives feedback. The feedback value is recorded as Q n , after iterative training, the global optimal solution is finally obtained to maximize the reward return, and as the number of iterations increases, the reward value gradually converges to one point. The optimization goal is:

[0099]

[0100] The optimization goal is to maximize the total throughput of the vehicle-to-infrastructure communication link under the condition of high reliability transmission of the inter-vehicle communication link; the continuous action variable space is divided into four discrete levels of normalized values: [23, 15, 5, -100] dBm;

[0101] The current environment state of agent n is γ is the discount factor of the reward, and the state-action value function of the MADDPG-A2C algorithm is obtained as:

[0102]

[0103] in, Indicates that for agent n, the current main Actor network takes action Total return obtained when Indicates that according to the next action and the next state The obtained Q value (i.e. calculated using formula 34) The value of ), represents the expected reward of the next batch of states; E is the mean function;

[0104] Loss function for the main Actor network Actor Loss And the loss function of the main critic network Critic Loss And the parameter update mechanism has been improved, the expressions are:

[0105]

[0106] Among them, y is the output Q value of the target Critic network, represents the expected total reward after taking N actions, advantages is the advantage function, and formula (33) has been given; Indicates taking action in a given state And the expected return that can be obtained by following the π strategy;

[0107] When Critic Loss It is continuously differentiable, ω can be updated with gradients, and the update strategy of the A2C algorithm is adopted:

[0108]

[0109] Among them, β critic is the learning rate of the main critic network, which is a hyperparameter; δ is the TD error;

[0110] In pursuit of the goal of maximizing the total reward, each agent adjusts the evaluation parameters of the main Actor network by optimizing the objective function. The objective function J θ The expression is:

[0111]

[0112] The advantage-driven asynchronous parallel strategy A2C is used to optimize parameters:

[0113]

[0114] Among them, α actor The learning rate of the main Actor network is a hyperparameter;

[0115] The update of the target Actor network and the target Critic network parameters are both soft updated, and the expression is:

[0116] θ′←τ*θ+(1-τ)*θ′ (40)

[0117] ω′←τ*ω+(1-τ)*ω′ (41).

[0118] Beneficial effects: The present invention provides a multi-agent deep reinforcement learning strategy, which is based on the A2C framework and uses the Markov decision process (MDP) for system modeling. The resource allocation problem of the Internet of Vehicles is concretized as a power control problem of the continuous actions of the agent. The algorithm is mainly divided into two stages, centralized training and distributed execution. The algorithm of the present invention addresses the problems of unstable training and low sample efficiency of MADDPG. It improves the AC network of the traditional MADDPG into a more stable A2C network architecture, and quotes the classic advantage function in the A2C algorithm to improve the overall stability and efficiency of the algorithm. Simulation results show that compared with the other four baseline algorithms, the proposed MADDPG-A2C algorithm can further maximize the throughput of the vehicle-to-infrastructure communication link while ensuring the reliability of V2V transmission (i.e., high-quality communication), and the convergence and stability are also improved. At the same time, the algorithm also alleviates the problem of huge signaling overhead and has good scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0120] Figure 1 It is a vehicle networking system model.

[0121] Figure 2 This is the MADDPG-A2C algorithm framework diagram.

[0122] Figure 3 It is a system optimization flowchart based on the MADDPG-A2C algorithm.

[0123] Figure 4 It is a schematic diagram of simulation parameter settings.

[0124] Figure 5 This is a comparison chart of reward under different algorithms.

[0125] Figure 6 This is a comparison chart of the convergence of MADDPG-A2C and MADDPG.

[0126] Figure 7 This is a comparison chart of the average transmission rate of the communication link from the vehicle to the infrastructure using different algorithms.

[0127] Figure 8 This is a comparison chart of the mean probability of successful V2V load transmission under different algorithms.

[0128] Fig. 9 It is a comparison chart of the probability of successful V2V load transmission under different algorithms (varies with the load size).

[0129] Fig.10 This is a comparison chart of the communication link throughput from the vehicle to the infrastructure under different algorithms (varies with the load size). DETAILED DESCRIPTION

[0130] The present invention provides a joint optimization method for resource allocation of an Internet of Vehicles based on a dominant strategy, which is used in an Internet of Vehicles environment under a high-speed dynamic network, and comprises the following steps:

[0131] Step 1: Establish the Internet of Vehicles environment;

[0132] like Figure 1 In the scenario shown, there is one base station (BS) and two or more vehicle users. Vehicle users are divided into vehicle-to-infrastructure communication V2I users and vehicle-to-vehicle communication V2V users according to different communication requirements. The vehicle-to-infrastructure communication link is used for communication between vehicles connected to the base station and roadside units, while the vehicle-to-vehicle communication (V2V) technology allows direct information exchange between vehicles. In the present invention, the vehicle-to-infrastructure communication link (V2I) is a set of S V2I , S V2I = {1, 2, ..., M}, the set of vehicle-to-vehicle communication links (V2V) is S V2V , S V2V ={1, 2, ..., N}; M represents the total number of vehicle-to-infrastructure communication links, N represents the total number of vehicle-to-vehicle communication links, and M and N are natural numbers;

[0133] It is assumed that all vehicles participating in the communication are equipped with a single antenna device, and each vehicle-to-infrastructure communication V2I user communicates only through one vehicle-to-infrastructure communication uplink. The spectrum resources of this link are evenly divided into M orthogonal sub-channels, and the number of these orthogonal sub-channels matches the number of vehicle-to-infrastructure communication links. In order to effectively utilize spectrum resources, more than two vehicle-to-infrastructure communication V2V users are allowed to share the spectrum resources of each vehicle-to-infrastructure communication link, thereby realizing spectrum sharing.

[0134] In the real scenario of the Internet of Vehicles, if the vehicle-to-infrastructure communication uplink of the mth subchannel is channel-multiplexed with the nth inter-vehicle communication link, the nth inter-vehicle communication link may be subject to interference signals from other inter-vehicle communication links or vehicle-to-infrastructure communication links, and the mth vehicle-to-infrastructure communication link may also be subject to interference from the transmitter of the inter-vehicle communication link. Taking into account the actual channel interference situation, the present invention defines the signal-to-interference plus noise ratio (SINR) of the mth vehicle-to-infrastructure communication link and the nth inter-vehicle communication link channel as follows:

[0135]

[0136] in, represents the transmission power of the mth vehicle-to-infrastructure communication link, g m，B represents the channel power gain of the mth vehicle-to-infrastructure communication link, σ 2 represents the Gaussian white noise power, It represents the interference degree caused by the nth vehicle-to-vehicle communication link to the mth vehicle-to-vehicle-to-infrastructure communication link, and the expression is:

[0137]

[0138] Among them, the symbol It is used to indicate whether the m-th vehicle-to-infrastructure communication link shares the channel with the n-th vehicle-to-vehicle communication link. The value is 1 if it shares and 0 if it does not. represents the transmission power of the nth inter-vehicle communication link, g n，B [m] represents the interference channel gain caused by the nth vehicle-to-vehicle communication link to the mth vehicle-to-infrastructure communication link. I1 represents the interference caused by the mth vehicle-to-infrastructure communication link to the nth vehicle-to-vehicle communication link, expressed as:

[0139]

[0140] Among them, g m，nrepresents the interference channel gain when the n-th vehicle-to-vehicle communication link is shared with the m-th vehicle-to-infrastructure communication link.

[0141] I2 represents the interference caused by other inter-vehicle communication links except the nth inter-vehicle communication link, and the expression is defined as:

[0142]

[0143] Among them, g k，n [m] represents the interference channel gain of the inter-vehicle communication link set K except the n-th inter-vehicle communication link on the n-th inter-vehicle communication link. The transmit calculation power g is defined as:

[0144] g=ah[m] (6)

[0145] Where a represents the frequency-independent large-scale fading, namely shadow fading and path loss. h[m] represents the frequency-dependent small-scale fading channel gain. This paper considers the direct path. The following definitions are given:

[0146] L V2V Denote the path loss of the V2V user in vehicle-to-vehicle communication, L V2I represents the path loss of the vehicle-to-infrastructure communication V2I user, and the expressions are:

[0147]

[0148] L V2I =128.1+37.6lg(d) (8)

[0149] Wherein, d is a distance parameter, which numerically represents the distance between the transmitting end and the receiving end of the vehicle user (calculated by the Euclidean norm in the present invention); f c Indicates the carrier frequency.

[0150] In the vehicle networking environment constructed by the present invention, large-scale and small-scale fading exist by default, and the channel fading is assumed to be almost the same in each orthogonal subchannel, and different orthogonal subchannels are distributed independently of each other. Therefore, according to the Shannon formula, the total throughput (total rate) of the mth vehicle-to-infrastructure communication link is obtained: and the total throughput (total rate) of the nth inter-vehicle communication link They are:

[0151]

[0152] Where B is the channel bandwidth.

[0153] As mentioned above, the optimization goal of the present invention is to maximize the total throughput of the vehicle-to-infrastructure communication link under the conditions of low latency and high-quality data transmission of the inter-vehicle communication link. Since the driving speed, traffic location status, etc. of the inter-vehicle communication V2V user belong to the key service safety information, it is necessary to ensure the effective transmission probability of the inter-vehicle communication link. Construct the effective transmission probability ψ n for:

[0154]

[0155] Among them, B1 represents the size of the periodically generated vehicle-to-vehicle communication link payload, in bits; △T is the coherence time of the channel.

[0156] Step 2: Establish optimization objectives and constraints;

[0157] Step 2-1, construct the problem;

[0158] The present invention takes improving the total throughput of the vehicle-to-infrastructure communication link and ensuring the transmission reliability of the inter-vehicle communication link as the optimization goal (here the mean probability of successful transmission of the inter-vehicle communication link is considered), and constructs the following optimization problem aimed at ensuring reliability under the premise of considering key service quality indicators such as delay and channel interference.

[0159] The following optimization goals are established:

[0160]

[0161] Where ∑n′ represents the total number of active inter-vehicle communication links; η represents the number of communicating vehicle pairs.

[0162] Construct the following constraints:

[0163]

[0164] Formula (14) represents the maximum transmission power allowed by the vehicle-to-infrastructure communication link and the vehicle-to-vehicle communication link; Formula (15) represents the interference constraint at the link receiving end; Formula (16) represents the minimum tolerable throughput constraint of the vehicle-to-infrastructure communication link; Formula (17) represents the effective transmission probability constraint of the vehicle-to-vehicle communication link; Formula (18) represents the maximum transmission delay allowed by the vehicle-to-vehicle communication link; Formula (19) represents whether the vehicle-to-vehicle communication link reuses the spectrum resources of the vehicle-to-infrastructure communication link, which is a binary function; Formula (20) represents the maximum tolerable throughput constraint of the vehicle-to-vehicle communication link, that is, the maximum transmission rate constraint.

[0165] In summary, in order to allocate spectrum resources of vehicle-to-infrastructure communication links more reasonably and efficiently so that channel conditions can be fully utilized, the present invention adopts a fully distributed resource allocation mechanism, which does not affect the performance of other links while ensuring the reliability of transmission of vehicle-to-vehicle communication links, and improves the total throughput of vehicle-to-infrastructure communication links.

[0166] According to the aforementioned constraints, it can be identified that this problem belongs to the category of non-convex optimization. Traditional optimization techniques may encounter the challenge of falling into local optimality rather than global optimality. In addition, due to the existence of constraint (18), the exponential complexity brought by the sequential decision of coherent time slots brings certain difficulties to traditional optimization methods. Therefore, this problem can be re-conceived as a reinforcement learning task involving multiple agents. Through the interaction between the agent and the environment, it aims to find an optimal strategy to maximize the reward. In this way, the appropriate transmission power is selected for data transmission.

[0167] Step 3: Establish a joint resource allocation optimization algorithm based on MADDPG (Multi-Agent Deep Deterministic Policy Gradient Algorithm, MADDPG) and A2C to achieve spectrum allocation of orthogonal sub-channels for vehicle-to-infrastructure communication and power selection for inter-vehicle communication in complex environments, and maximize the total throughput of the V2I (Vehicle-to-Infrastructure, V2I) link while ensuring high-reliability transmission of the V2V (Vehicle-to-Vehicle, V2V) link.

[0168] The joint optimization algorithm for resource allocation based on MADDPG and A2C regards the Internet of Vehicles scenario as a multi-agent deep reinforcement learning task, in which each communicating vehicle acts as an independent agent. These agents accumulate experience through interaction with the environment and learn to achieve the optimal strategy to optimize rewards. In the fully collaborative learning mode, the agent selects the transmission power for data transmission based on the real-time environmental state. The algorithm is divided into two stages: centralized learning and distributed execution. In the centralized learning stage, the agent trains the policy network and the value function network; in the distributed execution stage, the agent selects the optimal strategy π based on the state information and rewards. * To minimize the loss L(θ). The following is the core concept of the algorithm design for resource allocation using multi-agent deep reinforcement learning.

[0169] Step 3-1, design state space;

[0170] In time slot t, the environment state currently encountered by agent n is recorded as Agent n outputs corresponding actions according to the current strategy π In the entire connected vehicle system, the joint state space of all agents is defined as s t , after the strategy π output, we get the joint action A t , and get a common reward R after performing the action together t In the neural network, the state of agent n is By probability Transfer to the next state For each independent agent, here we take agent n as an example, it can only obtain the local state information and action space of its own environment, but cannot obtain the global state information and action space information of other agents, which are all unknown factors. The expression is:

[0171]

[0172] Therefore, the local channel information that agent n can observe includes the following elements: the channel gain g of its own link n [m], interference channel gain g caused by other inter-vehicle communication links k，n [m], interference channel gain from itself to the base station g n，B [m], interference channel gain g of the mth vehicle-to-infrastructure communication link to the nth vehicle-to-vehicle communication link m，n [m], The expression is:

[0173]

[0174] Step 3-2, design the action space;

[0175] The resource allocation problem of the Internet of Vehicles mainly involves the orthogonal sub-channel allocation and transmission power optimization of inter-vehicle communication. The spectrum resources of each vehicle-to-infrastructure communication link are evenly divided into M orthogonal sub-channels. N inter-vehicle communication links can select one of the M spectrum sub-channels for sharing and adjust the transmission power to perform transmission. Continuous power value The limit is defined as The action selected by agent n at time step t It is expressed as:

[0176]

[0177] Step 3-3, design reward function;

[0178] In the reinforcement learning framework, the reward function plays a core role. It is responsible for evaluating the decision-making performance of the agent, evaluating the behavior of the agent, guiding and shaping the agent's strategy. And provide rewards or penalties based on the effectiveness of the strategy. A reasonably designed reward function can help the agent learn to make effective decisions in a complex environment and improve system performance. The optimization goal of this invention is to improve the total throughput of the vehicle-to-infrastructure communication link and ensure the transmission reliability of the vehicle-to-vehicle communication link. In essence, this is a multi-objective optimization event, which can be simplified to a single-objective optimization problem by weighted linear combination (i.e., weight coefficient method). First of all, it is necessary to set the necessary reward elements. In the actual vehicle networking environment, the two reward elements such as the vehicle-to-infrastructure communication link rate and the vehicle-to-vehicle communication link payload transmission rate are mainly considered. In order to eliminate the influence of negative factors, this paper also considers adding appropriate penalty factors to get closer to the actual environment. For the objective function, consider the completion and incomplete transmission of the vehicle-to-vehicle communication link in a unit time slot t, and design two objective functions. The specific description is as follows: At time t, the total throughput of the vehicle-to-infrastructure communication link is the reward function of the vehicle-to-infrastructure communication link, which is the first objective function. For V2V users, the reward function is set according to the transmission payload rate for the uncompleted V2V communication link within time t. For V2V users that have completed the transmission, a scalar constant is set, which is greater than the payload transmission rate of the V2V communication link, to encourage V2V users to increase the transmission rate. This is the second objective function. In summary, the expressions of the two objective functions are as follows:

[0179]

[0180] Through the weight coefficient method, the final reward function Re is obtained total for:

[0181]

[0182] Among them, λ is the weight coefficient, and λ∈(0,1), indicating the importance of the element reward. is the reward factor, which is set to 0.3 here, V2V success Represents the probability of successful transmission of the inter-vehicle communication link. is the penalty factor, which is 0 when the specific constraint is met and ζ when the constraint is not met. The specific constraint is set to the minimum vehicle-to-infrastructure communication link rate tolerance. The expression of ζ is:

[0183]

[0184] Among them, χ is a tuning parameter used to control the degree of penalty, and η represents the number of communication link pairs.

[0185] During the training process, both the current reward and the expected future reward must be considered to ensure the stability of the reward. Therefore, the main goal of the algorithm is to find the optimal strategy π * , in order to maximize the cumulative expected reward:

[0186]

[0187] Step 3-4, implement the deep deterministic policy gradient algorithm (DDPG);

[0188] Specifically, DDPG mainly includes the following four common neural networks:

[0189] (1) Main Actor Network: As the core policy network, the main actor network is responsible for mapping states to actions and follows a deterministic policy. The network parameter is θ, which maps the state s of the environment at time step t to t As input, and generate corresponding action μ(s t |θ).

[0190] (2) Target Actor Network: The target actor network is responsible for generating target actions. Its initial parameters are set to θ′ and generates actions μ′(s t |θ′). The parameters of this network are gradually updated synchronously from the parameters θ of the main Actor network by adopting a smooth update strategy.

[0191] (3) Main Critic Network: As the core value assessment network, the Main Critic Network is responsible for learning the action value function. With ω as the initial parameter, it receives the state s of the environment. t and the corresponding action a t As input, and output the corresponding state value function, Q(s t , a t |ω). The parameters of the main critic network are updated by gradient descent to minimize the gap between the estimated action value function and the target action value.

[0192] (4) Target Critic Network: The target critic network is a network specifically used to evaluate the value of target actions. It is initially parameterized with ω′ and the output state value function is Q′(s t , a t|ω′). The parameters of the target critic network are gradually updated from the parameters of the main critic network through soft updates to maintain a smooth transition of the two network parameters. The update of the target critic network usually adopts exponential moving average or other smooth update methods.

[0193] DDPG algorithm updates four neural networks in the following ways:

[0194] According to the principle of neural network back propagation, in order to minimize the loss, the parameters θ of the Actor network can be updated by gradient descent. The following chain rule can be used:

[0195]

[0196] The update of the main critic network first uses the output of the target action network as the input of the target critic network to obtain the target value y of the state-value function:

[0197] y=r+γQ′(s′ t ,μ′(s t ′|θ′)|ω′) (30)

[0198] Where γ is a hyperparameter representing the discount rate, s′ t Represents the next state.

[0199] Next, the state-value function is calculated using the Critic network, and the Critic network is updated inversely according to the method of minimizing the loss function. The loss function is defined as follows:

[0200]

[0201] Among them, D n Represents the number of samples drawn. The loss function here is generally expressed as mean square error.

[0202] The target actor network and target critic network optimize their parameters through a progressive update method, which is updated as follows:

[0203] ω′←τω+(1-τ)ω′; θ′←τθ+(1-τ)θ′ (32)

[0204] Here, τ serves as a key hyperparameter to control the smoothness of the update.

[0205] Step 3-5, implement the multi-agent deep deterministic policy gradient (MADDPG) algorithm;

[0206] The MADDPG algorithm is designed for multi-agent scenarios. Compared with the DDPG algorithm, MADDPG implements a series of optimization measures and performance improvements in multi-agent interactions in single-agent scenarios:

[0207] (1) Multi-agent environment modeling: MADDPG takes into account the interactions and influences between multiple agents and regards the environment as a game environment containing multiple agents. The strategy of each agent needs to consider the influence of other agents.

[0208] (2) Shared experience replay buffer: MADDPG uses a shared experience replay buffer from which all agents can sample experience to train their own policy networks. This can improve the efficiency of sample utilization and reduce sample correlation during training.

[0209] (3) Centralized training and distributed execution: The MADDPG algorithm combines the strategies of centralized learning and distributed operation. It optimizes the strategy through centralized learning, while a distributed approach is adopted in the execution phase. That is, a global critic network is used to evaluate the behavior value of all agents, but the strategy network of each agent is trained independently, which can reduce the complexity of training.

[0210] (4) Adaptive Reward: In a multi-agent environment, the behavior of an agent may be affected by other agents. Therefore, MADDPG can design an adaptive reward function to consider the impact of the behavior of other agents on the current agent, thereby better guiding the agent to learn cooperative or competitive strategies.

[0211] Step 3-6, establish the advantage-based actor-critic A2C algorithm; use TD (Temporal Difference Error, TD) error to calculate the advantage function of the A2C algorithm

[0212]

[0213] in, Is to perform an action The total expected reward after is generated by the main Actor network; is in state The state value function under ;

[0214] Step 3-7, establish the MADDPG-A2C algorithm. Due to the defects of the traditional MADDPG algorithm such as low sample efficiency and unstable training, the present invention proposes a vehicle network communication resource allocation optimization algorithm based on multi-agent deep reinforcement learning. The algorithm integrates the traditional MADDPG algorithm and the A2C algorithm, introduces the classic advantage function in the A2C algorithm, and rewrites the loss function in the MADDPG algorithm using the idea of ​​the loss function in the A2C algorithm, thereby improving the training efficiency of the Actor network and the Critic network, thereby improving the performance of the entire vehicle network system.

[0215] The A2C algorithm is an improved version of the traditional AC algorithm. The main difference between this algorithm and the AC algorithm is that it introduces the concept of advantage function, which is used to measure the advantage of an action relative to the average or relative to the baseline. This makes the A2C algorithm more stable and efficient than the traditional AC algorithm.

[0216] Steps 3-7 include:

[0217] This paper proposes an improved MADDPG algorithm based on A2C. With the help of the idea of ​​A2C algorithm, the advantage function is introduced into the MADDPG algorithm. At the same time, the parameter adjustment of the Actor network and the Critic network follows the unique update strategy of the A2C algorithm, so as to improve the stability and efficiency of the system. The overall framework of the optimization algorithm is as follows Figure 2 As shown. Figure 2 It can be seen that the algorithm of the present invention still follows the idea of ​​centralized training and distributed execution of MADDPG in essence. Define each independent agent n, where n∈(1, 2, ..., N), according to their respective states Output corresponding actions through the Policy / Actor network The strategy function of the Actor network is denoted by π n Strategy, in this system, each agent is equipped with an independent Actor network, corresponding to its unique strategy function, and a Critic network. The function of the Critic network is to judge the value of the cumulative reward, usually called the value function network. According to the design of the algorithm, each Critic network must evaluate the strategy output generated by all Actor networks, which means that the input of each Critic network needs to integrate the strategy output of all N Actor networks and finally provide feedback. The feedback value is recorded as Q n After multiple iterations of training, the global optimal solution is finally obtained to maximize the reward return, and as the number of iterations increases, the reward value gradually converges to one point.

[0218] The system optimization flow chart based on MADDPG-A2C algorithm is as follows Figure 3 shown.

[0219] From the previous text, we can know that:

[0220]

[0221] The optimization goal is to maximize the total throughput of the vehicle-to-infrastructure communication link while ensuring high reliability of the inter-vehicle communication link. Since the transmission power of the V2V transmitter is a continuous variable, the continuous action variable space is divided into four discrete levels of normalized values, namely [23, 15, 5, -100] dBm. In the MADDPG algorithm based on the A2C architecture, the main idea is similar to that of MADDPG. Both use two typical network architectures - Actor and Critic. The former is responsible for obtaining environmental state information and taking corresponding actions according to the strategy π; the latter evaluates the pros and cons of the strategy through the state-action value function. The current environmental state of agent n is γ is the discount factor of the reward, ranging from 0 to 1. Therefore, the state-action value function of MADDPG-A2C is:

[0222]

[0223] Like the MADDPG algorithm, the MADDPG-A2C algorithm is essentially a policy gradient algorithm. The main idea is to obtain an optimal policy π′ n , execute the optimal action according to the strategy, and then maximize the expected reward. Unlike MADDPG, the main innovation of the algorithm of this invention is to improve the loss function and parameter update mechanism of the Actor network and the Critic network. The expressions are:

[0224]

[0225] Where y is the target Q value, advantages is the advantage function, and formula (33) has been given.

[0226] When Critic Loss It is continuously differentiable, and ω can be updated by gradient. Here, the update strategy of A2C algorithm is adopted:

[0227]

[0228] Among them, β critic is the learning rate of the Critic network, which is a hyperparameter. δ is the TD error, which has been defined above. In pursuit of the goal of maximizing the total reward, each agent adjusts the evaluation parameters of its Actor network by optimizing the objective function, and the objective function expression is:

[0229]

[0230] Similarly, J θ It is continuously differentiable, and θ can perform gradient updates. Like ω, it uses the advantage-driven asynchronous parallel strategy (Advantage Actor-Critic, A2C) to optimize parameters:

[0231]

[0232] Among them, α actor is the learning rate of the Actor network, which is a hyperparameter. δ is the TD error, which has been defined in the pseudocode above. The update of the two target network parameters adopts the soft update method, and the expression is:

[0233] θ′←τ*θ+(1-τ)*θ′ (40)

[0234] ω′←τ*ω+(1-τ)*ω′ (41).

[0235] The MADDPG-A2C algorithm includes the following steps:

[0236] Step 1: Initialize the Actor network and Critic network θ of all agents n and ω n ;

[0237] Step 2: Initialize the target network θ′ of all agents n and ω′ n ;

[0238] Step 3, initialize the experience replay buffer ReplayBuffer;

[0239] Step 4: Set hyperparameters such as learning rate and target network update rate;

[0240] Step 5: Loop through each training episode from 1 to M:

[0241] Step 6: Initialize the environment status

[0242] Step 7: Update vehicle position and large-scale fading;

[0243] Step 8: Loop from time step 1 to the maximum training set length:

[0244] Step 9. For each agent n, select an action Where ∈ is the exploration noise;

[0245] Step 10: Execute the action And observe the reward r n and the next state

[0246] Step 11: The experience tuple is stored in the experience replay buffer D;

[0247] Step 12: Update status:

[0248] Step 13: For each agent, loop from 1 to N:

[0249] Step 14: Randomly sample a batch of data from the experience replay buffer D;

[0250] Step 15: Define the target Q value as Where γ is the discount factor;

[0251] Step 16: Use the Critic network to estimate the current state value and state-action value And calculate the TD error δ.

[0252] Step 17: Update the critic network parameter ω to minimize the TD error;

[0253] Step 18. Define the advantage function:

[0254] Step 19: Minimize Losses

[0255] Step 20. Calculate the policy gradient of the actor network:

[0256] Step 21: Update actor network parameters θ n :

[0257] Step 22: Update critic network parameters ω n :

[0258] Step 23: Update the target network parameters using the soft update method:

[0259] θ′ n ←τ*θ n +(1-τ)*θ′ n ,

[0260] ω′ n ←τ*ω n +(1-τ)*ω′ n

[0261] The improved MADDPG-A2C algorithm introduces the classic advantage function of the A2C algorithm into the original MADDPG algorithm framework to reduce the mean square error and improve efficiency and stability. The updates of the policy network and the value function network also follow the update principle of the A2C algorithm.

[0262] In a specific embodiment of the present invention, in the constructed vehicle network communication environment, the 3GPP TR 36.885 standard is referred to, and the vehicle network simulator in the city is defined, including mobile stations (vehicles), communication lanes, base stations and other wireless communication network models. The main network simulation parameters are shown in the following table. In the environment, it is assumed that a vehicle-to-vehicle communication link can only correspond to a vehicle-to-infrastructure communication link, and the two constitute a group of communication unit pairs. In the MADDPG-A2C algorithm, the A2C dual network structure is adopted, and the learning rate is set to 0.001 by default. Secondly, in the calculation of the advantage function, in order to predict the stability of future rewards or losses, the cumulative advantage algorithm is adopted to further reduce the mean square error and improve the sample sampling efficiency. In this algorithm, the number of training episodes is set to 2000 times. When the environment is initialized each time, variable Gaussian noise is added to the action selected by each individual agent. The exploration rate uses the classic linear annealing algorithm, which is initially set to 1 and annealed to 0.01 at 2000 episodes, and then remains unchanged during subsequent training. When compared with the other four baseline algorithms, in order to verify the robustness, the present invention uses different sizes of inter-vehicle communication link loads. PyCharm software is used to run the simulation.

[0263] The simulation parameters are set as follows Figure 4 shown.

[0264] The remaining baseline algorithms are:

[0265] (1) MADQN strategy: MADQN uses multi-agent parallel learning, and each agent has its own experience pool and deep Q network. This distributed learning allows the agents to explore the environment and learn at the same time, thereby accelerating the learning speed of the entire system. At the same time, MADQN is suitable for non-steady-state environments, where the actions of the agents affect the state of the environment, and the state of the environment may change over time. By continuously updating Q values ​​and strategies, MADQN can adapt to this dynamic environment and learn to adjust strategies to obtain better returns.

[0266] (2) Random strategy: Under the random strategy, the way V2V vehicles choose to control power transmission follows the random principle of uniform distribution.

[0267] (3) DDPG algorithm: This algorithm has only one intelligent agent and also adopts the dual network structure of Actor network and Critic network. However, its disadvantage is that a single intelligent agent can only observe its own local state, action and reward, so there is a problem of local optimal solution.

[0268] (4) MADDPG algorithm: The biggest advantage of this algorithm is that it adopts a centralized training and distributed execution learning method. However, due to the concurrent learning of multiple agents, it will lead to defects such as poor stability and low sample efficiency.

[0269] Simulation results and analysis:

[0270] The present invention mainly compares the convergence and robustness of the improved algorithm and other baseline algorithms from the perspective of the total throughput of the vehicle-to-infrastructure communication link, the mean probability of successful transmission of the vehicle-to-vehicle communication link, and other indicators, so as to verify that the improved algorithm proposed in this paper has better performance and has certain research prospects.

[0271] Return reward simulation (varies with the number of training sets):

[0272] like Figure 5 As shown in the figure, the rewards of the five algorithms fluctuate and rise with the increase of Episodes (training rounds) and then tend to converge. Among them, the reward of the MADDPG-A2C algorithm is equivalent to that of MADDPG, but slightly higher than the traditional MADDPG algorithm, followed by the Random (random number optimization) algorithm, and then the MADQN algorithm and the DDPG algorithm. Observing the above figure, it is not difficult to find that since MADDPG-A2C uses a typical advantage function to reduce the mean square error, the theoretical reward will be higher than the traditional MADDPG algorithm. For example, the multi-agent deep Q network (MADQN) algorithm combines deep Q learning with multi-agent strategies. Each agent follows the DQN principle and uses a single Q network for learning, so there are problems such as training instability and inaccurate optimal strategy exploration. However, compared with the DDPG algorithm, since DDPG only trains a single agent, there is a problem of local optimal solution, so the performance of the MADQN algorithm is higher than that of the traditional DDPG algorithm.

[0273] Convergence comparison between MADDPG-A2C and MADDPG:

[0274] Since the MADDPG-A2C algorithm incorporates the classic advantage function to reduce the mean square error, it is different from the traditional MADDPG algorithm in the AC network. Therefore, in order to analyze the advantages of the MADDPG-A2C algorithm compared with MADDPG, such as Figure 6 shown. Figure 6The rewards of the two strategies are shown when the training rounds are 2000 times. It can be seen that with the increasing number of training times, the rewards tend to converge. Among them, the rewards obtained by MADDPG-A2C in the first 200 training rounds are roughly similar to those of MADDPG, but after more than 200 rounds, the MADDPG-A2C algorithm converges after 300 training rounds, and the convergence value fluctuates around 91. MADDPG tends to converge after 500 training rounds, and the convergence value fluctuates around 89. It is not difficult to see that the improved MADDPG that integrates the A2C algorithm has better sampling training efficiency than MADDPG, converges faster than MADDPG, and has a correspondingly high convergence value, indicating that the MADDPG-A2C algorithm has higher efficiency and stronger performance in dealing with complex unknown multi-agent environments.

[0275] V2I transmission rate simulation (varies with the number of training sets):

[0276] like Figure 7 As shown in the figure, the average vehicle-to-infrastructure communication link rate of the five algorithms after 2000 trainings. It can be seen that the average V2I rate of the MADDPG-A2C algorithm is higher than that of the other four baseline algorithms. The MADDPG strategy is higher than the Random, MADQN and DDPG algorithms. Since the neural network in the MADQN algorithm only contains one network, the Q network, its performance is naturally not as good as the MADDPG strategy and the improved algorithm. Due to the single-agent training and learning, DDPG cannot obtain the global optimal solution. Therefore, after the iteration of the training set, the average rate cannot obtain the solution corresponding to the optimal strategy, so the rate is low. Due to its own unbiasedness, the Random algorithm can avoid corresponding errors when exploring unknown environments, which helps to avoid falling into local optimal solutions. In theory, the effect is better than MADQN and DDPG. However, the Random algorithm is generally suitable for processing simple environments. The Internet of Vehicles environment built in this article belongs to a complex unknown environment, so the applicability of this algorithm is not better than the MADDPG algorithm.

[0277] Simulation of the probability of successful V2V load transmission (varies with the number of training sets):

[0278] like Figure 8 As shown in the figure, the mean probability of successful V2V load transmission of the five algorithms after 2000 training times is described. It is not difficult to see that the performance index of the MADDPG-A2C algorithm is greater than that of the other four algorithms, indicating that the improved MADDPG algorithm integrated with A2C can better ensure the transmission reliability of the inter-vehicle communication link, that is, the service quality performance is better.

[0279] Simulation of the probability of successful V2V load transmission (varies with load size):

[0280] Fig. 9 The relationship between the mean probability of successful transmission of the inter-vehicle communication link load and the remaining inter-vehicle communication link load size under different algorithms is shown. It can be found from the figure that the average transmission success rate of the five strategies will decrease with the increase of load. This is mainly because the higher the V2V load, the lower the utilization efficiency of the channel conditions, which leads to performance degradation, that is, the mean transmission success rate decreases. The experimental results show that the proposed MADDPG-A2C algorithm is better than MADDPG, Random, MADQN and DDPG algorithms in the mean probability of successful transmission of the inter-vehicle communication link load under different load sizes. With the increase of the remaining transmission load, the MADDPG-A2C algorithm gradually expands its advantage in the inter-vehicle communication link load transmission success rate compared with the other four strategies, showing better performance. In addition, the proposed algorithm can significantly alleviate the negative correlation between the inter-vehicle communication link load size and the transmission success probability, and has better robustness than other algorithms.

[0281] Simulation of vehicle-to-infrastructure communication link throughput (varying with load size):

[0282] Fig.10 The relationship between the total throughput of the vehicle-to-infrastructure communication link and the load size of the remaining inter-vehicle communication link under different algorithms is shown. It is not difficult to see that as the load of the remaining inter-vehicle communication link increases, the total throughput of the vehicle-to-infrastructure communication link shows a downward trend. This is because the increase in the load of the inter-vehicle communication link leads to increased channel contention and interference, which affects the performance of the vehicle-to-infrastructure communication link. When the load of the inter-vehicle communication link increases, the competition and interference of V2V communication will increase, which may lead to a decrease in the transmission efficiency of the vehicle-to-infrastructure communication link. Since the vehicle-to-infrastructure communication link and the inter-vehicle communication link share the same spectrum resources, the increase in the load of the inter-vehicle communication link will make the spectrum resources more crowded, thereby reducing the available bandwidth and transmission efficiency of the vehicle-to-infrastructure communication link. The experimental results show that under the same load conditions, the improved algorithm MADDPG-A2C proposed in the present invention continues to surpass the other four benchmark strategies in terms of the total throughput of the vehicle-to-infrastructure communication link. As the load continues to increase, except for MADDPG and MADDPG-A2C, the total throughput of the vehicle-to-infrastructure communication link of the other three baseline strategies shows a linear downward trend. Only MADDPG and the method of the present invention alleviate the negative correlation between V2V load and V2I total throughput, and have good robustness.

[0283] The present invention provides a joint optimization method for resource allocation of Internet of Vehicles based on advantage strategy. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A joint optimization method for resource allocation of Internet of Vehicles based on dominant strategy, characterized in that: The following steps are involved: Step 1: Establish the Internet of Vehicles environment; Step 2: Establish optimization objectives and constraints; Step 3: Establish a joint resource allocation optimization algorithm based on MADDPG and A2C to achieve spectrum allocation of orthogonal subchannels for vehicle-to-infrastructure communication and power selection for inter-vehicle communication in complex environments, and maximize the total throughput of the V2I link while ensuring high reliability transmission of the V2V link; Step 1 includes: setting a vehicle networking environment including one base station and more than two vehicle users, wherein the vehicle users include vehicle-to-infrastructure communication V2I users and vehicle-to-vehicle communication V2V users; the vehicle-to-infrastructure communication link is used for the communication between the vehicle and the base station and the roadside unit, and the vehicle-to-vehicle communication link is used for direct information exchange between vehicles; the vehicle-to-infrastructure communication link set is S V2I , S V2I = {1, 2, ..., M}, the set of inter-vehicle communication links is S V2V , S V2V ={1, 2, ..., N}; M represents the total number of vehicle-to-infrastructure communication links, N represents the total number of vehicle-to-vehicle communication links, and M and N are natural numbers; Step 1 also includes: assuming that all vehicles participating in the communication are equipped with a single antenna device, each vehicle-to-infrastructure communication V2I user communicates only through one vehicle-to-infrastructure communication uplink, and the spectrum resources of the vehicle-to-infrastructure communication uplink are evenly divided into M orthogonal sub-channels; allowing more than two vehicle-to-vehicle communication V2V users to share the spectrum resources of each vehicle-to-infrastructure communication link to achieve spectrum sharing; Step 1 also includes: defining the signal-to-noise-plus-interference ratio of the mth vehicle-to-infrastructure communication link and the nth vehicle-to-vehicle communication link channel as: in represents the signal-to-noise-and-interference ratio of the mth vehicle-to-infrastructure communication link, represents the transmission power of the mth vehicle-to-infrastructure communication link, g n [m] represents the channel power gain of the nth inter-vehicle communication link on the mth orthogonal subchannel, g m，B represents the channel power gain of the mth vehicle-to-infrastructure communication link, σ 2 represents the Gaussian white noise power, It represents the interference degree caused by the nth vehicle-to-vehicle communication link to the mth vehicle-to-vehicle-to-infrastructure communication link, and the expression is: Among them, the symbol It is used to indicate whether the m-th vehicle-to-infrastructure communication link shares the channel with the n-th vehicle-to-vehicle communication link. The value is 1 if they share the channel and 0 if they do not. represents the transmission power of the nth inter-vehicle communication link, g n，B [m] represents the interference channel gain caused by the nth vehicle-to-vehicle communication link to the mth vehicle-to-infrastructure communication link; I1 represents the interference caused by the mth vehicle-to-infrastructure communication link to the nth vehicle-to-vehicle communication link, which is expressed as: in, represents the transmission power of the mth vehicle-to-infrastructure communication link, g m，n represents the interference channel gain when the nth vehicle-to-vehicle communication link is shared with the mth vehicle-to-infrastructure communication link; I2 represents the interference caused by other inter-vehicle communication links except the nth inter-vehicle communication link, and the expression is: Among them, φ k [m] indicates whether the mth vehicle-to-infrastructure communication link shares a channel with the remaining k vehicle-to-vehicle communication links except the nth one. The value is 1 if it shares a channel, and 0 if it does not. represents the transmission power of the inter-vehicle communication link set k except the nth inter-vehicle communication link on the orthogonal subchannel of the mth vehicle-to-infrastructure communication link, g k，n [m] represents the interference channel gain of the inter-vehicle communication link set k except the n-th inter-vehicle communication link on the n-th inter-vehicle communication link; Define the transmit calculation power g as: g=ah[m] (6) Where a represents the large-scale fading that is independent of frequency; h[m] represents the small-scale fading channel gain that is dependent on frequency; Step 1 also includes: using L V2V Denote the path loss of the V2V user in vehicle-to-vehicle communication, L V2I represents the path loss of the vehicle-to-infrastructure communication V2I user, and the expressions are: L V2I =128.1+37.6lg(d) (8) Among them, d is a distance parameter; f c Indicates the carrier frequency; According to Shannon's formula, the total throughput of the mth vehicle-to-infrastructure communication link is obtained and the total throughput of the nth inter-vehicle communication link They are: Wherein, lb represents the logarithmic expression with base 2, and B is the channel bandwidth; Construct the effective transmission probability ψ n for: Where B1 represents the size of the periodically generated vehicle-to-vehicle communication link payload; △T is the coherence time of the channel; represents the total throughput of the nth inter-vehicle communication link within unit time t, and T is the maximum value of the parameter per unit time; Step 2 includes: establishing the following optimization objectives: Where ∑n′ represents the total number of active inter-vehicle communication links; η represents the number of communicating vehicles; max is the maximum value function; E represents the function expression of the expected value; Construct the following constraints: Among them, p max I represents the maximum transmission power allowed for the vehicle-to-infrastructure communication link and the vehicle-to-vehicle communication link; max represents the maximum interference allowed between the vehicle-to-infrastructure communication link and the vehicle-to-vehicle communication link; R min represents the minimum tolerable transmission rate of the mth vehicle-to-infrastructure communication link; ρ0 represents the minimum setting effective transmission probability; Indicates the minimum tolerable coherence time of the channel; Step 3 includes: A joint optimization algorithm for resource allocation based on MADDPG and A2C is established, and the Internet of Vehicles scenario is regarded as a multi-agent deep reinforcement learning task, in which each communicating vehicle is an independent agent. The agent accumulates experience through interaction with the environment and learns to achieve the optimal strategy to optimize the reward; in the fully collaborative learning mode, the agent selects the transmission power for data transmission according to the real-time environmental state; the algorithm is divided into two stages: centralized learning and distributed execution. In the centralized learning stage, the agent trains the policy network and the value function network; in the distributed execution stage, the agent selects the optimal strategy π according to the state information and reward * To minimize the loss L(θ), specifically: Step 3-1, design state space: In time slot t, the environment state currently encountered by agent n is recorded as Agent n outputs corresponding actions according to the current strategy π In the entire connected vehicle system, the joint state space of all agents is defined as s t , after the strategy π output, we get the joint action A t , after performing the action together, we get a common reward R t ; In the neural network, the state of agent n By probability Transfer to the next state The probability The expression is: in represents the action selection of agent n at time step t; p r represents the probability function; The local channel information observed by agent n is defined as follows: the channel gain g of its own link n [m], interference channel gain g caused by other inter-vehicle communication links k，n [m], interference channel gain from itself to the base station g n，B [m], interference channel gain g of the mth vehicle-to-infrastructure communication link to the nth vehicle-to-vehicle communication link m，n [m]; The expression is: Step 3-2, design action space: The spectrum resources of each vehicle-to-infrastructure communication link are evenly divided into M orthogonal sub-channels. N vehicle-to-vehicle communication links can select one of the M spectrum sub-channels to share and adjust the transmit power to perform transmission. The continuous power value The limit is defined as The action selected by agent n at time step t It is expressed as: Step 3-3, design reward function: Design two reward functions Re V2I and Re V2V : Where Φ is a constant; Through the weight coefficient method, the final reward function Re is obtained total for: Where λ is the weight coefficient, and λ∈(0,1); is the reward factor; V2V success represents the probability of successful transmission of the inter-vehicle communication link; is a penalty factor, which is 0 when a specific constraint condition is met and is a parameter ζ when the constraint condition is not met; the specific constraint condition is set to the minimum vehicle-to-infrastructure communication link rate tolerance; The expression of parameter ζ is: Among them, χ is a tuning parameter; The goal is to find the optimal strategy π * , so that the cumulative expected reward is maximized: Among them, Re max represents the maximum cumulative expected reward, E π is a function expression for the expected value under the π strategy, Represents the decay factor, which is used to control the discount of future rewards; τ represents the time step; when τ = 0, Re t+τ Represents the current reward return. When τ is not 0, Re t+τ represents the relevant rewards at other time steps; Represents the sum of current rewards and future rewards; Step 3-4, implement the deep deterministic policy gradient algorithm DDPG; Steps 3-5, implement the multi-agent deep deterministic policy gradient MADDPG algorithm; Step 3-6, build advantage-based actor-critic A2C algorithm; use TD error to calculate the advantage function of A2C algorithm Step 3-7, establish the MADDPG-A2C algorithm; Step 3-4 includes: the deep deterministic policy gradient algorithm DDPG includes the following neural network: Main Actor Network: The main Actor network is used to map states to actions. The parameters of the main Actor network are recorded as θ. The main Actor network maps the state s of the environment at time step t to t As input, and generate corresponding action μ(s t |θ); Target Actor Network: The target Actor network is used to generate target actions. The initial parameters are set to θ′ and generate actions μ′(s t |θ′); The parameters of the target Actor network are gradually updated synchronously from the parameters θ of the main Actor network by adopting a smooth update strategy; Main Critic Network: The main Critic network is used to learn the action value function. With ω as the initial parameter, the main Critic network will receive the state s of the environment at time step t t and the corresponding action a t As input, and output the corresponding state value function Q(s t , a t |ω); The parameters of the main critic network are updated by gradient descent to minimize the gap between the estimated action value function and the target action value; Target Critic Network: The target critic network is used to evaluate the value of the target action. It is initially parameterized with the parameter ω′ and the output state value function is Q′(s t , a t |ω′); The parameters of the target critic network are gradually updated from the parameters of the main critic network through soft updating; The parameters θ of the main Actor network are updated via gradient descent using the following chain rule: in, represents the gradient of the loss function with respect to the input parameter θ during back propagation, Represents the policy function about action a t The gradient of Q(s t , a t |ω) represents the output of the main Critic network, represents the gradient operator for the parameter θ, μ(s t |θ) represents the output of the main Actor network; The update of the main Critic network first uses the output of the target Actor network as the input of the target Critic network to obtain the target value y of the state value function: y=r+γQ′(s′ t ,μ′(s′ t |θ′)|ω′) (30) Among them, μ′(s′ t |θ′) represents the output of the target Actor network, γ is a hyperparameter, and s′ t Represents the next state; Next, the state value function is calculated using the main critic network, and the main critic network is reversely updated according to the method of minimizing the loss function. The loss function L is defined as: Among them, D n represents the number of samples drawn; Q(s t , a t |ω) represents the output of the main Critic network; The target actor network and the target critic network optimize their parameters through a progressive update method, expressed as: ω′←τω+(1-τ)ω′; θ′←τθ+(1-τ)θ′ (32) Here, τ is a key hyperparameter used to control the smoothness of the update.

2. The method according to claim 1, characterized in that Steps 3-5 include: Multi-agent environment modeling: The MADDPG algorithm regards the environment as a game environment containing more than two agents, and the strategy of each agent needs to consider the influence of other agents; Shared experience replay buffer: The MADDPG algorithm uses a shared experience replay buffer, from which all agents can sample experience to train their own policy networks; Centralized training and distributed execution: The MADDPG algorithm combines the strategies of centralized learning and distributed operation. It optimizes the strategy through centralized learning and uses a distributed approach in the execution phase, that is, a global main critic network is used to evaluate the behavior value of all agents, and the strategy network of each agent is trained independently. Adaptive Reward: The MADDPG algorithm designs an adaptive reward function.

3. The method according to claim 2, characterized in that In steps 3-6, the advantage function The calculation formula is: in, Is to perform an action The total expected reward after is generated by the main Actor network; is in state The state value function under ; Steps 3-7 include: Define each independent agent n, where n∈(1, 2, ..., N), according to their respective states Output corresponding actions through the Actor network The strategy function of the main Actor network is denoted as π n In the built car networking system, each intelligent agent is equipped with an independent main actor network, corresponding to the policy function, and a main critic network; each main critic network must evaluate the policy outputs generated by all actor networks. The input of each main critic network integrates the policy outputs of all N main actor networks and finally gives feedback. The feedback value is recorded as Q n , after iterative training, the global optimal solution is finally obtained to maximize the reward return, and as the number of iterations increases, the reward value gradually converges to one point. The optimization goal is: The optimization goal is to maximize the total throughput of the vehicle-to-infrastructure communication link under the condition of high reliability transmission of the inter-vehicle communication link; The current environment state of agent n is γ is the discount factor of the reward, and the state-action value function of the MADDPG-A2C algorithm is obtained as: in, Indicates that for agent n, the current main Actor network takes action Total return obtained when Indicates that according to the next action and the next state The obtained Q value represents the expected reward for the next batch of states; E is the mean function; Loss function for the main Actor network Actor loss And the loss function of the main critic network Critic Loss And the parameter update mechanism has been improved, the expressions are: Among them, y is the output Q value of the target Critic network, represents the expected total reward after taking N actions, and advantages is the advantage function; Indicates taking action in a given state And the expected return that can be obtained by following the π strategy; When Critic Loss It is continuously differentiable, ω can be updated with gradients, and the update strategy of the A2C algorithm is adopted: Among them, β criiic is the learning rate of the main critic network; δ is the TD error; In pursuit of the goal of maximizing the total reward, each agent adjusts the evaluation parameters of the main Actor network by optimizing the objective function. The objective function J θ The expression is: The advantage-driven asynchronous parallel strategy A2C is used to optimize parameters: Among them, α actor The learning rate of the main Actor network is a hyperparameter; The update of the target Actor network and the target Critic network parameters are both soft updated, and the expression is: θ′←τ*θ+(1-τ)*θ′ (40) ω′←τ*ω+(1-τ)*ω′ (41).

Images