C-V2X Internet of Vehicles resource allocation method based on deep reinforcement learning

Abstract

The invention discloses a C-V2X Internet of Vehicles resource allocation method based on deep reinforcement learning. Aiming at a distributed resource allocation problem in a cellular Internet of Vehicles environment, a dual-objective optimization problem is designed based on a multi-agent reinforcement learning technology according to application requirements of links. In a cooperative C-V2X scene, spectrum allocation and continuous power control are realized through deep reinforcement learning, and the problem of quantization errors caused by an existing motion space discrete resource allocation algorithm is solved. The method comprises the following steps: firstly, modeling an established distributed resource allocation problem into a Markov decision process; and after network parameters are randomly initialized, a resource allocation problem is solved based on an MAPPO algorithm framework. Furthermore, the Pareto frontier composed of a series of Pareto optimal solutions is efficiently solved through parameter migration and reinforcement learning so as to cope with a large amount of resource consumption in the training process caused by weight change. According to the method, the resource allocation result is finally returned, so that the user service quality is ensured, and an efficient and intelligent traffic system is realized.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and more specifically, to a C-V2X vehicle network resource allocation method based on deep reinforcement learning. Background Art

[0002] Vehicle-to-Everything (V2X) communication is a core component of smart cars and is of vital importance in improving road safety, realizing intelligent traffic management and autonomous driving. The types of vehicle-to-everything communication include: vehicle-to-vehicle (V2V), vehicle-to-network (V2N), vehicle-to-pedestrian (V2P), vehicle-to-infrastructure (V2I), etc. Through V2X communication, vehicles can exchange information with any entity that may affect them, which is a typical application of IoT technology in the transportation field.

[0003] The communication of the Internet of Vehicles is mainly based on two technologies: Dedicated Short Range Communication (DSRC) and Cellular-V2X (C-V2X). In the early days, vehicles mainly used DSRC technology for external communication. However, DSRC technology is limited by defects such as short transmission distance, small amount of transmission data, and high system deployment cost, and gradually cannot meet the increasing communication needs of the Internet of Vehicles. Compared with DSRC, C-V2X has many advantages, including pre-existing infrastructure, controllable latency, larger coverage area, strong scalability, etc., even in high mobility scenarios. Therefore, in recent years, C-V2X technology has gradually become the mainstream choice for realizing Internet of Vehicles communication and has attracted the attention of researchers. In C-V2X, the resource allocation (RA) mechanism plays an important role in the performance of the network. Therefore, it is of great practical significance to study the resource allocation algorithm for C-V2X.

[0004] 3GPP standardized LTE-V2X in its Rel.14, which introduced two modes of C-V2X, Mode 3 and Mode 4. Both modes enable vehicles to perform D2D communications through licensed cellular frequency bands, but differ in the method of radio resource allocation. In Mode 3, the cellular network (usually the BS) manages the radio resources used by the vehicle for its direct vehicle communications, which is equivalent to centralized resource allocation at the BS. In Mode 4, the vehicle autonomously selects radio resources, which is equivalent to distributed resource allocation at the vehicle. Mode 4 greatly improves the autonomy of the vehicle, allowing the vehicle to work with or without a cellular network.

[0005] It is worth noting that in the distributed resource allocation of Mode 4, interference caused by multiplexing still exists. Therefore, it is still necessary to adopt a reasonable resource allocation algorithm to control interference in order to improve the performance of the C-V2X network.

[0006] At present, there have been many studies on distributed resource allocation methods in the Internet of Vehicles. Reinforcement learning methods have performed very well in wireless resource management. The emergence of multi-agent reinforcement learning methods has made distributed management of wireless resources in the Internet of Vehicles possible. "Deep reinforcement learning based resource allocation for V2Vcommunications"[H.Ye, GYLi and B.-HFJuang, IEEE Trans.Veh.Technol., vol.68, no.4, pp.3163-3173, Apr.2019.] proposed a new resource allocation strategy for heterogeneous C-V2X scenarios that meets practical applications based on the DQN algorithm to find the optimal subband and transmission power level for each V2V user in V2V communication while meeting the requirements of low latency. "Distributed deep deterministic policy gradient for power allocation control in D2D-based V2Vcommunications"[KKNguyen, TQDuong, NAVien, N.-A.Le-Khac and LDNguyen, IEEE Access, vol.7, pp.164533-164543, 2019] Aiming at the power allocation problem in D2D-based V2V communication, two new methods based on deep deterministic policy gradient algorithm are proposed with energy efficiency as the optimization goal. "Intelligent Energy Efficient Resource Allocation for URLLC Services in IoV Networks" [RM Sohaib, O. Onireti, Y. Sambo, R. Swash and M. Imran, PIMRC, Kyoto, Japan, 2022.] proposes a new intelligent resource allocation method based on deep reinforcement learning (DRL), which combines V2V user delay and reliability constraints to maximize weighted comprehensive efficiency.

[0007] However, the existing DRL-based IoV resource allocation algorithms still have shortcomings, mainly in the following two aspects: First, most existing works discretize continuous power, which leads to quantization errors and reduces algorithm performance. Second, based on the actual communication link requirements, some works have studied the weighted optimization of V2I link rate and V2V link reliability, but these works have preset weights in training, so as to solve for a specific pair of weights. If the weights change, the retraining process will consume a lot of resources. Summary of the invention

[0008] The present invention discloses a C-V2X vehicle network resource allocation method based on deep reinforcement learning. Aiming at the distributed resource allocation problem in the cellular vehicle network environment, the present invention designs a dual-objective optimization problem based on multi-agent reinforcement learning technology and according to the application requirements of the link. In the scenario of cooperative C-V2X, spectrum allocation and continuous power control are realized through deep reinforcement learning, which solves the quantization error problem caused by the existing resource allocation algorithm with discrete action space. First, the established distributed resource allocation problem is modeled as a Markov decision process. After randomly initializing the network parameters, the resource allocation problem is solved based on the MAPPO algorithm framework. Furthermore, the Pareto frontier composed of a series of Pareto optimal solutions is efficiently solved through parameter migration and reinforcement learning to cope with the large amount of resource consumption in the training process caused by the change of weights. The present invention finally returns the resource allocation result, which helps to ensure the quality of user service and realize an efficient and intelligent transportation system.

[0009] To achieve the above objectives, the technical solution of the present invention is as follows:

[0010] A C-V2X vehicle network resource allocation method based on deep reinforcement learning, characterized by comprising the following steps:

[0011] S1: Initialize the position and channel characteristics of each vehicle user and calculate the signal-to-interference-noise ratio of each link;

[0012] S2: Based on the actual application of the link, construct the distributed resource allocation problem in the Internet of Vehicles environment;

[0013] S3: Modeling the distributed resource allocation problem as a Markov decision process;

[0014] S4: Randomly initialize network parameters;

[0015] S5: Solve the constructed resource allocation problem based on the MAPPO algorithm framework;

[0016] S6: Efficient solution of the Pareto frontier of bi-objective optimization problems based on parameter transfer and reinforcement learning.

[0017] Preferably, the calculation of the signal to interference and noise ratio of each link in step S1 is specifically as follows:

[0018] S1.1: The set of M cellular vehicle users (CUEs) communicating with the base station via a cellular link, i.e., a V2I link, is defined as The set of K pairs of vehicle users (DUE) communicating via D2D, i.e., V2V links is defined as Assuming that all V2I links are pre-assigned with orthogonal spectrum sub-bands, and the mth sub-band corresponds to the mth V2I link, the transmission power of the mth V2I link is The V2V link achieves spectrum sharing through frequency band selection and power control. represents the control of the transmission power of the kth V2V link. In addition, a binary decision variable ρ is introduced k [m] is the frequency band allocation variable for the kth V2V link: k When [m] = 1, it means that the kth V2V link and the mth V2I link share the mth sub-band; when ρ k When [m] = 0, it means that the corresponding two links do not share the sub-band. Therefore, the channel gain of each communication link can be expressed as follows: represents the channel gain from the transmitting device to the base station of the mth V2I link, represents the interference channel gain from the transmitter of the mth V2I link to the receiver of the kth V2V link, expressed as g k [m] represents the channel gain from the sending device to the receiving device of the kth V2V link, expressed in g k′,k [m] represents the interference channel gain from the transmitting device of the k′th V2V link to the receiving device of the kth V2V link, expressed as g k,B [m] represents the interference channel gain from the transmitting device of the kth V2V link to the base station on the mth sub-band;

[0019] S1.2: The signal-to-interference-to-noise ratio expression of the mth V2I link is:

[0020]

[0021] The signal-to-interference-to-noise ratio expression of the kth V2V link is:

[0022]

[0023] In the formula, σ 2 is the ambient noise power.

[0024] Preferably, the distributed resource allocation problem in the Internet of Vehicles environment constructed according to the actual application of the link in step S2 is specifically:

[0025] S2.1: The transmission rate of the mth V2I link can be calculated according to the Shannon formula as follows:

[0026]

[0027] Similarly, the transmission rate of the kth V2V link is expressed as follows:

[0028]

[0029] S2.2: Since V2I links are often used to support vehicles to obtain in-vehicle entertainment-related content from base stations (BS), such links require high transmission rates. For V2V links, the regular transmission of safety-related information requires high reliability. Therefore, by limiting the transmission time period T to a fixed period, the kth V2V link needs to complete a transmission of size B. k The data transmission of V2V link is guaranteed to ensure the high reliability of periodic transmission of V2V link. Let t represent the tth time slot in the time period T, then the reliability constraint of V2V link can be expressed as:

[0030]

[0031] Among them, Δ T is the channel coherence time, is the transmission rate of the kth V2V link in time slot t.

[0032] S2.3: Further transform the reliability constraints into a form that is convenient for problem modeling:

[0033]

[0034] B k (t) is the size of data that has not been transmitted yet at time slot t. β is a constant greater than 0, and its value is slightly greater than the maximum possible transmission rate of each communication link.

[0035] S2.4: The established bi-objective optimization problem can be expressed as follows:

[0036]

[0037] Among them, the optimization goal represents the transmission rate of the V2I link within the transmission period T, and the optimization target Indicates the reliability of the V2V link to complete the transmission requirements. The weight factor λ∈(0,1) is used to balance the optimization objectives f1 and f2. Constraint (a) indicates that the value of the frequency band allocation variable is 0 or 1. Constraint (b) indicates that in the same time slot, a V2V link can only share the frequency band of a V2I link at most. Constraint (c) limits the maximum and minimum values ​​of the V2V link transmission power.

[0038] Preferably, in step S3, the distributed resource allocation problem is modeled as a Markov decision process, specifically:

[0039] S3.1: The sending device of each V2V link is used as an agent. Each agent observes the environment and makes decisions on actions and gains benefits, thereby updating the policy network and learning its own resource sharing strategy. Each V2V link together forms a multi-agent system. At each time slot t, each V2V agent k obtains the observation value of the environment Then take action After all agents take action, they receive a reward R t , the environment with probability p(S t+1 |S t ,A)Transfer to the next state S t+1 As the elements of reinforcement learning, we will next clarify the state space, action space, and reward function.

[0040] S3.2: The local observation space of a single agent k contains: V2V link channel gain Interference channel gain of the corresponding V2I link Interference channel gain of other V2V links And the interference power caused by the interference link Interference channel gain of V2V link to base station Remaining transmission time T k , the size of data to be transmitted is B k . Then the observation space of a single agent k can be expressed as:

[0041]

[0042] S3.3: Each agent autonomously selects the transmission band and controls the transmission power according to the strategy to achieve good system performance. There are a total of M optional sub-bands, each of which is already occupied by a V2I link. The agent needs to choose to share spectrum resources with a V2I link to improve the utilization of precious spectrum resources. The transmission power of the V2V link takes values ​​in the continuous interval [-100,23]dBm, rather than discretizing the action space by quantization. This solves the quantization error caused by the existing resource allocation problem of discretizing the action space. The combination of transmission band selection and transmission power control forms the hybrid action space of the agent as follows:

[0043]

[0044] S3.4: Since the agents that make up the multi-agent system need to collaborate to achieve good performance of the system, we let the agents share the total reward value of the system. The reward value is related to the optimization objectives f1 and f2, and the reward function expression is as follows:

[0045]

[0046] In the above formula, λ1 and λ2 are weight factors of optimization targets f1 and f2 respectively, and satisfy λ1+λ2=1.

[0047] Preferably, in step S4, the agent is initialized based on the Proximal Policy Optimization (PPO) algorithm, which includes two main neural networks: the Actor network and the Critic network. Specifically:

[0048] S4.1: Construct a Critic network, which is used to evaluate the value function of the state. The input of the network is the observation vector obtained by the agent, and the output is the estimated value of the state. It is a three-layer fully connected neural network and uses the ReLU activation function. The corresponding loss function is the mean square error between the state value estimate and the actual return, which is optimized using the Adam optimizer.

[0049] S4.2: Construct the Actor network, which is used for the strategy generation of the intelligent agent. It consists of two sub-networks: one for selecting sub-bands and the other for transmitting power control. Each sub-network consists of three fully connected layers. The channel selection network outputs a category distribution, which is sampled to obtain the channel selection; the power control network outputs the mean and standard deviation of a normal distribution, which is sampled to obtain the power action. Similar to the Critic network, the Actor network is trained using the Adam optimizer.

[0050] S4.3: Before training begins, randomly initialize the network’s model parameters.

[0051] Preferably, the process of solving the constructed resource allocation problem based on the MAPPO algorithm framework in step S5 is specifically as follows:

[0052] S5.1: In each training cycle T, the agent interacts with the environment to obtain observation variables, perform actions, and reward data, that is, (O t ,a t ,R t ). Calculate the discounted return R′ t .

[0053] S5.2: Take the output V(S of the Critic network) t ) measures the long-term benefits that each state can bring. The expression of the advantage function is sdv = R′ t -V(S t ). The loss function of the Critic network is the predicted value function V(S t ) and the actual discounted return R′ t The mean square error between them is expressed as: The training of the Critic network is done by minimizing the loss function Loss c accomplish.

[0054] S5.3: Update the parameters of the old Actor policy network to the parameters of the current Actor policy network so that the current policy network can learn in a new round of training, while the old policy network remains unchanged.

[0055] S5.4: Since the parameters of the new and old Actor strategy networks are different, the action probabilities obtained for the same state and action are different. Assuming that the action probability obtained by the old strategy is p1 and the action probability obtained by the new strategy is p2, calculate the parameter update ratio. The expression of the ratio is as follows:

[0056]

[0057] S5.5: The expression of the loss function of the Actor network obtained from the update ratio is:

[0058] Loss A =-E[ratio.adv]

[0059] S5.6: In order to limit the excessive update range of the new strategy and ensure good convergence, the PPO algorithm trims the loss of the Actor network. Assuming that the trimming factor is a constant ∈, the loss functions of the discrete band selection action and the continuous power control part of the Actor network after trimming are:

[0060]

[0061] S5.7: The Critic network and the Actor network respectively update the network parameters by minimizing the loss function to complete a training process.

[0062] Preferably, in step S6, the training process for a specific weight is extended to the solution of all possible weights through parameter migration, specifically:

[0063] S6.1: For the objective function of the dual-objective optimization problem to be solved, the weights λ1 and λ2 are used to balance the index f1 of the V2I link and the index f2 of the V2V link. n The decomposed nth sub-objective function is as follows:

[0064]

[0065] If n = 1, the first possible weight is If n = 2, the possible weight of the second pair is If n = 3, the third possible weight is By analogy, when n=N, the possible weights of the Nth pair are

[0066] S6.2: For the N sub-optimization problems with different weights, each sub-problem is modeled as a Markov process.

[0067] S6.3: Since the weights of two adjacent sub-problems are similar, the optimal solutions are similar, and the neural network parameters that can obtain the optimal solutions are similar. For two adjacent sub-problems, the neural network parameters for the latter pair of weights are initialized to the neural network parameters trained for the former pair of weights.

[0068] S6.4: Solve N sub-problems in sequence, and greatly improve the efficiency of solving N-1 sub-problems through parameter migration. Finally, a Pareto front consisting of a series of Pareto optimal solutions for the constructed dual-objective optimization problem is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0070] Figure 2 Provided as an example, the reward value of the MAPPO method of the present invention varies with the training cycle.

[0071] Figure 3 In the test environment provided by the embodiment, the average transmission success rate of the V2V link changes with the transmission load.

[0072] Figure 4 As provided in the embodiment, the remaining transmission load of each V2V link of the MAPPO method of the present invention varies with time.

[0073] Figure 5 In the embodiment provided, the residual transmission load of each V2V link based on the Random algorithm for comparison varies with time. DETAILED DESCRIPTION

[0074] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0075] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0076] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0077] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0078] Example 1

[0079] This example provides a C-V2X vehicle network resource allocation method based on deep reinforcement learning. Figure 1 As shown, the following steps are included:

[0080] S1: Initialize the position and channel characteristics of each vehicle user and calculate the signal-to-interference-noise ratio of each link;

[0081] S2: Based on the actual application of the link, construct the distributed resource allocation problem in the Internet of Vehicles environment;

[0082] S3: Modeling the distributed resource allocation problem as a Markov decision process;

[0083] S4: Randomly initialize network parameters;

[0084] S5: Solve the constructed resource allocation problem based on the MAPPO algorithm framework;

[0085] S6: Efficient solution of the Pareto frontier of bi-objective optimization problems based on parameter transfer and reinforcement learning.

[0086] Example 2

[0087] This embodiment further discloses the following contents based on the embodiment 1:

[0088] The calculation of the signal to interference and noise ratio of each link in step S1 is specifically as follows:

[0089] S1.1: The set of M cellular vehicle users (CUEs) communicating with the base station via a cellular link, i.e., a V2I link, is defined as The set of K pairs of vehicle users (DUE) communicating via D2D, i.e., V2V links is defined as Assuming that all V2I links are pre-assigned with orthogonal spectrum sub-bands, and the mth sub-band corresponds to the mth V2I link, the transmission power of the mth V2I link is The V2V link achieves spectrum sharing through frequency band selection and power control. represents the control of the transmission power of the kth V2V link. In addition, a binary decision variable ρ is introduced k [m] is the frequency band allocation variable for the kth V2V link: k When [m] = 1, it means that the kth V2V link and the mth V2I link share the mth sub-band; when ρ k When [m] = 0, it means that the corresponding two links do not share the sub-band. Therefore, the channel gain of each communication link can be expressed as follows: represents the channel gain from the transmitting device to the base station of the mth V2I link, represents the interference channel gain from the transmitter of the mth V2I link to the receiver of the kth V2V link, expressed as g k [m] represents the channel gain from the sending device to the receiving device of the kth V2V link, expressed in g k′,k [m] represents the interference channel gain from the transmitting device of the k′th V2V link to the receiving device of the kth V2V link, expressed as g k,B [m] represents the interference channel gain from the transmitting device of the kth V2V link to the base station on the mth sub-band;

[0090] S1.2: The signal-to-interference-to-noise ratio expression of the mth V2I link is:

[0091]

[0092] The signal-to-interference-to-noise ratio expression of the kth V2V link is:

[0093]

[0094] In the formula, σ 2 is the ambient noise power.

[0095] The distributed resource allocation problem in the Internet of Vehicles environment constructed according to the actual application of the link in step S2 is specifically:

[0096] S2.1: The transmission rate of the mth V2I link can be calculated according to the Shannon formula as follows:

[0097]

[0098] Similarly, the transmission rate of the kth V2V link is expressed as follows:

[0099]

[0100] S2.2: Since V2I links are often used to support vehicles to obtain in-vehicle entertainment-related content from base stations (BS), such links require high transmission rates. For V2V links, the regular transmission of safety-related information requires high reliability. Therefore, by limiting the transmission time period T to a fixed period, the kth V2V link needs to complete a transmission of size B. k The data transmission of V2V link is guaranteed to ensure the high reliability of periodic transmission of V2V link. Let t represent the tth time slot in the time period T, then the reliability constraint of V2V link can be expressed as:

[0101]

[0102] Among them, Δ T is the channel coherence time, is the transmission rate of the kth V2V link in time slot t.

[0103] S2.3: Further transform the reliability constraints into a form that is convenient for problem modeling:

[0104]

[0105] B k (t) is the size of data that has not been transmitted yet at time slot t. β is a constant greater than 0, and its value is slightly greater than the maximum possible transmission rate of each communication link.

[0106] S2.4: The established bi-objective optimization problem can be expressed as follows:

[0107]

[0108] Among them, the optimization goal represents the transmission rate of the V2I link within the transmission period T, and the optimization target Indicates the reliability of the V2V link to complete the transmission requirements. The weight factor λ∈(0,1) is used to balance the optimization objectives f1 and f2. Constraint (a) indicates that the value of the frequency band allocation variable is 0 or 1. Constraint (b) indicates that in the same time slot, a V2V link can only share the frequency band of a V2I link at most. Constraint (c) limits the maximum and minimum values ​​of the V2V link transmission power.

[0109] In step S3, the distributed resource allocation problem is modeled as a Markov decision process, specifically:

[0110] S3.1: The sending device of each V2V link is used as an agent. Each agent observes the environment and makes decisions on actions and gains benefits, thereby updating the policy network and learning its own resource sharing strategy. Each V2V link together forms a multi-agent system. At each time slot t, each V2V agent k obtains the observation value of the environment Then take action After all agents take action, they receive a reward R t , the environment with probability p(S t+1 |S t ,A)Transfer to the next state S t+1 As the elements of reinforcement learning, we will next clarify the state space, action space, and reward function.

[0111] S3.2: The local observation space of a single agent k contains: V2V link channel gain Interference channel gain of the corresponding V2I link Interference channel gain of other V2V links And the interference power caused by the interference link Interference channel gain of V2V link to base station Remaining transmission time T k , the size of data to be transmitted is B k . Then the observation space of a single agent k can be expressed as:

[0112]

[0113] S3.3: Each agent autonomously selects the transmission band and controls the transmission power according to the strategy to achieve good system performance. There are a total of M optional sub-bands, each of which is already occupied by a V2I link. The agent needs to choose to share spectrum resources with a V2I link to improve the utilization of precious spectrum resources. The transmission power of the V2V link is taken in the continuous interval [-100,23]dBm, rather than discretizing the action space by quantization, which improves the performance of the existing resource allocation algorithm with discrete action space. The combination of transmission band selection and transmission power control forms the hybrid action space of the agent as follows:

[0114]

[0115] S3.4: Since the agents that make up the multi-agent system need to collaborate to achieve good performance of the system, we let the agents share the total reward value of the system. The reward value is related to the optimization objectives f1 and f2, and the reward function expression is as follows:

[0116]

[0117] In the above formula, λ1 and λ2 are weight factors of optimization targets f1 and f2 respectively, and satisfy λ1+λ2=1.

[0118] In step S4, the agent is initialized based on the Proximal Policy Optimization (PPO) algorithm, which includes two main neural networks: the Actor network and the Critic network. Specifically:

[0119] S4.1: Construct a Critic network, which is used to evaluate the value function of the state. The input of the network is the observation vector obtained by the agent, and the output is the estimated value of the state. It is a three-layer fully connected neural network and uses the ReLU activation function. The corresponding loss function is the mean square error between the state value estimate and the actual return, which is optimized using the Adam optimizer.

[0120] S4.2: Construct the Actor network, which is used for the strategy generation of the intelligent agent. It consists of two sub-networks: one for selecting sub-bands and the other for transmitting power control. Each sub-network consists of three fully connected layers. The channel selection network outputs a category distribution, which is sampled to obtain the channel selection; the power control network outputs the mean and standard deviation of a normal distribution, which is sampled to obtain the power action. Similar to the Critic network, the Actor network is trained using the Adam optimizer.

[0121] S4.3: Before training begins, randomly initialize the network’s model parameters.

[0122] The process of solving the constructed resource allocation problem based on the MAPPO algorithm framework in step S5 is specifically as follows:

[0123] S5.1: In each training cycle T, the agent interacts with the environment to obtain observation variables, perform actions, and reward data, that is, (O t ,a t ,R t ). Calculate the discounted return R′ t .

[0124] S5.2: Take the output V(S of the Critic network) t ) measures the long-term benefits that each state can bring. The expression of the advantage function is adv = R′ t -V(S t ). The loss function of the Critic network is the predicted value function V(S t ) and the actual discounted return R′ t The mean square error between them is expressed as: The training of the Critic network is done by minimizing the loss function Loss c accomplish.

[0125] S5.3: Update the parameters of the old Actor policy network to the parameters of the current policy network so that the current policy network can learn in a new round of training, while the old policy network remains unchanged.

[0126] S5.4: Since the parameters of the new and old Actor strategy networks are different, the action probabilities obtained for the same state and action are different. Assuming that the action probability obtained by the old strategy is p1 and the action probability obtained by the new strategy is p2, calculate the parameter update ratio. The expression of the ratio is as follows:

[0127]

[0128] S5.5: The expression of the loss function of the Actor network obtained from the update ratio is:

[0129]

[0130] S5.6: In order to limit the excessive update range of the new strategy and ensure good convergence, the PPO algorithm trims the loss of the Actor network. Assuming that the trimming factor is a constant ∈, the loss functions of the discrete band selection action and the continuous power control part of the Actor network after trimming are:

[0131]

[0132] S5.7: The Critic network and the Actor network respectively update the network parameters by minimizing the loss function to complete a training process.

[0133] In step S6, the training process for a specific weight is extended to the solution of all possible weights through parameter migration, specifically:

[0134] S6.1: For the objective function of the dual-objective optimization problem to be solved, the weights λ1 and λ2 are used to balance the index f1 of the V2I link and the index f2 of the V2V link. n The decomposed nth sub-objective function is as follows:

[0135]

[0136] If n = 1, the first possible weight is If n = 2, the possible weight of the second pair is If n = 3, the third possible weight is By analogy, when n=N, the possible weights of the Nth pair are

[0137] S6.2: For the N sub-optimization problems with different weights, each sub-problem is modeled as a Markov process.

[0138] S6.3: Since the weights of two adjacent sub-problems are similar, the optimal solutions are similar, and the neural network parameters that can obtain the optimal solutions are similar. For two adjacent sub-problems, the neural network parameters for the latter pair of weights are initialized to the neural network parameters trained for the former pair of weights.

[0139] S6.4: Solve N sub-problems in sequence, and greatly improve the efficiency of solving N-1 sub-problems through parameter migration. Finally, a Pareto front consisting of a series of Pareto optimal solutions for the constructed dual-objective optimization problem is obtained.

[0140] Example 3

[0141] This embodiment provides the following specific embodiments based on Embodiment 1 and Embodiment 2:

[0142] The simulation parameters are set as follows: the number of V2I links M = 4, the number of V2V links K = 4, the carrier frequency f c =2GHz, single link bandwidth is 1MHz, base station antenna height is 25m, base station antenna gain is 8dBi, base station receiver noise factor is 5dB, vehicle antenna height is 1.5m, vehicle antenna gain is 3dBi, vehicle receiver noise factor is 9dB, vehicle speed v=36km / h, V2I link transmission power Noise power σ 2 =-114dBm, the transmission period of the V2V link is T = 100ms, the transmission load of the V2V link is B = [1,2,…,8]*1060Bytes, the shadow fading of the V2I link is a log-normal distribution with a mean of 0 and a standard deviation of 8dB, the shadow fading of the V2V link is a log-normal distribution with a mean of 0 and a standard deviation of 3dB, and the fast fading update period is 1ms. The transmission power of the V2V link Take the value in the interval [-100,23]dBm.

[0143] Figure 2 In the figure, the reward value of the MAPPO algorithm changes with the training cycle during the training process, reflecting the convergence performance of the algorithm. During the training process, the transmission load of the V2V link is set to B = 2*1060Bytes, that is, in each cycle, each V2V needs to complete the transmission of 2*1060Bytes of data. In order to facilitate training, the reward value is normalized to a negative value. Figure 2 It can be seen that this method can converge to a high reward value relatively quickly.

[0144] Figure 3In the paper, the variation of V2V link transmission success rate with transmission load of three different resource allocation methods under the test environment is shown. Compared with the single-agent reinforcement learning method SARL and the random selection algorithm Random, the MAPPO method of the present invention can achieve a higher V2V link transmission success rate at each load. And the transmission load of the V2V link is fixed to B=2*1060Bytes during the training process, and the proposed MAPPO resource allocation method still shows a certain robustness for larger transmission loads.

[0145] Figure 4 In the figure, the variation of the remaining transmission load of each V2V link of the method proposed in the present invention within a transmission cycle T under the test environment is shown. It can be seen that in the simulation test of the method proposed in the present invention, all V2V links can quickly complete their own transmission tasks. Among them, V2V link 1 and V2V link 4 first wait for V2V link 2 and V2V link 3 to quickly complete their own transmission tasks, and then perform their own transmission after V2V link 2 and V2V link 3 complete the transmission, thereby completing the transmission of all links in about 40ms. This reflects the good cooperation between the various intelligent agents and the effectiveness of the method of the present invention.

[0146] Figure 5 In the figure, the variation of the remaining transmission load of each V2V link in a transmission cycle T of the Random method is shown in the test environment. It can be seen that in the simulation test of the Random method, only two V2V links can complete their own transmission tasks. Figure 5 and Figure 4 The comparison can better reflect the excellent performance of the method proposed in the present invention.

[0147] The above examples show that the method of the present invention is aimed at the distributed resource allocation problem in the cellular vehicle networking environment. In the cooperative C-V2X scenario, spectrum allocation and continuous power control are realized through deep reinforcement learning, which solves the quantization error problem caused by the existing resource allocation algorithm with discrete action space, and can ensure good V2V link reliability and vehicle networking system performance. Through parameter migration, the method of the present invention copes with the large amount of resource consumption in the training process caused by the change of weights.

[0148] The same or similar reference numerals correspond to the same or similar components;

[0149] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present patent.

[0150] Obviously, the above examples of the present invention are only examples for clearly explaining the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A C-V2X vehicle network resource allocation method based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Initialize the position and channel characteristics of each vehicle user and calculate the signal-to-interference-noise ratio of each link; S2: Based on the actual application of the link, construct the distributed resource allocation problem in the Internet of Vehicles environment; S3: Modeling the distributed resource allocation problem as a Markov decision process; S4: Randomly initialize network parameters; S5: Solve the constructed resource allocation problem based on the MAPPO algorithm framework; S6: Efficient solution of the Pareto frontier of bi-objective optimization problems based on parameter transfer and reinforcement learning.

2. The C-V2X vehicle network resource allocation method based on deep reinforcement learning according to claim 1 is characterized in that: The calculation of the signal to interference and noise ratio of each link in step S1 is specifically as follows: S1.1: The set of M cellular vehicle users (CUEs) communicating with the base station via a cellular link, i.e., a V2I link, is defined as The set of K pairs of vehicle users (DUE) communicating via D2D, i.e., V2V links is defined as Assuming that all V2I links are pre-assigned with orthogonal spectrum sub-bands, and the mth sub-band corresponds to the mth V2I link, the transmission power of the mth V2I link is The V2V link achieves spectrum sharing through frequency band selection and power control. represents the control of the transmission power of the kth V2V link. In addition, a binary decision variable ρ is introduced k [m] is the frequency band allocation variable for the kth V2V link: k When [m] = 1, it means that the kth V2V link and the mth V2I link share the mth sub-band; when ρ k When [m] = 0, it means that the corresponding two links do not share the sub-band. Therefore, the channel gain of each communication link can be expressed as follows: represents the channel gain from the transmitting device to the base station of the mth V2I link, represents the interference channel gain from the transmitter of the mth V2I link to the receiver of the kth V2V link, expressed as g k [m] represents the channel gain from the sending device to the receiving device of the kth V2V link, expressed in g k′,k [m] represents the interference channel gain from the transmitting device of the k′th V2V link to the receiving device of the kth V2V link, expressed as g k,B [m] represents the interference channel gain from the transmitting device of the kth V2V link to the base station on the mth sub-band; S1.2: The signal-to-interference-to-noise ratio expression of the mth V2I link is: The signal-to-interference-to-noise ratio expression of the kth V2V link is: In the formula, σ 2 is the ambient noise power.

3. The C-V2X vehicle network resource allocation method based on deep reinforcement learning according to claim 2 is characterized in that: The distributed resource allocation problem in the Internet of Vehicles environment constructed according to the actual application of the link in step S2 is specifically: S2.1: The transmission rate of the mth V2I link can be calculated according to the Shannon formula as follows: Similarly, the transmission rate of the kth V2V link is expressed as follows: S2.2: Since V2I links are often used to support vehicles to obtain in-vehicle entertainment-related content from base stations (BS), such links require high transmission rates. For V2V links, the regular transmission of safety-related information requires high reliability. Therefore, by limiting the transmission time period T to a fixed period, the kth V2V link needs to complete a transmission of size B. k The data transmission of V2V link is guaranteed to ensure the high reliability of periodic transmission of V2V link. Let t represent the tth time slot in the time period T, then the reliability constraint of V2V link can be expressed as: Among them, Δ T is the channel coherence time, is the transmission rate of the kth V2V link in time slot t. S2.3: Further transform the reliability constraints into a form that is convenient for problem modeling: B k (t) is the size of data that has not been transmitted yet at time slot t. β is a constant greater than 0, and its value is slightly greater than the maximum possible transmission rate of each communication link. S2.4: The established bi-objective optimization problem can be expressed as follows: Among them, the optimization goal represents the transmission rate of the V2I link within the transmission period T, and the optimization target Indicates the reliability of the V2V link to complete the transmission requirements. The weight factor λ∈(0,1) is used to balance the optimization objectives f1 and f2. Constraint (a) indicates that the value of the frequency band allocation variable is 0 or 1. Constraint (b) indicates that in the same time slot, a V2V link can only share the frequency band of a V2I link at most. Constraint (c) limits the maximum and minimum values ​​of the V2V link transmission power.

4. The C-V2X vehicle network resource allocation method based on deep reinforcement learning according to claim 3 is characterized in that: In step S3, the distributed resource allocation problem is modeled as a Markov decision process, specifically: S3.1: The sending device of each V2V link is used as an agent. Each agent observes the environment and makes decisions on actions and gains benefits, thereby updating the policy network and learning its own resource sharing strategy. Each V2V link together forms a multi-agent system. At each time slot t, each V2V agent k obtains the observation value of the environment Then take action After all agents take action, they receive a reward R t , the environment with probability p(S t+1 |S t ,A)Transfer to the next state S t+1 As the elements of reinforcement learning, we will next clarify the state space, action space, and reward function. S3.2: The local observation space of a single agent k contains: V2V link channel gain Interference channel gain of the corresponding V2I link Interference channel gain of other V2V links And the interference power caused by the interference link Interference channel gain of V2V link to base station Remaining transmission time T k , the size of data to be transmitted is B k . Then the observation space of a single agent k can be expressed as: S3.3: Each agent autonomously selects the transmission band and controls the transmission power according to the strategy to achieve good system performance. There are a total of M optional sub-bands, each of which is already occupied by a V2I link. The agent needs to choose to share spectrum resources with a V2I link to improve the utilization of precious spectrum resources. The transmission power of the V2V link takes values ​​in the continuous interval [-100,23]dBm, rather than discretizing the action space by quantization, which improves the performance of the existing resource allocation algorithm with discrete action space. The combination of transmission band selection and transmission power control forms the hybrid action space of the agent as follows S3.4: Since the agents that make up the multi-agent system need to collaborate to achieve good performance of the system, we let the agents share the total reward value of the system. The reward value is related to the optimization objectives f1 and f2, and the reward function expression is as follows: In the above formula, λ1 and λ2 are weight factors of optimization targets f1 and f2 respectively, and satisfy λ1+λ2=1.

5. The C-V2X vehicle network resource allocation method based on deep reinforcement learning according to claim 4 is characterized in that: In step S4, the agent is initialized based on the Proximal Policy Optimization (PPO) algorithm, which includes two main neural networks: the Actor network and the Critic network. Specifically: S4.1: Construct a Critic network, which is used to evaluate the value function of the state. The input of the network is the observation vector obtained by the agent, and the output is the estimated value of the state. It is a three-layer fully connected neural network and uses the ReLU activation function. The corresponding loss function is the mean square error between the state value estimate and the actual return, which is optimized using the Adam optimizer. S4.2: Construct the Actor network, which is used for the strategy generation of the intelligent agent. It consists of two sub-networks: one for selecting sub-bands and the other for transmitting power control. Each sub-network consists of three fully connected layers. The channel selection network outputs a category distribution, which is sampled to obtain the channel selection; the power control network outputs the mean and standard deviation of a normal distribution, which is sampled to obtain the power action. Similar to the Critic network, the Actor network is trained using the Adam optimizer. S4.3: Before training begins, randomly initialize the network’s model parameters.

6. The C-V2X vehicle network resource allocation method based on deep reinforcement learning according to claim 5 is characterized in that: The process of solving the constructed resource allocation problem based on the MAPPO algorithm framework in step S5 is specifically as follows: S5.1: In each training cycle T, the agent interacts with the environment to obtain observation variables, perform actions, and reward data, that is, (O t ,a t ,R t ). Calculate the discounted return R′ t . S5.2: Take the output V(S of the Critic network) t ) measures the long-term benefits that each state can bring. The expression of the advantage function is adv = R′ t -V(S t ). The loss function of the Critic network is the predicted value function V(S t ) and the actual discounted return R′ t The mean square error between them is expressed as: The training of the Critic network is done by minimizing the loss function Loss c accomplish. S5.3: Update the parameters of the old Actor policy network to the parameters of the current policy network so that the current policy network can learn in a new round of training, while the old policy network remains unchanged. S5.4: Since the parameters of the new and old Actor strategy networks are different, the action probabilities obtained for the same state and action are different. Assuming that the action probability obtained by the old strategy is p1 and the action probability obtained by the new strategy is p2, calculate the parameter update ratio. The expression of the ratio is as follows: S5.5: The expression of the loss function of the Actor network obtained from the update ratio is: S5.6: In order to limit the excessive update range of the new strategy and ensure good convergence, the PPO algorithm trims the loss of the Actor network. Assuming that the trimming factor is a constant ∈, the loss functions of the discrete band selection action and the continuous power control part of the Actor network after trimming are: S5.7: The Critic network and the Actor network respectively update the network parameters by minimizing the loss function to complete a training process.

7. The C-V2X vehicle network resource allocation method based on deep reinforcement learning according to claim 6 is characterized in that: In step S6, the training process for a specific weight is extended to the solution of all possible weights through parameter migration, specifically: S6.1: For the objective function of the dual-objective optimization problem to be solved, the weights λ1 and λ2 are used to balance the index f1 of the V2I link and the index f2 of the V2V link. n The decomposed nth sub-objective function is as follows: If n = 1, the first possible weight is If n = 2, the possible weight of the second pair is If n = 3, the third possible weight is By analogy, when n=N, the possible weights of the Nth pair are S6.2: For the N sub-optimization problems with different weights, each sub-problem is modeled as a Markov process. S6.3: Since the weights of two adjacent sub-problems are similar, the optimal solutions are similar, and the neural network parameters that can obtain the optimal solutions are similar. For two adjacent sub-problems, the neural network parameters for the latter pair of weights are initialized to the neural network parameters trained for the former pair of weights. S6.4: Solve N sub-problems in sequence, and greatly improve the efficiency of solving N-1 sub-problems through parameter migration. Finally, a Pareto front consisting of a series of Pareto optimal solutions for the constructed dual-objective optimization problem is obtained.

Images