V2X resource allocation method based on graph neural network and reinforcement learning

Abstract

The invention discloses a V2X resource allocation method based on a graph neural network and reinforcement learning, and belongs to the field of resource allocation in the Internet of Vehicles C-V2X technology. The method specifically comprises the following steps: (1) setting a system model, including setting a channel model and setting a performance index; (2) constructing the vehicle network into a graph, and determining a neighbor of each node and node features included in each node in the graph; (3) obtaining an optimal channel and power distribution scheme by training the GNN-DDQN network, and maximizing the transmission rate of the V2I link and the communication success rate of the V2V link; and (4) carrying out numerical value and simulation evaluation. According to the method, a graph construction mode that vehicle neighbor information is transmitted in a network is achieved without additionally constructing a communication link, a distributed resource allocation method is constructed by utilizing GNN and DDQN networks, and experimental results show that the method provided by the invention is superior to a single DQN model and also has better performance compared with other reference methods.

Description

Technical Field

[0001] The present invention belongs to the field of resource allocation in C-V2X technology of Internet of Vehicles, and specifically relates to a V2X resource allocation method based on graph neural network and reinforcement learning. Background Art

[0002] C-V2X technology is believed to provide higher speed, lower latency and higher reliability than IEEE802.11P technology. Recently, the Third Generation Partnership Project (3GPP) standardized NR-V2X technology in its 16th version of the standard [4-5] (formerly 7-8). In actual deployment, in order to support the large amount of wireless communication requirements brought by V2X communication, resource allocation becomes a crucial part of IoV technology. However, resource allocation is usually NP-hard. In IoV environments, it is necessary to simultaneously meet the reliability requirements of vehicle-to-vehicle (V2V) links and the rate requirements of vehicle-to-infrastructure (V2I) links, and it is difficult to find the optimal solution. In addition, most traditional resource allocation methods require accurate channel state information (CSI), which is very expensive in IoV environments where vehicles move at high speeds. With the development of deep learning and reinforcement learning, some researchers have begun to use the powerful function approximation characteristics of deep reinforcement learning to solve resource allocation problems. Under inaccurate CSI conditions, deep reinforcement learning can accumulate experience through trial and error and learn a universal strategy that can achieve better performance in distributed resource allocation scenarios.

[0003] In a distributed scenario, the local observation of a vehicle is usually affected by various noises, making it difficult to characterize channel conditions and insufficient to support the vehicle in making good decisions. When the vehicle autonomously selects resources, it is easy to cause resource collisions and conflicts, affecting the success rate of communication. The agents in reinforcement learning only make decisions based on their own local observations. Without sufficient information, vehicles cannot achieve effective cooperation. This seriously affects the quality of global resource scheduling, and even easily causes problems such as resource collisions and congestion. Mutual interference between communication links is also difficult to control. In addition, in an environment where vehicles are moving at high speeds, the CSI based on the local observations of the vehicle contains a lot of noise, making it difficult to accurately describe the channel environment in which the vehicle is located, which also affects the effect of DRL to a certain extent. Summary of the invention

[0004] The present invention aims to overcome the limitations of the prior art and proposes a distributed resource allocation scheme using the GNN-DDQN algorithm in Mode 2 of the NR-V2X standard. The scheme combines the introduction of graph neural networks (GNNs) and reinforcement learning (RL), uses GNNs to extract low-dimensional features containing global structural information from local observations of vehicles, helps the intelligent agent obtain richer local information, and then uses a dual deep Q network (DDQN) model to make decisions, thereby achieving a more efficient resource allocation strategy.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The V2X resource allocation method based on graph neural network and reinforcement learning includes:

[0007] Step 1: System model setting; the system model setting includes channel model setting and performance index setting;

[0008] Step 2: Construct the vehicle network into a graph, determine the neighbors of each node and the node features contained in each node in the graph;

[0009] Step 3: By training the GNN-DDQN network, the optimal channel and power allocation scheme is obtained to maximize the transmission rate of the V2I link and the communication success rate of the V2V link;

[0010] Step 4: Conduct numerical and simulation evaluations.

[0011] Furthermore, in step 1, the specific steps of setting the system model are:

[0012] (1) Channel model setting: A traffic model simulating an intersection is constructed. The base station is located at the center of the intersection. Vehicles are added to the road in a randomly distributed manner and travel at a given initial speed. Assume that the environment contains M cellular users (CUEs), which use V2I links to communicate with the base station to transmit entertainment life information with high speed requirements required for daily use, represented by M = {1, 2, 3, ..., m}; there are K pairs of V2V users (VUEs), which use V2V links to communicate and transmit necessary safety information during vehicle driving, represented by K = {1, 2, 3, ..., k}. Since the uplink resource usage is not so intensive, it is assumed that the V2V link and the V2I link share the orthogonally allocated uplink spectrum to improve the spectrum utilization. The resource selection of the vehicle is divided into channel selection and power selection. Each vehicle selects a channel and a power level in the corresponding channel. Assume that the number of subchannels is equal to the number of V2I links m, the number of powers in the power list is n, and each vehicle can only select one channel and one transmission power level at the same time. Therefore, there are m*n resource selection schemes for each link.

[0013] The V2I link and the V2V link are managed separately. Considering that each V2V link autonomously selects resources when the resource selection of the V2I link is given in advance, the learning goal is to minimize the interference to the V2I link as much as possible while meeting the latency requirements and reliability of the V2V link, thereby maximizing the transmission rate of the V2I link. For the convenience of description, the reliability requirements of the V2V link are converted into interruption probability.

[0014] Consider that there are s vehicles in the environment, and each vehicle performs device-to-device (D2D) communication with the three surrounding vehicles, so the number of VUEs k is equal to 3s; each vehicle is equipped with an antenna and loaded with the Q network model and GraphSAGE model; since the V2V link and the V2I link share the orthogonally allocated uplink spectrum, the signal-to-noise ratio (SINR) of the i-th CUE is expressed as:

[0015]

[0016] in, represents the transmit power of the i-th CUE, and Refers to the transmission power of the jth VUE. σ 2 represents the noise power; for the i-th CUE, the channel power gain is given by h i For the jth VUE, the power gain of the interference is expressed as To represent; In addition, ρ j [i] is used as an indicator of frequency allocation. If the jth VUE uses the spectrum resources of the ith CUE, then ρ j [i] is assigned a value of 1, otherwise it is 0; according to Shannon's formula, the communication capacity of the i-th CUE is expressed as:

[0017] C c [i] = B·log(1+γ c [i]) (28)

[0018] Where B represents bandwidth; similarly, the SINR of the jth VUE can be expressed as:

[0019]

[0020] in:

[0021]

[0022] Represents the interference power of the V2I link sharing the same resource block (RB);

[0023]

[0024] represents the total interference power of all V2V links in the same RB, g j represents the power gain of the jth VUE, represents the interference power gain of the i-th CUE, represents the interference power gain of the j'th VUE, and the capacity of the j'th VUE is expressed as follows:

[0025] C v [j] = B·log(1+γ v [j]) (32);

[0026] (2) Performance indicator setting: The interruption probability of V2V communication and the transmission rate of V2I communication are used as performance indicators. Considering the delay requirements of the V2V link, the corresponding constraints are expressed as follows:

[0027]

[0028] in represents the V2V transmission rate of link j, D j represents the load of link j, that is, the amount of data required to be transmitted, T max Indicates the maximum transmission delay;

[0029] The reliability requirement of the V2V link is converted into the interruption probability; the interruption threshold γ0 and the interruption probability p0 are set, and the reliability requirement of the V2V link j∈K is expressed as:

[0030]

[0031] in, Denotes the SINR of V2V pair j. With Rayleigh fading, the reliability constraint (8) becomes:

[0032]

[0033] where γ eff is the effective outage probability; it is assumed that the tolerable outage probability and the maximum tolerable delay on all V2V links are the same;

[0034] In addition, since each V2V link can only be assigned one resource block and one power level, using a j ∈[0,1],j∈K to indicate whether the jth V2V link is allocated a resource block. If a j =1, it means that the jth V2V link has been allocated resource blocks; similarly, b j ∈[0,1],j∈K is used to mark the power selection of the jth V2V link; accordingly, the optimization problem is formulated as follows:

[0035]

[0036] Among them, λ represents the weight ratio representing the importance of the V2V link, C1-C2 represents the reliability and delay requirements for the V2V link, C3-C4 represents the resource block selection mark and power selection mark of the V2V link, and C5 represents the constraint on the maximum power to avoid using too large a transmission power and causing greater interference to the V2I link. Since the maximization goal includes the rate of the V2I link, and the main factor affecting the V2I rate is the interference caused by the signal of the V2V link, which is mainly determined by the transmission power of the V2V link signal, this problem statement will reduce the use of power to reduce interference to the V2I link while ensuring the reliability and delay of the V2V link as much as possible.

[0037] Furthermore, the specific method of step 2 is: regard the V2V links as nodes in the graph, the interference between the links as edges, and the physical characteristics of the links as node characteristics, so as to construct a complete graph.

[0038] Furthermore, the specific operation steps of constructing the graph are:

[0039] Consider each V2V pair as a node, denoted by N g =[v1,v2,v3,…,v k ], the interference relationship between links is regarded as an edge; for node v, it contains an initial feature x v and a list N(v) storing the index of neighbor nodes; the initial feature of the node is the information of the channel and interference observed locally by the vehicle. According to the previous assumptions of the present invention, the number of subchannels is equal to the number of CUEs m. For the i-th subchannel, record its instantaneous channel power gain for the V2V link, expressed as Gt[i],i∈M, and the subchannel power gain from the transmitter to the receiver in the V2I link, denoted as H t [i], i∈M and the interference signal strength I of the previous time slot t-1 [i], i∈M; therefore, the feature representation of node v is

[0040] x v = {G t PH t PI t-1} (37)

[0041] Where || represents the concatenation of vectors;

[0042] In order to achieve distributed deployment and save computing resources, the present invention constructs a graph based on the communication relationship between vehicles; the neighbors of each node are determined based on the communication relationship between the vehicles, the number of neighbors is 10-14, and does not change with the number of vehicles in the environment.

[0043] Furthermore, the specific algorithm of step three is: design a GNN-DDQN framework including state, action, and reward, and obtain the optimal strategy through training iteration;

[0044] (1) State space: The state space consists of two parts: one is the aggregated features extracted by GNN, and the other is the local observation of the vehicle. Assuming that the number of neighbors sampled is S, for node v, the set of neighbors sampled is N s (v), the node feature obtained by aggregation is z v , then the feature aggregation process is expressed as follows:

[0045] z v =f aggregate ({x u |u∈N(v)}) (38)

[0046] Among them, f aggregate The aggregation function of the model is represented. The present invention adopts the average aggregation function. Different from the general average aggregation, the present invention considers the weight of the edge to reflect the importance of each neighbor node feature, so it is expressed as:

[0047]

[0048] For the feature updating process, the present invention uses the aggregation result of neighbor features and merges it with the initial features of the node itself to obtain the final node feature embedding;

[0049] h v =f update (z v ,x v ) (40)

[0050] The aggregation result of neighbor features is combined with the initial features of the node by summing, and features are extracted through a set of parameters obtained through training. v represents the final node embedding, so f update It is expressed as follows:

[0051] f update (z v ,x v )=σ(W(z v +x v )+b) (41)

[0052] Where W represents the training weight parameter, b represents the bias vector, and σ represents the activation function;

[0053] Therefore, a complete round of aggregation process is expressed as:

[0054] h v =f update(x v ,f aggregate ({x u |u∈N s (v)})) (42)

[0055] Similarly, the aggregation process of two layers is expressed as:

[0056] h v =f update (x v ,f aggregate ({f update (x u ,f aggregate ({x n |n∈N s (u)}))|u∈N s (v)}))(43)

[0057] After obtaining the low-dimensional features of each node, the state of the agent in reinforcement learning is defined; for the V2X environment considered in the present invention, the real state information is mainly composed of the local observation of the vehicle on the environment and the low-dimensional features extracted from the CSI by the GraphSAGE model. The local observation of the vehicle is divided into two parts. The first part is the information of the channel and interference, including the instantaneous channel power gain G of the V2V link of each subchannel t , used to represent the subchannel power gain H from the transmitter to the receiver of the V2I link channel information t , and the interference signal strength I of the previous time slot t―1 ; The second part is the shared information from the surrounding vehicles, including the number of times N the neighboring vehicles selected each channel in the previous time slot t―1 , and the ratio L between the number of bits remaining to be sent by the vehicle and the total number of bits that need to be sent t and the remaining sending time U under the delay constraint t The low-dimensional features extracted by the GraphSAGE model are then used Representation; therefore, the state observed locally by the agent is represented as follows:

[0058]

[0059] (2) Action space: Based on the locally observed state, the DDQN network selects action a according to strategy π. t ∈A; Since the agent needs to select subchannel and power at the same time, the two actions are combined into a composite action, and the composite action selected by the agent is mapped to two dimensions, which are used as the selection of subchannel and power level respectively; the present invention considers three relatively simple power levels, and the number of resource blocks is m, so there are 3*m actions in total; assuming that the action selected by the agent is a t, decompose it in the following way:

[0060]

[0061] Among them, % represents the modulo operation, / represents division and rounding down, represents the decomposed sub-channel selection action, It indicates the selection of power level;

[0062] (3) Reward function: The goal of reinforcement learning is to enable the V2V link in the environment to meet the low latency and high reliability requirements of V2V communication to the greatest extent possible, and on this basis, to minimize the interference to the V2I link to maximize the transmission rate of the V2I link. Previously, the present invention converted the requirements of the V2V link for latency and reliability into requirements for interruption probability. Therefore, for the design of rewards, the present invention only needs to consider the interruption probability of the V2V link and the transmission rate of the V2I link. In addition, the present invention sets a penalty term based on the time the link has been transmitted. The reward function is expressed as follows:

[0063]

[0064] Where λc represents the weight of the V2I link, λp represents the weight of the used transmission time, Ut represents the remaining time, T0 represents the transmission delay limit, and thus (T0-Ut) represents the time that has been transmitted;

[0065] (4) GNN-DDQN training phase: The GNN and DDQN networks are updated separately. For the GNN network, the present invention stores the reward information obtained by the agent for selecting each subchannel in a matrix corresponding to the number of subchannels, denoted as Rg = {r1, r2, r3, …, rm}, and uses this matrix as the label of the corresponding node. The label is softened in the following way:

[0066]

[0067] in, represents the weight of the network aggregation result, represents the label of node v, represents the aggregation result of the lagged network;

[0068] Update the graph network using the mean squared error function:

[0069]

[0070] Among them, θ represents the weight parameter in the GraphSAGE model, y v represents the smoothed label used for network update;

[0071] For the DDQN network, what needs to be learned in essence is an optimal strategy π* that can maximize the long-term discounted reward and a deep network model that can predict the Q value of the corresponding action based on the state; for a given state-action pair (st, at), its corresponding Q value Q(st, at) is the expected cumulative discounted reward obtained after taking the action at∈A according to the strategy π. Therefore, the Q value can be used to measure the quality of an action in a given state; once the present invention can obtain a more accurate Q value, the action can be selected according to the following formula:

[0072] a t = argmax a∈A Q(s t ,a) (50)

[0073] This means that the present invention selects the action with the largest Q value;

[0074] In the DDQN network, the Q value Q* corresponding to the optimal strategy is obtained according to the following update equation:

[0075]

[0076] in, is the learning rate, the second term on the right side of the equation is the TD error for updating the Q value, the β discount factor, Q old represents the Q value predicted by the Q network, represents the Q value predicted by the target Q network;

[0077] Assume that the weight parameters of the Q network are It means that when the input is the state observed by the agent, the output will be the Q value corresponding to each action; the present invention uses TD error to optimize the parameters of the Q network It is expressed as follows:

[0078]

[0079] until both networks converge.

[0080] Compared with the prior art, the present invention has the following beneficial effects: BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 This is the relationship between the total V2I rate of all sub-channels and the number of vehicles in the solution of the present invention.

[0082] Figure 2 It is the relationship between the transmission success rate of all V2V links and the number of vehicles in the solution environment of the present invention.

[0083] Figure 3 This is the statistical probability that the intelligent agent of the solution of the present invention selects each power level under different remaining transmission times.

[0084] Figure 4 The difference in single decision time between using a complete graph or an incomplete graph when constructing a GNN network in the solution of the present invention.

[0085] Figure 5 It is the sum of the number of vehicles in the dynamic environment of the solution of the present invention, the instantaneous average success rate of V2V communication and the instantaneous V2I communication rate of all channels.

[0086] Figure 6 Box plot analysis of the number of vehicles in different time periods, the instantaneous average success rate of V2V communication, and the V2I communication rate of the solution of the present invention. DETAILED DESCRIPTION

[0087] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0088] The present invention is based on a V2X resource allocation scheme based on graph neural network and reinforcement learning, and its specific steps are as follows:

[0089] Step (1): In this solution, the present invention mainly considers V2I communication and V2V communication in V2X communication. Figure 1 As shown, the present invention constructs a traffic model simulating an intersection, where the base station is located at the center of the intersection, and vehicles are added to the road in a randomly distributed manner and travel at a constant speed according to a given initial speed. Assume that the environment contains M cellular users (CUEs), which use V2I links to communicate with the base station and transmit entertainment life information with high speed requirements required for daily use, represented by M = {1, 2, 3, ..., m}. There are K pairs of V2V users (VUEs), which use V2V links to communicate and transmit necessary safety information during vehicle driving, represented by K = {1, 2, 3, ..., k}. Since the uplink resource usage is not so intensive, the present invention assumes that the V2V link and the V2I link share the orthogonally allocated uplink spectrum to improve the utilization of the spectrum, and the resource selection of the vehicle is divided into channel selection and power selection. Each vehicle selects a channel and selects a power level in the corresponding channel. The present invention assumes that the number of subchannels is equal to the number of V2I links m, and the number of powers in the power list is n. Each vehicle can only select one channel and one transmission power level at the same time, so there are m*n resource selection schemes for each link.

[0090] In addition, the base station does not participate in the resource selection of the V2V link. Therefore, the present invention manages the V2I link and the V2V link separately, and considers that each V2V link autonomously selects resources when the resource selection of the V2I link is given in advance. The learning goal is to minimize the interference to the V2I link as much as possible while meeting the latency requirements and reliability of the V2V link, thereby maximizing the transmission rate of the V2I link. For the convenience of description, the present invention converts the reliability requirements of the V2V link into the interruption probability.

[0091] Step (2): Consider that there are s vehicles in the environment, and each vehicle performs device-to-device (D2D) communication with the three surrounding vehicles, so the number of VUEs k is equal to 3s. Each vehicle is equipped with an antenna and loaded with the Q network model and GraphSAGE model. Since the V2V link and the V2I link share the orthogonally allocated uplink spectrum, the signal-to-noise ratio (SINR) of the i-th CUE can be expressed as:

[0092]

[0093] in, represents the transmit power of the i-th CUE, and Refers to the transmission power of the jth VUE. σ 2 represents the noise power. For the i-th CUE, the channel power gain is given by h i For the jth VUE, the power gain of the interference is expressed as In addition, ρ j [i] is used as an indicator of frequency allocation. If the jth VUE uses the spectrum resources of the ith CUE, then ρ j [i] is assigned a value of 1, otherwise it is 0. According to Shannon's formula, the communication capacity of the i-th CUE can be expressed as:

[0094] C c [i] = B·log(1+γ c [i]) (54)

[0095] Where B represents the bandwidth. Similarly, the SINR of the jth VUE can be expressed as:

[0096]

[0097] in:

[0098]

[0099] Represents the interference power of the V2I link sharing the same resource block (RB).

[0100]

[0101] represents the total interference power of all V2V links in the same RB, g j represents the power gain of the jth VUE, represents the interference power gain of the i-th CUE, represents the interference power gain of the j'th VUE. The capacity of the j'th VUE can be expressed as follows:

[0102] C v [j] = B·log(1+γ v [j]) (58)

[0103] Step (3): In order to evaluate the performance of the resource allocation scheme, the present invention uses the interruption probability of V2V communication and the transmission rate of V2I communication as performance indicators. Considering the delay requirement of the V2V link, the corresponding constraints can be expressed as:

[0104]

[0105] in represents the V2V transmission rate of link j, D j represents the load of link j, that is, the amount of data required to be transmitted, T max Indicates the maximum transmission delay.

[0106] The reliability requirement of the V2V link is converted into the interruption probability. Setting the interruption threshold γ0 and the interruption probability p0, the reliability requirement of the V2V link j∈K can be expressed as:

[0107]

[0108] in, Denotes the SINR of V2V pair j. With Rayleigh fading, the reliability constraint (8) can be transformed into:

[0109]

[0110] where γ eff is the effective outage probability. It is assumed that the tolerable outage probability and the maximum tolerable delay on all V2V links are the same.

[0111] In addition, since each V2V link can only be assigned one resource block and one power level, using a j ∈[0,1],j∈K to indicate whether the jth V2V link is allocated a resource block. If a j =1, it means that the jth V2V link has been allocated resource blocks. Similarly, b j ∈[0,1],j∈K is used to mark the power selection of the jth V2V link.

[0112] Based on this, the optimization problem can be expressed as follows:

[0113]

[0114] Among them, λ represents the weight ratio representing the importance of the V2V link, C1-C2 represents the reliability and delay requirements for the V2V link, C3-C4 represents the resource block selection mark and power selection mark of the V2V link, and C5 represents the constraint on the maximum power to avoid using too large a transmission power and causing greater interference to the V2I link. Since the maximization goal includes the rate of the V2I link, and the main factor affecting the V2I rate is the interference caused by the signal of the V2V link, which is mainly determined by the transmission power of the V2V link signal, this problem statement will reduce the use of power to reduce interference to the V2I link while ensuring the reliability and delay of the V2V link as much as possible.

[0115] This problem is a combinatorial optimization problem, which is usually NP-hard and difficult to solve. Most traditional methods require accurate channel state information (CSI), so the present invention mainly uses reinforcement learning to obtain suboptimal solutions and uses GNN to assist the vehicle in obtaining better local observations.

[0116] Step (4): On a large time scale, first construct the vehicle network into a graph. Consider each V2V pair as a node, denoted by N g =[v1,v2,v3,…,v k ], the interference relationship between links is regarded as an edge. For node v, it contains an initial feature x v and a list N(v) storing neighbor node indexes. The initial features of the node are the channel and interference information observed locally by the vehicle. According to the previous assumptions of the present invention, the number of subchannels is equal to the number of CUEs m. For the i-th subchannel, the present invention records its instantaneous channel power gain for the V2V link, expressed as Gt[i], i∈M, and the subchannel power gain from the transmitter to the receiver in the V2I link, recorded as H t [i], i∈M and the interference signal strength I of the previous time slot t-1 [i], i∈M. Therefore, the feature of node v can be expressed as

[0117] x v = {G t PH t PI t-1} (63)

[0118] Where || represents the concatenation of vectors.

[0119] In order to achieve distributed deployment and save computing resources, the present invention constructs a graph based on the communication relationship between vehicles. Since each V2V pair consists of a sending vehicle and a receiving vehicle, assuming that there are s vehicles in the environment, represented by V = {V1, V2, V3, ..., V s The present invention gives each V2V link a unique label, denoted as V t to V r , V t , V r Belongs to V. Where Vt represents the sending vehicle and Vr represents the receiving vehicle.

[0120] Assume that the three destination vehicles of vehicle Vx are Vm, Vp, and Vo, and vehicle Vx is also the target vehicle of vehicles Vy and Vz. For the V2V link nodes formed by these vehicles, their labels are: Vx to Vm, Vx to Vp, Vx toVo, Vy to Vx, and Vz to Vx. For each node, taking node Vx to Vm as an example, the present invention regards all nodes containing Vx and Vm in the labels as neighbors of node Vx to Vm, that is, all Vx to Z, Z to Vx, Vm to Z, and Z to Vm nodes, where Z represents any qualified vehicle.

[0121] The present invention stipulates that each vehicle will select three sending destinations from the list of adjacent vehicles with a relatively close distance, that is, each vehicle will serve as the starting point of three V2V links. Therefore, the number of V2V links in the environment is 3s. For each node, its number of neighbors can be approximately equal to 12, because Vx to Z and Vm to Z each represent 3 links, and the number of links represented by Z to Vx and Z to Vm is uncertain, but because the total number of links is three times the number of vehicles, the average number of times Vx and Vm serve as receivers is 3. Therefore, the number of neighbors of each node is about 12, and will not change with the number of vehicles in the environment.

[0122] The graph construction method proposed in the present invention ignores some interference relationships between links, but it basically retains the neighbors with the strongest interference. Moreover, GNN is a multi-layer aggregation process, and information will be continuously transmitted along the edges, and even distant nodes will have an impact on the current node. In addition, the reason why the present invention chooses to construct a graph based on communication relationships rather than directly constructing a graph based on physical distances is because such a graph construction method is conducive to the implicit construction of graphs between vehicles. For the six neighbor nodes represented by Vx to Z and Vm to Z, vehicles Vx and Vm have all the relevant information, and for the neighbor nodes represented by Z to Vx and Z toVm, relevant information such as node features can be carried when sending information to Vx and Vm. In this way, only a two-way connection needs to be established between Vx and Vm to realize information transmission in the graph without adding additional communication burden.

[0123] Step (5): On a small time scale, first extract features from each node. The feature extraction process can be divided into three steps: neighbor sampling, feature aggregation, and feature update.

[0124] Neighbor sampling can be applied to large graphs by presetting the number of neighbors to be collected at different levels for each node, thereby randomly sampling the neighbor nodes of the node. This random sampling step limits the amount of computation required for each node when aggregating features, ensuring that the computational resources required by the model are still controllable even when facing extremely complex graph structures.

[0125] Assume that the number of sampled neighbors is S. For node v, the sampled neighbor set is N s (v), the node feature obtained by aggregation is z v , then the feature aggregation process can be expressed as follows:

[0126] z v =f aggregate ({x u |u∈N(v)}) (64)

[0127] Among them, f aggregate Represents the aggregation function of the model. Here, the present invention adopts the average aggregation function. Different from the general average aggregation, the present invention considers the weight of the edge to reflect the importance of each neighbor node feature, so it can be expressed as:

[0128]

[0129] For the feature updating process, the present invention uses the aggregation result of the neighbor features to merge with the initial features of the node itself to obtain the final node feature embedding.

[0130] h v =f update(z v ,x v ) (66)

[0131] Here, the present invention uses the summation method to combine the aggregation result of neighbor features with the initial features of the node, and extracts features through a set of parameters obtained through training, h v represents the final node embedding. Therefore, f update It can be expressed as follows:

[0132] f update (z v ,x v )=σ(W(z v +x v )+b) (67)

[0133] Where W represents the training weight parameter, b represents the bias vector, and σ represents the activation function.

[0134] Therefore, a complete round of aggregation process can be expressed as:

[0135] h v =f update (x v ,f aggregate ({x u |u∈N s (v)})) (68)

[0136] Similarly, the aggregation process of two layers can be expressed as:

[0137] h v =f update (x v ,f aggregate ({f update (x u ,f aggregate ({x n |n∈N s (u)}))|u∈N s (v)}))(69)

[0138] Step (6): After obtaining the low-dimensional features of each node, define the state of the agent in reinforcement learning. For the V2X environment considered in this paper, the real state information mainly consists of the vehicle's local observation of the environment and the low-dimensional features extracted from the CSI by the GraphSAGE model. As mentioned before, the local observation of the vehicle can be divided into two parts. The first part is the information of the channel and interference, including the instantaneous channel power gain G of the V2V link of each subchannel t , used to represent the subchannel power gain H from the transmitter to the receiver of the V2I link channel information t, and the interference signal strength I of the previous time slot t―1 The second part is the shared information from surrounding vehicles, including the number of times N the neighbor vehicle selected each channel in the previous time slot. t―1 , and the ratio L between the number of bits remaining to be sent by the vehicle and the total number of bits that need to be sent t and the remaining sending time U under the delay constraint t The low-dimensional features extracted by the GraphSAGE model are then used Therefore, the state observed locally by the agent can be expressed as follows:

[0139]

[0140] Step (7): Define the action. Based on the locally observed state, the DDQN network selects action a according to strategy π t ∈A. Since the agent needs to select both the subchannel and the power, the two actions are combined into a composite action. The composite action selected by the agent is mapped to two dimensions, which are used as the selection of the subchannel and the power level. In this paper, three relatively simple power levels are considered, and the number of resource blocks is m. Therefore, there are 3*m actions in total. Assume that the action selected by the agent is a t , decompose it in the following way:

[0141]

[0142] Among them, % represents the modulo operation, and / represents division and rounding down. represents the decomposed sub-channel selection action, It indicates the selection of power level.

[0143] Step (8): At each time t, the agent obtains the current state s from the state space of the environment t , and obtain the corresponding action a from the action space A according to the existing strategy π t To select the resource block occupancy and signal transmission power of the vehicle. The strategy π is determined by the state-action function, also known as the Q function, denoted as Q(s t ,a t ). In deep reinforcement learning, the Q function is usually approximated by deep learning to adapt to complex environmental changes. After the agent takes an action, the environment changes and moves to a new state s t+1 At the same time, according to the results of the action selected by the agent, a reward value r is given to evaluate the quality of the action t .

[0144] The goal of reinforcement learning in this article is to enable the V2V link in the environment to meet the low latency and high reliability requirements of V2V communication to the maximum extent, and on this basis, minimize the interference to the V2I link to maximize the transmission rate of the V2I link. Previously, the present invention converted the V2V link's requirements for latency and reliability into requirements for interruption probability. Therefore, for the design of rewards, the present invention only needs to consider the interruption probability of the V2V link and the transmission rate of the V2I link. In addition, the present invention sets a penalty term based on the time the link has been transmitted. The reward function is expressed as follows:

[0145]

[0146] Where λc represents the weight of the V2I link, λp represents the weight of the used transmission time, Ut represents the remaining time, T0 represents the transmission delay limit, and (T0-Ut) represents the time that has been transmitted. The long-term discounted benefit of the agent can be expressed as:

[0147]

[0148] Step (9): The present invention stores the reward information obtained by the agent for selecting each subchannel in a matrix corresponding to the number of subchannels, denoted as Rg = {r1, r2, r3, ..., rm}, and uses this matrix as the label of the corresponding node. The label is softened in the following way:

[0149]

[0150] in, represents the weight of the network aggregation result, represents the label of node v, Represents the aggregation result of the lagged network.

[0151] Update the graph network using the mean squared error function:

[0152]

[0153] Among them, θ represents the weight parameter in the GraphSAGE model, y v Represents the smoothing label used for network update.

[0154] Step (10): Perform joint training and testing of the GNN-DDQN network. After the environment starts, on a large time scale, the agent determines its neighbors based on the communication relationship between vehicles, implicitly constructs a graph, and calculates the edge weight with each neighbor based on the distance. On a small time scale, the agent obtains local observations from the environment. Perform feature aggregation and update operations based on the collected neighbor features to obtain low-dimensional features h v . The local observation With low-dimensional features h v The Q network predicts the action with the largest Q value and obtains rewards from the environment. During the training process, the Q value corresponding to each action predicted by the storage network is used as the basic evaluation value of the channel. After label softening, y is obtained. v , guiding the update of the GNN model. At the same time, the strategy of the agent is optimized according to the Q value predicted by the target Q network, and the Q network is updated. In the test phase, the maximum Q value action predicted by the Q network is directly used as the decision output.

[0155] In deep reinforcement learning, what needs to be learned is essentially an optimal strategy π* that can maximize the long-term discounted reward and a deep network model that can predict the Q value of the corresponding action based on the state. For a given state-action pair (st, at), the corresponding Q value Q(st, at) is the expected cumulative discounted reward obtained after taking the action at∈A according to the strategy π. Therefore, the Q value can be used to measure the quality of an action in a given state. Once the present invention can obtain a more accurate Q value, the action can be selected according to the following formula:

[0156] a t = argmax a∈A Q(s t ,a) (77)

[0157] This means that the present invention selects the action with the largest Q value.

[0158] In the DDQN network, the Q value Q* corresponding to the optimal strategy can be obtained according to the following update equation:

[0159]

[0160] in, is the learning rate, and the second term on the right side of the equation is the TD error for updating the Q value, the β discount factor. old represents the Q value predicted by the Q network, Represents the Q value predicted by the target Q network.

[0161] Assume that the weight parameters of the Q network are It means that when the input is the state observed by the agent, the output will be the Q value corresponding to each action. The present invention uses TD error to optimize the parameters of the Q network. It is expressed as follows:

[0162]

[0163] Example:

[0164] Figure 1The relationship between the total V2I rate of all subchannels and the number of vehicles is shown. As can be seen from the figure, as the number of vehicles increases, the number of V2V links in the environment increases, and the interference of V2V communication links on V2I links increases accordingly, causing the rate of V2I links to decrease. The DQN method is essentially reinforcement learning like the DDQN-GNN method proposed in this invention. However, the addition of GNN enables the agent to obtain a richer state containing more global information, thus producing better results.

[0165] Figure 2 The relationship between the transmission success rate of all V2V links in the environment and the number of vehicles is shown. Previously, the present invention mentioned that the number of V2V links in the environment is three times the number of vehicles. As can be seen from the figure, as the number of vehicles increases, the V2V communication success rate of the random strategy decreases the fastest, and the other schemes have a certain effect of slowing down the decline rate, and the utilization rate of resources is significantly increased. It can be seen that the DDQN-GNN scheme proposed by the present invention is better than the simple DQN network.

[0166] Figure 3 The effect of GNN on the strategy learned by the agent is shown in the figure. As can be seen from the figure, when the transmission time is relatively sufficient, the agent tends to choose a lower power to reduce the interference to the V2I link. When the transmission time is relatively tight, it tends to choose a high power to ensure the successful transmission of the V2V link. In addition, the agent also tends to choose a higher power when it starts to transmit. This may be because the agent does not reselect the channel in every time slot, and it is safe to give a larger power at the beginning.

[0167] Figure 4 The performance difference caused by using a complete graph or an incomplete graph when building a GNN network is demonstrated. When using a complete graph, as the number of vehicles increases, the number of nodes in the environment increases linearly, and the time required for training increases slightly. In order to ensure the reliability of the comparison, the present invention omits the data preprocessing process performed in the CPU in the case of an incomplete graph. This is because the incomplete graph has an additional process of obtaining neighbors from the graph compared to the complete graph. The present invention does not use an efficient high-level API dedicated to GNN to accelerate it, but uses the original code to implement it. Therefore, processing in the CPU may consume a certain amount of time, including the delay caused by sending data from the CPU to the GPU. The present invention compares the speed of the part calculated in the GPU. For the curve of the incomplete graph, since the number of neighbors is small and does not increase with the increase in the number of vehicles, the time required for a single decision is relatively constant and is less affected by the number of vehicles in the environment.

[0168] Figure 5The figure shows how the number of vehicles, the instantaneous average success rate of V2V communication, and the sum of the instantaneous V2I communication rates of all channels change over time. The number of vehicles continues to increase over time. At the beginning of the environment, the success rate of V2V communication has a large oscillation. After testing, it does not always occur. It is caused by the randomness of the environment and quickly stabilizes. As can be seen from the figure, the success rate of V2V communication is very stable and will not fluctuate with the change of vehicle position, but will gradually decrease as the number of vehicles in the environment increases. For the V2I rate, since the strategy is trained in a static environment, it cannot fully adapt to the changing environment. Under the premise of ensuring the success rate of V2V communication, it cannot well reduce the interference to the V2I link, so the V2I rate fluctuates greatly. Overall, it shows a downward trend with the increase of the number of vehicles.

[0169] Figure 6 The performance of the number of vehicles, V2I communication rate, and V2V communication success rate for each time period is shown. In the first time period, the V2V communication success rate exhibits a high frequency of outliers, which can be attributed to the oscillations in the environment observed from the instantaneous values. Although the V2I communication rate occasionally shows outliers, these outliers are still within the controllable range.

[0170] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A V2X resource allocation method based on graph neural network and reinforcement learning, characterized by: Specifically include: Step 1: System model setting; the system model setting includes channel model setting and performance index setting; Step 2: Construct the vehicle network into a graph, determine the neighbors of each node and the node features contained in each node in the graph; Step 3: By training the GNN-DDQN network, the optimal channel and power allocation scheme is obtained to maximize the transmission rate of the V2I link and the communication success rate of the V2V link; Step 4: Conduct numerical and simulation evaluations.

2. The V2X resource allocation method based on graph neural network and reinforcement learning as claimed in claim 1, characterized in that: In step 1, the specific steps of setting the system model are: (1) Channel model setting: A traffic model simulating an intersection is constructed. The base station is located at the center of the intersection. Vehicles are added to the road in a randomly distributed manner and travel at a given initial speed. Assume that the environment contains M cellular users (CUEs), which use V2I links to communicate with the base station to transmit entertainment life information with high speed requirements required for daily use, represented by M = {1, 2, 3, ..., m}; there are K pairs of V2V users (VUEs), which use V2V links to communicate and transmit necessary safety information during vehicle driving, represented by K = {1, 2, 3, ..., k}. Since the uplink resource usage is not so intensive, it is assumed that the V2V link and the V2I link share the orthogonally allocated uplink spectrum to improve the spectrum utilization. The resource selection of the vehicle is divided into channel selection and power selection. Each vehicle selects a channel and a power level in the corresponding channel. Assume that the number of subchannels is equal to the number of V2I links m, the number of powers in the power list is n, and each vehicle can only select one channel and one transmission power level at the same time. Therefore, there are m*n resource selection schemes for each link. The V2I link and the V2V link are managed separately. Considering that each V2V link autonomously selects resources when the resource selection of the V2I link is given in advance, the learning goal is to minimize the interference to the V2I link as much as possible while meeting the latency requirements and reliability of the V2V link, thereby maximizing the transmission rate of the V2I link. For the convenience of description, the reliability requirements of the V2V link are converted into interruption probability. Consider that there are s vehicles in the environment, and each vehicle performs device-to-device (D2D) communication with the three surrounding vehicles, so the number of VUEs k is equal to 3s; each vehicle is equipped with an antenna and loaded with the Q network model and GraphSAGE model; since the V2V link and the V2I link share the orthogonally allocated uplink spectrum, the signal-to-noise ratio (SINR) of the i-th CUE is expressed as: in, represents the transmit power of the i-th CUE, and refers to the transmission power of the jth VUE, σ 2 represents the noise power; for the i-th CUE, the channel power gain is given by h i For the jth VUE, the power gain of the interference is expressed as To represent; In addition, ρ j [i] is used as an indicator of frequency allocation. If the jth VUE uses the spectrum resources of the ith CUE, then ρ j [i] is assigned a value of 1, otherwise it is 0; according to Shannon's formula, the communication capacity of the i-th CUE is expressed as: C c [i]=B·log(1+γ c [i]) (2) Where B represents bandwidth; similarly, the SINR of the jth VUE can be expressed as: in: Represents the interference power of the V2I link sharing the same resource block (RB); represents the total interference power of all V2V links in the same RB, g j represents the power gain of the jth VUE, represents the interference power gain of the i-th CUE, represents the interference power gain of the j'th VUE, and the capacity of the j'th VUE is expressed as follows: (2) Performance indicator setting: The interruption probability of V2V communication and the transmission rate of V2I communication are used as performance indicators. Considering the delay requirements of the V2V link, the corresponding constraints are expressed as follows: in represents the V2V transmission rate of link j, D j represents the load of link j, that is, the amount of data required to be transmitted, T max Indicates the maximum transmission delay; The reliability requirement of the V2V link is converted into the interruption probability; the interruption threshold γ0 and the interruption probability p0 are set, and the reliability requirement of the V2V link j∈K is expressed as: in, Denotes the SINR of V2V pair j. With Rayleigh fading, the reliability constraint (8) becomes: where γ eff is the effective outage probability; it is assumed that the tolerable outage probability and the maximum tolerable delay on all V2V links are the same; In addition, since each V2V link can only be assigned one resource block and one power level, using a j ∈[0,1],j∈K to indicate whether the jth V2V link is allocated a resource block. If a j =1, it means that the jth V2V link has been allocated resource blocks; similarly, b j ∈[0,1],j∈K is used to mark the power selection of the jth V2V link; accordingly, the optimization problem is formulated as follows: Among them, λ represents the weight ratio representing the importance of the V2V link, C1-C2 represents the reliability requirement and delay requirement for the V2V link, C3-C4 represents the resource block selection mark and power selection mark of the V2V link, and C5 represents the constraint on the maximum power to avoid using too large a transmission power and causing greater interference to the V2I link. Since the maximization target includes the rate of the V2I link, and the main factor affecting the V2I rate is the interference caused by the signal of the V2V link, which is mainly determined by the transmission power of the V2V link signal, this problem statement will reduce the use of power to reduce interference to the V2I link while ensuring the reliability and delay of the V2V link as much as possible.

3. The V2X resource allocation method based on graph neural network and reinforcement learning as claimed in claim 2, characterized in that: The specific method of step 2 is: regard the V2V links as nodes in the graph, the interference between the links as edges, and the physical characteristics of the links as node characteristics, so as to construct a complete graph.

4. The V2X resource allocation method based on graph neural network and reinforcement learning as claimed in claim 3, characterized in that: The specific steps of constructing the graph are: Consider each V2V pair as a node, denoted by N g =[v1,v2,v3,…,v k ], the interference relationship between links is regarded as an edge; for node v, it contains an initial feature x v and a list N(v) storing the index of neighbor nodes; the initial feature of the node is the information of the channel and interference observed locally by the vehicle. According to the previous assumptions of the present invention, the number of subchannels is equal to the number of CUEs m. For the i-th subchannel, record its instantaneous channel power gain for the V2V link, expressed as Gt[i],i∈M, and the subchannel power gain from the transmitter to the receiver in the V2I link, denoted as H t [i], i∈M and the interference signal strength I of the previous time slot t-1 [i], i∈M; therefore, the feature representation of node v is x v ={G t PH t PI t-1 } (11) Where || represents the concatenation of vectors; In order to achieve distributed deployment and save computing resources, the present invention constructs a graph based on the communication relationship between vehicles; the neighbors of each node are determined based on the communication relationship between the vehicles, the number of neighbors is 10-14, and does not change with the number of vehicles in the environment.

5. The V2X resource allocation method based on graph neural network and reinforcement learning as claimed in claim 4, characterized in that: The specific algorithm of step 3 is: design a GNN-DDQN framework including state, action, and reward, and obtain the optimal strategy through training iteration; (1) State space: The state space consists of two parts: one is the aggregated features extracted by GNN, and the other is the local observation of the vehicle. Assuming that the number of neighbors sampled is S, for node v, the set of neighbors sampled is N s (v), the node feature obtained by aggregation is z v , then the feature aggregation process is expressed as follows: z v =f aggregate ({x u |u∈N(v)}) (12) Among them, f aggregate The aggregation function of the model is represented. The present invention adopts the average aggregation function. Different from the general average aggregation, the present invention considers the weight of the edge to reflect the importance of each neighbor node feature, so it is expressed as: For the feature updating process, the present invention uses the aggregation result of neighbor features and merges it with the initial features of the node itself to obtain the final node feature embedding; h v = f update ( z v , x v ) (14) The aggregation result of neighbor features is combined with the initial features of the node by summing, and features are extracted through a set of parameters obtained through training. v represents the final node embedding, so f update It is expressed as follows: f update (z v ,x v )=σ(W(z v +x v )+b) (15) Where W represents the training weight parameter, b represents the bias vector, and σ represents the activation function; Therefore, a complete round of aggregation process is expressed as: h v =f update (x v ,f aggregate ({x u |u∈N s (v)})) (16) Similarly, the aggregation process of two layers is expressed as: h v =f update (x v ,f aggregate ({f update (x u ,f aggregate ({x n |n∈N s (u)}))|u∈N s (v)})) (17) After obtaining the low-dimensional features of each node, the state of the agent in reinforcement learning is defined; for the V2X environment considered in the present invention, the real state information is mainly composed of the local observation of the vehicle on the environment and the low-dimensional features extracted from the CSI by the GraphSAGE model. The local observation of the vehicle is divided into two parts. The first part is the information of the channel and interference, including the instantaneous channel power gain G of the V2V link of each subchannel t , used to represent the subchannel power gain H from the transmitter to the receiver of the V2I link channel information t , and the interference signal strength I of the previous time slot t―1 ; The second part is the shared information from the surrounding vehicles, including the number of times N the neighboring vehicles selected each channel in the previous time slot t―1 , and the ratio L between the number of bits remaining to be sent by the vehicle and the total number of bits that need to be sent t and the remaining sending time U under the delay constraint t The low-dimensional features extracted by the GraphSAGE model are used Representation; therefore, the state observed locally by the agent is represented as follows: (2) Action space: Based on the locally observed state, the DDQN network selects action a according to strategy π. t ∈A; Since the agent needs to select subchannel and power at the same time, the two actions are combined into a composite action, and the composite action selected by the agent is mapped to two dimensions, which are used as the selection of subchannel and power level respectively; the present invention considers three relatively simple power levels, and the number of resource blocks is m, so there are 3*m actions in total; assuming that the action selected by the agent is a t , decompose it in the following way: Among them, % represents the modulo operation, / represents division and rounding down, represents the decomposed sub-channel selection action, It indicates the selection of power level; (3) Reward function: The goal of reinforcement learning is to enable the V2V link in the environment to meet the low latency and high reliability requirements of V2V communication to the greatest extent possible, and on this basis, to minimize the interference to the V2I link to maximize the transmission rate of the V2I link. Previously, the present invention converted the requirements of the V2V link for latency and reliability into requirements for interruption probability. Therefore, for the design of rewards, the present invention only needs to consider the interruption probability of the V2V link and the transmission rate of the V2I link. In addition, the present invention sets a penalty term based on the time the link has been transmitted. The reward function is expressed as follows: Where λc represents the weight of the V2I link, λp represents the weight of the used transmission time, Ut represents the remaining time, T0 represents the transmission delay limit, and thus (T0-Ut) represents the time that has been transmitted; (4) GNN-DDQN training phase: The GNN and DDQN networks are updated separately. For the GNN network, the present invention stores the reward information obtained by the agent for selecting each subchannel in a matrix corresponding to the number of subchannels, denoted as Rg = {r1, r2, r3, …, rm}, and uses this matrix as the label of the corresponding node. The label is softened in the following way: in, represents the weight of the network aggregation result, represents the label of node v, represents the aggregation result of the lagged network; Update the graph network using the mean squared error function: Among them, θ represents the weight parameter in the GraphSAGE model, y v represents the smoothed label used for network update; For the DDQN network, what needs to be learned in essence is an optimal strategy π* that can maximize the long-term discounted reward and a deep network model that can predict the Q value of the corresponding action based on the state; for a given state-action pair (st, at), its corresponding Q value Q(st, at) is the expected cumulative discounted reward obtained after taking the action at∈A according to the strategy π. Therefore, the Q value can be used to measure the quality of an action in a given state; once the present invention can obtain a more accurate Q value, the action can be selected according to the following formula: a t =argmax a∈A Q(s t ,a) (24) This means that the present invention selects the action with the largest Q value; In the DDQN network, the Q value Q* corresponding to the optimal strategy is obtained according to the following update equation: in, is the learning rate, the second term on the right side of the equation is the TD error for updating the Q value, the β discount factor, Q old represents the Q value predicted by the Q network, represents the Q value predicted by the target Q network; Assume that the weight parameters of the Q network are It means that when the input is the state observed by the agent, the output will be the Q value corresponding to each action; the present invention uses TD error to optimize the parameters of the Q network It is expressed as follows:

Images