A resource allocation method and system based on graph reinforcement learning for cellular internet of vehicles

Abstract

The application belongs to the technical field of Internet of Vehicles control, and specifically discloses a resource allocation method and system for cellular Internet of Vehicles based on graph reinforcement learning. First, a two-dimensional Markov chain model is established to describe the resource allocation and data packet transmission process in the coexistence scenario of CAM and DENM. Then, a data packet reception ratio model is proposed to evaluate the successful reception probability of CAM and DENM data packets. Meanwhile, a delay model is proposed to depict the time interval between the successful reception of two consecutive CAM data packets or DENM data packets of the same vehicle. On this basis, a multi-agent deep reinforcement learning framework is established to realize intelligent resource selection decision of vehicles. Furthermore, a resource allocation method based on graph reinforcement learning is proposed to guide the agents to more effectively extract the spatial topological features of vehicle nodes. Based on the extracted features, the application can learn and obtain the optimal resource allocation decision for each vehicle, thereby improving the successful reception rate of CAM and DENM data packets and reducing the delay.

Description

Technical Field

[0001] This invention belongs to the field of vehicle network control technology, and specifically relates to a resource allocation method and system for cellular vehicle networks based on graph reinforcement learning. Background Technology

[0002] Vehicle-to-everything (V2X) communication is a crucial supporting technology for intelligent transportation systems and autonomous driving, improving traffic efficiency and reducing traffic accidents. Currently, the main technologies supporting V2X communication include Dedicated Short-Range Communications (DSRC) and Cellular Vehicle-to-Everything (C-V2X) technology. The C-V2X standard, as an alternative to DSRC, was proposed by the 3GPP (3rd Generation Partnership Project) in Release 14. 3GPP defines four C-V2X communication forms: vehicle-to-vehicle, vehicle-to-infrastructure, vehicle-to-pedestrian, and vehicle-to-network. C-V2X introduces Mode 4, enabling direct V2X communication. Mode 4 employs a distributed resource mechanism, allowing vehicles to autonomously select channel resources via the PC5 interface when no base station support is available. Mode 4 uses a perception-based semi-persistent scheduling (SPS) protocol for resource allocation. In C-V2X, vehicle-to-vehicle (V2V) communication includes two important messages: Cooperative Awareness Messages (CAM) and Distributed Environmental Notification Messages (DENM). CAM is a periodic message containing information such as vehicle position, speed, acceleration, and direction of travel, reflecting the real-time operating status of surrounding vehicles. Unlike CAM, DENM is a non-periodic message, generated only when a vehicle detects abnormal traffic conditions. Due to its time-sensitive requirements, DENM faces stricter reliability and latency constraints than CAM in V2V communication, thus giving it higher transmission priority than CAM.

[0003] The performance of the SPS protocol is crucial for reliable V2V communication, but it suffers from a serious resource allocation conflict problem. All vehicles autonomously select channel resources from a shared resource pool. When multiple vehicles simultaneously select the same channel resource, resource conflicts occur undetectable, leading to packet collisions. As traffic density increases, resource conflicts become more frequent, resulting in more packet collisions and ultimately degrading communication performance. Furthermore, due to the semi-persistent nature of the SPS protocol, vehicles also face persistent packet collisions. To mitigate packet collisions caused by resource conflicts, many V2V channel resource allocation methods have been proposed. For example, existing research incorporates the time-frequency coordinates of future reserved resources into vehicle broadcast messages. This reservation information is published approximately one second before actual resource use and repeated multiple times to ensure reliability. When a vehicle selects a resource, it excludes resources already reserved by other vehicles, thus reducing resource conflicts. However, this research only considers resource allocation for CAM transmission and neglects the resource selection process for DENM transmission. Therefore, existing research proposes a resource pool separation scheme. This scheme divides the entire resource pool into two parts: one for DENM transmission and the other for CAM transmission. Vehicles randomly generate DENMs, and the ratio of resources in the two parts is dynamically adjusted based on the number of DENMs. However, within each resource pool, vehicles still randomly select resources based on the SPS algorithm, which still leads to frequent packet collisions. In recent years, deep reinforcement learning methods have been introduced to solve the resource allocation problem. For example, existing research has proposed a partition-based deep reinforcement learning method, which divides all vehicles into multiple regions and uses the state information collected by all vehicles in each region to train an independent neural network for each region. However, in this method, vehicles make decisions only based on their own observation information, ignoring the spatial interference relationships between vehicles, resulting in a decline in communication performance. Summary of the Invention

[0004] The purpose of this invention is to propose a resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning, which helps to improve the success rate of data packet reception for Collaborative Awareness Messages (CAM) and Distributed Environmental Notification Messages (DENM) and reduce latency.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning includes the following steps: Step 1. Establish a two-dimensional Markov chain model to describe the resource allocation and data packet transmission process in the coexistence scenario of CAM and DENM; Step 2. Establish a data packet reception ratio model to evaluate the successful reception probability of CAM and DENM data packets, and use a time delay model to characterize the time interval between two consecutive CAM or DENM data packets successfully received by the same vehicle; Step 3. Establish a multi-agent deep reinforcement learning framework to realize resource selection decisions for vehicle intelligence, and propose a resource allocation method based on graph reinforcement learning to adapt to the frequently changing topology in the Internet of Vehicles. By using a resource allocation method based on graph reinforcement learning, the agent is guided to extract the spatial topological features of vehicle nodes, and based on the extracted spatial topological features, the agent learns and obtains the optimal resource allocation decision for each vehicle.

[0006] Furthermore, based on the aforementioned resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning, this invention also proposes a corresponding resource allocation system for cellular V2X networks based on graph reinforcement learning, which adopts the following technical solution: A resource allocation system for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning includes the following modules: The Markov chain model building module is used to build a two-dimensional Markov chain model to describe the resource allocation and data packet transmission process in the coexistence scenario of CAM and DENM. The packet reception ratio model building module is used to build a packet reception ratio model to evaluate the probability of successful reception of CAM and DENM packets, and to characterize the time interval between two consecutive CAM or DENM packets successfully received by the same vehicle through a delay model. And a resource allocation module is proposed to establish a multi-agent deep reinforcement learning framework to realize resource selection decision-making for vehicle intelligence, and a resource allocation method based on graph reinforcement learning is proposed to adapt to the frequently changing topology in the Internet of Vehicles. By using a resource allocation method based on graph reinforcement learning, the agent can be guided to extract the spatial topological features of vehicle nodes, and based on the extracted spatial topological features, learn and obtain the optimal resource allocation decision for each vehicle.

[0007] The present invention has the following advantages: As described above, this invention discloses a resource allocation method for cellular vehicular networks based on graph reinforcement learning. To improve the packet reception ratio (PRR) of vehicle-to-everything (V2X) and reduce the latency of vehicle-to-everything (DENM) networks, this method first establishes a two-dimensional Markov chain model to describe the resource allocation and packet transmission process in a scenario where CAM and DENM coexist. Then, a packet reception ratio (PRR) model is proposed to evaluate the successful reception probability of CAM and DENM packets. Furthermore, this invention proposes a latency model to characterize the time interval between two consecutive CAM or DENM packets successfully received by the same vehicle. Based on this, this invention establishes a multi-agent deep reinforcement learning (DRL) framework to achieve intelligent resource selection decisions for vehicles. However, traditional DRL methods rely solely on their own observation state during the decision-making process, failing to fully utilize spatial topology information between neighbors. Therefore, this invention further proposes a resource allocation method based on graph reinforcement learning to adapt to the frequently changing topology in V2X networks. The proposed method can guide agents to more effectively extract the spatial topology features of vehicle nodes. Based on the extracted spatial topology features, the reinforcement learning method can learn and obtain the optimal resource allocation decision for each vehicle. Simulation results show that the method of the present invention significantly improves the data packet success rate of Collaborative Awareness Message (CAM) and Distributed Environmental Notification Message (DENM) and reduces latency in a highly dynamic vehicle network environment. Attached Figure Description

[0008] Figure 1 This is a flowchart of a resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of the perception-based semi-persistent scheduling algorithm SPS in an embodiment of the present invention; Figure 3 This is a two-dimensional Markov chain model in the scenario where CAM and DENM coexist in this embodiment of the invention; Figure 4 This is a schematic diagram of excluded resources and candidate resources for vehicles in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating three scenarios in which resource R is selected in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the state of channel resources in an embodiment of the present invention; Figure 7 This is a diagram illustrating the architecture of the GRL-RA method in an embodiment of the present invention. Figure 8 This is a schematic diagram of a vehicle network in an embodiment of the present invention; Figure 9 For different numbers of vehicles in the embodiments of the present invention, at Γ=50ms and A schematic diagram of PRR when = 0.8; Figure 10 For different numbers of vehicles in the embodiments of the present invention =1 and A schematic diagram of PRR when = 0.8; Figure 11 For different numbers of vehicles in the embodiments of the present invention =0.8 and A schematic diagram of the average delay when the latency is 20ms; Figure 12 This invention provides examples of different resource reservation probabilities. =100ms and A schematic diagram of the average time delay when =1; Figure 13 This is a schematic diagram of the training process of the GRL-RA method proposed in this embodiment of the invention; Figure 14 This is a schematic diagram of the PRR for three methods with different numbers of vehicles in an embodiment of the present invention; Figure 15 This is a schematic diagram illustrating the collision probability of three methods under different numbers of vehicles in an embodiment of the present invention. Figure 16 This is a schematic diagram illustrating the average time delay of three methods under different numbers of vehicles in this embodiment of the invention. Detailed Implementation

[0009] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 The 3GPP third-generation partnership introduced Cellular Vehicle-to-Everything (C-V2X) Mode 4 in Release 14, supporting direct communication between vehicles. In Mode 4, vehicles use the Sensing-Based Semi-Persistent Scheduling (S-SPS) protocol to autonomously select channel resources and broadcast Cooperative Sensing Messages (CAM) and Distributed Environment Notification Messages (DENM).

[0010] To improve the packet reception ratio of CAM and reduce the latency of DENM, this invention proposes a resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning in scenarios where CAM and DENM coexist. The inventive concept is as follows: First, a two-dimensional Markov chain model was designed to describe the resource selection and packet transmission process in C-V2X scenarios where CAM and DENM coexist. Unlike other models that only use a reselection counter (RC), this model adds a retransmission counter (RT) specifically for DENM packet transmission.

[0011] Secondly, a packet reception PRR model is proposed to evaluate the reliability of C-V2X Mode 4 communication. The PRR model characterizes the proportion of CAM and DENM packets successfully decoded out of all packets sent by all vehicles. A delay model is also proposed to describe the time interval between two consecutive successful receptions of CAM or DENM packets by the receiving vehicle.

[0012] Finally, a resource allocation method based on graph reinforcement learning, GRL-RA, is proposed. GRL-RA first utilizes a graph-based network (GCN) to aggregate features of neighboring vehicles and obtain spatial topology information between neighbors. Then, it employs an LSTM to capture the periodic characteristics of resource occupancy and predict the channel resources that neighboring vehicles will occupy, thereby avoiding resource conflicts. Finally, the extracted spatiotemporal feature vectors are input into a Q-network to assist vehicles in making optimal resource selection decisions.

[0013] like Figure 1 As shown, the resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning in this embodiment includes the following steps: Step 1. Establish a two-dimensional Markov chain model to describe the resource allocation and data packet transmission process in the coexistence scenario of CAM and DENM.

[0014] This invention presents a perception-based semi-persistent scheduling (SPS) algorithm that enables vehicles to autonomously select resources without base station assistance. The SPS-based resource reservation process can be divided into the following three steps: I. When a vehicle needs to send a data packet and the reselection counter (RC) is zero, the vehicle must reserve new resources within the selection window. For example... Figure 2 As shown, parameters and The lower and upper bounds of the selection window are defined respectively. Determined by vehicle configuration, Constrained by latency requirements, within the selection window, the vehicle identifies possible sub-channel resources in all subframes.

[0015] II. Initial available resource set for vehicle construction . Includes all sub-channel resources within the selection window, but excludes the following resources: a. During data packet transmission, the vehicle continuously receives sidelink control information from other vehicles. Based on the received sidelink control information, it excludes subchannel resources that have been reserved or occupied by other vehicles.

[0016] b. The vehicle continuously monitors the Reference Signal Received Power (RSRP) and Received Signal Strength Indication (RSSI) of all resources within the perception window. Sub-channel resources with RSRP measurements exceeding a threshold are excluded. If If the number of sub-channel resources in step 1 is less than 20% of the total number of sub-channel resources in step 1, then the RSRP threshold is iteratively increased by 3dB, and step 2 is repeated.

[0017] when The resource exclusion process stops when at least 20% of the total subchannel resources are included.

[0018] III. The vehicle calculates the average RSSI for each resource in the first 10 subframes and sorts these resources according to the magnitude of the average RSSI. The 20% of sub-channel resources with the lowest average RSSI are added to the new resource set. middle.

[0019] Vehicle from A resource is randomly and uniformly selected and reserved for the next transmission, while an RC value is set. RC is decremented by 1 after each data packet transmission. When RC reaches 0, the vehicle... The probability of choosing a new resource, where Otherwise, the vehicle continues to use the original resources and a new RC value is set.

[0020] To analyze and quantify the packet reception ratio and latency in scenarios where CAM and DENM coexist, this invention first establishes a two-dimensional Markov chain model, which is illustrated in... Figure 3 The model introduces an "Idle" state, indicating that the vehicle currently has no data packets to send or no available sub-channel resources. This represents the probability that a vehicle has at least one available subchannel resource. and These represent the probabilities of generating CAM and DENM data packets, respectively.

[0021] When a new data packet is generated while the vehicle is idle, it first determines whether the data packet is a CAM data packet or a DENM data packet. If it is a DENM data packet, the DENM selection window is opened, and its duration is [duration missing]. ,in =20ms. If it is a CAM data packet, the CAM selection window will be opened, and its duration will be recorded as . When RC is between [5, 15], =100ms; when RC is between [10,30] =50ms; when RC is between [25, 75] =20ms.

[0022] The vehicle selects channel resources from a selection window based on a semi-persistent scheduling mechanism. When CAM and DENM packets are generated simultaneously, the state transition process is consistent with the case when the vehicle is idle and generates DENM packets. Priority scheduling of DENM packets is achieved through differentiated selection window sizes and state transition rules. If a DENM packet is generated in the idle state, the vehicle opens the DENM selection window and selects available channel resources according to the SPS algorithm, as follows: Figure 3 As shown on the right. The latency before transmission is modeled as... A state of equal probability ,in After the waiting time ends, the vehicle begins transmitting DENM data packets.

[0023] To improve the reliability of DENM data transmission, a retransmission counter RT is introduced, with its initial value set to... DENM packets are retransmitted at fixed time intervals. Next. Status This indicates the status of the initial transmission of a DENM data packet, while the retransmission status of DENM is determined by... It means that among them , This represents the current retransmission count. The RT value is decremented by 1 for each completed transmission. After this transmission, the vehicle returns to an idle state. If a CAM data packet is generated while the vehicle is idle, the vehicle opens the CAM selection window, the process of which is as follows: Figure 3 As shown on the left.

[0024] The vehicle selects a subchannel resource based on the SPS algorithm. The waiting process before transmission is modeled as Γ-1 equally probable states (w, j), where j [0, Γ-2]. After the waiting time ends, the vehicle randomly selects a resource counter RC value from (1+ht) values, where h represents the upper limit of the RC value range and t represents its lower limit.

[0025] In state ( , ) below, among which [1, h], the vehicle has one transmission opportunity. This indicates the current RC value. If no DENM data packets are generated at this time, the transmission opportunity will be used for CAM data packets, and the RC value will be decremented by 1 after the transmission is completed.

[0026] If no DENM data packet is generated, the process will continue to repeat until the vehicle reaches state (R,1). If a DENM data packet is generated during the periodic CAM transmission, the periodic CAM transmission will be interrupted.

[0027] The vehicle is in state (R, ),in ∈ [1, h], transition to state ( The DENM data packet transmission begins. After each DENM data packet transmission, the retransmission count m is decremented by 1, and the vehicle waits for the next retransmission opportunity.

[0028] During the waiting period, the vehicle maintains the same RT value. After completing a total of s DENM packet transmissions, the vehicle returns to state (R, The CAM data packets continue to be transmitted periodically. The RC value remains constant throughout the entire DENM transmission process.

[0029] When the vehicle is in state (R,1) and new CAM data packets are still being generated, the vehicle can choose to continue using the original channel resources or reselect new sub-channel resources. The probability that the vehicle will continue to use the original resources is... If the vehicle chooses to continue using the original channel resources, it will need to wait. ms, then reset the RC value for packet transmission. If the vehicle selects a new channel resource, its state transition process is the same as the process of generating CAM packets when the vehicle is idle.

[0030] In the Markov chain model proposed in this invention, the message generation process follows a Poisson distribution.

[0031] Assuming all data packets are of the same length and all vehicles have the same communication range, the vehicles are arranged at an average speed of... and The Poisson distribution generates CAM and DENM data packets.

[0032] Therefore, in the time period The probability of generating at least one CAM data packet within is expressed as: ; in time period The probability of generating at least one DENM packet within is expressed as: .

[0033] in and These represent the probabilities of generating CAM and DENM data packets, respectively.

[0034] based on Figure 3 The Markov chain model constructed in the model obtains the steady-state probability for each state as follows: definition Indicates idle The steady-state probability of a state. Defined as: (1) in, , This represents the probability that a vehicle has at least one available subchannel resource. This indicates the probability that the vehicle will continue to use the original channel resources. express( The steady-state probability of a state.

[0035] state( The ) indicates the waiting time before CAM data packet transmission. ms. Status ( steady-state probability Represented as: (2)

[0036] in Indicates the maximum allowable delay for CAM. Representing state The steady-state probability.

[0037] Define state ( ) indicates the waiting time before DENM data packet transmission. ms.

[0038] state( steady-state probability Represented as: (3) in This indicates the maximum allowable delay for DENM.

[0039] Define state ( , This indicates that the vehicle's RC value has reached... The state, state (R, steady-state probability Represented as: (4) Where RC represents the reselection counter, h represents the upper limit of the range of RC values, and t represents its lower limit. Indicates state ( The steady-state probability of ,1). This indicates that the vehicle's RC value at this time is , Representing state The steady-state probability.

[0040] Define state ( (,m) represents the state where the vehicle's RT value reaches m. steady-state probability of m) Represented as: (5) in This represents the initial value of the retransmission counter RT. Representing state The steady-state probability is obtained by summing the steady-state probabilities of all states according to the normalization property of Markov chains. Therefore, the steady-state probability of each state can be derived.

[0041] Step 2. Establish a data packet reception ratio model to evaluate the successful reception probability of CAM and DENM data packets, and use a time delay model to characterize the time interval between two consecutive CAM or DENM data packets successfully received by the same vehicle.

[0042] Errors that cause packet reception failures in C-V2X Mode 4 can be mainly divided into four categories.

[0043] The first type is half-duplex error, which occurs when multiple vehicles transmit data packets in the same subframe. The second type is perception error, which occurs when the received signal power is below the perception power threshold. The third type is propagation error, which occurs when the received signal-to-noise ratio is insufficient for the receiving vehicle to successfully decode the data packet. The last type is packet collision error, which occurs when multiple vehicles simultaneously select the same channel resource for broadcasting, causing packet collisions and preventing the receiving vehicle from correctly receiving the data packet.

[0044] I. Half-duplex error.

[0045] In C-V2X Mode 4, vehicles operate in half-duplex mode. A half-duplex error occurs when multiple vehicles broadcast in the same subframe. Because vehicles cannot simultaneously send and receive data packets within the same subframe, data packets are lost.

[0046] The probability of half-duplex error occurring The probability that at least two vehicles will choose the same subframe for transmission is given by the following formula: (6) in This indicates the total number of subframes in a selection window. The formula for representing the number of vehicles within the communication range is: (7) in , and These represent the number of lanes, vehicle density, and communication radius, respectively.

[0047] II. Perception error. When a vehicle broadcasts within its communication range, it cannot decode data packets if the received signal power is below the sensing power threshold. Considering path loss and shadow fading, the received signal power... Represented as: (8) in Indicates the distance between the sender and receiver. Indicates the transmission power. and These represent path loss and shadow fading, respectively. During signal propagation, path loss occurs as the propagation distance increases, and its magnitude depends on the distance between the transmitter and receiver. Shadow fading is an unpredictable attenuation of the signal caused by obstacles such as buildings and terrain during propagation, and it follows a log-normal distribution.

[0048] probability density function of received signal power Represented as: (9) in The standard deviation represents the log-normal distribution. This indicates a possible value for the received signal power.

[0049] Received signal power Below the perceived power threshold probability for: (10) in This represents the error function.

[0050] Will Defined as the average probability of failing to receive data packets due to perceptual errors, expressed as: (11) in and These represent the maximum and minimum distances between the sender and receiver, respectively.

[0051] III. Spreading errors.

[0052] The received signal power at the receiving end is not lower than the sensing power threshold. However, the received signal-to-noise ratio This is still insufficient for correctly decoding data packets. Path loss should be considered. Shadow decay And noise factors, received signal-to-noise ratio Represented as: (12) in, Indicates the received signal power. Indicates noise power. Indicates the transmission power.

[0053] Due to shadow decay It follows a log-normal distribution, therefore the probability density function of the received signal-to-noise ratio is... Represented as: (13) in This indicates the possible values ​​of the received signal-to-noise ratio. It represents the standard deviation of a log-normal distribution.

[0054] Received signal power not less than And the received signal-to-noise ratio is below the threshold. probability Represented as: (14) Will Defined as the average probability of failing to receive a data packet correctly due to propagation errors, it is expressed as: (15)

[0055] in and These represent the maximum and minimum distances between the sender and receiver, respectively.

[0056] IV. Packet collision error.

[0057] Vehicles autonomously select sub-channel resources to transmit data packets using the SPS mechanism. When multiple vehicles within communication range select the same sub-channel for transmission in the same subframe, a data packet collision occurs.

[0058] To calculate the probability that multiple vehicles will choose the same resource, it is first necessary to determine the number of candidate resources for a vehicle.

[0059] exist Figure 4 In the middle, considering vehicles Open the CAM selection window. The grid in the image represents all resources within this selection window. Orange grids represent resources excluded in SPS Step II, while white grids represent available resources.

[0060] definition For the total number of resources, The number of excluded resources (orange squares).

[0061] For communication range The probability that a certain resource is not selected by any vehicle is . Therefore, the probability that at least one vehicle chooses this resource is... .

[0062] thus, Represented as: (16) Total resources Represented as: (17) in This indicates the number of subframes in the selection window. This indicates the number of sub-channels in each subframe.

[0063] To calculate the packet collision probability, it is necessary to determine how many vehicles contain the same resource in their candidate resource set.

[0064] like Figure 5 As shown, resources It is a vehicle A candidate resource. First, it is necessary to determine whether the candidate resource set contains... The number of vehicles. In the proposed Markov chain model, the resource selection window opens under three conditions: 1) When the vehicle reaches the state When a new resource selection is required, the CAM selection window will be launched; 2) When a vehicle generates a CAM data package in Idle mode, the CAM selection window is activated; 3) When a vehicle generates a DENM data packet in the Idle state, the DENM selection window is launched.

[0065] First, deduce the state the vehicle reaches. When the CAM selection window is launched, the resources in its candidate resource set include ( Figure 5 The probability of situation ①). The vehicle in state... The duration is .

[0066] definition For a vehicle in the vehicle Entering state during selection window The probability is expressed as: (18) Therefore, the probability that the candidate resource set contains resource R is... Represented as: (19) in This indicates the probability that the vehicle will continue to use the original channel resources.

[0067] When the vehicle is in an idle state, the CAM selection window is activated, and the resources in the vehicle's candidate resource set include... The probability that the vehicle remains idle for a certain duration is given. ; Representing state The steady-state probability.

[0068] When the vehicle is in an idle state, the CAM selection window is activated, and the resources in the vehicle's candidate resource set include... The probability of ( Figure 5 (Situation ②) The duration of the vehicle in an idle state is . The candidate resource set for vehicles contains resources probability Represented as: (20) Finally, it is derived that when a vehicle launches the DENM selection window in the Idle state, the resources in its candidate resource set include... probability ( Figure 5 In case ③, the probability can be expressed as: (twenty one) Using the three probabilities mentioned above, we can obtain the resources contained in the candidate resource set. Number of vehicles for: (twenty two)

[0069] vehicle Each resource in the candidate resource set presents three states, such as... Figure 6 As shown.

[0070] The first state is when no vehicle selects the resource; this type of resource is represented by a white square.

[0071] definition The probability of the first state occurring is calculated as follows: (twenty three)

[0072] in This indicates that the candidate resource set contains resources. The number of vehicles.

[0073] The second scenario is that it is selected by exactly one vehicle; this type of resource is... Figure 6 The text is represented by red squares.

[0074] definition The probability of the second state occurring is calculated as follows: (twenty four)

[0075] in Indicates the number of vehicles within the communication range. From Randomly select one vehicle from the available vehicles.

[0076] The third state is when it is selected by at least two vehicles; this type of resource is... Figure 6 The middle section is represented by green squares.

[0077] definition The probability of the third state occurring is calculated as follows: (25) Using the total number of candidate resources, we obtain the expected number of resources in each state: (26) (27) (28) in, , and These represent the quantities of resources in the first, second, and third states, respectively. and The sum represents the total number of resources selected by at least one vehicle within the CAM selection window.

[0078] Therefore, the probability of a packet collision caused by a resource selection conflict is expressed as: (29) If none of the four errors occur, the data packet is received correctly. Therefore, the Packet Receiver Ratio (PRR) can be expressed as: .

[0079] Latency is another important metric affecting V2V communication. Latency is defined as the time interval between two consecutive successful data packet receptions by a receiving vehicle; this time interval is determined by resource-reserved latency. Delay caused by data packet collision It consists of two parts.

[0080] In this invention, the propagation delay and processing time are extremely small and can be ignored.

[0081] The resource reservation delay is determined by both the size of the selection window and the data packet transmission time. The resource reservation delay is calculated as follows: (30) in, and These represent the number of CAM data packets and DENM data packets generated within a resource reservation interval, respectively. and The value depends on the resource reservation interval and the packet generation interval, and is expressed as: , (31) in Reserve intervals for resources, Generate intervals for CAM data packets. The interval for generating DENM packets; in formula (32) and The transmission times of CAM and DENM data packets are respectively defined as:

[0082] , (32)

[0083] in and These represent the sizes of the CAM and DENM data packets, respectively. This indicates the data packet transmission rate.

[0084] When vehicles select resources, multiple vehicles may simultaneously select the same resource, leading to packet collisions. When a collision occurs, the latency will increase by the resource reservation interval. Therefore, the latency caused by packet collisions... Represented as:

[0085] (33)

[0086] in This represents the probability of a packet collision caused by a resource selection conflict.

[0087] Therefore, average delay Represented as:

[0088] (34)

[0089] Step 3. Establish a multi-agent deep reinforcement learning framework to realize intelligent resource selection decisions for vehicles, and propose a resource allocation method based on graph reinforcement learning to adapt to the frequently changing topology in the Internet of Vehicles.

[0090] By using a resource allocation method based on graph reinforcement learning, the agent is guided to extract the spatial topological features of vehicle nodes, and based on the extracted spatial topological features, the agent learns and obtains the optimal resource allocation decision for each vehicle.

[0091] The objective of this invention is to find the optimal resource allocation strategy that maximizes the packet reception ratio while satisfying V2V communication latency constraints. (The time step is not specified in the original text.) The resource selection action of all vehicles is defined as follows: .

[0092] in Indicates vehicle The resource selection vector. If Indicates the time step Vehicle Use the Each sub-channel resource is used for data packet transmission. If This indicates that the first [unspecified action] was not used. Sub-channel resources. The optimization problem of sub-channel resource allocation in C-V2X is expressed as:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] in Indicates time step The percentage of data packets successfully decoded out of all data packets sent by all vehicles at that time. Indicates time step The time interval between two consecutive successful receptions of two data packets by the receiving vehicle.

[0098] Constraint C1 is a time delay constraint. The maximum tolerable transmission delay is given. Constraint C2 indicates that each vehicle can only select one channel resource during transmission. Constraint C3 indicates that the vehicle... Binary decision variables.

[0099] The resource allocation decision-making process can be modeled as a Markov decision process. Each vehicle, acting as an agent, learns the optimal resource selection strategy. The Markov decision model is defined as a quintuple. .

[0100] in It represents a set of intelligent agents. Represents a set of states, where Indicates time step The state space. Represents a set of actions, where For time step The action space.

[0101] Represents the set of state transition functions. Indicates the vehicle is in a certain state. Take action After transitioning to state The probability of. For the reward set, where Indicates time step The reward function.

[0102] At time step At that time, all agents act according to their states. Take action The environmental state is determined by... Transfer to The intelligent agent receives a reward. The state space, action space, and reward function of the Markov decision model in C-V2X Mode 4 are defined as follows.

[0103] State space: It includes global channel status information and motion status information of all vehicles.

[0104] However, each vehicle agent can only acquire a portion of the observable environmental information. (Single vehicle agent) The observation space includes its local channel information and its own motion state. Therefore, the vehicle The observation information can be defined as:

[0105] ;

[0106] in, Indicates the time step Vehicle The actions taken. Indicates vehicle The location coordinates. and Representing vehicles time step RC and RT values ​​at time Indicates vehicle The RSSI index.

[0107] Action space: within the time step At that time, intelligent agent From size The agent selects a suitable sub-channel resource from the candidate resource set. Resource selection corresponds to an action. The action space is represented as:

[0108] ;in, It is a binary decision variable.

[0109] like , indicating the time step Vehicle Use the Each subchannel resource is used for data packet transmission; otherwise , This means that each agent can only choose one resource at each time step.

[0110] Reward Function: In high-density scenarios, a reward mechanism incentivizes neighboring agents to use different resources, while allowing distant agents to reuse the same sub-channel resources. Agents receive higher rewards when they choose sub-channel resources not used by other agents. Therefore, agents... time step Rewards received Represented as:

[0111] .

[0112] in, Indicates the time step Selecting resources The number of intelligent agents, .when At that time, it indicates that there are only intelligent agents. Select Resources .

[0113] intelligent agent The transmitted data packets do not collide, therefore the agent receives a positive reward. .when At that time, it indicated that some other vehicles had chosen to interact with the intelligent agent. Identical resources lead to data packet collisions, therefore the agent... They should receive a negative reward. Indicating in intelligent agents The set of intelligent agents within the communication range, Indicates the time step Use resources A set of agents broadcasting. The penalty is represented by the number of vehicles that receive the signal together among the agents, encouraging distant agents to choose the same resources. Consider the objective of maximizing PRR and the time delay constraint, and adjust the time step accordingly. The reward at that time is defined as:

[0114] .

[0115] In multi-agent reinforcement learning, each agent makes decisions based solely on its own local observations, ignoring the spatial topological relationships between vehicles. Specifically, due to mutual interference on shared sub-channels, a vehicle's resource selection decision may be influenced by the decisions of its neighboring vehicles. Furthermore, due to the use of the SPS mechanism, vehicles periodically occupy the same sub-channels during data packet transmission, and this periodic multiplexing behavior introduces temporal correlation into resource occupancy.

[0116] If temporal dependencies are ignored, vehicles may repeatedly select subchannel resources already occupied by neighboring vehicles, leading to continuous collisions. To address this issue, a C-V2X Mode 4 resource allocation method, GRL-RA, based on graph reinforcement learning, is proposed.

[0117] like Figure 7 As shown, the C-V2X network is modeled as a multi-agent graph model. Each vehicle in C-V2X is modeled as a node, and the edges between nodes and their weights reflect the dependencies and importance between agents.

[0118] The GRL-RA method employs a graph convolutional network (GCN) to capture spatial interference relationships between agents. It utilizes a long short-term memory (LSTM) network to extract temporal features from the vehicle's historical states, capturing the periodic patterns of resource usage.

[0119] The GRL-RA method deploys a target network and a training network for each agent, employing centralized training and distributed execution to guide the vehicle in making autonomous resource selection decisions. By using a centralized Q-function, the agent's policy can be optimized globally. The process of the graph reinforcement learning-based resource allocation method GRL-RA is as follows:

[0120] First, a graph convolutional network (GCN) is used to aggregate the features of neighboring vehicles to obtain the spatial topological features between neighbors. Then, an LSTM is used to capture the periodic characteristics of resource occupancy and predict the channel resources that neighboring vehicles will occupy. Finally, the spatiotemporal feature vectors extracted by the GCN and LSTM are input into the Q network to assist vehicles in making optimal resource selection decisions.

[0121] Graph Convolutional Networks (GCNs) capture spatial topological relationships between neighbors by aggregating feature information of neighboring vehicles.

[0122] First, the vehicle communication network is represented as a graph; each vehicle can be considered a node in the graph, and the communication link between two vehicles can be considered an edge; given the graph... ,in For a set of nodes, Let it be the set of edges.

[0123] The diagram is represented as follows Figure 8 As shown, node The observation is defined as a vehicle status edge weight Determined by the connection relationship between the two vehicles, it is represented as:

[0124] (35)

[0125] in, Indicates vehicle With vehicles The physical distance between them and Representing vehicles and vehicles Its position in the Internet of Vehicles This represents the vehicle's maximum communication radius.

[0126] After representing the network as a graph, GCN is used to aggregate node features.

[0127] node state vector First, it is encoded into a fixed-length feature vector:

[0128] (36)

[0129] in Represents a node The initial feature vector, This represents the encoding function.

[0130] definition Aggregating information from neighbors is represented as:

[0131] (37)

[0132] in, Represents intelligent agents The neighborhood group, Represents a node In the Feature vectors after GCN aggregation. Represents aggregate functions, Indicates the layer number of the GCN. The convolution process is represented as:

[0133] (38)

[0134] in Represents a node go through Feature vectors after GCN aggregation. Represents a node In the Feature vectors after GCN aggregation. This represents vector concatenation. This represents the trainable weights in the convolutional layer. This represents the activation function.

[0135] The intelligent agent obtained after GCN aggregation with a fixed number of layers The feature vector, will Defined as spatial topological features .

[0136] To capture the dynamic characteristics of time, the GRL-RA method uses LSTM and incorporates historical information.

[0137] intelligent agent The feature vector is updated to The formula is as follows:

[0138] (39)

[0139] in, Indicates time step The hidden state, The weight matrix is ​​a learnable matrix. This represents the final spatiotemporal feature vector, where Indicates the number of observations; This represents the process of extracting time features.

[0140] This reinforcement learning method uses GCN for spatial feature extraction and LSTM for temporal feature extraction, combining the feature extraction network with a deep Q-network to learn the optimal channel resource selection strategy for each vehicle.

[0141] The agent's goal is to maximize the cumulative discount reward. , Defined as:

[0142] (40)

[0143] in, Indicates at time step The instant reward received; The discount factor balances immediate and future rewards; the Q-function is used to quantify the agent's state. Execute actions according to the strategy The expected cumulative reward.

[0144] The expression for the Q function is as follows:

[0145] (41)

[0146] in Represents the action value function. This indicates cumulative discount rewards. Indicates a known state Current action In the case of [the specific circumstances], and the expected value when future actions are chosen according to the strategy.

[0147] Based on the extracted spatiotemporal feature vectors, the input of DQN, i.e., a deep Q-network, is... Represented as:

[0148] (42)

[0149] To achieve a stable training process in reinforcement learning, two identical neural networks are used: a training network and a target network. The training network is responsible for selecting actions based on the current state, while the target network periodically synchronizes the parameters of the training network and calculates the target Q-value to mitigate the problem of overestimation of value.

[0150] This invention will have parameters The target Q-network is represented as , will have parameters The trained Q-network is represented as The agent's experience at each time step Add to playback buffer .

[0151] During training, samples are uniformly taken from the buffer. A small batch of experience, used in each The model is updated in each time step. Parameter optimization is achieved by minimizing the loss function, typically using stochastic optimization methods. The loss function is defined as follows:

[0152] (43)

[0153] in, This represents the learning parameters in GCN and LSTM. This represents the target Q-value output by the target Q-network. This represents the predicted Q-value output by the currently trained Q-network.

[0154] This represents the spatiotemporal feature vector extracted by GCN and LSTM, which depends on the parameters. , Representing the trainable parameters in the GCN network and LSTM network, respectively.

[0155] Indicates time step The state and the chosen action, This represents the target network parameters.

[0156] This represents the state-action pairs sampled in the experience replay buffer. Seeking expectations.

[0157] The target Q value, obtained from the Bellman optimality equation, is defined as follows:

[0158] (44)

[0159] in This represents the learning rate.

[0160] Indicates the next observation state. Indicates the next state Consider the possible candidate actions.

[0161] To verify the correctness of the analysis model, the open-source C-V2X Mode 4 simulator implemented in ns-3 was used.

[0162] This invention compares the results obtained from the analysis model with those obtained from the ns-3 simulator. It assumes that each vehicle sends only two types of messages: CAM and DENM. Vehicles periodically broadcast CAM data packets. When various emergencies such as traffic accidents, road construction, or weather occur, vehicles begin broadcasting DENM data packets. In this invention, DENM is given a higher priority than CAM. The vehicles are placed on a 2-kilometer-long, 4-lane highway (2 lanes in each direction). The simulation settings for the highway scenario are based on the 3GPP TR 36.885 standard, and the specific parameters are shown in Table 1.

[0163] Table 1 Simulation Parameters

[0164]

[0165] This invention uses Python 3.8 to evaluate GRL-RA. To reduce the computational complexity of the algorithm, the highway is set as a two-lane highway (one in each direction). A two-layer GCN is used to aggregate the features of neighboring nodes. The feature dimension of the input LSTM is 128. For the DQN model, a two-layer neural network model is used, with each layer containing 256 neurons. The final output is the Q-value of an 80-dimensional action. The non-linear activation function between layers is ReLU. Detailed parameter settings are shown in Table 2.

[0166] Table 2 DRL Model Parameters

[0167]

[0168] This invention verifies the accuracy of the PRR model and the delay model by comparing the analysis results with the simulation results. This applies to different numbers of sub-channels. CAM selection window size Next, assess the PRR.

[0169] Figure 9 Comparison in and Under the given conditions, the proposed analysis model (solid line) and the C-V2X Mode 4 simulator (dashed line) yielded the PRR. The analysis results and simulation results are in high agreement, verifying the accuracy of the PRR model.

[0170] When the number of vehicles As the number of vehicles increases, the PRR decreases significantly. The main reason is that more vehicles compete for limited channel resources, leading to packet collisions and a decrease in PRR. This invention also compares different... The PRR obtained from the analysis model and simulator under the specified conditions is analyzed. The results show that when At that time, PRR was always better than The PRR is below. The reason is that, with With the increase in the number of available resources for vehicles, the probability of multiple vehicles selecting the same channel resource for transmission at the same time decreases, leading to an increase in the number of successfully received data packets, thus improving the PRR. When hour, The PRR under the condition is approximately greater than The increase of 18% indicates that the number of sub-channels in each subframe significantly affects the vehicle's PRR.

[0171] Figure 10 Shown in and Under these conditions, with the number of vehicles The PRR was increased, and the results showed that the PRR obtained from the analysis model was in high agreement with the simulation results, further validating the effectiveness of the PRR model.

[0172] This invention compares different CAM selection window sizes. The PRR was obtained from the analysis model and simulator under the given conditions. The results show that... PRR was always lower than The PRR at that time. There are two main reasons:

[0173] First, when When the number is reduced from 100 to 50, the number of available resources decreases by almost 50%, making it more likely that multiple vehicles will simultaneously choose to broadcast to the same resource, leading to more data packet collisions; secondly, The reduction in size decreases the probability of overlapping selection windows between vehicles, but due to the change in selection window size, the corresponding resource reservation interval also decreases, increasing the frequency of vehicle resource selection broadcasts and leading to more frequent packet collisions. hour, PRR under the condition is approximately equal to The 34% decrease indicates that the selection window size has a significant impact on PRR performance.

[0174] Another key performance indicator is latency. Figure 11 Shown in and Under these conditions, with the number of vehicles The change in latency is observed with increasing latency. Firstly, it can be observed that the results obtained from the analysis model are consistent with the simulator results, indicating that the proposed latency model is correct. As... As the number of vehicles increases, latency rises significantly. This is because as the number of vehicles increases, the probability of vehicles simultaneously selecting the same channel resource increases, leading to more data packet collisions and a large number of data packets failing to be successfully received. Therefore, the time interval between two successful data packet receptions at the receiver increases. Furthermore, when... As latency increases, it decreases. For example... Figure 11 As shown, The latency under the (orange curve) condition is lower than (Red curve). The main reason is that as the number of sub-channels increases, the number of sub-channel resources within the selection window increases, and the number of candidate resources that the vehicle can choose increases, reducing the probability of data packet collisions and thus reducing latency.

[0175] This invention studies the impact of resource reservation probability on latency through experiments. Figure 12 Shown in and Under different conditions The simulation results again demonstrate a high degree of consistency with the analysis model. This invention evaluates latency in scenarios with 160 and 200 vehicles.

[0176] It can be observed that, Better than the latency under the given conditions Delay under certain conditions. Variations were made within the specified range of 0 to 0.8. The results show that, as... The decrease in RC leads to an increase in latency. This is because the RC value decreases by 1 each time a vehicle broadcasts a data packet. When the RC value reaches zero, the vehicle will... Select a new sub-channel resource. If Higher latency means vehicles will reselect resources more frequently, leading to more data packet collisions and thus increasing latency.

[0177] To verify the effectiveness of the GRL-RA method, it is compared with the following two benchmark methods:

[0178] The semi-persistent scheduling algorithm RL-SPS, based on multi-agent deep reinforcement learning, aims to maximize distance-weighted transmission utility while satisfying Quality of Service (QoS) constraints for secure message transmission. This algorithm employs a multi-head attention mechanism to ensure vehicles pay attention to neighbor information. Each vehicle agent updates its decisions based on its own information combined with observations from neighboring vehicles.

[0179] The Adaptive Resource Reservation Semi-Persistent Scheduling Algorithm (ARR-SPS) aims to minimize the probability of channel access collisions and latency interruptions. This method dynamically adjusts the sensing range and transmit power based on vehicle density and beacon message transmission requirements, and introduces an adaptive resource reservation algorithm to optimize the value range of the reselection counter (RC), achieving efficient vehicle-to-vehicle communication while maintaining high reliability and low latency.

[0180] Figure 13 The training process of the GRL-RA algorithm of this invention is demonstrated. The number of vehicles is set to 100. It can be observed that as the number of training rounds increases, the average reward per round initially decreases slightly and then gradually increases. The main reason is that in the early exploration phase, the agent adopts a random resource selection strategy, which may lead to multiple vehicles selecting the same channel resource and sending data packets to a certain receiver simultaneously, resulting in resource conflicts and preventing the receiver from decoding correctly, thus causing the PRR to decrease. As training progresses, the agent learns from accumulated experience and gradually masters a better resource selection strategy, thus the average reward increases, proving the effectiveness of the proposed algorithm. The dynamic changes in the vehicle environment cause the agent to continuously adjust its strategy, resulting in fluctuations during the training process. When the number of training rounds reaches approximately 450, the average reward gradually stabilizes, proving that the GRL-RA algorithm has converged. In this invention, the Q network is trained for 600 rounds, and all agents share the same network parameters.

[0181] Figure 14 This demonstrates how, under three resource allocation methods, the number of vehicles... The increase and the change in PRR. It can be seen that... The PRR is significantly affected. As the number of vehicles increases, the PRR obtained by all three methods decreases. The increase in the number of vehicles leads to more resource conflicts, causing more packet collisions, thus affecting the vehicle's receiving performance.

[0182] However, among the three methods, the GRL-RA method proposed in this invention consistently maintains the highest PRR. When At that time, the PRR obtained by the RL-SPS and ARR-SPS methods were similar and relatively low, mainly because these two methods could not adapt to high-density scenarios. In contrast, the PRR of the GRL-RA method was about 5% higher than the other two methods. Therefore, the GRL-RA method can more effectively reduce packet collisions, thereby improving communication reliability in large-scale vehicular networks.

[0183] Figure 15 The diagram shows how the collision probability changes with the number of vehicles under three resource allocation methods. It can be seen that the collision probability increases with the number of vehicles in all three methods. The increased number of vehicles exacerbates resource conflicts, leading to more data packet collisions. It can be observed that, under different vehicle numbers, the collision probability of the method in this invention is lower than the other two benchmark methods. At that time, the collision probabilities of the three methods were not significantly different. However, when At that time, the collision probability obtained by the proposed method was approximately 4% and 8% lower than that of the RL-SPS and ARR-SPS methods, respectively. More packet collisions lead to higher latency. Figure 16 As shown, it can be observed that the latency of the GRL-RA method is lower than that of the other two benchmark methods as the number of vehicles increases. This indicates that with the GRL-RA method, all agents can select more suitable channel resources for transmission. Therefore, the GRL-RA method achieves superior performance in terms of reliability and latency.

[0184] This invention proposes an analysis model for SPS performance in C-V2X Mode 4. First, a two-dimensional Markov chain model is designed to describe the resource selection and data packet transmission process in a CAM and DENM coexistence scenario. By analyzing four types of errors that may lead to data packet loss, a PRR model is established to evaluate communication reliability. Furthermore, a latency model is proposed to characterize the time interval between two consecutive CAM or DENM data packets successfully received by the receiving vehicle. Simulations verify the accuracy and effectiveness of the proposed analysis model, providing a theoretical basis for further optimization. To enhance communication reliability and reduce latency in C-V2X Mode 4, this invention also proposes a graph reinforcement learning-based resource allocation method, GRL-RA, which introduces a spatiotemporal feature extraction module to capture the spatiotemporal relationships between vehicles. Simulation results show that the proposed method reduces communication latency while maintaining high communication reliability, ensuring the performance of vehicle-to-everything (V2X) security applications.

[0185] Example 2

[0186] This embodiment 2 describes a resource allocation system for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning. This system is based on the same inventive concept as the resource allocation method for cellular V2X networks based on graph reinforcement learning in embodiment 1 above.

[0187] The resource allocation system for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning in this embodiment includes the following modules:

[0188] The Markov chain model building module is used to build a two-dimensional Markov chain model to describe the resource allocation and data packet transmission process in the coexistence scenario of CAM and DENM.

[0189] The packet reception ratio model building module is used to build a packet reception ratio model to evaluate the probability of successful reception of CAM and DENM packets, and to characterize the time interval between two consecutive CAM or DENM packets successfully received by the same vehicle through a delay model.

[0190] And a resource allocation module is proposed to establish a multi-agent deep reinforcement learning framework to realize resource selection decision-making for vehicle intelligence, and a resource allocation method based on graph reinforcement learning is proposed to adapt to the frequently changing topology in the Internet of Vehicles.

[0191] By using a resource allocation method based on graph reinforcement learning, an agent can be guided to extract the spatial topological features of vehicle nodes, and based on the extracted spatial topological features, learn and obtain the optimal resource allocation decision for each vehicle.

[0192] It should be noted that any content not mentioned in the above-mentioned functional modules of the system described in this embodiment 2 can be referred to the step description of the corresponding method in embodiment 1 above, and will not be repeated in detail here.

[0193] Example 3

[0194] This embodiment 3 describes a computer device including a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it implements the steps of the resource allocation method for cellular vehicle-to-everything (V2X) based on graph reinforcement learning described in embodiment 1 above.

[0195] In this embodiment, the computer device can be any device or apparatus with data processing capabilities, and will not be described in detail here.

[0196] Example 4

[0197] This embodiment 4 describes a computer-readable storage medium storing a program that, when executed by a processor, implements the steps of the resource allocation method for cellular vehicle-to-everything (V2X) based on graph reinforcement learning in embodiment 1.

[0198] The computer-readable storage medium can be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc.

[0199] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning, characterized in that, Includes the following steps: Step 1. Establish a two-dimensional Markov chain model to describe the resource allocation and data packet transmission process in the coexistence scenario of CAM and DENM; Step 2. Establish a data packet reception ratio model to evaluate the successful reception probability of CAM and DENM data packets, and use a time delay model to characterize the time interval between two consecutive CAM or DENM data packets successfully received by the same vehicle; Step 3. Establish a multi-agent deep reinforcement learning framework to realize resource selection decisions for vehicle intelligence, and propose a resource allocation method based on graph reinforcement learning to adapt to the frequently changing topology in the Internet of Vehicles. In step 3, the resource allocation method based on graph reinforcement learning proceeds as follows: First, the features of neighboring vehicles are aggregated using a graph convolutional network (GCN) to obtain the spatial topology features between neighbors; then, LSTM is used to capture the periodic characteristics of resource occupancy and predict the channel resources that neighboring vehicles will occupy. Finally, the spatiotemporal feature vectors extracted by GCN and LSTM are input into the Q network for optimal resource allocation; By using a resource allocation method based on graph reinforcement learning, the agent is guided to extract the spatial topological features of vehicle nodes, and based on the extracted spatial topological features, the agent learns and obtains the optimal resource allocation decision for each vehicle.

2. The resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning according to claim 1, characterized in that, In step 1, it is assumed that all data packets have the same length and all vehicles have the same communication range. The vehicles are arranged according to an average speed of... and The Poisson distribution generates CAM and DENM data packets; In time period The probability of generating at least one CAM data packet within is expressed as: ; In time period The probability of generating at least one DENM packet within is expressed as: ; in and These represent the probabilities of generating CAM data packets and DENM data packets, respectively. Based on the constructed Markov chain model, the steady-state probability of each state is obtained, as follows: definition Indicates idle The steady-state probability of the state. Defined as: (1) in, , This represents the probability that a vehicle has at least one available subchannel resource. This indicates the probability that the vehicle will continue to use the original channel resources. express( The steady-state probability of a state; state( The ) indicates the waiting time before CAM data packet transmission. ms; status ( steady-state probability Represented as: (2) in Indicates the maximum allowable delay for CAM. Indicates state ( The steady-state probability of ). Define state ( ) indicates the waiting time before DENM data packet transmission. ms; status ( steady-state probability Represented as: (3) in Indicates the maximum allowable delay of DENM; Define state ( , This indicates that the vehicle's RC value has reached... The state, state (R, steady-state probability Represented as: (4) Where RC represents the reselection counter, h represents the upper limit of the range of RC values, and t represents its lower limit. Indicates state ( The steady-state probability of ,1). This indicates that the vehicle's RC value at this time is , Representing state The steady-state probability; Define state ( (,m) represents the state where the vehicle's RT value reaches m. steady-state probability of m) Represented as: (5) in This represents the initial value of the retransmission counter RT. Representing state The steady-state probability is obtained by summing the steady-state probabilities of all states according to the normalization property of Markov chains. Therefore, the steady-state probability of each state can be derived.

3. The resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning according to claim 2, characterized in that, In step 2, four types of errors are defined: half-duplex error, perception error, propagation error, and packet collision error. If none of the four errors occur, it means that the data packet was correctly received. The formula for the data packet reception ratio is as follows: ； in Indicates the data packet reception ratio. This represents the probability of a half-duplex error occurring. This represents the average probability that a data packet cannot be received correctly due to a perception error. This represents the average probability that a data packet cannot be received correctly due to a propagation error; This represents the probability of a packet collision caused by a resource selection conflict.

4. The resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning according to claim 3, characterized in that, In step 2, the probability of a half-duplex error occurring... The formula is expressed as follows: (6) in This indicates the total number of subframes in a selection window. The formula for representing the number of vehicles within the communication range is: (7) in , and These represent the number of lanes, vehicle density, and communication radius, respectively. In step 2, the process of calculating the average probability of failing to receive data packets due to perception errors is as follows: Considering the effects of path loss and shadow fading, the received signal power Represented as: (8) in Indicates the distance between the sender and receiver. Indicates the transmission power. and Representing path loss and shadow fading, respectively; the probability density function of the received signal power. Represented as: (9) in The standard deviation represents the log-normal distribution. Indicates possible values ​​for the received signal power; Received signal power Below the sensing power threshold probability for: (10) in Represents the error function; Will Defined as the average probability of failing to receive data packets due to perceptual errors, expressed as: (11) in and These represent the maximum and minimum distances between the sender and receiver, respectively. In step 2, the process of calculating the average probability of failing to receive data packets correctly due to propagation errors is as follows: Consider path loss Shadow decay And noise factors, received signal-to-noise ratio Represented as: (12) in, Indicates the received signal power. Indicates noise power. Indicates the transmission power; Due to shadow decay It follows a log-normal distribution, therefore the probability density function of the received signal-to-noise ratio is... Represented as: (13) in This indicates the possible values ​​of the received signal-to-noise ratio. This represents the standard deviation of the log-normal distribution. Received signal power not less than And the received signal-to-noise ratio is below the threshold. probability Represented as: (14) Will Defined as the average probability of failing to receive a data packet correctly due to propagation errors, it is expressed as: (15) in and These represent the maximum and minimum distances between the sender and receiver, respectively.

5. The resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning according to claim 3, characterized in that, In step 2, the process of calculating the probability of packet collisions caused by resource selection conflicts is as follows: definition For the total number of resources, The number of excluded resources; for those within the communication range The probability that a certain resource is not selected by any vehicle is . ; Therefore, the probability that at least one car chooses this resource is ;thus, Represented as: (16) Total resources Represented as: (17) in This indicates the number of subframes in the selection window. This indicates the number of sub-channels in each subframe; To calculate the packet collision probability, it is necessary to determine how many vehicles contain the same resource in their candidate resource sets; in the proposed Markov chain model, the resource selection window opens under three conditions: 1) When the vehicle reaches the state When a new resource selection is required, the CAM selection window will be launched; 2) When a vehicle generates a CAM data packet in Idle mode, the CAM selection window is activated; 3) When a vehicle generates a DENM data packet in Idle mode, the DENM selection window is launched; First, deduce the state the vehicle reaches. When the CAM selection window is launched, the resources in its candidate resource set include The probability of the vehicle in a certain state; The duration is ; It is a vehicle A candidate resource; definition For a vehicle in the vehicle Entering state during selection window The probability is expressed as: (18) Therefore, the probability that the candidate resource set contains resource R is... Represented as: (19) in This indicates the probability that the vehicle will continue to use the original channel resources; When the vehicle is in an idle state, the CAM selection window is activated, and the resources in the vehicle's candidate resource set include... The probability that the vehicle remains idle for a certain duration is given. ; Representing state The steady-state probability; The candidate resource set for vehicles contains resources probability Represented as: (20) in and These represent the probabilities of generating CAM data packets and DENM data packets, respectively. Indicates idle The steady-state probability of a state; Indicates the maximum allowable delay for CAM; The derivation shows that when a vehicle starts the DENM selection window in an idle state, the resources in its candidate resource set include... probability : (21) Using the three probabilities mentioned above, we can obtain the resources contained in the candidate resource set. Number of vehicles for: (22) vehicle Each resource in the candidate resource set presents three states: The first state is when not selected by any vehicle, defined as follows: The probability of the first state occurring is calculated as follows: (23) The second state is being selected by exactly one car, defined as follows: The probability of the second state occurring is calculated as follows: (24) in Indicates the number of vehicles within the communication range. For combination, From Randomly select one vehicle from the vehicles; The third state is when at least two vehicles select the vehicle, defined as follows: The probability of the third state occurring is calculated as follows: (25) Using the total number of candidate resources, we obtain the expected number of resources in each state: (26) (27) (28) in, , and These represent the quantities of resources in the first, second, and third states, respectively. and The sum represents the total number of resources selected by at least one vehicle within the CAM selection window; Therefore, the probability of a packet collision caused by a resource selection conflict is expressed as: (29) 。 6. The resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning according to claim 1, characterized in that, In step 2, the delay is defined as the time interval between two consecutive successful data packet receptions by the receiving vehicle; this time interval is determined by the resource-reserved delay. Delay caused by data packet collision It consists of two parts; The resource reservation delay is determined by both the size of the selection window and the data packet transmission time. The resource reservation delay is calculated as follows: (30) in, and These represent the number of CAM data packets and DENM data packets generated within a resource reservation interval, respectively. and The value depends on the resource reservation interval and the packet generation interval, and is expressed as: , (31) in Reserve intervals for resources, Generate intervals for CAM data packets. The interval for generating DENM packets; in formula (32) and The transmission times of CAM and DENM data packets are respectively defined as: , (32) in and These represent the sizes of the CAM and DENM data packets, respectively. Indicates the data packet transmission rate; When vehicles select resources, multiple vehicles may simultaneously select the same resource, leading to data packet collisions. When a collision occurs, the latency will increase by a resource reservation interval; therefore, the latency caused by data packet collisions is significant. Represented as: (33) in This represents the probability of packet collisions caused by resource selection conflicts; therefore, the average latency... Represented as: (34)。 7. The resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning according to claim 1, characterized in that, In step 3, the process by which GCN obtains the spatial topological features between neighbors is as follows: First, the vehicle communication network is represented as a graph; each vehicle can be considered a node in the graph, and the communication link between two vehicles can be considered an edge; given the graph... ,in For a set of nodes, Let it be the set of edges; node The observation is defined as a vehicle status edge weight Determined by the connection relationship between the two vehicles, it is represented as: (35) in, Indicates vehicle With vehicles The physical distance between them and Representing vehicles and vehicles Its position in the Internet of Vehicles This represents the vehicle's maximum communication radius. After representing the network as a graph, GCN is used to aggregate node features; node state vector First, it is encoded into a fixed-length feature vector: (36) in Represents a node The initial feature vector, Indicates the encoding function; definition Aggregating information from neighbors is represented as: (37) in, Represents intelligent agents The neighborhood group, Represents a node In the Feature vectors aggregated by layer GCN Represents aggregate functions, Indicates the layer number of the GCN; the first The convolution process is represented as: (38) in Represents a node go through Feature vectors after GCN aggregation. Represents a node In the Feature vectors after GCN aggregation. This represents vector concatenation. This represents the trainable weights in the convolutional layer. Indicates the activation function; The intelligent agent obtained after GCN aggregation with a fixed number of layers The feature vector is defined as the spatial topological feature. .

8. The resource allocation method for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning according to claim 1, characterized in that, In step 3, to capture the dynamic characteristics of time, LSTM is used in conjunction with historical information. intelligent agent The feature vector is updated to The formula is as follows: (39) in, Indicates time step The hidden state, The weight matrix is ​​a learnable matrix. This represents the final spatiotemporal feature vector, where Indicates the number of observations; This represents the process of extracting time features; Spatial feature extraction is performed using GCN, and temporal feature extraction is performed using LSTM. The feature extraction network is combined with a deep Q-network to learn the optimal channel resource selection strategy for each vehicle. The agent's goal is to maximize the cumulative discount reward. , Defined as: (40) in, Indicates at time step The instant reward received; The discount factor balances immediate and future rewards; the Q-function is used to quantify the agent's state. Execute actions according to the strategy Expected cumulative reward; The expression for the Q function is as follows: (41) in Represents the action value function. This indicates cumulative discount rewards. Indicates a known state Current action In the case of [the specific circumstances], and the expected value when future actions are chosen according to the strategy; Based on the extracted spatiotemporal feature vectors, the input of DQN, i.e., a deep Q-network, is... Represented as: (42)。 9. A resource allocation system for a graph reinforcement learning-based cellular vehicle network for implementing the resource allocation method for a graph reinforcement learning-based cellular vehicle network as described in claim 1, characterized in that, The resource allocation system for cellular vehicle-to-everything (V2X) networks based on graph reinforcement learning includes the following modules: The Markov chain model building module is used to build a two-dimensional Markov chain model to describe the resource allocation and data packet transmission process in the coexistence scenario of CAM and DENM. The packet reception ratio model building module is used to build a packet reception ratio model to evaluate the probability of successful reception of CAM and DENM packets, and to characterize the time interval between two consecutive CAM or DENM packets successfully received by the same vehicle through a delay model. And a resource allocation module is proposed to establish a multi-agent deep reinforcement learning framework to realize resource selection decision-making for vehicle intelligence, and a resource allocation method based on graph reinforcement learning is proposed to adapt to the frequently changing topology in the Internet of Vehicles. By using a resource allocation method based on graph reinforcement learning, an agent can be guided to extract the spatial topological features of vehicle nodes, and based on the extracted spatial topological features, learn and obtain the optimal resource allocation decision for each vehicle.

Images