Cognitive Internet of Vehicles spectrum allocation method in uncertain environment

Abstract

The invention belongs to the technical field of mobile communication, and particularly relates to a cognitive Internet of Vehicles spectrum allocation method in an uncertain environment, which comprises the following steps: constructing a cognitive vehicle network system model comprising a base station, a plurality of cellular users and a plurality of cognitive vehicles; based on a cognitive vehicle network system model, constructing an imperfect CSI model, a spectrum sensing error model and a communication model; setting constraint conditions, and constructing a target optimization function according to the cognitive vehicle network system model, the imperfect CSI model, the spectrum sensing error model, the communication model and the constraint conditions; establishing a Markov decision process for the target optimization function; based on a Markov decision process, a multi-agent reinforcement learning algorithm is adopted to solve and obtain an optimal spectrum allocation strategy of the cognitive vehicle; according to the invention, under the scene of the Internet of Vehicles, the cognitive radio is used to solve the problem of spectrum scarcity of the Internet of Vehicles.

Description

Technical Field

[0001] This invention belongs to the field of mobile communication technology, and specifically relates to a method for spectrum allocation in cognitive vehicle-to-everything (V2X) networks under uncertain environments. Background Technology

[0002] With the rapid development of modern automotive technology, future automotive technologies will drive real-time communication and collaboration between vehicles (V2V) and between vehicles and infrastructure (V2I), thereby enabling functions such as data exchange, intelligent traffic management, and dynamic route scheduling. With the surge in the number of connected vehicles and the stringent requirements of intelligent driving vehicles and data-intensive applications, such as ultra-low latency, high reliability, and high security, vehicle-to-everything (V2X) technology is developing rapidly, promising to bring entirely new solutions to society and provide seamless, reliable, and efficient large-scale connectivity for countless users and connected vehicles.

[0003] The massive number of network connections and diverse data services generated by the Internet of Vehicles (IoV) have led to a severe spectrum scarcity problem. Limited spectrum resources significantly restrict the bandwidth and quality of service in IoV communication, further exacerbating the difficulty of ensuring reliable communication between vehicles in high-traffic environments. Spectrum congestion often results in data transmission delays or even communication interruptions, thereby affecting the efficient operation of intelligent transportation systems and the safety of autonomous driving.

[0004] To address this issue, cognitive radio technology is considered a key technology for solving the spectrum scarcity problem in vehicle-to-everything (V2X) communication. By sensing the activities of primary users in real time, cognitive radio ensures that secondary users dynamically access available spectrum for data transmission without interfering with primary users. Through intelligent spectrum allocation mechanisms, cognitive radio can effectively guarantee the quality of service (QoS) of V2X communication and significantly improve spectrum utilization. While effectively alleviating the problem of insufficient spectrum resources in V2X, it also provides a feasible solution for spectrum optimization in V2X communication.

[0005] However, current research on spectrum allocation algorithms for cognitive vehicle-to-everything (V2X) networks still faces many challenges:

[0006] 1. Traditional heuristic and optimization-based methods are limited by specific assumptions and the requirements of structured problems, making them difficult to adapt to the dynamic and uncertain Internet of Vehicles environment, especially the inherent difficulties in the mathematical representation of advanced services such as the periodic propagation of safety messages;

[0007] 2. Existing methods are mostly based on the assumptions of ideal CSI and perfect spectrum perception, while the time-varying characteristics of CSI and spectrum perception errors in the mobile vehicle-to-everything (V2X) environment can lead to uneven spectrum allocation and degraded communication quality.

[0008] 3. The centralized allocation strategy relies too heavily on the central node. In scenarios where vehicles can only obtain local information, a large amount of communication overhead is required to achieve global information synchronization, resulting in insufficient real-time performance. Summary of the Invention

[0009] To address the above problems, this invention provides a cognitive vehicle-to-everything (V2X) spectrum allocation method under uncertain environments, comprising the following steps:

[0010] S1. Construct a cognitive vehicle network system model that includes base stations, multiple cellular users, and multiple cognitive vehicles; wherein each cognitive vehicle communicates with its neighboring cognitive vehicles within its communication range, forming multiple V2V user pairs;

[0011] S2. Based on the cognitive vehicle network system model, construct an imperfect CSI model, a spectrum sensing error model, and a communication model;

[0012] S3. Set constraints and construct the objective optimization function based on the cognitive vehicle network system model, the imperfect CSI model, the spectrum sensing error model, the communication model, and the constraints.

[0013] S4. Establish a Markov decision process for the objective optimization function;

[0014] S5. Based on Markov decision process, a multi-agent reinforcement learning algorithm is used to solve for the optimal spectrum allocation strategy of cognitive vehicle.

[0015] The beneficial effects of this invention are:

[0016] This invention provides a cognitive vehicular network (V2V) spectrum allocation method under uncertain environments, considering the use of cognitive radio to address the spectrum scarcity problem in V2V scenarios. Addressing the dynamic and partially observable nature of the cognitive V2V environment, and comprehensively considering vehicle mobility, main user activity characteristics, and main network access constraints, a cognitive V2V environment model based on spectrum sensing error and imperfect CSI is established. A spectrum allocation problem is proposed with the objective of maximizing V2V communication payload delivery rate. A partially observable Markov decision process is used to model the problem, and a multi-agent algorithm is designed based on this to achieve intelligent decision-making for spectrum allocation. Finally, simulations verify the feasibility of the proposed scheme. Compared with other algorithms, the proposed scheme can effectively adapt to the dynamic cognitive V2V environment and improve the V2V user payload success delivery rate. Attached Figure Description

[0017] Figure 1 This is a system model diagram in the present invention;

[0018] Figure 2 This is a comparison chart of the average reward values ​​obtained by different algorithms in this invention;

[0019] Figure 3 This is a graph showing the change in V2V user payload success rate with the number of payloads in this invention;

[0020] Figure 4 This is a graph showing the change in V2V user payload success rate with vehicle speed in this invention;

[0021] Figure 5 This is a graph showing the change in V2V user payload success rate with spectrum sensing error in this invention;

[0022] Figure 6 This is a graph showing the change in V2V user payload success rate with CU transmission rate in this invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Regarding the spectrum allocation problem in cognitive vehicle networks, some research results include: (1) QoS-Aware Uplink Spectrum Allocation for Vehicular Communications (Thakur KP, Palit B. A QoS-Aware Uplink Spectrum and Power Allocation With Link Adaptation for Vehicular Communications in 5G Networks[J]. IEEE Transactions on Network and Service Management, 2024.) proposed a spectrum and power allocation algorithm based on link adaptation. It adopts a greedy strategy to allocate resources to cellular users according to the link conditions of cellular users, uses the maximum weighted binary matching Hungarian algorithm to pair cellular users and vehicle users, and uses power allocation to make them meet their respective rate constraints and coverage probability constraints. (2) Selective Spectrum Sensing and Allocation in Cognitive Vehicular Radio Networks Based on Deep Learning (Paul A, Choi K. Deep learning-based selective spectrum sensing and allocation in cognitive vehicular radio networks[J]. Vehicular Communications, 2023, 41: 100606.) proposes a selective spectrum sensing strategy based on the LSTM algorithm, prioritizing the sensing of channels with higher PU activity probabilities to address the problem of blind sensing. It also proposes a global signal state decision-making strategy based on DRL to cope with the uncertain spectrum environment in cognitive vehicle networks. Based on this, a spectrum allocation strategy based on deep Q-learning is proposed to allocate the best channel to the SU from the idle channel pool.

[0025] In addition, some research results include: (3) Joint collaborative big spectrum data sensing and reinforcement learning based dynamic spectrum access for cognitive internet of vehicles (Liu X, Sun C, Yau KLA, et al. Joint collaborative big spectrum data sensing and reinforcement learning based dynamic spectrum access for cognitive internet of vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(1): 805-815.) introduces a collaborative large-scale spectrum sensing mechanism, utilizing multiple vehicles to collect spectrum data in different areas, which is then fused by the base station. For the three spectrum access modes, a dynamic spectrum access scheme based on Q-learning is adopted to optimize spectrum utilization. (4) Joint power control and time allocation for UAV-assisted IoV networks over licensed and unlicensed spectrum[J]. IEEE Internet of Things Journal, 2024, 11(1): 1522-1533. This paper studies the resource allocation problem of introducing unlicensed spectrum in UAV-assisted IoV networks. For ground-based IoV systems, a joint power control and time allocation scheme based on licensed and unlicensed spectrum is proposed to maximize the total system capacity of UAV-assisted IoV systems and WiFi systems, while reducing mutual interference between air-to-ground IoV systems and ground-based IoV systems.

[0026] However, the aforementioned research on cognitive vehicle network spectrum allocation algorithms still faces problems such as difficulty in adapting to dynamic and uncertain vehicle network environments and insufficient real-time performance.

[0027] Therefore, some embodiments of the present invention propose a cognitive vehicle-to-everything (V2X) spectrum allocation method under uncertain environments, comprising the following steps:

[0028] S1. Construct a cognitive vehicle network system model that includes base stations, multiple cellular users, and multiple cognitive vehicles; wherein each cognitive vehicle communicates with its neighboring cognitive vehicles within its communication range, forming multiple V2V user pairs;

[0029] S2. Based on the cognitive vehicle network system model, construct an imperfect CSI model, a spectrum sensing error model, and a communication model;

[0030] S3. Set constraints and construct the objective optimization function based on the cognitive vehicle network system model, the imperfect CSI model, the spectrum sensing error model, the communication model, and the constraints.

[0031] S4. Establish a Markov decision process for the objective optimization function;

[0032] S5. Based on Markov decision process, a multi-agent reinforcement learning algorithm is used to solve for the optimal spectrum allocation strategy of cognitive vehicle.

[0033] Figure 1 This is a system model diagram of a cognitive vehicle-to-everything (V2X) spectrum allocation method under uncertain conditions, as shown in some embodiments of the present invention.

[0034] In some embodiments, the cognitive vehicle network system model is as follows Figure 1 As shown, it includes base stations (BS), cellular users (CU), and cognitive vehicles, with the cognitive vehicles acting as secondary users that move freely within the CU network coverage area.

[0035] In the cognitive vehicle network system model, OFDM is used to divide the overall uplink available frequency bandwidth into M orthogonal sub-channels with a bandwidth of W, and the channel set is defined as follows. ={1,2,…,M}, these M channels are allocated to the uplink communication between the M CUs and the BS. Assume there are N cognitive vehicles in the network, and represent the vehicle set as... ={1,2,…,N}, where each cognitive vehicle communicates with its neighboring cognitive vehicles within its communication range, representing the set of V2V user pairs in the network as follows: ={1,2,…,K}, where K represents the number of V2V user pairs; each V2V pair consists of a V2V transmitter and a V2V receiver, and K V2V user pairs and M cellular users share M channels.

[0036] V2V users access channels via a hybrid spectrum access method while ensuring normal communication with the CU. In overlay mode, the cognitive vehicle can only access idle channels not used by the CU; in underlay mode, the cognitive vehicle can share channels with the CU, but its transmit power needs to be limited to avoid interfering with the CU. Before spectrum access, each V2V user needs to perform spectrum awareness and use binary variables s. m Let (t) represent the state of channel m at time slot t, s m (t)=1 indicates that channel m is occupied by CU, that is, channel m is busy, s m(t)=0 indicates that channel m is not occupied by CU, that is, channel m is idle. Therefore, each channel has two states, and its state transitions follow a two-state Markov chain, represented as:

[0037]

[0038] In the formula p represents the state transition probability matrix of channel m; 00 p represents the probability that channel m remains idle; 01 p represents the probability that channel m transitions from an idle state to a busy state. 10 p represents the probability that channel m transitions from a busy state to an idle state. 11 This represents the probability that channel m remains busy.

[0039] The system's time resources are divided into multiple time slots, and the time slot set... ={1,2,…,T}. Assuming each V2V user pair selects only one channel for transmission within a single time slot, if the k-th V2V user pair selects access channel m in the t-th time slot, then the binary representation a... k,m (t)=1, otherwise a k,m (t)=0, and the transmit power of the kth V2V user pair on channel m at time slot t is represented as p. k,m (t).

[0040] In some embodiments, a traffic model is constructed based on a cognitive vehicle network system model. The traffic model in this embodiment adopts the Manhattan model, referencing the traffic scenario settings of 3GPP TR 36.885. The network topology uses a grid layout, with horizontal and vertical roads intersecting each other. The base station is located at the center of network coverage. It is assumed that the network adopts a bidirectional traffic model, the initial distribution of cognitive vehicles on the lanes follows a spatial Poisson process, and the cognitive vehicles move in the network at a constant speed v∈[v...]. min ,v max Free movement, v min v max Let p represent the minimum speed and p represent the maximum speed, respectively. At each intersection, the cognitive vehicle randomly selects a direction of travel: uphill, downhill, left turn, or right turn, with probabilities p, p, and p, respectively. u p d p l p r And satisfy p u +p d +p l +p r =1. After selecting a direction, the vehicle continues to travel in a straight line until it reaches the next intersection.

[0041] In some embodiments, in the imperfect CSI model, the channel gain g of the k-th V2V user pair in slot t on channel m is defined. k,m (t) is represented as:

[0042]

[0043] In the formula, ω k,m (t) represents the path loss of the k-th V2V user pair in time slot t on channel m, ξ k,m (t) represents the shadowing fading of the k-th V2V user pair in time slot t, which follows a log-normal distribution on channel m; h k,m (t) represents the small-scale fading component of the k-th V2V user pair in time slot t on channel m.

[0044] Specifically, the small-scale fading component h k,m (t) exhibits frequency selectivity and changes dynamically with time slots. This embodiment assumes that the channel state information (CSI) of the V2V user-to-user link is estimated by the receiver, while the CSI of the cellular user-to-base station link is directly estimated at the BS. Since the aforementioned CSI needs to be transmitted to the transmitter of the V2V user pair through a feedback period τ, and considering that the high mobility of cognitive vehicles will cause significant Doppler frequency shift, leading to time-varying characteristics of small-scale fading, a first-order Gaussian-Markov random process is used to model the fast fading time-varying characteristics within the feedback period τ, resulting in:

[0045]

[0046] in, Let represent the outdated small-scale fading components, assuming they are independent and identically distributed and follow CN(0,1), such that It follows an exponential distribution with a mean of 1; e k,m (t)~CN(0,1) represents the independent complex Gaussian noise component in time slot t that is uncorrelated with the outdated small-scale fading component; λ k,m (t) represents the time-varying correlation coefficient between time slot t and the outdated small-scale fading component, which is constructed using the Jakes statistical model as follows:

[0047]

[0048] In the formula, J0(·) represents the zeroth-order Bessel function of the first kind; v k f represents the relative velocity of the k-th V2V user pair; m τ represents the carrier frequency; τ represents the feedback period; and c represents the speed of light.

[0049] Specifically, shadow decay ξ k,m(t) is typically spatially independent, primarily depending on the relative location distribution of the transceiver; that is, the calculation of shadow fading depends on the relative location of the mobile cognitive vehicle. In this embodiment, the update formula for shadow fading for any V2V user on any channel in any time slot is:

[0050]

[0051] In the formula, ξ new and ξ old These represent the shadow fading values ​​before and after the update, respectively; △d represents the distance traveled by the cognitive vehicle; d rel Let Z represent the relevant distance of shadow fading; Z represents a Gaussian random variable; where the vehicle position can be updated based on the vehicle's real-time speed and direction of motion, and the cognitive vehicle position calculation is defined as follows:

[0052]

[0053]

[0054] In the formula, x n (t), y n (t) represents the horizontal and vertical coordinates of the nth cognitive vehicle in time slot t, respectively. n Let θ represent the speed of the nth cognitive vehicle. n (t) represents the direction of motion of the nth cognitive vehicle in time slot t, and △t represents the time step.

[0055] In some embodiments, in the spectrum sensing error model, considering imperfect spectrum sensing, the true state of channel m at time slot t may not be accurately reflected. Therefore, the probability ε of the sensing error of the k-th V2V user on channel m in time slot t is defined. k,m (t) ~U(0, ε max Then we have:

[0056]

[0057] In the formula, Pr{s m (t)= } represents the actual occupancy status of channel m in time slot t. The probability, This represents the sensing state of a V2V user in time slot t relative to channel m, s m (t) represents the actual state of channel m in time slot t; ε k,m (t)~U(0,ε max ) represents ε k,m (t) is uniformly distributed in [0, ε max ] within; ε max This represents the maximum perceptual error probability.

[0058] In some embodiments, in the communication model, the signal-to-interference-plus-noise ratio γ is defined when the k-th V2V user selects channel m. k,m (t) is:

[0059]

[0060] In the formula, a k,m (t) represents the channel allocation indication for channel m by the k-th V2V user in time slot t, p k,m (t) represents the transmit power of the k-th V2V user in time slot t on channel m, g k,m (t) represents the channel gain of the k-th V2V user pair in time slot t on channel m, σ 2 Indicates noise power; I m,k (t) represents the interference generated by cellular users on channel m to the kth V2V user pair; when channel m is idle, I m,k (t)=0, when channel m is busy. s m (t) represents the actual state of time slot m in time slot t; p m (t) represents the transmit power of the cellular user on channel m; g m,k (t) represents the interference channel gain from the cellular user to the receiver of the k-th V2V user pair; This represents the interference channel gain from the transmitter of the k'-th V2V user pair to the receiver of the k-th V2V user pair on channel m. This represents a set of V2V user pairs.

[0061] Define the interference I generated by the k-th V2V user accessing channel m to the base station. k,m (t) is:

[0062]

[0063] In the formula, g k,m,B (t) represents the interference channel gain from transmitter to base station for the k-th V2V user pair.

[0064] Define the transmission rate R of the k-th V2V user pair on channel m. k,m (t) is:

[0065]

[0066] In the formula, W represents the channel bandwidth, b k,m (t) represents the reliability performance index of the k-th V2V user pair for data transmission on channel m, used to ensure effective communication between vehicles, where,

[0067]

[0068] In the formula, γ k,min (t) represents the SINR threshold for communication between the k-th V2V user.

[0069] To prevent harmful interference from V2V users to the CU, in this embodiment of the invention, V2V users share the channel with the CU simultaneously by limiting their transmit power. ={1,2,…,K'} represents the set of V2V user pairs accessing channel m, and the interference of V2V user pairs accessing channel m to the CU must be less than the interference threshold. , represented as ,Right now:

[0070]

[0071] To meet the minimum transmission rate C of the CU m (t), defined as:

[0072]

[0073] Therefore, Represented as:

[0074]

[0075] In the formula, G represents the transmit power of the m-th CU. m,B (t) represents the channel gain from the cellular user to the base station on channel m.

[0076] In some embodiments, the V2V link is responsible for reliably transmitting safety-critical messages. To ensure the safe and reliable operation of the vehicle-to-everything (V2X) system, vehicles need to continuously receive safety-critical messages from surrounding vehicles, such as vehicle status information, traffic environment information, and cooperative driving assistance information. These messages are generated periodically at varying frequencies based on vehicle mobility, imposing stringent requirements on the real-time performance and reliability of transmission. To quantify this requirement, this embodiment establishes the following mathematical model: [The text then describes a message generation period.] Within the network, each vehicle needs to receive a signal of size B from its neighboring vehicles. k In addition to the data, there are delay constraints, requiring the vehicle to be within the time constraint T. k Data transfer is completed within the scope, described as {B k T k The reliability of such message transmission can be mathematically modeled as follows: under time constraint T... k The size of the successfully transmitted data is B. k The probability of grouping (i.e., V2V payload delivery rate) is expressed as:

[0077]

[0078] Therefore, the system optimization objective is to help V2V users rationally select their own transmission parameters, namely the access spectrum and the transmission power used, to improve their communication performance. Based on this, the cognitive vehicle-to-everything (V2V) spectrum resource allocation problem is defined as follows:

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086] In the formula, This represents the set of binary channel allocation variables for V2V user pairs. Let T represent the set of power allocation variables for V2V user pairs, Pr() represent the probability function used to represent the success probability of the task within the parentheses, K represent the number of V2V user pairs, and T represent the number of V2V user pairs. k This represents the data transmission time constraint for the k-th V2V user pair, where M represents the number of channels, and a k,m (t) represents the channel allocation indication for the k-th V2V user in time slot t to channel m, R k,m (t) represents the transmission rate of the k-th V2V user pair on channel m, B k γ represents the size of the security data, in bits, periodically generated by the k-th V2V user, where Δt represents the time step and γ represents the value of the security data. k,m γ(t) represents the signal-to-interference-plus-noise ratio (SIR) of the k-th V2V user when selecting channel m, where γ k,min (t) represents the SINR threshold for communication between the k-th V2V user and p. k,m p(t) represents the transmit power of the k-th V2V user in time slot t on channel m, where p max G represents the maximum transmit power of a V2V user pair. k,m,B (t) represents the interference channel gain from transmitter to base station for the k-th V2V user pair. R represents the interference threshold of channel m in time slot t. k,m (t) represents the transmission rate of the k-th V2V user pair on channel m, C m (t) represents the minimum transmission rate required by cellular users on channel m, Nn (t) represents the distance threshold l between the nth cognitive vehicle and the other vehicle. th An orderly collection of vehicles within the area. Represents a set of vehicles, l n (t) represents the position coordinates of the nth cognitive vehicle in time slot t, l th N represents the distance threshold for neighbor selection. neighbor This represents the maximum number of neighboring vehicles allowed for each cognitive vehicle.

[0087] C1 indicates that each V2V user pair can only occupy one channel at a time; C2 indicates the reliability constraint of V2V user pair in data transmission; C3 indicates the power constraint; C4 indicates the interference threshold constraint of active CU; C5 indicates the minimum transmission rate constraint of CU; and C6 indicates the neighbor selection constraint of cognitive vehicle.

[0088] In some embodiments, for the objective optimization function, each V2V user pair is used as an agent in reinforcement learning and modeled as a Markov decision process, specifically including:

[0089] 1) Observation space: The true environmental state S(t) (including accurate channel state information and the behavior of other agents) is unknown for each individual V2V user pair, and the understanding of the environment can only be obtained through observation functions.

[0090] Define the local observation O of the k-th V2V user pair in time slot t. k (t) is:

[0091]

[0092]

[0093] (t) represents the remaining V2V payload of the k-th V2V user pair. (t) represents the remaining time budget for data transmission by the k-th V2V user, φ represents the number of training iterations, and ε represents the instantaneous exploration rate. (t) represents the sensing state of channel m in time slot t, I k,m (t) represents the interference caused to the base station by the k-th V2V user accessing channel m, and g k,m (t) represents the channel gain of the k-th V2V user pair on channel m, g k,m,B (t) represents the transmitter-to-base station interference channel gain for the k-th V2V user pair, g m,k (t) represents the interference channel gain from the cellular user to the receiver of the k-th V2V user pair, G k,m (t) represents the channel information for time slot t. Represents the set of channels.

[0094] The number of training iterations φ and the instantaneous exploration rate ε are used to track the policy adjustments of other V2V user pairs during the training phase, thereby reducing the impact of the non-steady-state characteristics of the multi-agent environment.

[0095] 2) Action space: Each V2V user autonomously makes decisions on spectrum access and power control based on local observations of each time slot.

[0096] Define the action A of the k-th agent in time slot t. k (t) is:

[0097]

[0098] a k,m (t) represents the channel allocation indication for channel m by the k-th V2V user in time slot t, p k,m (t) represents the transmit power of the k-th V2V user in time slot t on channel m.

[0099] 3) Reward function: Given the current observation vector O k (t), V2V users select action a according to the policy in time slot t. k,m (t), at which point it will receive a reward from the environment.

[0100] Specifically, since V2V users experience different levels of interference on idle channels and channels occupied by CUs, the reward function needs to differentiate between them based on channel conditions. Furthermore, to encourage efficient data transmission by V2V users, the reward function also needs to consider whether data packet delivery is completed ahead of schedule. Therefore, this embodiment of the invention divides the reward function for the k-th V2V user in time slot t into the following cases:

[0101] (1) If the k-th V2V user fails to complete the data packet transmission in advance and the channel it accesses is not occupied by cellular users, then the transmission rate of the k-th V2V user on the channel it accesses in time slot t shall be used as the reward for time slot t, and the reward r k (t) is represented as:

[0102]

[0103] (2) If the k-th V2V user pair fails to complete data packet transmission in advance and the channel it accesses is occupied by the CU, then the difference between the interference threshold of the CU on the channel and the interference generated by the CU on the channel to the k-th V2V user pair at time slot t is taken as the reward for time slot t. At this time, the reward r k (t) is represented as:

[0104]

[0105]

[0106] In the formula, L k (t) represents the penalty term for the k-th V2V user after accessing the channel, due to interference with the CU; This represents the interference penalty factor.

[0107] (3) If the k-th V2V user pair has a data transmission time constraint T k If a user completes data packet transmission ahead of schedule and remains silent thereafter without accessing the channel for data packet transmission, then the k-th V2V user will receive a constant reward β for each remaining time slot. f (Rewards are still applied during data packet transmission as in the previous two scenarios), thus encouraging V2V users to complete data transmission as efficiently as possible. The reward at this time is r. k (t) is represented as:

[0108] r k (t)= β f

[0109] (4) If the k-th V2V user pair has a data transmission time constraint T k If no data packets are transmitted within the channel, meaning no data is sent to any channel, the user will not receive any reward or penalty. In this case, the reward r... k (t):

[0110] r k (t)=0.

[0111] The fourth scenario involves V2V user pairs that have not yet completed their transmission. To prevent multiple V2V user pairs from accessing the same channel, they can choose not to access the channel at this time to avoid collisions.

[0112] To address the non-stationary nature of distributed solutions in multi-agent environments, adopting a mechanism where all V2V user pairs share the total reward transforms the competitive game into a fully cooperative one. However, this strategy is prone to convergence to local optima and may sacrifice the individual V2V user pairs' gains to improve global performance. Therefore, from an individual performance perspective, this algorithm is not ideal. To solve these problems, an instantaneous reward function for the k-th V2V user pair is defined. The structure is as follows:

[0113]

[0114] Where λ (0≤λ≤1) is the trade-off coefficient for balancing individual goals and global goals.

[0115] In reinforcement learning algorithms, the agent's objective is to maximize cumulative reward. Therefore, for a V2V user pair, at time slot t, it will, based on its observations of the local environment and reward feedback, select a channel and transmit power level to maximize the reward from time slot t up to time slot T. k Cumulative reward G t , will G t Represented as:

[0116]

[0117] In the formula, γ t Discount factor (0≤ ≤1)) to the power of t, T k This represents the total number of time slots accumulated as rewards.

[0118] Quantifying state-action pairs The expected return, i.e., in state O k (t) take action A k After (t), the expected cumulative reward that V2V users can obtain is expressed by introducing the core reinforcement learning metric, namely the state-action-value function (Q-value function):

[0119]

[0120] In the formula, Indicates local observation O k (t) Select action A k The Q value of (t), This represents the mathematical expectation operator.

[0121] In some embodiments, the traditional DQN (Deep Q-Network) algorithm fits the Q-value function through a deep neural network to achieve action decision-making in a high-dimensional state space. However, it has two major drawbacks: First, it suffers from overestimation of Q-values. Since a single network is responsible for both action selection and value assessment, it is prone to overly optimistic estimates of the Q-values ​​of advantageous actions, leading to decision bias. Second, it suffers from low efficiency in value assessment. It assesses the Q-values ​​of all actions independently, failing to utilize the inherent correlation between the value of the state itself and the relative advantage of the action, and thus cannot meet the low-latency requirements of V2V user pairs.

[0122] To address the problem of "overestimation of Q-value function" in traditional DQN algorithms, this invention introduces a Double DQN architecture, separating the action selection and value evaluation stages by deploying two parallel neural networks (DRN1 and DRN2) with identical structures but independent parameters. DRN1 is responsible for selecting the optimal action based on the current state, while DRN2 is responsible for objectively evaluating the Q-value of the action selected by DRN1, thereby reducing the bias caused by overestimation.

[0123] To address the low efficiency of value assessment in traditional DQN algorithms, theoretically, for some states, the chosen action will not significantly affect subsequent rewards, thus eliminating the need to evaluate the value of all actions. Therefore, the Dueling DQN architecture is further introduced. The Dueling DQN architecture consists of two sub-networks: one network estimates the state-value function, representing the long-term value of the state, and the other estimates the action advantage function, measuring the value of actions in the observed state. Select action The advantage over the average action is considered. The outputs of these two sub-networks are aggregated to approximate the Q-value function. Therefore, the proposed DRN1 and DRN2 have completely identical network structures, both containing an input layer, a shared feature extraction hidden layer, and two parallel output branches (state value branch and action advantage branch). The input layer receives the state feature vector of the V2V user observation; the shared feature extraction hidden layer extracts high-order abstract features of the state through a 3-layer fully connected network; the output layer is divided into a state value sub-network and an action advantage sub-network, which output the inherent value of the state and the relative advantage of the action, respectively.

[0124] In addressing the aforementioned optimization problem, V2V users must be able to extract valid information from outdated interference state information and imperfect spectrum sensing results. To this end, an LSTM layer can be added between the input layer and the shared feature extraction hidden layer to mitigate the impact of outdated channel state information and sensing errors.

[0125] The flowchart of the multi-agent deep reinforcement learning algorithm proposed in this embodiment is shown in Table 1. The training process of V2V user pairs includes:

[0126] S51. Obtain current local observations O k (t) and input it into the DRN1 network to obtain the Q values ​​corresponding to all actions;

[0127] S52. Using the greedy strategy to select action A k (t), the agent uses action A k (t) Receive the reward and obtain the next local observation O k (t+1), update the termination flag "done". t+1 ;

[0128] S53. Move the next local observation O k (t+1) Input to DRN1 and DRN2 networks, store the trajectory { O k (t),A k (t), r k (t), O k (t+1), done t+1} to replay buffer D k ;

[0129] S54. From replay buffer D k Extract mini-batch trajectories of size B and sequence length L from the data, update all parameters of the DRN1 network through backpropagation of neural network gradients, and update the parameters of the DRN2 network every C steps.

[0130] S55. Repeat steps S51-S54 until the maximum number of training iterations is reached to complete the training.

[0131] Table 1

[0132] Algorithm 1: CIoV Spectrum Allocation Scheme Based on LSTM-D3QN 1: Initialize the DRN1 and DRN2 network parameters for each agent, training epochs E, experience replay buffer D, and discount factor. Training iteration count φ, exploration rate ε, target network update frequency C, batch size B, etc. 2: for Episode e = 1,2,...,E do 3: for Time slot t = 1,2,...,T do 4: for Agent k = 1,2,...,K do <![CDATA[5: Initialize the environment and obtain the initial state , set the termination flag done0 = False]]> 6: Set the currently observed variable Inputting the data into the online Q-network DRN1 yields the Q-values ​​for all actions. 7: Use the greedy strategy to select actions <![CDATA[8: The agent takes an action , obtains a reward , obtains the next state observation vector , updates the termination flag done t+1 > 9: Receive the next state observation vector Input it into the DRN1 and DRN2 networks. 10: Storage Track To replay buffer 11: From Extract mini-batch trajectories of size B and sequence length L. 12: Update all parameters of the DRN1 network through backpropagation of neural network gradients. 13: Update the target network DRN2 parameters every C steps. 14: End for 15: End for 16: End for

[0133] In some embodiments, a single cell is considered, where the BS is located at the center of the cell, M cellular links are formed between each CU and the BS, and K V2V links are established between each vehicle and its neighboring vehicles. The simulation setup follows the detailed description in the 3GPP urban scenario case study, including several specific parameters such as speed and channel model. The deep reinforcement learning simulation environment is implemented using the TensorFlow and Keras libraries in Python. The agent's DRN1 and DRN2 network structures consist of three fully connected layers (250, 180, and 120 neurons respectively), using the ReLU function as the activation function, and the Adam optimizer is selected to update the network parameters. The LSTM layer has 50 neurons, while the value and advantage layers in the duel layer each have 30 neurons. The exploration factor is slowly annealed from 1 to 0.02 during training.

[0134] Figure 2 The graph illustrates the cumulative reward for each training round as the number of training iterations increases, with the horizontal axis representing the number of training rounds and the vertical axis representing the average cumulative reward for each round. As can be seen from the graph, the cumulative reward for each round gradually increases with continued training. The proposed algorithm achieves the maximum cumulative reward after 1500 training rounds. Although the mobility of the vehicular network environment causes some fluctuations due to channel fading, the overall performance gradually converges.

[0135] Figure 3The delivery rate is shown under different V2V payload sizes. As can be seen from the figure, the delivery rate of V2V users gradually decreases with the increase of V2V payload. This is because the increase in V2V payload leads to longer V2V transmission duration and potentially higher V2V transmit power, which inevitably causes stronger interference to CU users, thus affecting the V2V payload delivery rate. Nevertheless, experimental results show that the proposed algorithm outperforms the comparative scheme under different V2V payload conditions. Furthermore, although the proposed algorithm uses a fixed payload size of 16×1060 bytes during the training phase, it still exhibits good adaptability to different payload sizes during the testing phase, which fully demonstrates the robustness of the algorithm to changes in V2V payload.

[0136] Figure 4 The impact of vehicle speed on V2V user payload delivery rate is demonstrated. The proposed algorithm is trained with a fixed payload of 16×1060 bytes within the speed range of [10, 15] m / s, and then generalized to the intervals of [15, 20] m / s, [20, 25] m / s, and [25, 30] m / s. As shown in the figure, the increased vehicle motion leads to more significant changes in CSI, and the increase in vehicle speed causes a performance degradation. However, the performance degradation of the proposed algorithm is always controlled within an acceptable range, which demonstrates the adaptability of the proposed algorithm to dynamic environments.

[0137] Figure 5 The impact of the maximum perception error probability on V2V user payload delivery rate is illustrated in the figure. As can be seen from the figure, the delivery rate gradually decreases with increasing maximum perception error probability. This phenomenon is because, although the proposed algorithm can predict the CU state using historical experience, the accuracy of the agent's estimation of the CU state decreases with increasing perception error, leading to a decline in the reliability of spectrum allocation decisions. It is noteworthy that even under ideal perception conditions where the maximum perception error probability is 0, the delivery rate of the proposed algorithm is still significantly better than that of the DQN and PPO algorithms. This is because the agents in the DQN and PPO algorithms struggle to obtain accurate channel state information, while the proposed algorithm, by effectively utilizing historical experience to learn the agent's state information, can better overcome the adverse effects of high-speed vehicle movement.

[0138] Figure 6This paper demonstrates how the V2V user payload delivery rate varies with the minimum transmission rate threshold of the CU under different algorithms. As the minimum transmission rate threshold of the CU increases, the system needs to prioritize the communication quality of the CU, which limits the spectrum allocation for V2V users. With limited spectrum resources, V2V users can only avoid interference by reducing transmit power or switching channels, but both of these adjustments lead to a decrease in transmission success rate. Experimental results show that, under the constraint of CU service quality, the proposed algorithm consistently outperforms the other two algorithms.

[0139] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "setting," "connection," "fixing," "rotation," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Unless otherwise explicitly limited, those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0140] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for spectrum allocation in cognitive vehicle-to-everything (V2X) networks under uncertain environments, characterized in that, Includes the following steps: S1. Construct a cognitive vehicle network system model that includes base stations, multiple cellular users, and multiple cognitive vehicles; wherein each cognitive vehicle communicates with its neighboring cognitive vehicles within its communication range, forming multiple V2V user pairs; S2. Based on the cognitive vehicle network system model, construct an imperfect CSI model, a spectrum sensing error model, and a communication model; S3. Set constraints and construct the objective optimization function based on the cognitive vehicle network system model, the imperfect CSI model, the spectrum sensing error model, the communication model, and the constraints. S4. Establish a Markov decision process for the objective optimization function; S5. Based on Markov decision process, a multi-agent reinforcement learning algorithm is used to solve for the optimal spectrum allocation strategy of cognitive vehicle.

2. The method for spectrum allocation in a cognitive vehicle-to-everything (V2X) network under uncertain conditions according to claim 1, characterized in that, In the cognitive vehicle network system model: Define the channel set. ={1,2,…,M}, where M represents the number of channels; the vehicle set is defined as... ={1,2,…,N}, where N represents the number of cognitive vehicles; define the V2V user pair set as... ={1,2,…,K}, where K represents the number of V2V user pairs; the number of cellular users is the same as the number of channels, and M channels are allocated to M cellular users for uplink communication with the base station; K V2V user pairs and M cellular users share M channels.

3. The method for spectrum allocation in a cognitive vehicle-to-everything (V2X) network under uncertain conditions according to claim 1, characterized in that, In the imperfect CSI model, the channel gain g of the k-th V2V user pair in slot t on channel m is defined. k,m (t) is represented as: In the formula, ω k,m (t) represents the path loss of the k-th V2V user pair in time slot t on channel m, ξ k,m (t) represents the shadowing fading of the k-th V2V user pair in time slot t on channel m, h k,m (t) represents the small-scale fading component of the k-th V2V user pair in time slot t on channel m; where... The fast fading time-varying characteristics within the feedback period τ are modeled using a first-order Gaussian-Markov random process, yielding the following results: in, Let represent the outdated small-scale fading components, which are independent and identically distributed and follow CN(0,1), such that It follows an exponential distribution with a mean of 1; e k,m (t)~CN(0,1) represents the independent complex Gaussian noise component in time slot t that is uncorrelated with the outdated small-scale fading component; λ k,m (t) represents the time-varying correlation coefficient between time slot t and the outdated small-scale fading component, which is constructed using the Jakes statistical model as follows: In the formula, J0(·) represents the zeroth-order Bessel function of the first kind; v k f represents the relative velocity of the k-th V2V user pair; m τ represents the carrier frequency; τ represents the feedback period; c represents the speed of light. The calculation of shadow fading depends on the relative position of the mobile cognitive vehicle. The update formula for shadow fading for any V2V user on any channel in any time slot is as follows: In the formula, ξ new and ξ old These represent the shadow fading values ​​before and after the update, respectively; △d represents the distance traveled by the cognitive vehicle; d rel Let Z represent the correlation distance of shadow fading; Z represents a Gaussian random variable; where the cognitive vehicle position is calculated as follows: In the formula, x n (t), y n (t) represents the horizontal and vertical coordinates of the nth cognitive vehicle in time slot t, respectively. n Let θ represent the speed of the nth cognitive vehicle. n (t) represents the direction of motion of the nth cognitive vehicle in time slot t, and △t represents the time step.

4. The method for spectrum allocation in a cognitive vehicle-to-everything (V2X) network under uncertain conditions according to claim 1, characterized in that, In the spectrum sensing error model, considering imperfect spectrum sensing, the probability ε of the sensing error of the k-th V2V user pair in time slot t on channel m is defined. k,m (t) ~U(0, ε max Then we have: In the formula, Pr{s m (t)= } represents the actual occupancy status of channel m in time slot t. The probability, This represents the sensing state of time slot m in time slot t, s m (t) represents the true state of channel m in time slot t; ε k,m (t)~U(0, ε max ) represents ε k,m (t) is uniformly distributed in [0, ε max ] within; ε max This represents the maximum perceptual error probability.

5. The method for spectrum allocation in a cognitive vehicle-to-everything (V2X) network under uncertain conditions according to claim 1, characterized in that, In the communication model, the signal-to-interference-plus-noise ratio γ is defined when the k-th V2V user selects channel m. k,m (t) is: In the formula, a k,m (t) represents the channel allocation indication for the k-th V2V user in time slot t to channel m, p k,m (t) represents the transmit power of the k-th V2V user pair in time slot t on channel m, g k,m (t) represents the channel gain of the k-th V2V user pair in time slot t on channel m, σ 2 Indicates noise power; I m,k (t) represents the interference generated by cellular users on channel m to the kth V2V user pair; when channel m is idle, I m,k (t)=0, when channel m is busy. s m (t) represents the actual state of time slot m in time slot t; p m (t) represents the transmit power of the cellular user on channel m; g m,k (t) represents the interference channel gain from the cellular user to the receiver of the k-th V2V user pair; This represents the interference channel gain from the transmitter of the k'-th V2V user pair to the receiver of the k-th V2V user pair on channel m. Represents a set of V2V user pairs; Define the interference I generated by the k-th V2V user accessing channel m to the base station. k,m (t) is: In the formula, g k,m,B (t) represents the interference channel gain from transmitter to base station for the k-th V2V user pair; Define the transmission rate R of the k-th V2V user pair on channel m. k,m (t) is: In the formula, W represents the channel bandwidth, b k,m (t) represents the reliability performance index of the k-th V2V user pair transmitting data on channel m. in, In the formula, γ k,min (t) represents the SINR threshold for communication between the k-th V2V user.

6. The method for spectrum allocation in a cognitive vehicle-to-everything (V2X) network under uncertain conditions according to claim 1, characterized in that, The objective optimization function is expressed as: In the formula, This represents the set of binary channel allocation variables for V2V user pairs. Let T represent the set of power allocation variables for V2V user pairs, Pr() represent the probability function used to represent the success probability of the task within the parentheses, K represent the number of V2V user pairs, and T represent the number of V2V user pairs. k This represents the data transmission time constraint for the k-th V2V user pair, where M represents the number of channels, and a k,m (t) represents the channel allocation indication for the k-th V2V user in time slot t to channel m, R k,m (t) represents the transmission rate of the k-th V2V user pair on channel m, B k γ represents the size of the security data, in bits, periodically generated by the k-th V2V user, where Δt represents the time step and γ represents the value of the security data. k,m γ(t) represents the signal-to-interference-plus-noise ratio (SIR) of the k-th V2V user when selecting channel m, where γ k,min (t) represents the SINR threshold for communication between the k-th V2V user and p. k,m p(t) represents the transmit power of the k-th V2V user in time slot t on channel m, where p max g represents the maximum transmit power of a V2V user pair. k,m,B (t) represents the interference channel gain from transmitter to base station for the k-th V2V user pair. R represents the interference threshold of channel m in time slot t. k,m (t) represents the transmission rate of the k-th V2V user pair on channel m, C m (t) represents the minimum transmission rate required by cellular users on channel m, N n (t) represents the distance threshold l between the nth cognitive vehicle and the other vehicle. th An orderly collection of vehicles within the area. Represents a set of vehicles, l n (t) represents the position coordinates of the nth cognitive vehicle in time slot t, l th N represents the distance threshold for neighbor selection. neighbor This represents the maximum number of neighboring vehicles allowed for each cognitive vehicle.

7. The method for spectrum allocation in a cognitive vehicle-to-everything (V2X) network under uncertain conditions according to claim 1, characterized in that, Using each V2V user pair as an agent, the Markov decision process for the objective optimization function includes: Define the local observation O of the k-th V2V user pair in time slot t. k (t) is: (t) represents the remaining V2V payload of the k-th V2V user pair. (t) represents the remaining time budget for data transmission by the k-th V2V user, φ represents the number of training iterations, and ε represents the instantaneous exploration rate. (t) represents the sensing state of channel m in time slot t, I k,m (t) represents the interference caused to the base station by the k-th V2V user accessing channel m. G represents the channel information for time slot t. k,m (t) represents the channel gain of the k-th V2V user pair on channel m, g k,m,B (t) represents the transmitter-to-base station interference channel gain for the k-th V2V user pair, g m,k (t) represents the interference channel gain from the cellular user to the receiver of the k-th V2V user pair. Represents the channel set; Define the action A of the k-th V2V user pair in time slot t. k (t) is: a k,m (t) represents the channel allocation indication for channel m by the k-th V2V user in time slot t, p k,m (t) represents the transmit power of the k-th V2V user in time slot t on channel m; Define the instantaneous reward function for the k-th V2V user pair. for: Where, r k (t) represents the reward for the k-th V2V user pair in time slot t, λ represents the tradeoff coefficient, and K represents the number of V2V user pairs.

8. The method for spectrum allocation in a cognitive vehicle-to-everything (V2X) network under uncertain conditions according to claim 7, characterized in that, The reward r for the k-th V2V user in time slot t k (t) includes: If the k-th V2V user pair fails to complete packet transmission in advance, and the channel it accesses is not occupied by cellular users, then the reward r in time slot t is... k (t) is represented as: If the k-th V2V user fails to complete packet delivery in advance and its access channel is occupied by the CU, then the reward r in time slot t will be... k (t) is represented as: In the formula, L k (t) represents the penalty term. Indicates the interference penalty factor; I represents the interference threshold of channel m in time slot t. k,m (t) represents the interference that the kth V2V user causes to the base station by accessing channel m; If the k-th V2V user completes packet transmission early and does not subsequently access the channel for packet transmission, then the reward r for each remaining time slot t after packet transmission is completed is... k (t) is represented as: r k (t)= β f In the formula, represents β f Constant reward value; If the k-th V2V user pair has a data transmission time constraint T k If no data packet is transmitted within time slot t, then the reward r is... k (t) is represented as: r k (t)=0.

9. The method for spectrum allocation in a cognitive vehicle-to-everything (V2X) network under uncertain conditions according to claim 1, characterized in that, Step S5 employs a multi-agent reinforcement learning algorithm based on DQN, combining Double DQN and Dueling DQN. The training process for the k-th agent includes: S51. Obtain current local observations O k (t) and input it into the DRN1 network to obtain the Q values ​​corresponding to all actions; where the Q value function is expressed as: In the formula, Indicates local observation O k (t) Select action A k The Q value of (t), γ t Discount factor The power of t, T k This represents the total number of time slots accumulated for the rewards. Represents the mathematical expectation operator. Let G represent the instantaneous reward function for the k-th V2V user pair. t Indicates cumulative rewards; S52. Using the greedy strategy to select action A k (t), the agent uses action A k (t) Receive the reward and obtain the next local observation O k (t+1), update the termination flag "done". t+1 ; S53. Move the next local observation O k (t+1) Input to DRN1 and DRN2 networks, store the trajectory { O k (t),A k (t), r k (t), O k (t+1), done t+1 } to replay buffer D k ; S54. From replay buffer D k Extract mini-batch trajectories of size B and sequence length L from the data, update all parameters of the DRN1 network through backpropagation of neural network gradients, and update the parameters of the DRN2 network every C steps. S55. Repeat steps S51-S54 until the maximum number of training iterations is reached to complete the training.

Images