Dynamic spectrum access algorithm for evaluating spectrum stability in cognitive Internet of Vehicles

Abstract

The invention requests to protect a dynamic spectrum access algorithm for evaluating spectrum stability in cognitive Internet of Vehicles. The invention provides a dynamic spectrum access algorithm for evaluating spectrum stability. The method comprises the following steps: firstly, based on a signal-to-noise ratio, received signal strength and bandwidth parameters, predicting values of the parameters at multiple moments in a future period by utilizing a long-short-term memory neural network, calculating a change rate of each parameter in one period, and taking a result as an evaluation index of spectrum stability; secondly, clustering the change rate vectors by using a K-Means algorithm, and constructing a stability evaluation model; thirdly, reconstructing a state space and a reward function according to a stability evaluation result, and proposing a dynamic spectrum access algorithm based on reinforcement learning; finally, an experiment result shows that the algorithm can meet the stability requirements of different vehicle terminal services, the utilization rate of spectrum resources is improved, and meanwhile the collision probability in the spectrum access process is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of mobile communications, and in particular to a dynamic spectrum access algorithm in a cognitive vehicle-connected network. Background Art

[0002] With the popularization of a large number of Internet of Vehicles applications, the demand for spectrum resources by vehicle terminals has increased rapidly. As an emerging network environment, cognitive Internet of Vehicles combined with cognitive radio technology has shown great potential. However, in this network, the stability of spectrum resources is affected by the environment and motion state of the vehicle terminal, and the channel parameters change quickly and in real time, so spectrum stability is difficult to evaluate. Spectra with poor access stability will seriously affect the efficiency of vehicle terminal transmission services, and thus affect the overall performance of the network. In addition, high concurrency of vehicle terminal access requests makes the competition for spectrum resources more intense, which will increase the collision probability of the access process. Therefore, in cognitive Internet of Vehicles, how to evaluate the stability of spectrum resources, achieve effective use of spectrums with different stability, improve spectrum utilization and reduce the probability of access collision has become a hot issue in this field.

[0003] At present, research on cognitive Internet of Vehicles environments mainly focuses on four aspects: wireless access, spectrum sensing, routing protocols, and security issues. In the field of wireless spectrum access, few literatures have conducted quantitative evaluations of spectrum stability in order to improve the performance of spectrum access. In the literature [Chuang MC. Cooperation-Assisted Spectrum Handover Mechanism in Vehicular Ad Hoc Networks [A]. 2021 Wireless Telecommunications Symposium (WTS) [C]. CA, USA: IEEE, 2021: 1–7.], the author abstracted the number of available spectrum resources into a numerical value, calculated its variance after three rounds of spectrum scanning, and used the calculated result as the spectrum stability index. This paper uses the variance of a single parameter to measure stability and gives the concept of stability index. Reference [N.Cheng, N.Zhang, N.Lu, X.Shen, JWMark, and F.Liu, “Opportunistic spectrum access for CR-VANETs: A game-theoretic approach,” IEEE Trans.Veh.Technol., vol.63, no.1, pp.237–251, Jan.2014.] analyzes the impact of the primary user’s channel usage behavior and vehicle mobility on channel availability. On this basis, the congestion game model is used to model the spectrum access process of vehicle terminals. Reference [Niyato D, Hossain E, Wang P. Optimal Channel Access Management with QoS Support for Cognitive Vehicular Networks [J]. IEEE Transactions on Mobile Computing, 2011, 10 (4): 573–591.] analyzes the impact of spectrum status on access results from multiple perspectives, including abnormal spectrum sensing results, packet transmission status, and weighted scheduling priority. This paper only measures spectrum availability by the spectrum status at a single moment.The literature [Xiang P, Shan H, Wang M,. Multi-Agent RL Enables Decentralized Spectrum Access in Vehicular Networks[J]. IEEE Transactions on Vehicular Technology, 2021, 70(10): 10750–10762.] considers the different delay and throughput requirements of vehicle-to-infrastructure users and vehicle-to-vehicle users for spectrum, and sets different reward functions to meet the user's service quality and maximize network performance.

[0004] The above access algorithms analyze the impact of spectrum status or availability on spectrum access from different perspectives. Different models are used to model the dynamic spectrum access process in order to maximize spectrum utilization and minimize the probability of spectrum collision. When evaluating the spectrum status, most of the above literature does not consider the long-term performance of multiple spectrum parameters and their impact on access. It also ignores the different requirements for spectrum stability for services transmitted by different vehicle terminals. In order to solve the above problems, how to comprehensively evaluate spectrum stability and achieve rational use of spectrum with different stability has become the core issue of this study. Therefore, this paper proposes a spectrum stability evaluation model and designs a dynamic spectrum access algorithm based on reinforcement learning. Summary of the invention

[0005] The present invention aims to solve the above problems of the prior art. A dynamic spectrum access algorithm for cognitive vehicle networking is proposed. The technical solution of the present invention is as follows:

[0006] An access algorithm considering spectrum stability in cognitive vehicle networking includes the following steps:

[0007] Firstly, based on the signal-to-noise ratio, received signal strength and bandwidth parameters, a stability parameter prediction model is established using a long short-term memory neural network. The values ​​of the parameters at multiple moments in the future are predicted, and the rate of change of the parameters within a prediction period is used as an evaluation index for spectrum stability. Secondly, the K-Means algorithm is used to cluster the predicted value change rate vector, realize stability evaluation, and construct a stability evaluation model. Thirdly, combining the spectrum stability index and channel throughput, a dynamic spectrum access algorithm based on reinforcement learning is proposed.

[0008] Furthermore, the construction of the stability parameter prediction model specifically includes: firstly, the historical data and real-time data of bandwidth, signal-to-noise ratio and received signal strength are used as inputs of the long short-term memory neural network. Through the step-by-step prediction algorithm, the predicted values ​​of the above parameters at multiple moments are obtained, and a stability parameter prediction model is constructed in the above manner.

[0009] Furthermore, the process of predicting the stability parameters described is mainly to analyze the monitored historical spectrum data and the collected real-time data, obtain the predicted data of the spectrum at multiple moments in the future, and calculate the rate of change within a cycle. It mainly includes the following three steps:

[0010] 1) The three features that serve as the input vector and output vector of the long short-term memory neural network together constitute the initial input matrix, where the matrix consists of real-time data at time t and historical data from time t-1 to time tT-1.

[0011] 2) Secondly, transform the matrix X t After inputting into the prediction model, the prediction data y at time t+1 is obtained t+1 The historical data from time t-1 to time tT, the real-time data at time t. and the predicted data at time t+1 form a new input matrix. After the model prediction, the predicted data y at time t+2 is obtained. t+2 .

[0012] 3) Repeat the process in the previous step to get the final prediction value moment Y T =[y t+1 ,y t+2 ,....y t+T ]The matrix consists of the predicted data from time t+1 to time t+T.

[0013] Furthermore, the signal-to-noise ratio, bandwidth, and received signal strength specifically include:

[0014] Signal-to-Noise Ratio:

[0015]

[0016] in, It indicates the transmission power of channel j detected by vehicle terminal i at time t, represents the channel gain of channel j accessed by vehicle terminal i, μ t represents background noise with mean 0 and standard deviation σ. Represents the distance between vehicle terminal i and the base station to which channel j belongs at time t.

[0017] Received signal strength:

[0018]

[0019] in, It indicates the transmission power of channel j detected by vehicle terminal i at time t, μ represents the distance between vehicle terminal i and the base station to which channel j belongs at time t. trepresents background noise with mean 0 and standard deviation σ.

[0020] bandwidth:

[0021]

[0022] in represents the highest frequency of channel j accessed by vehicle terminal i at time t, Indicates the lowest frequency of channel j accessed by vehicle terminal i at time t.

[0023] Furthermore, the K-Means algorithm is used in the stability evaluation model to complete the stability evaluation. The K value assumed in this paper is 3, that is, there are three clusters classified by stability in the end, and the change rate vectors of three channels are randomly selected from the channel set M as the initial centroid. According to the stability parameter prediction model mentioned above, the predicted value vectors of each parameter of each channel at time t+T are obtained, and the change rate vectors of each channel from time t to time t+T are obtained by calculation. As the input of the stability assessment model, three clusters distinguished by stability are obtained, and the stability assessment is completed.

[0024] Furthermore, the calculated signal-to-noise ratio change rate, received signal strength change rate, and bandwidth change rate specifically include:

[0025] Signal-to-noise ratio change rate:

[0026]

[0027] in, represents the real-time signal-to-noise ratio of channel j measured by vehicle terminal i at time t, It represents the predicted value of the signal-to-noise ratio of channel j at time t+T output by LSTM. The calculation of the received signal strength change rate and bandwidth change rate is similar.

[0028] Further use of the K-Means algorithm to implement stability assessment is divided into the following steps:

[0029] 1) Normalization of predicted value change rate: Since the channel signal-to-noise ratio change rate, received signal strength change rate, and bandwidth change rate vary in different intervals, this paper uses the Min-Max normalization method to normalize the above parameters. The normalization calculation of the signal-to-noise ratio change rate is shown in the following formula:

[0030]

[0031] in represents the rate of change of the signal-to-noise ratio of channel j within the time step T, It represents the minimum value of the signal-to-noise ratio change rate in the channel set M within the time step T. represents the maximum value of the signal-to-noise ratio change rate in the channel set M within the time step T, Indicates the normalized result of the signal-to-noise ratio change rate. The bandwidth change rate and the received signal strength change rate are normalized in the same way.

[0032] 2) Stability clustering: Calculate the Euclidean distance between the rate of change vector of each channel and each centroid, and assign the rate of change vector of the channel to the cluster with the smallest centroid distance. Randomly select three channels from the channel set M, and use their rate of change vectors as the initial centroids. Assume that channel m is one of the initial centroids. The Euclidean distance between channel j and centroid m is dis m,j The calculation formula is as follows:

[0033]

[0034] 3) Update the centroid: For each cluster, calculate the average value of the rate of change vectors of all its channels and use the average value as the new centroid. If channel j+1, channel j+2, and channel j+3 are assigned to the same cluster, the new centroid is expressed as:

[0035]

[0036] 4) Complete stability assessment: After repeated iterations, until the centroid no longer changes, the final three clusters and centroids are obtained. Assume that the final centroid is centroid i i∈{1,2,3}, its norm is:

[0037] ω 1 ,ω 2 ,ω 3 Sorting from small to large, the larger the value, the higher the stability of the cluster. 1 <ω 2 <ω 3 , then the centroid 1 、centroid 2 、centroid 3 The stability of the clusters is low, medium, and high respectively. If channel i belongs to the centroid 1 The cluster in which it is located has a stability index

[0038] Furthermore, in the network scenario of this paper, the service requests of vehicle terminals are divided into two types: high stability type services and low stability type services. For vehicle terminals, their goal is to access a channel that meets their stability and throughput requirements while competing with other vehicle terminals for access channels. Therefore, this paper proposes an access algorithm based on reinforcement learning, which redesigns the state space and reward function according to the service type requested by the terminal and the spectrum stability evaluation results.

[0039] Furthermore, when constructing the state space, this paper aims to enable the vehicle terminal to interact effectively with channels of different stabilities, and then obtain effective samples for reinforcement learning training. Therefore, this chapter selects channel state, stability index and throughput to construct the state space of reinforcement learning. The vehicle terminal is prompted to select the most suitable channel for data transmission based on the requirements of the channel stability and throughput of its own transmitted business through continuous iterative training, so as to achieve the rational use of spectrum resources with different stabilities. In summary, the state space of this paper is: The state space is composed of a channel state vector, a stability index vector and a throughput vector from vehicle terminal 1 to vehicle terminal i.

[0040] Furthermore, the channel occupancy vector Represents the status of channel 1 to channel m detected by vehicle terminal i at time t. It means that at time t, vehicle terminal i detects that channel j is occupied. It means that at time t, vehicle terminal i detects that channel j is idle.

[0041] Furthermore, the stability index vector Represents the stability index of channel 1 to channel m evaluated by the system side at time t.

[0042] Furthermore, the throughput vector It represents the throughput that can be obtained when vehicle terminal i accesses channel 1 to channel m at time t, where the throughput calculation formula is as follows:

[0043]

[0044] Furthermore, is the action space of this paper, including the channel selection behavior from vehicle terminal 1 to vehicle terminal n. Indicates that at time t, vehicle terminal i does not choose to access any channel. It indicates that at time t, vehicle terminal i chooses to access channel j.

[0045] Furthermore, considering the different requirements of spectrum stability and throughput for high-stability services and low-stability services, this paper uses the trade-off coefficient λ to distinguish the reward function of the vehicle terminal transmitting these two services. Its expression is as follows:

[0046]

[0047] Among them, if the service transmitted by vehicle terminal i at time t is a high-stability service, then λ>0.5; if the service transmitted is a low-stability service, then λ<0.5. It represents the normalized stability index of channel j accessed by vehicle terminal i at time t. The normalization method is the same as the previous one, using the Min-Max method. It represents the normalized throughput of channel j accessed by vehicle terminal i at time t.

[0048] Furthermore, this paper uses the experience playback protocol to replay the experience tuples used by the vehicle terminal at each moment. Stored in the experience replay pool. At the same time, the vehicle terminal uses the mini-batch algorithm to randomly extract L experience tuples from the experience replay pool each time, and input them into the neural network composed of the gated recurrent unit (GRU) and the Dueling Network for stochastic gradient descent to update the neural network parameters θ t , which enables the access algorithm in this paper to converge faster.

[0049] Furthermore, this paper uses GRU as the first hidden layer because channel parameters are usually time series data and the channel state changes continuously in time. GRU can memorize and utilize historical information through its internal loop structure, thereby effectively tracking the changing trend of channel parameters. Therefore, GRU is introduced to solve the problem of rapid changes in channel parameters in cognitive Internet of Vehicles.

[0050] Furthermore, this paper uses Dueling Network as the second hidden layer. Dueling Network divides the Q value function into the state value function With advantage function In this way, on the one hand, the problem of overestimation of Q value is solved and the convergence speed is accelerated. On the other hand, the value of each channel selection behavior in different environments is correctly estimated. For example, the Q value of a low-stability channel is low in an environment where most high-stability channels are idle. When all high-stability channels are occupied and there are fewer idle channels, the Q value should be high to encourage vehicle terminals to access. Dueling Network enables vehicle terminals to dynamically adjust their access strategies according to different environments, so that channels with different stabilities can be reasonably utilized.

[0051] Furthermore, the access algorithm based on reinforcement learning proposed in this paper continuously adjusts the loss function L t (θ t ) to perform gradient descent to update the neural network parameters, so that the estimated value of the Q value function is constantly approaching the actual value, and finally an optimal θ is obtained. t , so that the access strategy of each vehicle terminal converges to the optimal. The loss function L t (θ t ) is expressed as:

[0052]

[0053] in This article will The value is taken as the timing error (Temporal Difference Error, TDR) By continuously learning from the error, the Q value is updated. The Q value is updated as follows:

[0054]

[0055] Furthermore, this paper uses the ε strategy in channel selection, that is, each time the behavior with the largest Q value is selected with a probability of 1-ε. The strategy is defined as:

[0056]

[0057] The advantages and beneficial effects of the present invention are as follows:

[0058] 1. Based on the historical and real-time data of signal-to-noise ratio, received signal strength and bandwidth parameters, the traditional step-by-step prediction algorithm is improved, and LSTM is used to predict channel parameters at multiple times within a cycle, and the rate of change within the prediction cycle is used as a spectrum stability evaluation indicator. The K-Means algorithm is used to achieve stability clustering of the rate of change vector and build a spectrum stability evaluation model.

[0059] 2. According to the stability evaluation results and the stability requirements of the services transmitted by the vehicle terminals, the state space, action space and reward function of reinforcement learning were redesigned, and an access algorithm based on reinforcement learning was proposed to achieve rational utilization of spectrums with different stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is the algorithm flow chart;

[0061] Figure 2 Neural network structure for prediction model;

[0062] Figure 3 Predict results for the bandwidth training set;

[0063] Figure 4 Predict results for the bandwidth test set;

[0064] Figure 5 is the loss function curve of the training set;

[0065] Figure 6 is the test set accuracy curve;

[0066] Figure 7 It is the Q value update process;

[0067] Figure 8 This is a simulation diagram of the city cognitive Internet of Vehicles;

[0068] Fig. 9 This is the comparison result of spectrum utilization;

[0069] Fig.10 is the collision probability comparison result;

[0070] Fig.11 is the cumulative stability index;

[0071] Fig.12 is the cumulative throughput;

[0072] Fig.13 Convergence speed comparison under different numbers of vehicles and channels; DETAILED DESCRIPTION

[0073] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and in detail describe the technical solutions in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0074] The technical solution of the present invention is as follows:

[0075] Firstly, based on the signal-to-noise ratio, received signal strength and bandwidth parameters, the long short-term memory neural network is used to predict the values ​​of the parameters at multiple moments in the future cycle, and the change rate of each parameter in one cycle is calculated, and the result is used as the evaluation index of spectrum stability. Secondly, the K-Means algorithm is used to cluster the change rate vector and construct a stability evaluation model. Thirdly, the state space and reward function are reconstructed according to the stability evaluation results, and a dynamic spectrum access algorithm based on reinforcement learning is proposed. Finally, the experimental results show that the proposed algorithm can meet the stability requirements of different vehicle terminal services, improve the utilization of spectrum resources, and reduce the collision probability during spectrum access.

[0076] The switching algorithm considering network security proposed in the present invention comprises the following steps:

[0077] Step 1: Use the historical data and real-time data of bandwidth, signal-to-noise ratio, and received signal strength as the input of LSTM. Through the step-by-step prediction algorithm, the predicted values ​​of the above parameters at multiple moments in the future prediction cycle are obtained. The Drop-out algorithm is introduced to balance the impact of historical data and real-time data on the prediction results.

[0078] Step 2: The above parameters, the predicted value change rate from the start of the prediction to the end of a prediction period, are used as the input of the K-Means algorithm. After normalizing the predicted value change rate, allocating samples, and updating the centroid, stability clustering is achieved. A stability assessment model is constructed to achieve stability assessment.

[0079] Step 3: Based on the spectrum stability evaluation results, the state space, action space and reward function of reinforcement learning are redesigned to enable vehicle terminals that transmit services with different stability requirements to access channels that meet their needs. The Dueling-Network and GRU neural network layers are introduced to solve the problem of overestimation and utilization of historical channel information.

[0080] In order to evaluate the performance of the algorithm proposed in this paper, a network simulation scenario of a 3000m long and 30m wide one-way three-lane network in the core area of ​​the city was simulated using Matlab simulation tools. In this simulation scenario, two 5G macro base stations (5GM) and several vehicles are deployed. Assuming that the coverage range of each base station is 40m and the speed of the vehicle is 10-15m / s, in order to simplify the simulation settings, this paper sets the number of vehicles and channels to 10 and 5 respectively.

[0081] In the simulation experiment, the dynamic spectrum access algorithm for evaluating spectrum stability proposed in this paper is compared with the access algorithm based on Q-learning [Chen L, Zhao Q, Fu K, et al. Multi-user reinforcement learning based multi-reward for spectrum access in cognitive vehicular networks[J]. Telecommunication Systems, 2023, 83(1): 51-65.], the access algorithm based on Deep Q-learning [Chen L, Wang Z, Zhao X, et al. A dynamic spectrum access algorithm based on deep reinforcement learning with novel multi-vehicle reward functions in cognitive vehicular networks[J]. Telecommunication Systems, 2024: 1-25.] and the time-slot Aloha algorithm [K. Kar, S. Sarkar, and L. Tassiulas, Achieving proportional fairness using local information in aloha networks,[J]. IEEE Trans. Autom. Control, 2004, 49(10):1858–1863].

[0082] Fig. 9The spectrum utilization curves of the four algorithms during 100,000 iterations of reinforcement learning of the vehicle terminal are shown. The performance is calculated every 5,000 times. The vehicle terminal using the slotted Aloha protocol randomly selects channels for access. Therefore, as the number of iterations increases, its utilization rate will stabilize around 0.4. The Q-learning algorithm, Deep Q-learning algorithm and the algorithm proposed in this article are all based on reinforcement learning. In the initial state of this type of algorithm, the neural network parameters are random, which will result in a low initial utilization rate. By continuously learning from the environment and iteratively updating the neural network parameters, the channel access strategy will be continuously optimized. Therefore, as the number of iterations increases, the utilization rate of vehicle terminals using these three types of algorithms will continue to increase. Compared with the Q-learning algorithm and the Deep Q-learning algorithm, the algorithm proposed in this article has the most obvious improvement in spectrum utilization. After 20 rounds of iterations, the utilization rate will eventually converge to about 0.9. Because the algorithm in this article introduces the GRU layer and the experience replay protocol, the vehicle terminal can maximize the use of the access experience accumulated during the iteration process. From Fig. 9 It can be seen that the utilization rate of the Deep Q-learning algorithm is higher than that of the Q-learning algorithm in each stage. Since the Q-learning algorithm updates the Q value by maintaining the Q table, as the number of iterations increases, the environmental information becomes more complex, and the overhead of maintaining the Q table also increases exponentially, so its performance is lower than that of the Deep Q-learning algorithm.

[0083] Fig.10 The collision probability curves of the four algorithms are shown during 100,000 iterations of reinforcement learning. The performance results are calculated every 5,000 times. Fig.10It can be seen from the figure that except for the time slot Aloha algorithm, whose collision probability always fluctuates around 0.6, the collision probability of the other algorithms is relatively high in the initial stage of training, but as the number of iterations increases, the collision probability will gradually decrease and converge. The reason is that in the reinforcement learning algorithms such as Q-learning algorithm, Deep Q-learning algorithm and the algorithm proposed in this paper, the exploration probability of each vehicle terminal in the initial stage of training is to select a channel for access at any time, which is particularly easy to cause multiple terminals to access the same channel. As the number of iterations increases, it gradually decreases to 0, and the access strategy of each vehicle terminal gradually reaches the optimal. The terminal will most likely choose the channel with the largest Q value for access each time, so the collision probability will gradually decrease. The collision probability of the algorithm proposed in this paper decreases faster than that of the Q-learning algorithm and the Deep Q-learning algorithm, and basically converges to 0 when the number of iterations reaches 12. The reason is that the algorithm in this paper differentiates the reward function according to the service type of the vehicle terminal. In this way, as the number of iterations increases, the vehicle terminal transmitting high-stability services is more inclined to access the idle high-stability channel. Vehicle terminals transmitting low-stability services tend to access medium- and low-stability channels, so the collision probability converges to 0 faster than other algorithms.

[0084] Fig.11 The cumulative stability index curve is shown when multiple vehicle terminals select different algorithms to transmit high-stability services. Fig.11 It can be seen that as the number of iterations increases, the cumulative stability indexes of the three algorithms are gradually increasing. This shows that during the training process, when the vehicle terminal is transmitting high-stability services, it is more and more inclined to access a channel with high stability. The cumulative stability index obtained by the algorithm proposed in this paper is much higher than that of the other two algorithms. The reason is that this paper makes the trade-off coefficient λ>0.5 in formula (10), so that the vehicle terminal that accesses the high-stability channel can obtain more rewards.

[0085] Fig.12 The curves of cumulative throughput changes when multiple vehicle terminals select different algorithms to transmit low-stability services are shown. As the number of iterations increases, the cumulative throughput of the three algorithms gradually increases, and the algorithm proposed in this article makes the throughput obtained by the vehicle terminal much higher than the other two algorithms. The reason is that this article uses the GRU layer and the experience replay protocol to maximize the use of dynamic spectrum access experience during the training process. As the number of iterations increases, the vehicle terminals that are transmitting low-stability services no longer compete for high-stability channels, but access medium- and low-stability channels. Although these channels have low stability, they can guarantee basic throughput. This not only improves the utilization of the channel, but also reduces the probability of collision while ensuring the throughput requirements.

[0086] Fig.13 The proposed algorithm is compared with the two contrast algorithms in terms of convergence speed under different numbers of vehicle terminals and channels. When the number of vehicle terminals is 3, 5, 7, and 9, the number of channels is 2, 3, 4, and 5. Fig.13 It can be observed that as the number of vehicle terminals increases, the number of iterations required for the algorithm to converge also increases accordingly. The algorithm proposed in this paper correctly estimates the value of each channel selection behavior in different environments by introducing Dueling-Network, solves the problem of overestimation of Q value, and accelerates the convergence speed.

[0087] In summary, this paper is more suitable for cognitive Internet of Vehicles with a large number of vehicles, and its convergence speed in small-scale networks does not lag behind the comparison algorithms too much.

[0088] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0089] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0090] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0091] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. An access algorithm for evaluating spectrum stability in cognitive vehicle networks, characterized in that: The following steps are involved:

101. Based on the signal-to-noise ratio, received signal strength and bandwidth parameters, the long short-term memory neural network is used to predict the values ​​of the parameters at multiple times in the future cycle, and the change rate of each parameter in one cycle is calculated, and the result is used as an evaluation indicator of spectrum stability; 102. According to the data results obtained in step 101, the change rate vectors are clustered using the K-Means algorithm to construct a stability assessment model; 103. According to the stability evaluation result obtained in step 102, the state space and reward function are reconstructed, and a dynamic spectrum access algorithm based on reinforcement learning is proposed; 104. According to the access algorithm proposed in step 103, the stability requirements of different vehicle terminal services can be met, the utilization rate of spectrum resources can be improved, and the collision probability during spectrum access can be reduced.

2. An access algorithm for evaluating spectrum stability in a cognitive vehicle network as described in claim 1, characterized in that: The construction of the stability assessment model specifically includes:

201. Use the historical data and real-time data of bandwidth, signal-to-noise ratio and received signal strength as inputs of the long short-term memory neural network; 202. According to the input of step 201, the predicted values ​​of the above parameters at multiple moments in a future prediction period are obtained through a step-by-step prediction algorithm; 203. Using the predicted value change rate calculated in step 202 as an evaluation index of spectrum stability; 204. Based on the stability evaluation result obtained in step 203, the state space, action space and reward function of reinforcement learning are redesigned so that vehicle terminals transmitting services with different stability requirements can access channels that meet their requirements.

3. An access algorithm for evaluating spectrum stability in a cognitive vehicle network according to claim 2, characterized in that: Analyze the historical spectrum data and real-time data collected, predict the spectrum status at multiple moments in the future, calculate the rate of change within a cycle, and use it as a stability evaluation indicator:

301. Will Together, the three features of the input vector and output vector of the long short-term memory neural network constitute the initial input matrix X t , where the matrix consists of real-time data at time t and historical data from time t-1 to time tT-1; 302. According to the input matrix X in step 301 t , get the predicted data y at time t+1 t+1 The historical data from time t-1 to time tT, the real-time data at time t, and the predicted data at time t+1 form a new input matrix. The predicted data y at time t+2 is obtained through the prediction of the model. t+2 ; 303. By repeating step 302, the final prediction value matrix Y is obtained. T =[y t+1 ,y t+2 ,....y t+T ]The matrix consists of the predicted data from time t+1 to time t+T.

4. An access algorithm for evaluating spectrum stability in a cognitive vehicle network as described in claim 2, characterized in that: The definition of the three parameters based on the collected data specifically includes: Signal-to-Noise Ratio: in, It indicates the transmission power of channel j detected by vehicle terminal i at time t, represents the channel gain of channel j accessed by vehicle terminal i, μ t represents background noise with mean 0 and standard deviation σ. Represents the distance between vehicle terminal i and the base station to which channel j belongs at time t. Received signal strength: in, It indicates the transmission power of channel j detected by vehicle terminal i at time t, μ represents the distance between vehicle terminal i and the base station to which channel j belongs at time t. t represents background noise with mean 0 and standard deviation σ. bandwidth: in represents the highest frequency of channel j accessed by vehicle terminal i at time t, Indicates the lowest frequency of channel j accessed by vehicle terminal i at time t.

5. An access algorithm for evaluating spectrum stability in a cognitive vehicle network according to claim 4, characterized in that: The calculation of the signal-to-noise ratio change rate, the bandwidth change rate, and the received signal strength change rate specifically includes: Signal-to-noise ratio change rate: in, represents the real-time signal-to-noise ratio of channel j measured by vehicle terminal i at time t, Represents the predicted signal-to-noise ratio of channel j at time t+T output by the long short-term memory neural network. Bandwidth change rate: in, represents the real-time bandwidth of channel j measured by vehicle terminal i at time t, Represents the bandwidth prediction value of channel j at time t+T output by the long short-term memory neural network. Received signal strength change rate: in, represents the real-time received signal strength of channel j measured by vehicle terminal i at time t, Represents the predicted value of the received signal strength of channel j at time t+T output by the long short-term memory neural network.

501. According to the access algorithm for evaluating spectrum stability in a cognitive Internet of Vehicles described in claim 4, the feature is that the K-Means algorithm is used in the stability evaluation model to complete the stability evaluation. The K value assumed in this article is 3, that is, there are ultimately three clusters classified by stability, and the rate of change vectors of three channels are randomly selected from the channel set M as the initial centroid. According to the stability parameter prediction model above, the predicted value vectors of each parameter of each channel at time t+T are obtained, and the rate of change vectors of each channel from time t to time t+T are obtained by calculation. After the rate of change vector of the channel i to be evaluated is As the input of the stability assessment model, three clusters distinguished by stability are obtained, and the stability assessment is completed. It mainly includes the following four steps:

502. Normalization of predicted value change rate: Since the channel signal-to-noise ratio change rate, received signal strength change rate, and bandwidth change rate vary in different intervals, this paper uses the Min-Max normalization method to normalize the above parameters.

503. Stability clustering: Calculate the Euclidean distance between the rate of change vector of each channel and each centroid, and assign the channel's rate of change vector to the cluster with the smallest Euclidean distance to the centroid.

504. Update the centroid: For each cluster, calculate the average value of the change rate vectors of all its channels, and use the average value as the new centroid.

505. Completion of stability assessment: After repeated iterations, until the centroid no longer changes, the final three clusters and centroids are obtained.

6. An access algorithm for evaluating spectrum stability in a cognitive vehicle network as described in claim 5, characterized in that The channel state, stability index and throughput are selected to construct the state space of reinforcement learning. The vehicle terminal is prompted to select the most suitable channel for data transmission through continuous iterative training according to the channel stability and throughput requirements of its own transmission business, so as to achieve the rational use of spectrum resources with different stability. Therefore, the state space of this paper is: The state space is composed of the channel state vector, stability index vector and throughput vector from vehicle terminal 1 to vehicle terminal i. The action space of this paper is: The reward function of this article is: The channel occupancy vector Represents the status of channel 1 to channel m detected by vehicle terminal i at time t. It means that at time t, vehicle terminal i detects that channel j is occupied. It means that at time t, vehicle terminal i detects that channel j is idle; The stability index vector represents the stability index of channel 1 to channel m evaluated by the system side at time t; The throughput vector It represents the throughput that can be obtained when vehicle terminal i accesses channel 1 to channel m at time t, where the throughput calculation formula is as follows: in in Indicates that at time t, vehicle terminal i does not choose to access any channel. It indicates that at time t, vehicle terminal i chooses to access channel j. Considering the different requirements of spectrum stability and throughput for high-stability services and low-stability services, this paper uses the trade-off coefficient λ to distinguish the reward function of the vehicle terminal transmitting these two services. Its expression is as follows: Among them, if the service transmitted by vehicle terminal i at time t is a high-stability service, then λ>0.5; if the service transmitted is a low-stability service, then λ<0.

5. It represents the normalized stability index of channel j accessed by vehicle terminal i at time t. The normalization method is the same as the previous one, using the Min-Max method. It represents the normalized throughput of channel j accessed by vehicle terminal i at time t.

7. An access algorithm for evaluating spectrum stability in a cognitive vehicle network according to claim 7, characterized in that This paper uses the experience replay protocol to store the experience tuples used by the vehicle terminal at each moment in the experience replay pool. At the same time, the vehicle terminal uses the mini-batch algorithm to randomly extract experience tuples from the experience replay pool each time, and input them into the neural network composed of the gated recurrent unit (GRU) and the Dueling Network for stochastic gradient descent, thereby updating the neural network parameters, so that the access algorithm in this paper can converge faster.

8. An access algorithm for evaluating spectrum stability in a cognitive vehicle network as described in claim 8, characterized in that This paper uses GRU as the first hidden layer because channel parameters are usually time series data and the channel state changes continuously in time. GRU can memorize and utilize historical information through its internal loop structure, thereby effectively tracking the changing trend of channel parameters. Therefore, GRU is introduced to solve the problem of rapid changes in channel parameters in cognitive Internet of Vehicles.

9. According to the access algorithm for evaluating spectrum stability in a cognitive vehicle network described in claim 9, this paper uses Dueling Network as the second hidden layer. Dueling Network divides the Q value function into a state value function and an advantage function. In this way, on the one hand, the problem of overestimation of the Q value is solved and the convergence speed is accelerated. On the other hand, the value of each channel selection behavior in different environments is correctly estimated. For example, the Q value of a low-stability channel is relatively low in an environment where most high-stability channels are idle. When all high-stability channels are occupied and there are fewer idle channels, the Q value should be relatively high to encourage vehicle terminals to access. Dueling Network enables vehicle terminals to dynamically adjust their access strategies according to different environments, so that channels with different stabilities can be reasonably utilized. The Q value is expressed as: 。 10. According to the access algorithm for evaluating spectrum stability in a cognitive vehicle network described in claim 10, by continuously calculating the loss function L t (θ t ) to perform gradient descent to update the neural network parameters, so that the estimated value of the Q value function is constantly approaching the actual value, and finally an optimal θ is obtained. t , so that the access strategy of each vehicle terminal converges to the optimal. The loss function L t (θ t ) is expressed as: in This article will The value is taken as the timing error (TemporalDifference Error, TDR) By continuously learning from the error, the Q value is updated. The Q value is updated as follows: This paper uses the ε strategy in channel selection, that is, each time the behavior with the largest Q value is selected with a probability of 1-ε. The strategy is defined as: 。

Images