A spectrum access method based on deep reinforcement learning

Abstract

The present invention discloses a spectrum access method based on deep reinforcement learning, which sets a Beta distribution for each channel; determines the action of the secondary user, that is, the channel and time slot number to be accessed, through deep reinforcement learning; the secondary user accesses the channel, and if the primary user exists, the access fails, otherwise it is successful, and the parameters of the Beta distribution of the channel are updated according to whether the access is successful or not; if it fails, a random number is generated according to the Beta distribution for each channel in the next time slot, and the random numbers are arranged in descending order as a perception order; the channels are sensed in sequence until an idle channel is found; the idle channel is accessed, and if successful, the original channel is accessed in the next time slot, otherwise, the next time slot is sensed first and then accessed; when all designated time slots are connected, the average throughput of these time slots accessing the channel is calculated as a reward value; repeat the above steps, and save each state, action and reward for network training; repeat the above operations until all time slots are completed; the advantage is that the perception interval can be determined while selecting the access channel.

Description

Technical Field

[0001] The present invention relates to a cognitive radio technology, and in particular to a spectrum access method based on deep reinforcement learning, which learns a dynamic spectrum access strategy through deep reinforcement learning. Background Art

[0002] With the rapid development of 5G and the Internet of Things, the number of wireless users has increased dramatically, people's demand for wireless communications has increased, and spectrum resources have become increasingly important. The shortage of wireless spectrum resources and the substantial increase in people's demand for wireless spectrum resources constitute a major contradiction in the development of wireless communications today. Although there are more and more studies on millimeter wave communications, millimeter wave communications have disadvantages such as being easily blocked and having a small coverage range. The use of millimeter wave communications requires the establishment of a large number of base stations, which is very costly and difficult to popularize in a short period of time. Therefore, improving the spectrum utilization rate of medium and low frequency bands is crucial to the development of 5G and the Internet of Things. Cognitive radio technology allows secondary users to access unused licensed frequency bands to support higher data rates and access more users, which can greatly improve the utilization rate of the spectrum. Therefore, it has attracted widespread attention once it was proposed. In the context of cognitive radio, secondary users need to sense the channel before accessing the channel, and will only access it after sensing that the channel is idle to avoid interference with authorized users.

[0003] In order to maximize spectrum utilization and increase system throughput, a good spectrum access strategy is very important. Since channel sensing before secondary users access the channel also requires time, energy and other resources, the more times the sensing is performed, the less time is needed for access. Reasonably reducing the number of sensings can effectively increase the access time of secondary users, thereby improving spectrum utilization. Therefore, a good spectrum access strategy should not only consider which channel to access in each time slot, but also consider the frequency of sensing, that is, how many time slots are sensed once and the order of sensing. The traditional random sensing strategy senses multiple channels in a random order in each time slot until an idle channel is found and then access is performed. The existing Bayesian-based method, the existing Thompson sampling-based strategy, and the existing deep reinforcement learning-based strategy mainly focus on considering which channel to access without considering the frequency of sensing. In addition, there are methods that consider optimizing the sensing time under a fixed time slot length or optimizing the time slot size under a fixed sensing time length in order to maximize throughput under the constraint of detection probability. Although these methods all study the frequency of sensing, they are all aimed at single-channel scenarios, and they all need to know the parameters of the authorized user's busy / idle. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a spectrum access method based on deep reinforcement learning, which can not only select which channel to access, but also determine how many time slots to perform a perception, thereby reducing the resources spent on perception and effectively improving spectrum utilization.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a spectrum access method based on deep reinforcement learning, characterized by comprising the following steps:

[0006] Step 1: In the cognitive radio system, set the total number of time slots to T and the total number of channels to K; set the length of the historical time slot, that is, the total number of historical time slots, to L; let t represent the sequence number of the time slot, which is numbered sequentially from 0 to T-1, that is, the initial value of t is 0; assume that there is only one secondary user, and the secondary user selectively senses the channel in part of the time slot according to the sequence number of the time slot; let n renew Indicates the parameter used to record the number of updates, n renew The initial value of is 0; the update frequency of the neural network parameters is set to N renew , N renew ∈[50,200]; where T>10, K≥1, L≥10, L<T;

[0007] Step 2: Traverse each time slot in order of the time slot number. The number of the currently traversed time slot is t. Compare t with L. If t<L, execute step 3; if t=L, execute step 4;

[0008] Step 3: The secondary user sets a Beta distribution for each channel in the time slot numbered t; then generates a random number based on the Beta distribution of each channel, and the random number generated based on the Beta distribution of the i-th channel is recorded as d i ; Then sort the K random numbers in descending order, and use the sorting result as the perception order, that is, if d i In the sorting result, the i-th channel is the k-th channel to be sensed. Then the channels are sensed in the sensed order. After the sensed process is completed, the state marks of all channels are obtained. The state marks of K channels are arranged in the order of the channel numbers to form a vector with a dimension of K×1 as the state mark vector of all channels in the time slot with the sequence number t, which is recorded as s t ; Finally, let t = t + 1, and then return to step 2 to continue; where the two parameters of the Beta distribution of the i-th channel are α i and β i , the parameter group composed is recorded as (α i ,β i ), α i and βi The initial value of is 1, 1≤i≤K, 1≤k≤K. If there are two or more identical random numbers among the K random numbers, then these identical random numbers are sorted in the order of the channel numbers. The channel status is marked as 0, 1 or 2. When the channel status is marked as 0, it means that the channel is perceived to be idle. When the channel status is marked as 1, it means that the channel is perceived to be busy. When the channel status is marked as 2, it means that the channel is not perceived. The "=" in t=t+1 is an assignment symbol;

[0009] Step 4: Construct the initial state for deep reinforcement learning, denoted as S t-1 , S t-1 =[s t-1 ,s t-2 ,…,s0]; and construct a neural network, which includes three fully connected layers, the input end of the first fully connected layer is the input end of the neural network, the data output by the output end of the first fully connected layer is input to the input end of the second fully connected layer after passing through the tanh activation function, the data output by the output end of the second fully connected layer is input to the input end of the third fully connected layer after passing through the tanh activation function, and the data output by the output end of the third fully connected layer is used as the data output by the output end of the neural network; then execute step 5; wherein, S t-1 The dimension is K×L. The symbol “[]” represents a vector or matrix. t-1 represents the state mark vector of all channels in the time slot numbered t-1, s t-2 represents the state label vector of all channels in the time slot numbered t-2, s0 represents the state label vector of all channels in the time slot numbered 0, the number of neurons in the first fully connected layer is 64, the number of neurons in the second fully connected layer is 64, the number of neurons in the third fully connected layer is K×M+1, M represents the maximum number of time slots that a secondary user can access each time to access a channel, M∈[2,50];

[0010] Step 5: S t-1 Input into the constructed neural network and get the output of the neural network, recorded as a t , a t ∈{0,1,2,…,K×M}; where a t ≠K×M t Indicates that the secondary user can access the first time slot starting from the time slot numbered t. channels and the number of accessible time slots n is equal to n = a t %M+1, c∈[1,K], symbol is the rounding down operator, the symbol "%" is the remainder operator, a t =K×M tIt means that the secondary user does not access any channel in the time slot numbered t;

[0011] Step 6: Determine a t Is the value of equal to K×M? If so, it is determined that the secondary user does not access any channel in the time slot with sequence number t, and the state marks of all channels are set to 2. The state marks of K channels are arranged in the order of the channel sequence numbers to form a state mark vector s of all channels in the time slot with sequence number t with a dimension of K×1 t , then let S t =[s t ,s t-1 ,…,s t-L+1 ], t = t + 1, n renew =n renew +1, then let R t-1 =0, then S t-1 、a t-n , R t-1 , S t-2 Constitute a set {S t-1 ,a t-n ,R t-1 ,S t-2} and save it for neural network training, and then execute step 12; otherwise, determine that the secondary user can access the first time slot starting from the time slot numbered t channels and the number of accessible time slots n is equal to n = a t %M+1, then let n0=n, let q represent whether the secondary user and the primary user collide during the access process, the initial value of q is 0, and then execute step 7; where S t represents the t-time slot state for deep reinforcement learning, S t The dimension is K×L, s t-L+1 Represents the state mark vector of all channels in the time slot numbered t-L+1, R t-1 represents the reward value of the secondary user in the time slot numbered t-1, S t-1 represents the t-1 time slot state for deep reinforcement learning, when a t-1 When the value of a is equal to K×M t-n n=1, that is, a t-n for a t-1 , S t-2 represents the t-2 time slot state for deep reinforcement learning, n renew =n renew The "=" in t=t+1 and t=t+1 is an assignment symbol, and n0 is an introduced intermediate variable;

[0012] Step 7: The secondary user accesses the cth tchannels; then determine whether the secondary user collides with the primary user during the access process. If no collision occurs, calculate the throughput of the secondary user in the time slot numbered t, denoted as r t , r t =log2(1+snr), and at the same time t The state mark of the K channels is set to 0, and the state marks of all other channels are set to 2. The state marks of the K channels are arranged in the order of the channel numbers to form a state mark vector s of all channels in the time slot with the sequence number t of dimension K×1. t , and then let S t =[s t ,s t-1 ,…,s t-L+1 ]、 q = 0, n = n-1, t = t+1, and then execute step 8; if a collision occurs, the throughput of the secondary user in the time slot with sequence number t is recorded as r t , let r t =0, and at the same time t The state mark of the K channel is set to 1, and the state marks of all other channels are set to 2. The state marks of the K channels are arranged in the order of the channel numbers to form a state mark vector s of all channels in the time slot with the sequence number t of dimension K×1. t , and then let S t =[s t ,s t-1 ,…,s t-L+1 ]、 q=1、n=n-1、t=t+1,then execute step 8; where c t =c, snr represents the signal-to-noise ratio of the secondary user’s receiving end, S t represents the t-time slot state for deep reinforcement learning, s t-L+1 represents the state mark vector of all channels in the time slot numbered t-L+1, Indicates the c t The parameter group is composed of two parameters of the Beta distribution of the channel. q indicates whether the secondary user collides with the primary user during the access process. q = 0 means that the secondary user does not collide with the primary user during the access process. q = 1 means that the secondary user collides with the primary user during the access process. n=n-1、t=t+1、 The "=" in is the assignment symbol;

[0013] Step 8: If t<T, n>0 and q=0, return to step 7 to continue execution; if t<T, n>0 and q=1, directly execute step 9; if t<T and n=0, directly execute step 11; if t=T, directly execute step 13;

[0014] Step 9: The secondary user sets a Beta distribution for each channel in the time slot numbered t; then generates a random number based on the Beta distribution of each channel, and the random number generated based on the Beta distribution of the i-th channel is recorded as d i ; Then sort the K random numbers in descending order, and use the sorting result as the perception order, that is, if d i In the sorting result, the i-th channel is the k-th channel to be sensed. Then the channels are sensed in the sensed order. After the sensed process is completed, the state marks of all channels are obtained. The state marks of K channels are arranged in the order of the channel numbers to form a vector with a dimension of K×1 as the state mark vector of all channels in the time slot with the sequence number t, which is recorded as s t , and record the total number of perceptions in the time slot numbered t, denoted as η t ; Then let S t =[s t ,s t-1 ,…,s t-L+1 ]; Finally, determine whether the secondary user senses an idle channel in the time slot numbered t. If the secondary user senses the idle channel in the time slot numbered c, t If a channel is idle, the secondary user accesses the channel and t = c and there is no collision between the secondary user and the primary user during the access process, calculate the throughput of the secondary user in the time slot with sequence number t, denoted as r t , make q=0、n=n-1、t=t+1, then execute step 10; when c t = c and when the secondary user collides with the primary user during the access process, let the throughput of the secondary user in the time slot with sequence number t be r t For r t =0, let q=1、n=n-1、t=t+1, then execute step 10; when c t ≠c and there is no collision between the secondary user and the primary user during the access process, let the throughput of the secondary user in the time slot with sequence number t be r t For r t =0, let q=0、n=n-1、t=t+1, then execute step 10; when c t ≠c and when a secondary user collides with a primary user during access, let the throughput of the secondary user in the time slot with sequence number t be r t For r t =0, let q=1、n=n-1、t=t+1, then execute step 10; if no idle channel is sensed, then let the throughput of the secondary user in the time slot with sequence number t be rt For r t =0, let n=n-1, t=t+1, and repeat step 9; where T f represents the length of each time slot, and τ represents the duration required for each perception;

[0015] Step 10: If t<T, n>0 and q=0, return to step 7 to continue execution; if t<T, n>0 and q=1, return to step 9 to continue execution; if t<T and n=0, directly execute step 11; if t=T, directly execute step 13;

[0016] Step 11: Let n renew =n renew +1; then calculate the reward value of the secondary user in the time slot numbered t-1, denoted as R t-1 , Then S t-1 、a t-n , R t-1 , S t-2 Constitute a set {S t-1 ,a t-n ,R t-1 ,S t-2} and save it for neural network training, and then execute step 12; where r j represents the throughput of the secondary user in the time slot with sequence number j, S t-1 represents the t-1 time slot state for deep reinforcement learning, S t-2 represents the t-2 time slot state for deep reinforcement learning, when a t-1 When the value of is not equal to K×M t-n n=a t %M+1;

[0017] Step 12: Determine n renew %N renew = 0 is true, if true, then use {S t ,a t-n+1 ,R t ,S t-1} Train the neural network, set the batch size to be greater than or equal to 100, use Adam as the optimizer, use the cross entropy function as the loss function, update the parameters of the neural network, and set n renew =0, then return to step 5 to continue execution; otherwise, directly return to step 5 to continue execution;

[0018] Step 13: Spectrum access is completed.

[0019] In the step 3 and the step 9, s t The acquisition process is:

[0020] Step a1: sense the channels in sequence according to the sensing order, and define the currently sensed channel as the current channel;

[0021] Step a2: if the current channel is perceived to be idle, the state flag of the current channel is set to 0, and then step a3 is executed; if the current channel is perceived to be busy, the state flag of the current channel is set to 1, and then step a4 is executed;

[0022] Step a3: determine whether there are any channels that have not been sensed. If yes, set the status flags of all channels that have not been sensed to 2, end the sensing process, and then execute step a5; otherwise, directly end the sensing process and then execute step a5;

[0023] Step a4: Determine whether the time of the time slot with sequence number t has been used up. If the time of the time slot with sequence number t has been used up, determine whether there are still unperceived channels. If so, set the status flags of all unperceived channels to 2, end the perception process, and then execute step a5; otherwise, directly end the perception process, and then execute step a5; if the time of the time slot with sequence number t has not been used up, determine whether the current channel is the last channel to be perceived. If so, perceive the channels again in the perception order, define the currently perceived channel as the current channel, and then return to step a2 to continue execution; if not, set the next channel to be perceived as the current channel, and then return to step a2 to continue execution;

[0024] Step a5: Arrange the status marks of K channels in the order of channel numbers to form s t .

[0025] Compared with the prior art, the advantages of the present invention are:

[0026] 1) The deep reinforcement learning method of the present invention selects both the access channel and the number of time slots that can be continuously accessed in the channel in each action, which greatly reduces the time required for perception and increases the access time of secondary users, thereby improving spectrum utilization and increasing the throughput of secondary users.

[0027] 2) The method of the present invention only needs to know whether there is an authorized user, ie, a primary user, but does not need to know the busy / idle parameters of the authorized user, ie, the primary user, which is more in line with actual scenarios and has a wider application range.

[0028] 3) The method of the present invention is not limited to the channel access of a single secondary user, and each secondary user can be regarded as an intelligent agent, thereby being extended to solve the channel access problem of multiple secondary users. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of the overall implementation process of the method of the present invention;

[0030] Figure 2 A curve comparison diagram of the average normalized throughput of each time slot obtained by the method of the present invention and the existing Bayesian-based method over time;

[0031] Figure 3 The following is a curve comparison diagram showing the average number of collisions per time slot obtained by using the method of the present invention and the existing Bayesian-based method over time. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below with reference to the accompanying drawings.

[0033] The present invention proposes a spectrum access method based on deep reinforcement learning, and its overall implementation flow chart is as follows: Figure 1 As shown, it includes the following steps:

[0034] Step 1: In the cognitive radio system, set the total number of time slots to T and the total number of channels to K; set the length of the historical time slot, that is, the total number of historical time slots, to L; let t represent the sequence number of the time slot, which is numbered sequentially from 0 to T-1, that is, the initial value of t is 0; assume that there is only one secondary user, and the secondary user selectively senses the channel in part of the time slot according to the sequence number of the time slot; let n renew Indicates the parameter used to record the number of updates, n renew The initial value of is 0; the update frequency of the neural network parameters is set to N renew , N renew ∈[50,200], in this embodiment, N renew =100; wherein, T>10, such as T=50, K≥1, in this embodiment, K=5, L≥10, L<T, in this embodiment, L=40.

[0035] Step 2: Traverse each time slot in order of the time slot number. The number of the currently traversed time slot is t. Compare t with L. If t<L, execute step 3; if t=L, execute step 4.

[0036] Step 3: The secondary user sets a Beta distribution for each channel in the time slot numbered t; then generates a random number based on the Beta distribution of each channel, and the random number generated based on the Beta distribution of the i-th channel is recorded as d i ; Then sort the K random numbers in descending order, and use the sorting result as the perception order, that is, if d iIn the sorting result, the i-th channel is the k-th channel to be sensed. Then the channels are sensed in the sensed order. After the sensed process is completed, the state marks of all channels are obtained. The state marks of K channels are arranged in the order of the channel numbers to form a vector with a dimension of K×1 as the state mark vector of all channels in the time slot with the sequence number t, which is recorded as s t ; Finally, let t = t + 1, and then return to step 2 to continue; where the two parameters of the Beta distribution of the i-th channel are α i and β i , the parameter group composed is recorded as (α i ,β i ), α i and β i The initial value of is 1, 1≤i≤K, 1≤k≤K. If there are two or more identical random numbers among the K random numbers, then these identical random numbers are sorted in the order of the channel numbers. The channel status is marked as 0, 1 or 2. When the channel status is marked as 0, it means that the channel is perceived to be idle. When the channel status is marked as 1, it means that the channel is perceived to be busy. When the channel status is marked as 2, it means that the channel is not perceived. The "=" in t=t+1 is an assignment symbol.

[0037] Step 4: Construct the initial state for deep reinforcement learning, denoted as S t-1 , S t-1 =[s t-1 ,s t-2 ,…,s0]; and construct a neural network, which includes three fully connected layers, the input end of the first fully connected layer is the input end of the neural network, the data output by the output end of the first fully connected layer is input to the input end of the second fully connected layer after passing through the tanh activation function, the data output by the output end of the second fully connected layer is input to the input end of the third fully connected layer after passing through the tanh activation function, and the data output by the output end of the third fully connected layer is used as the data output by the output end of the neural network; then execute step 5; wherein, S t-1 The dimension is K×L. The symbol “[]” represents a vector or matrix. t-1 represents the state mark vector of all channels in the time slot numbered t-1, s t-2 represents the state mark vector of all channels in the time slot numbered t-2, s0 represents the state mark vector of all channels in the time slot numbered 0, the number of neurons in the first fully connected layer is 64, the number of neurons in the second fully connected layer is 64, the number of neurons in the third fully connected layer is K×M+1, M represents the maximum number of time slots that a secondary user can access each time he accesses a channel, M∈[2,50], and the specific value of M is set artificially.

[0038] Step 5: St-1 Input into the constructed neural network and get the output of the neural network, recorded as a t , a t ∈{0,1,2,…,K×M}; where a t ≠K×M t Indicates that the secondary user can access the first time slot starting from the time slot numbered t. channels and the number of accessible time slots n is equal to n = a t %M+1, c∈[1,K], symbol is the rounding down operator, the symbol "%" is the remainder operator, a t =K×M t It means that the secondary user does not access any channel in the time slot numbered t.

[0039] Step 6: Determine a t Is the value of equal to K×M? If so, it is determined that the secondary user does not access any channel in the time slot with sequence number t, and the state marks of all channels are set to 2. The state marks of K channels are arranged in the order of the channel sequence numbers to form a state mark vector s of all channels in the time slot with sequence number t with a dimension of K×1 t , then let S t =[s t ,s t-1 ,…,s t-L+1 ], t = t + 1, n renew =n renew +1, then let R t-1 =0, then S t-1 、a t-n , R t-1 , S t-2 Constitute a set {S t-1 ,a t-n ,R t-1 ,S t-2} and save it for neural network training, and then execute step 12; otherwise, determine that the secondary user can access the first time slot starting from the time slot numbered t channels and the number of accessible time slots n is equal to n = a t %M+1, then let n0=n, let q represent whether the secondary user and the primary user collide during the access process, the initial value of q is 0, and then execute step 7; where S t represents the t-time slot state for deep reinforcement learning, S t The dimension is K×L, s t-L+1 Represents the state mark vector of all channels in the time slot numbered t-L+1, R t-1 represents the reward value of the secondary user in the time slot numbered t-1, S t-1represents the t-1 time slot state for deep reinforcement learning, when a t-1 When the value of a is equal to K×M t-n n=1, that is, a t-n for a t-1 , S t-2 represents the t-2 time slot state for deep reinforcement learning, n renew =n renew The "=" in t=t+1 and t=t+1 is an assignment symbol, and n0 is an introduced intermediate variable.

[0040] Step 7: The secondary user accesses the cth t channels; then determine whether the secondary user collides with the primary user during the access process. If no collision occurs, calculate the throughput of the secondary user in the time slot numbered t, denoted as r t , r t =log2(1+snr), and at the same time t The state mark of the K channels is set to 0, and the state marks of all other channels are set to 2. The state marks of the K channels are arranged in the order of the channel numbers to form a state mark vector s of all channels in the time slot with the sequence number t of dimension K×1. t , and then let S t =[s t ,s t-1 ,…,s t-L+1 ]、 q = 0, n = n-1, t = t+1, and then execute step 8; if a collision occurs, the throughput of the secondary user in the time slot with sequence number t is recorded as r t , let r t =0, and at the same time t The state mark of the K channel is set to 1, and the state marks of all other channels are set to 2. The state marks of the K channels are arranged in the order of the channel numbers to form a state mark vector s of all channels in the time slot with the sequence number t of dimension K×1. t , and then let S t =[s t ,s t-1 ,…,s t-L+1 ]、 q=1、n=n-1、t=t+1,then execute step 8; where c t =c, snr represents the signal-to-noise ratio of the secondary user’s receiving end, S t represents the t-time slot state for deep reinforcement learning, s t-L+1 represents the state mark vector of all channels in the time slot numbered t-L+1, Indicates the c tThe parameter group is composed of two parameters of the Beta distribution of the channel. q indicates whether the secondary user collides with the primary user during the access process. q = 0 means that the secondary user does not collide with the primary user during the access process. q = 1 means that the secondary user collides with the primary user during the access process. n=n-1、t=t+1、 The “=” in is the assignment symbol.

[0041] Step 8: If t<T, n>0 and q=0, return to step 7 to continue execution; if t<T, n>0 and q=1, directly execute step 9; if t<T and n=0, directly execute step 11; if t=T, directly execute step 13.

[0042] Step 9: The secondary user sets a Beta distribution for each channel in the time slot numbered t; then generates a random number based on the Beta distribution of each channel, and the random number generated based on the Beta distribution of the i-th channel is recorded as d i ; Then sort the K random numbers in descending order, and use the sorting result as the perception order, that is, if d i In the sorting result, the i-th channel is the k-th channel to be sensed. Then the channels are sensed in the sensed order. After the sensed process is completed, the state marks of all channels are obtained. The state marks of K channels are arranged in the order of the channel numbers to form a vector with a dimension of K×1 as the state mark vector of all channels in the time slot with the sequence number t, which is recorded as s t , and record the total number of perceptions in the time slot numbered t, denoted as η t , for example: i The 5th channel in the sorting result is the 5th channel. When the 5th channel is perceived to be idle, the perception process ends. Then η t =5; then let S t =[s t ,s t-1 ,…,s t-L+1 ]; Finally, determine whether the secondary user senses an idle channel in the time slot numbered t. If the secondary user senses the idle channel in the time slot numbered c, t If a channel is idle, the secondary user accesses the channel and t = c and there is no collision between the secondary user and the primary user during the access process, calculate the throughput of the secondary user in the time slot with sequence number t, denoted as r t , make q=0、n=n-1、t=t+1, then execute step 10; when c t = c and when the secondary user collides with the primary user during the access process, let the throughput of the secondary user in the time slot with sequence number t be rt For r t =0, let q=1、n=n-1、t=t+1, then execute step 10; when c t ≠c and there is no collision between the secondary user and the primary user during the access process, let the throughput of the secondary user in the time slot with sequence number t be r t For r t =0, let q=0、n=n-1、t=t+1, then execute step 10; when c t ≠c and when a secondary user collides with a primary user during access, let the throughput of the secondary user in the time slot with sequence number t be r t For r t =0, let q=1、n=n-1、t=t+1, then execute step 10; if no idle channel is sensed, then let the throughput of the secondary user in the time slot with sequence number t be r t For r t =0, let n=n-1, t=t+1, and repeat step 9; where T f represents the length of each time slot, and τ represents the duration required for each perception.

[0043] Step 10: If t<T, n>0 and q=0, return to step 7 to continue execution; if t<T, n>0 and q=1, return to step 9 to continue execution; if t<T and n=0, directly execute step 11; if t=T, directly execute step 13.

[0044] Step 11: Let n renew =n renew +1; then calculate the reward value of the secondary user in the time slot numbered t-1, denoted as R t-1 , Then S t-1 、a t-n , R t-1 , S t-2 Constitute a set {S t-1 ,a t-n ,R t-1 ,S t-2} and save it for neural network training, and then execute step 12; where r j represents the throughput of the secondary user in the time slot with sequence number j, S t-1 represents the t-1 time slot state for deep reinforcement learning, S t-2 represents the t-2 time slot state for deep reinforcement learning, when a t-1 When the value of is not equal to K×M t-n n=a t %M+1.

[0045] Step 12: Determine n renew %N renew = 0 is true, if true, then use {S t ,a t-n+1 ,R t ,S t-1} Train the neural network, set the batch size to be greater than or equal to 100, use Adam as the optimizer, use the cross entropy function as the loss function, update the parameters of the neural network, and set n renew =0, then return to step 5 to continue execution; otherwise, directly return to step 5 to continue execution.

[0046] Step 13: Spectrum access is completed.

[0047] In this specific embodiment, in step 3 and step 9, s t The acquisition process is:

[0048] Step a1: sense the channels in sequence according to the sensing order, and define the currently sensed channel as the current channel.

[0049] Step a2: If the current channel is perceived to be idle, the status flag of the current channel is set to 0, and then step a3 is executed; if the current channel is perceived to be busy, the status flag of the current channel is set to 1, and then step a4 is executed.

[0050] Step a3: Determine whether there are any unperceived channels. If so, set the status flags of all unperceived channels to 2, end the perception process, and then execute step a5; otherwise, directly end the perception process and then execute step a5.

[0051] Step a4: Determine whether the time of the time slot with sequence number t has been used up. If the time of the time slot with sequence number t has been used up, determine whether there are any channels that have not been sensed. If so, set the status marks of all channels that have not been sensed to 2, end the sensing process, and then execute step a5; otherwise, directly end the sensing process, and then execute step a5; if the time of the time slot with sequence number t has not been used up, determine whether the current channel is the last channel that needs to be sensed. If so, sense the channels again in the sensing order, define the currently sensed channel as the current channel, and then return to step a2 to continue; if not, set the next channel to be sensed as the current channel, and then return to step a2 to continue.

[0052] Step a5: Arrange the status marks of K channels in the order of channel numbers to form s t .

[0053] The feasibility and effectiveness of the spectrum access method of the present invention are further illustrated by the following simulation.

[0054] Figure 2 The following is a comparison of the average normalized throughput of each time slot obtained by the method of the present invention and the existing Bayesian-based method over time. In the simulation, the number of channels K = 5 and the length of the historical time slot L = 40. Figure 2 It can be seen from the figure that the average normalized throughputs of the two methods increase continuously with time, and the throughput of the method of the present invention is always higher than the throughput of the Bayesian-based method.

[0055] Figure 3 The following is a comparison of the average collision times of each time slot obtained by the method of the present invention and the existing Bayesian-based method. In the simulation, the number of channels K = 5 and the length of the historical time slot L = 40. Figure 3 It can be seen that the average collision times of the two methods are decreasing with the increase of time, and the collision times of the method of the present invention are always lower than that of the Bayesian method; it can also be found that the collision rate of the method of the present invention is always very low. Figure 2 It can be seen that the throughput of the method of the present invention is much higher than that of the Bayesian method just because the collision rate is very low. In summary, the access performance of the method of the present invention is better than that of the existing Bayesian method.

Claims

1. A spectrum access method based on deep reinforcement learning, characterized in that The following steps are involved: Step 1: In the cognitive radio system, set the total number of time slots to T and the total number of channels to K; set the length of the historical time slots, i.e. the total number of historical time slots, to L; let t represent the sequence number of the time slots, which are numbered sequentially from 0 to T-1, i.e. the initial value of t is 0; Assume that there is only one secondary user, and the secondary user selectively senses the channel in part of the time slots according to the sequence of the time slots; let n renew Indicates the parameter used to record the number of updates, n renew The initial value of is 0; the update frequency of the neural network parameters is set to N renew , N renew ∈[50,200]; where T>10, K≥1, L≥10, L<T; Step 2: Traverse each time slot in order of the time slot number. The number of the currently traversed time slot is t. Compare t with L. If t<L, execute step 3; if t=L, execute step 4; Step 3: The secondary user sets a Beta distribution for each channel in the time slot numbered t; then generates a random number based on the Beta distribution of each channel, and the random number generated based on the Beta distribution of the i-th channel is recorded as d i ; Then sort the K random numbers in descending order, and use the sorting result as the perception order, that is, if d i In the sorting result, the i-th channel is the k-th channel to be sensed. Then the channels are sensed in the sensed order. After the sensed process is completed, the state marks of all channels are obtained. The state marks of K channels are arranged in the order of the channel numbers to form a vector with a dimension of K×1 as the state mark vector of all channels in the time slot with the sequence number t, which is recorded as s t ; Finally, let t = t + 1, and then return to step 2 to continue; where the two parameters of the Beta distribution of the i-th channel are α i and β i , the parameter group composed is recorded as (α i ,β i ), α i and β i The initial value of is 1, 1≤i≤K, 1≤k≤K. If there are two or more identical random numbers among the K random numbers, then these identical random numbers are sorted in the order of the channel numbers. The channel status is marked as 0, 1 or 2. When the channel status is marked as 0, it means that the channel is perceived to be idle. When the channel status is marked as 1, it means that the channel is perceived to be busy. When the channel status is marked as 2, it means that the channel is not perceived. The "=" in t=t+1 is an assignment symbol; Step 4: Construct the initial state for deep reinforcement learning, denoted as S t-1 , S t-1 =[s t-1 ,s t-2 ,…,s0]; and construct a neural network, which includes three fully connected layers, the input end of the first fully connected layer is the input end of the neural network, the data output by the output end of the first fully connected layer is input to the input end of the second fully connected layer after passing through the tanh activation function, the data output by the output end of the second fully connected layer is input to the input end of the third fully connected layer after passing through the tanh activation function, and the data output by the output end of the third fully connected layer is used as the data output by the output end of the neural network; then execute step 5; wherein, S t-1 The dimension is K×L, the symbol "[]" is the symbol for vector or matrix, s t-1 represents the state mark vector of all channels in the time slot numbered t-1, s t-2 represents the state label vector of all channels in the time slot numbered t-2, s0 represents the state label vector of all channels in the time slot numbered 0, the number of neurons in the first fully connected layer is 64, the number of neurons in the second fully connected layer is 64, the number of neurons in the third fully connected layer is K×M+1, M represents the maximum number of time slots that a secondary user can access each time to access a channel, M∈[2,50]; Step 5: S t-1 Input into the constructed neural network and get the output of the neural network, recorded as a t , a t ∈{0,1,2,…,K×M}; where a t ≠K×M t Indicates that the secondary user can access the first time slot starting from the time slot numbered t. channels and the number of accessible time slots n is equal to n = a t %M+1, c∈[1,K], symbol is the rounding down operator, the symbol "%" is the remainder operator, a t =K×M t It means that the secondary user does not access any channel in the time slot numbered t; Step 6: Determine a t Is the value of equal to K×M? If so, it is determined that the secondary user does not access any channel in the time slot with sequence number t, and the state marks of all channels are set to 2. The state marks of K channels are arranged in the order of the channel sequence numbers to form a state mark vector s of all channels in the time slot with sequence number t with a dimension of K×1 t , then let S t =[s t ,s t-1 ,…,s t-L+1 ], t = t + 1, n renew =n renew +1, then let R t-1 =0, then S t-1 、a t-n , R t-1 , S t-2 Constitute a set {S t-1 ,a t-n ,R t-1 ,S t-2 } and save it for neural network training, and then execute step 12; otherwise, determine that the secondary user can access the first time slot starting from the time slot numbered t channels and the number of accessible time slots n is equal to n = a t %M+1, then let n0=n, let q represent whether the secondary user and the primary user collide during the access process, the initial value of q is 0, and then execute step 7; where S t represents the t-time slot state for deep reinforcement learning, S t The dimension is K×L, s t-L+1 Represents the state mark vector of all channels in the time slot numbered t-L+1, R t-1 represents the reward value of the secondary user in the time slot numbered t-1, S t-1 represents the t-1 time slot state for deep reinforcement learning, when a t-1 When the value of a is equal to K×M t-n n=1, that is, a t-n for a t-1 , S t-2 represents the t-2 time slot state for deep reinforcement learning, n renew =n renew The "=" in t=t+1 and t=t+1 is an assignment symbol, and n0 is an introduced intermediate variable; Step 7: The secondary user accesses the cth t channels; then determine whether the secondary user collides with the primary user during the access process. If no collision occurs, calculate the throughput of the secondary user in the time slot numbered t, denoted as r t , r t =log2(1+snr), and at the same time t The state mark of the K channels is set to 0, and the state marks of all other channels are set to 2. The state marks of the K channels are arranged in the order of the channel numbers to form a state mark vector s of all channels in the time slot with the sequence number t of dimension K×1. t , and then let S t =[s t ,s t-1 ,…,s t-L+1 ]、 q = 0, n = n-1, t = t+1, and then execute step 8; if a collision occurs, the throughput of the secondary user in the time slot with sequence number t is recorded as r t , let r t =0, and at the same time t The state mark of the K channel is set to 1, and the state marks of all other channels are set to 2. The state marks of the K channels are arranged in the order of the channel numbers to form a state mark vector s of all channels in the time slot with the sequence number t of dimension K×1. t , and then let S t =[s t ,s t-1 ,…,s t-L+1 ]、β ct =β ct +1, q=1, n=n-1, t=t+1, and then execute step 8; wherein c t =c, snr represents the signal-to-noise ratio of the secondary user’s receiving end, S t represents the t-time slot state for deep reinforcement learning, s t-L+1 represents the state mark vector of all channels in the time slot numbered t-L+1, Indicates the c t The parameter group is composed of two parameters of the Beta distribution of the channel. q indicates whether the secondary user collides with the primary user during the access process. q = 0 means that the secondary user does not collide with the primary user during the access process. q = 1 means that the secondary user collides with the primary user during the access process. n=n-1、t=t+1、 The "=" in is the assignment symbol; Step 8: If t<T, n>0 and q=0, return to step 7 to continue execution; if t<T, n>0 and q=1, directly execute step 9; if t<T and n=0, directly execute step 11; if t=T, directly execute step 13; Step 9: The secondary user sets a Beta distribution for each channel in the time slot numbered t; then generates a random number based on the Beta distribution of each channel, and the random number generated based on the Beta distribution of the i-th channel is recorded as d i ; Then sort the K random numbers in descending order, and use the sorting result as the perception order, that is, if d i In the sorting result, the i-th channel is the k-th channel to be sensed. Then the channels are sensed in the sensed order. After the sensed process is completed, the state marks of all channels are obtained. The state marks of K channels are arranged in the order of the channel numbers to form a vector with a dimension of K×1 as the state mark vector of all channels in the time slot with the sequence number t, which is recorded as s t , and record the total number of perceptions in the time slot numbered t, denoted as η t ; Then let S t =[s t ,s t-1 ,…,s t-L+1 ]; Finally, determine whether the secondary user senses an idle channel in the time slot numbered t. If the secondary user senses the idle channel in the time slot numbered c, t If a channel is idle, the secondary user accesses the channel and t = c and there is no collision between the secondary user and the primary user during the access process, calculate the throughput of the secondary user in the time slot with sequence number t, denoted as r t , make q=0、n=n-1、t=t+1, then execute step 10; when c t = c and when the secondary user collides with the primary user during the access process, let the throughput of the secondary user in the time slot with sequence number t be r t For r t =0, let q=1、n=n-1、t=t+1, then execute step 10; when c t ≠c and there is no collision between the secondary user and the primary user during the access process, let the throughput of the secondary user in the time slot with sequence number t be r t For r t =0, let q=0、n=n-1、t=t+1, then execute step 10; when c t ≠c and when a secondary user collides with a primary user during access, let the throughput of the secondary user in the time slot with sequence number t be r t For r t =0, let q=1、n=n-1、t=t+1, then execute step 10; if no idle channel is sensed, then let the throughput of the secondary user in the time slot with sequence number t be r t For r t =0, let n=n-1, t=t+1, and repeat step 9; where T f represents the length of each time slot, and τ represents the duration required for each perception; Step 10: If t<T, n>0 and q=0, return to step 7 to continue execution; if t<T, n>0 and q=1, return to step 9 to continue execution; if t<T and n=0, directly execute step 11; if t=T, directly execute step 13; Step 11: Let n renew =n renew +1; then calculate the reward value of the secondary user in the time slot numbered t-1, denoted as R t-1 , Then S t-1 、a t-n , R t-1 , S t-2 Constitute a set {S t-1 ,a t-n ,R t-1 ,S t-2 } and save it for neural network training, and then execute step 12; where r j represents the throughput of the secondary user in the time slot with sequence number j, S t-1 represents the t-1 time slot state for deep reinforcement learning, S t-2 represents the t-2 time slot state for deep reinforcement learning, when a t-1 When the value of is not equal to K×M t-n n=a t %M+1; Step 12: Determine n renew %N renew = 0 is true, if true, then use {S t ,a t-n+1 ,R t ,S t-1 } Train the neural network, set the batch size to be greater than or equal to 100, use Adam as the optimizer, use the cross entropy function as the loss function, update the parameters of the neural network, and set n renew =0, then return to step 5 to continue execution; otherwise, directly return to step 5 to continue execution; Step 13: Spectrum access is completed.

2. A spectrum access method based on deep reinforcement learning according to claim 1, characterized in that In the step 3 and the step 9, s t The acquisition process is: Step a1: sense the channels in sequence according to the sensing order, and define the currently sensed channel as the current channel; Step a2: if the current channel is perceived to be idle, the state flag of the current channel is set to 0, and then step a3 is executed; if the current channel is perceived to be busy, the state flag of the current channel is set to 1, and then step a4 is executed; Step a3: determine whether there are any channels that have not been sensed. If yes, set the status flags of all channels that have not been sensed to 2, end the sensing process, and then execute step a5; otherwise, directly end the sensing process and then execute step a5; Step a4: Determine whether the time of the time slot with sequence number t has been used up. If the time of the time slot with sequence number t has been used up, determine whether there are still unperceived channels. If so, set the status flags of all unperceived channels to 2, end the perception process, and then execute step a5; otherwise, directly end the perception process, and then execute step a5; if the time of the time slot with sequence number t has not been used up, determine whether the current channel is the last channel to be perceived. If so, perceive the channels again in the perception order, define the currently perceived channel as the current channel, and then return to step a2 to continue execution; if not, set the next channel to be perceived as the current channel, and then return to step a2 to continue execution; Step a5: Arrange the status marks of K channels in the order of channel numbers to form s t .

Images