Dynamic spectrum access method based on interpretable deep reinforcement learning

Abstract

The invention discloses a dynamic spectrum access method based on interpretable deep reinforcement learning, and belongs to the field of communication. The method comprises the following steps: constructing a DQN dynamic spectrum access model based on MLP; taking a spectrum sensing result of the SU in the cognitive network as input data of the model, controlling the SU to access different channels according to a Q value output by the model, and providing a feedback reward of the SU on each channel to the model to update the model; the weight distribution of the input features of the model is visualized to reflect the cognition and preference of the dynamic spectrum access model for different channels. According to the method, the DQN is combined with the MLP to construct the dynamic spectrum access model, so that the processing capability of the state space can be enhanced, and feedback rewards can be provided for model training according to the actual wireless transmission quality, so that the SU can continuously optimize the spectrum use efficiency and the weight distribution of visual input characteristics, and the transparency of the spectrum access decision process is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and more specifically, to a dynamic spectrum access method based on interpretable deep reinforcement learning. Background Art

[0002] In recent years, as the next generation of wireless networks develop towards higher density and dynamism, the management and optimization of spectrum resources have become crucial. Dynamic spectrum access technology integrates advanced intelligent algorithms to enable wireless devices to autonomously adjust spectrum usage according to environmental changes, greatly improving spectrum utilization efficiency and network service quality. In addition, dynamic spectrum access technology also helps to reduce the cost and complexity of electromagnetic spectrum management, opening up new possibilities for the intelligent use of spectrum resources. However, existing technologies face many challenges in practical applications, especially in terms of imperfect spectrum perception, strong user and channel dynamics, etc.

[0003] The rapid development of machine learning has enabled dynamic spectrum access users to better adapt to dynamic and unknown spectrum environments, making spectrum access not only based on current spectrum sensing results, but also increasing learning of past spectrum states, effectively alleviating the negative impact of imperfect spectrum sensing. However, existing models generally lack sufficient interpretability. This lack of transparency makes it difficult for spectrum access users to understand the reasoning process within the model, affecting their understanding and trust in the model output results, thus limiting the widespread adoption of these technologies in key applications. Summary of the invention

[0004] 1. Technical problem to be solved by the invention

[0005] Existing models often lack sufficient explainability. This lack of transparency makes it difficult for spectrum access users to understand the reasoning process within the model, affecting their understanding and trust in the model output results, thus limiting the widespread adoption of these technologies in critical applications.

[0006] 2. Technical solution

[0007] To achieve the above-mentioned purpose, in a first aspect, the technical solution provided by the present invention is: a dynamic spectrum access method based on interpretable deep reinforcement learning, comprising the following steps: constructing a dynamic spectrum access model, wherein the dynamic spectrum access model is a DQN (Deep Q-Learning Network) model based on MLP (Multilayer Perceptron); using the spectrum perception results of SU (Secondary User) in the cognitive network as input data of the dynamic spectrum access model, and obtaining the Q values ​​corresponding to different SU actions output by the dynamic spectrum access model, wherein the cognitive network is composed of a main network composed of N PUs (Primary Users) and a slave network composed of L SUs; controlling SU to access different channels according to the Q value, and providing the feedback reward of the SU on each channel to the dynamic spectrum access model to update the dynamic spectrum access model; visualizing the weight distribution of the input features of the dynamic spectrum access model to reflect the cognition and preference of the dynamic spectrum access model for different channels.

[0008] As a further improvement of the present invention, the feedback reward of the SU on each channel is obtained according to the following formula:

[0009] Among them, r l (t+1) represents the reward function of the lth SU at time slot (t+1), C is the channel capacity, represents the signal-to-noise ratio of the signal received by the lth SU receiver on the nth wireless channel, Γ represents the gap between the channel capacity and the actual modulation and coding scheme, and l and n are positive integers.

[0010] As a further improvement of the present invention, a heat map is generated to visualize the weight distribution of the input features of the dynamic spectrum access model.

[0011] As a further improvement of the present invention, the following formula is used to obtain the weight distribution of the input features: Among them, W ij Indicates the influence of the i-th input feature on the j-th action, is the Q-value function of taking action a in state s, x i is the i-th input feature.

[0012] As a further improvement of the present invention, in the process of training the dynamic spectrum access model, weight updating is completed by a gradient descent algorithm.

[0013] To achieve the above-mentioned purpose, in a second aspect, the technical solution provided by the present invention is: a dynamic spectrum access device based on interpretable deep reinforcement learning, comprising: a construction module, the construction module is used to construct a dynamic spectrum access model, and the dynamic spectrum access model is a DQN model based on MLP; an acquisition module, the acquisition module is used to use the spectrum perception results of SU in the cognitive network as input data of the dynamic spectrum access model to obtain the Q values ​​corresponding to different SU actions output by the dynamic spectrum access model, and the cognitive network is composed of a main network composed of N main users PU and a slave network composed of L secondary users SU; an update module, the update module is used to control SU to access different channels according to the Q value, and provide the feedback reward of the SU on each channel to the dynamic spectrum access model to update the dynamic spectrum access model; a visualization module, the visualization module is used to visualize the weight distribution of the input features of the dynamic spectrum access model to reflect the cognition and preference of the dynamic spectrum access model for different channels.

[0014] As a further improvement of the present invention, the updating module obtains the feedback reward of the SU on each channel according to the following formula: Among them, r l (t+1) represents the reward function of the lth SU at time slot (t+1), C is the channel capacity, represents the signal-to-noise ratio of the signal received by the lth SU receiver on the nth wireless channel, Γ represents the gap between the channel capacity and the actual modulation and coding scheme, and l and n are positive integers.

[0015] As a further improvement of the present invention, the visualization module specifically visualizes the weight distribution of the input features of the dynamic spectrum access model by generating a heat map.

[0016] As a further improvement of the present invention, the visualization module uses the following formula to obtain the weight distribution of the input features: Among them, W ij Indicates the influence of the i-th input feature on the j-th action, is the Q-value function of taking action a in state s, x i is the i-th input feature.

[0017] As a further improvement of the present invention, in the process of training the dynamic spectrum access model, weight updating is completed by a gradient descent algorithm.

[0018] 3. Beneficial effects

[0019] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0020] (1) The present invention uses DQN combined with MLP to construct a dynamic spectrum access model, which can enhance the processing capability of the state space and improve the model performance;

[0021] (2) Providing feedback rewards based on the actual wireless transmission quality and using them as model training data to update spectrum access strategies, so that SUs can continuously optimize their spectrum usage efficiency and achieve more efficient spectrum utilization;

[0022] (3) An interpretable method of weight analysis is added to visualize the weight distribution of the input features of the dynamic spectrum access model, reflecting the learning status of the model network during training, enhancing the transparency of the dynamic spectrum access decision-making process, and realizing dynamic spectrum access based on an interpretable method. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of a cognitive network according to an embodiment of the present invention;

[0024] Figure 2 is a flowchart of a dynamic spectrum access method based on interpretable deep reinforcement learning according to an embodiment of the present invention;

[0025] Figure 3 A neural network structure diagram of a dynamic spectrum access model according to an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of simulation results of average success rates of different methods in a dynamic spectrum access scenario according to a specific example of the present invention;

[0027] Figure 5 A schematic diagram of simulation results of average collision rate between SUs under different methods in a dynamic spectrum access scenario according to a specific example of the present invention;

[0028] Figure 6 A schematic diagram of simulation results of average rewards under different methods in a dynamic spectrum access scenario according to a specific example of the present invention;

[0029] Figure 7 Schematic diagram of a block diagram of a dynamic spectrum access device based on interpretable deep reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and embodiments.

[0031] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention, so they have no substantial technical significance. Any modification of the structure, change of the proportion relationship or adjustment of the size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical content disclosed by the present invention. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and so on quoted in this specification are only for the convenience of description, and are not used to limit the scope of implementation. The change or adjustment of their relative relationship should also be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0032] like Figure 1 As shown, the dynamic spectrum access scenario of the present invention is composed of a cognitive network, including: a main network composed of N primary users (PU) and a slave network composed of L secondary users (SU). In order to avoid mutual interference between PUs, the same number of orthogonal wireless channels and PUs are set to ensure that each PU transmits data on a unique wireless channel. When a SU collides with a PU, the corresponding PU will send a broadcast warning to the SU, which includes the channel on which the collision occurred, so that the corresponding SU on the channel can be aware of the collision and reduce interference to the PU. All wireless channels are shared by all SUs, and the SU selects a suitable channel to enter and transmit data according to its spectrum access strategy.

[0033] like Figure 1 As shown, two-dimensional coordinates are set to represent the location of each transmitter and receiver. and represents the transmitter and receiver of the i-th SU, and Denote the transmitter and receiver of the jth PU, where i∈{1,2,...,L},j∈{1,2,...,N}. Based on the corresponding coordinates, the distance d of the signal link can be calculated ii for: When the jth PU and the kth SU use the same wireless channel, the propagation distances of the interference signals are: Where k∈{1,2,...,L},k≠i.

[0034] Based on the propagation distance d, the WINNERⅡ channel model, which can accurately model the path loss in different environments, is used to calculate the signal path loss PL(d, f c ):

[0035]

[0036] where f c represents the carrier frequency of the wireless channel, represents the path loss at the reference distance, A W and B w Respectively represent the path loss index and the path loss frequency dependence. Based on this, the path loss PL (d ii ,f c ), the path loss of the interference signal PL(d ji ,f c ) and PL(d ki ,f c ). Assuming that there is a strong line-of-sight path between the transmitter and the receiver in the scene, the channel model h in the scene of the present invention is derived by using the Rician channel model that can simulate an environment with a strong line-of-sight component:

[0037]

[0038] in is determined by the path loss, κ is the K factor, which represents the ratio of the received signal power on the strong line-of-sight path to that on the scattering path, θ~U(0,1) represents the phase of the arriving signal on the strong line-of-sight path, which is uniformly distributed between 0 and 1, and CN(·) represents a circularly symmetric complex Gaussian random variable.

[0039] Based on the above representation, the signal-to-noise ratio (SINR) of the signal received by the i-th SU receiver can be expressed as:

[0040]

[0041] Among them, |h ii | 2 ,|h ji | 2 and |h ki | 2 represent the channel gains between transmitter i and receiver i, transmitter j and receiver i, and transmitter k and receiver i respectively; p jj , p ij and p kj denote the transmission power of the jth PU, the ith SU and the kth SU on the jth wireless channel respectively; B and N0 denote the channel bandwidth and the noise spectral density respectively. The channel capacity of the i-th SU can be obtained by C i =B·log2(1+SINR i / Γ), where Γ represents the gap between the channel capacity and the actual modulation and coding scheme.

[0042] The present invention is described below in conjunction with specific embodiments. Figure 1The dynamic spectrum access process of the cognitive network is shown.

[0043] Figure 2 FIG. 1 is a flow chart of a dynamic spectrum access method based on interpretable deep reinforcement learning according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0044] S1, build a dynamic spectrum access model, which is a DQN model based on MLP.

[0045] Deep Q-learning network (DQN) is a key branch in the field of machine learning. The intelligent agent continuously learns and adjusts strategies through interaction with the environment, and formulates the optimal strategy to obtain the maximum long-term cumulative reward. Compared with other machine learning methods, reinforcement learning is particularly suitable for complex problems in dynamically changing environments and is suitable for distributed dynamic spectrum access scenarios. In the distributed dynamic spectrum access scenario, each SU acts as an independent intelligent agent, its perception decision is regarded as an action, the perception result constitutes the observation state, and the formulated perception strategy is the action guideline it follows.

[0046] Q-learning is a model-free method whose agent does not require prior knowledge about the environment and can handle various problems and environments more flexibly. Q-learning maximizes the expected reward value by selecting the best action among all possible actions in a specific state. The algorithm uses the Bellman equation to iteratively update the Q value:

[0047]

[0048] Where α∈(0,1) is the learning rate, which is used to measure the magnitude of the update; γ∈[0,1] is the discount factor, which is mainly used to strengthen the recent benefits, weaken the long-term benefits, and ensure the convergence of the Q function; Q(s t ,a t ) means in state s t Take action t The expected reward is an estimated value; r t+1 Indicates that in state s t Take action t actual rewards; Indicates that in state s t+1 The maximum Q value obtained under is the target value, is the estimated value Q(s t ,a t ) is the target to be approached, the estimated value and the target value gradually approach each other until the difference approaches 0, Q(s t ,a t ) will no longer change, and a converged Q table is obtained.

[0049] In actual scenarios, the state space faced by reinforcement learning tasks is often continuous, and as the number of states increases, the state space continues to expand, making the Q table in Q-learning too large to train and unable to store the corresponding value function. In order to solve the above problem, the deep Q learning network DQN approximates the function Q(s,a); Q(s,a;w) instead of the function Q(s,a), which is generated by a neural network, where w is the parameter of the neural network training.

[0050] MLP is a feedforward artificial neural network used to map a set of input vectors to a set of output vectors. MLP can be viewed as a directed graph consisting of multiple node layers, each of which is fully connected to the next layer. In addition to the input node, each node is a neuron with a nonlinear activation function. MLP overcomes the limitation of perceptrons that they cannot recognize linearly inseparable data by using the back-propagation algorithm for supervised learning training. Compared with a single-layer perceptron, MLP expands from one node to multiple nodes at the output end, and adds a hidden layer between the input and output ends. In a typical feedforward network structure, information is passed from the input layer to each hidden layer, and then to the output layer, layer by layer. The hidden layer implements a nonlinear mapping of the input space, while the output layer implements linear classification. In this way, MLP is able to learn nonlinear mapping and linear discriminant functions simultaneously. During the training process of MLP, the weights are adjusted by the back-propagation algorithm to minimize the prediction error, so that the network can accurately map the input to the desired output. This structure enables MLP to perform well in dealing with complex nonlinear problems and is widely used in fields such as classification, regression, and pattern recognition.

[0051] In the present invention, in dynamic spectrum access, DQN is combined with MLP to build a model to enhance the processing capability of the state space. Figure 3 The figure is a neural network structure diagram of the dynamic spectrum access model, which consists of an input layer, a hidden layer and an output layer.

[0052] S2, the spectrum sensing results of SU in the cognitive network are used as the input data of the dynamic spectrum access model, and the Q values ​​corresponding to different SU actions output by the dynamic spectrum access model are obtained. The cognitive network consists of a master network composed of N primary users PU and a slave network composed of L secondary users SU.

[0053] Specifically, Figure 3 As shown in the figure, the spectrum sensing result S(t) of SU is used as the input data of the input layer, then passes through the nonlinear calculation of the hidden layer, and finally outputs the Q value corresponding to different actions of SU through the output layer.

[0054] S3, controls the SU to access different channels according to the Q value, and provides the feedback reward of the SU on each channel to the dynamic spectrum access model to update the dynamic spectrum access model.

[0055] Specifically, the spectrum access strategy is jointly determined by DQN and spectrum sensing results, and SU accesses the wireless channel according to this spectrum access strategy to perform corresponding data transmission. In addition, the SU receiver provides feedback rewards based on the actual wireless transmission quality. These reward information are then stored in the SU transmitter and used as training data for the dynamic spectrum access model to update the spectrum access strategy. In this way, SU can continuously optimize its spectrum utilization efficiency and achieve more efficient spectrum utilization.

[0056] In the model, each wireless channel is occupied by a PU. Each PU has two states: Active and Inactive. Active means that the PU is transmitting data and the SU cannot access the corresponding wireless channel; Inactive means that the PU is in an inactive state, and the SU is allowed to access the corresponding wireless channel for data transmission. The active state of the PU is described as a two-state Markov chain. The transition probability P on the nth wireless channel is n for:

[0057]

[0058] where p bq Here b represents the current state and q represents the next state.

[0059] At the beginning of each time slot, each SU needs to perform spectrum sensing on all channels to detect the channel status. The sensing result at time slot t is:

[0060] S(t)=[S 1 (t),...,S L (t)] (6)

[0061] in, is an N-dimensional vector, It represents the perception result of the lth SU on the nth channel. Usually, there will be a certain error in spectrum perception. Let the correct probability of the lth SU perceiving the nth channel be:

[0062]

[0063] Among them, T n (t) is the true state of the nth channel, is the perception error probability. The channel switching probability and the perception error probability are unknown to SU. l (t) is the only known information of SU, which represents the observed state in the environment and the input of DQN.

[0064] According to the sensing results, each SU decides to access a channel or remain stationary. The action of the lth SU is expressed as:

[0065] a l (t)∈{0,...,N}(8)

[0066] where a l (t) = n (n>0) means that the lth SU accesses the nth channel in time slot t, a l (t)=0 means that the lth SU does not access any channel in time slot t.

[0067] After the SU transmitter enters the channel, the channel state changes according to the Markov chain, and the SU receiver feeds back the received signal-to-noise ratio to the transmitter. Based on the state and position of the SU and PU, the present invention considers four situations to determine the feedback reward:

[0068] (1) If the channel accessed by the SU is not occupied by other SUs or PUs, the SU will not receive any interference and the data transmission rate that can be achieved will be As a reward for feedback.

[0069] (2) If the channel accessed by the SU is occupied by a PU, a collision will occur with the PU. At this time, the PU will broadcast a warning signal to all SUs and set the feedback reward for this action to -C (C>0).

[0070] (3) If multiple SUs access a channel, collisions will occur between SUs. At this time, the data transmission rate can still be achieved. As a reward for feedback.

[0071] (4) If the SU does not access any channel, the reward will be fed back Set to 0.

[0072] Based on the above four situations, the present invention obtains the feedback reward of SU on each channel according to the following formula:

[0073]

[0074] Among them, r l (t+1) represents the reward function of the lth SU at time slot (t+1), C is the channel capacity, represents the signal-to-noise ratio of the signal received by the lth SU receiver on the nth wireless channel, Γ represents the gap between the channel capacity and the actual modulation and coding scheme, and l and n are positive integers.

[0075] The present invention adopts a distributed dynamic spectrum access strategy. The SU perception results and access decision information will not be shared with other SUs. Each SU has its own independent DQN to make access strategies. Without knowing the transition probability of the channel state and the perception error, its only input is the perception result of the SU. The corresponding access strategy can only be formulated based on the SINR fed back after access, so as to maximize the cumulative discount reward. The cumulative discount reward R l for:

[0076]

[0077] Where γ∈[0,1] is the discount factor.

[0078] S4, visualizes the weight distribution of the input features of the dynamic spectrum access model to reflect the dynamic spectrum access model’s cognition and preference for different channels.

[0079] Specifically, Figure 3 As shown in the figure, at the end of the dynamic spectrum access, weight analysis is added. The distribution of weights can reflect the learning situation during the training process of the dynamic spectrum access model. The area with larger weight values ​​represents which channel action decisions selected by SU in the neural network are given more attention.

[0080] By analyzing and visualizing the weight distribution in the dynamic spectrum access model, the present invention reveals how the model makes trade-offs between different input features and potential actions to reflect the dynamic spectrum access model's cognition and preference for different channels.

[0081] In a specific embodiment of the present invention, a heat map is generated to visualize the weight distribution of input features of the dynamic spectrum access model. Thus, it can be observed that some rows or columns have larger weights, implying the importance of these features or actions in the decision-making.

[0082] The specific steps of visualization are as follows:

[0083] In the dynamic spectrum access model, the weight matrix can reflect the relationship between different input features and actions. The following formula is used to represent the elements of the weight matrix:

[0084]

[0085] Among them, W ij Indicates the influence of the i-th input feature on the j-th action, is the Q-value function of taking action a in state s, x i is the i-th input feature.

[0086] It can be understood that the weight matrix can be obtained according to the elements of the weight matrix, and the weight distribution of the input features can be obtained according to the weight matrix.

[0087] During the training of the dynamic spectrum access model, the weight update is completed through the gradient descent algorithm. The update formula is as follows:

[0088]

[0089] Among them, W new is the updated weight, W old is the weight before updating, α is the learning rate, and L is the loss function.

[0090] By analyzing and visualizing the above weight update process, we can observe the changes and convergence of weight distribution, which not only reflects the neural network's cognition and preference for different channels, but also guides the further optimization of the model. Furthermore, the sparsity or concentration of weights can indicate potential overfitting or underfitting problems, thus providing direction for the regularization strategy of the model.

[0091] In order to verify the effect of the dynamic spectrum access method based on interpretable deep reinforcement learning proposed in the present invention, the inventors conducted relevant simulations and analyses, as follows:

[0092] First, the spatial range of SU and PU is set to 150m×150m, a medium-sized wireless network environment, such as a campus area, a small commercial area, etc. The distance between the SU transmitter and the corresponding receiver is 20~40m, simulating the actual situation in a dense user environment to ensure sufficient user distribution density while avoiding excessive interference. For the WINNERⅡ channel model, set f c =5GHz, A W =22.7, B w =20, f c represents the carrier frequency of the wireless channel, represents the path loss at the reference distance, A W and B w denote the path loss exponent and the path loss frequency dependence, respectively. For the Rician channel model, set κ = 8, σ 2 The path loss is determined by the WINNERⅡ channel model. The signal-to-noise ratio received by the SU receiver has a bandwidth of B = 1MHz and a noise spectral density of N0 = 10 -14.7 (mW / Hz), the transmission power is 20mW, and the PU transmission power is 40mW. The transition probability p in the two-state Markov chain 11 and p 00 We randomly select from the uniform distribution of [0.7, 1] and [0, 0.3], respectively, and based on this, we calculate p 10 =1-p 11 and p 01=1-p 00 .

[0093] The comparison algorithm selected a baseline method, a Q-learning algorithm and the MLP+DQN algorithm (the above-mentioned dynamic spectrum access model) of the present invention. The baseline method selects the channel with the largest instant reward and requires prior knowledge of the channel switching probability and the perception error probability information. The lth SU first calculates the channel state probability:

[0094]

[0095] in, is the perception result of the lth SU on the nth channel, represents the perception error probability of the lth SU on the nth channel, and the baseline method calculates the expected immediate reward of the nth channel:

[0096]

[0097] Therefore, the baseline method of this paper is that SU accesses R n (t) Maximum channel.

[0098] The number of SUs is set to 8, the number of channels is set to 10, the number of PUs is the same as the number of channels, and the penalty for collision between SU ​​and PU is set to -2. The number of neurons in the input layer and hidden layer of MLP is set to 64, and the learning rate of all methods is set to 0.01. In order to verify the effectiveness of the method, the present invention designs two DQN+MLP models, namely DQN+MLP with one hidden layer and DQN+MLP2 with two hidden layers.

[0099] Figure 4 , Figure 5 and Figure 6 The simulation results of the average success rate, average collision rate between SUs and average reward under different methods in a multi-user multi-channel dynamic spectrum access scenario are given respectively. It can be seen that although the baseline method (Myopic) has sufficient prior knowledge, it cannot enable SU to learn the strategies of other SUs, resulting in a stable state with low performance; in contrast, the learning-based method can draw experience from the interaction between the environment and the user, and the performance is improved compared with the baseline method. It can be seen from the figure that compared with other algorithms, the algorithm proposed in the present invention performs best in terms of average success rate, average collision rate between SUs and average reward. In addition, it can be found that the performance of DQN+MLP2 with an additional hidden layer is slightly improved compared to DQN+MLP with only one hidden layer.

[0100] In summary, according to the dynamic spectrum access method based on interpretable deep reinforcement learning in an embodiment of the present invention, DQN combined with MLP is used to construct a dynamic spectrum access model, which can enhance the processing capability of the state space and improve the model performance; feedback rewards are provided according to the actual wireless transmission quality and used as model training data to update the spectrum access strategy, so that SU can continuously optimize its spectrum utilization efficiency and achieve more efficient spectrum utilization; an interpretable method of weight analysis is added to visualize the weight distribution of the input features of the dynamic spectrum access model, reflecting the learning situation in the model network training process, enhancing the transparency of the dynamic spectrum access decision-making process, and realizing dynamic spectrum access based on an interpretable method.

[0101] Corresponding to the above-mentioned interpretable dynamic spectrum access method, the present invention also proposes an interpretable dynamic spectrum access device.

[0102] Figure 7 is a block diagram of a dynamic spectrum access device based on interpretable deep reinforcement learning according to an embodiment of the present invention. Figure 7 As shown, the device includes: a construction module 10, an acquisition module 20, an update module 30 and a visualization module 40.

[0103] Among them, the construction module 10 is used to construct a dynamic spectrum access model, which is a DQN model based on MLP; the acquisition module 20 is used to use the spectrum perception results of the SU in the cognitive network as the input data of the dynamic spectrum access model to obtain the Q values ​​corresponding to different SU actions output by the dynamic spectrum access model, and the cognitive network is composed of a main network composed of N main users PU and a slave network composed of L secondary users SU; the update module 30 is used to control the SU to access different channels according to the Q value, and provide the feedback reward of the SU on each channel to the dynamic spectrum access model to update the dynamic spectrum access model; the visualization module 40 is used to visualize the weight distribution of the input features of the dynamic spectrum access model to reflect the dynamic spectrum access model's cognition and preference for different channels.

[0104] According to one embodiment of the present invention, the updating module 30 obtains the feedback reward of the SU on each channel according to the following formula:

[0105]

[0106] Among them, r l (t+1) represents the reward function of the lth SU at time slot (t+1), C is the channel capacity, represents the signal-to-noise ratio of the signal received by the lth SU receiver on the nth wireless channel, Γ represents the gap between the channel capacity and the actual modulation and coding scheme, and l and n are positive integers.

[0107] According to an embodiment of the present invention, the visualization module 40 specifically visualizes the weight distribution of the input features of the dynamic spectrum access model by generating a heat map.

[0108] According to one embodiment of the present invention, the visualization module 40 uses the following formula to obtain the weight distribution of the input features:

[0109]

[0110] Among them, W ij Indicates the influence of the i-th input feature on the j-th action, is the Q-value function of taking action a in state s, x i is the i-th input feature.

[0111] According to an embodiment of the present invention, during the dynamic spectrum access model training process, weight updating is completed by a gradient descent algorithm.

[0112] In summary, according to the dynamic spectrum access device based on interpretable deep reinforcement learning in the embodiment of the present invention, DQN combined with MLP is used to construct a dynamic spectrum access model, which can enhance the processing capability of the state space and improve the model performance; feedback rewards are provided according to the actual wireless transmission quality and used as model training data to update the spectrum access strategy, so that SU can continuously optimize its spectrum utilization efficiency and achieve more efficient spectrum utilization; an interpretable method of weight analysis is added to visualize the weight distribution of the input features of the dynamic spectrum access model, reflecting the learning situation in the model network training process, enhancing the transparency of the dynamic spectrum access decision-making process, and realizing dynamic spectrum access based on an interpretable method.

[0113] The present invention and its embodiments are described schematically above, and the description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by it and designs a structural method and an embodiment similar to the technical solution without creativity without departing from the purpose of the invention, they shall all fall within the protection scope of the present invention.

Claims

1. A dynamic spectrum access method based on interpretable deep reinforcement learning, characterized in that: The following steps are involved: Constructing a dynamic spectrum access model, wherein the dynamic spectrum access model is a DQN model based on MLP; The spectrum sensing result of the SU in the cognitive network is used as input data of the dynamic spectrum access model, and the Q value corresponding to different SU actions output by the dynamic spectrum access model is obtained, wherein the cognitive network is composed of a master network composed of N primary users PU and a slave network composed of L secondary users SU; Controlling the SU to access different channels according to the Q value, and providing the feedback reward of the SU on each channel to the dynamic spectrum access model to update the dynamic spectrum access model; The weight distribution of the input features of the dynamic spectrum access model is visualized to reflect the cognition and preference of the dynamic spectrum access model for different channels.

2. The dynamic spectrum access method based on interpretable deep reinforcement learning according to claim 1, characterized in that: The feedback reward of the SU on each channel is obtained according to the following formula: Among them, r l (t+1) represents the reward function of the lth SU at time slot (t+1), C is the channel capacity, represents the signal-to-noise ratio of the signal received by the lth SU receiver on the nth wireless channel, Γ represents the gap between the channel capacity and the actual modulation and coding scheme, and l and n are positive integers.

3. The dynamic spectrum access method based on interpretable deep reinforcement learning according to claim 1, characterized in that: The weight distribution of the input features of the dynamic spectrum access model is visualized by generating a heat map.

4. The dynamic spectrum access method based on interpretable deep reinforcement learning according to claim 1, characterized in that: The following formula is used to obtain the weight distribution of input features: Among them, W ij Indicates the influence of the i-th input feature on the j-th action, is the Q-value function of taking action a in state s, x i is the ith input feature.

5. The dynamic spectrum access method based on interpretable deep reinforcement learning according to claim 4 is characterized in that: During the training process of the dynamic spectrum access model, weight update is completed through the gradient descent algorithm.

Images