A dynamic spectrum adaptive access method for dynamically switching interference modes

Abstract

The present invention discloses a dynamic spectrum adaptive access method for coping with dynamic switching of interference modes. A sensor is first configured for a dynamic spectrum access device to sense the switching of the interference mode of a jammer. Before sensing the switching of the interference mode, the dynamic spectrum access device first collects a sufficient amount of source experience samples and stores them in a source memory bank. When the sensor senses the switching of the interference mode, the device continues to collect a small amount of target experience samples and stores them in a target memory bank. Subsequently, the device leaves the spectrum environment and carries out an offline pre-training process to learn an initial dynamic spectrum access strategy for the new interference mode. Finally, the device re-accesses the spectrum environment and further optimizes based on the initial dynamic spectrum access strategy until the strategy converges, thereby obtaining an optimal dynamic spectrum access strategy suitable for the new interference mode.

Description

Technical Field

[0001] The present invention belongs to the technical field of dynamic spectrum access, and more specifically, relates to a dynamic spectrum adaptive access method for coping with dynamic switching of interference modes. Background Art

[0002] With the continuous development of wireless communication technology, limited spectrum resources have been allocated to more and more frequency users, resulting in increasingly tight spectrum resources available to users. This tense situation is mainly caused by the traditional static spectrum allocation method, which is achieved by allocating different frequency bands to frequency users. Under this spectrum allocation method, only users who are granted the right to use can use the authorized frequency band.

[0003] Dynamic Spectrum Access (DSA) is an opportunistic spectrum access technology that flexibly utilizes spectrum holes of authorized users to expand frequency usage. Figure 1 The DSA technology was demonstrated to be an opportunistic frequency utilization method. DSA can achieve near real-time spectrum management, giving cognitive users the ability to access and obtain the required spectrum resources, thereby making full use of limited spectrum resources in the time and space domains, and ultimately greatly improving spectrum utilization efficiency without causing harmful interference to primary users. DSA technology is one of the effective ways to solve the shortage of frequency utilization and is also a research hotspot in the current wireless communication field.

[0004] Currently, dynamic spectrum access technology can be applied to the field of anti-interference communications. Most existing work is based on a static and stable interference environment. However, in actual applications, due to the influence of factors such as time and space, the interference signal pattern in the spectrum environment often changes dynamically. The current mainstream dynamic spectrum access technology based on reinforcement learning is often difficult to adapt to the dynamic changes of the interference environment, which not only reduces the stability of spectrum access efficiency, but also greatly reduces the user's communication experience. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a dynamic spectrum adaptive access method for dynamically switching interference modes. By introducing an offline pre-training algorithm based on a memory library, cognitive users can effectively transfer knowledge related to the source electromagnetic environment to the learning process of the target dynamic spectrum access strategy, which can improve the learning speed, stability and strategy performance of the dynamic spectrum access strategy under the new interference mode.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a dynamic spectrum adaptive access method for dynamically switching interference modes, characterized by comprising the following steps:

[0007] (1) Environment construction: Build a dynamic spectrum access user D based on the DQN algorithm and configure a sensor for user D to sense whether the interference pattern of the interference signal has changed;

[0008] (2) Obtaining the source memory bank and the target memory bank;

[0009] (2.1) Obtain the observation values ​​of all channels: User D observes all channels and obtains the state observation values ​​of all channels at the current time t S(t) = {s1(t), s2(t), …, s k (t),…,s K (t)}, where s k (t) represents the state observation value of the kth channel obtained by user D at time t, and K represents the number of channels;

[0010] (2.2) Action selection: The user obtains the Q value of all channel selection actions under the current channel state observation value S(t) through the evaluation neural network according to the current channel state observation value S(t), and selects the next action a(t) according to the ε-greedy strategy;

[0011] (2.3) Obtaining the reward value: After executing the channel selection action a(t), user D will obtain the reward value r(t+1) at time t+1. At the same time, the channel state will be transferred to the new state of the next step, and the observation value S(t+1) of all channels at time t+1 will be obtained;

[0012] (2.4) Store in the memory bank: store all channel observation values ​​S(t) at the current moment, channel selection action a(t), reward value r(t+1), and observation values ​​S(t+1) of all channels at the next moment in the form of experience sample x(t) = {S(t), a(t), r(t+1), S(t+1)} in the memory bank;

[0013] (2.5) Obtain source memory library: When the sensor does not sense the interference signal and its interference pattern changes, repeat steps (2.1) to (2.4) for a total of M times, and the memory library that stores M experience samples is called source memory library X source ={x(t)}={(S(t),a(t),r(t+1),S(t+1))}, t=1,2,…,M;

[0014] (2.6) Obtain the target memory library: When the sensor senses that the interference pattern of the interference signal has changed, repeat steps (2.1) to (2.4) N times in total. The memory library that stores N experience samples is called the target memory library X. target={x(t)}={(S(t),a(t),r(t+1),S(t+1))}, t=M+1,M+2,…,M+N;

[0015] (3) Offline pre-training initialization: The source memory library X source With target memory X target Merge into total memory X = {X souirce ,X target}={x(t)}={S(t),a(t),r(t+1),S(t+1)}, t=1,2,…,N+M, and then initialize a weight p for each empirical sample x(t) t , the weight initialization formula is as follows:

[0016]

[0017] Among them, κ≥1, represents the importance factor of the empirical sample;

[0018] (4) Perform offline pre-training process:

[0019] (4.1) Sample extraction: User D extracts batch_size experience samples from the total memory library X according to the sample weights. The extracted set of experience samples is used as X batch The sampling formula is as follows:

[0020]

[0021] Among them, P(l) represents the probability of extracting the experience sample x(l), and the extracted experience sample set X batch It is expressed as:

[0022] X batch ={x(l)}={S(l),a(l),r(l+1),S(l+1)},l∈{1,2,...,M+N},crad(X batch ) = batch_size

[0023] Among them, crad(X batch ) = batch_size represents X batch The total number of experience samples in is batch_size;

[0024] (4.2) Strategy update: based on X batch Perform a round of evaluation of the neural network parameters θ update, and based on the custom update step size Update the network parameters of the target neural network

[0025] First, X batchFor each empirical sample x(l), a normalized weight correction coefficient ω is calculated. l , the formula is described as follows:

[0026]

[0027] Among them, β represents a hyperparameter with a value range of 0 to 1; Represents X batch All empirical samples in correspond to ((M+N)·P(l)) -β The maximum value of the value;

[0028] Calculate X batch The loss function value δ of each empirical sample x(l) l ;

[0029]

[0030] Where γ is the discount factor, which ranges from 0 to 1. target is the Q reality function obtained by the target neural network, are the parameters of the target neural network. Q eval is the Q estimation function obtained by evaluating the neural network, and θ is the parameter of the evaluation neural network. The table can make Q target The channel with the largest value selects action a.

[0031] Then X batch For each empirical sample x(l), a network weight adjustment is calculated and accumulated to obtain the total network weight adjustment Δ, which is calculated as follows:

[0032]

[0033] in, It means to evaluate the gradient of the parameter θ of the neural network;

[0034] Finally, the neural network parameter θ is updated based on the learning rate η, and the update formula is θ←θ+ηΔ. The total network weight adjustment Δ is reset to zero. The step size copies the evaluation neural network parameters θ to the target neural network, that is,

[0035] (4.3) Weight adjustment: adjust the weight of x(l) in the total memory library X and renormalize it;

[0036] When X batch The x(l) in the source memory X souirce When , the corresponding weight in X is updated as:

[0037] p l=1 / |δ l |

[0038] When X batch The x(l) in the target memory is from X target When , the corresponding weight in X is updated as:

[0039] p l =κ·|δ l |

[0040] After updating all weights, all experience samples in X are re-normalized. The normalization formula is as follows:

[0041]

[0042] Among them, p t represents the normalized weight of the t-th empirical sample in X;

[0043] (4.4) Repeat steps (4.1) to (4.3) until the evaluation neural network converges, thereby completing the offline pre-training process and obtaining the initial dynamic spectrum access strategy;

[0044] (5) The initial dynamic spectrum access strategy is optimized online by using the online strategy learning method of the DQN algorithm until the dynamic spectrum access strategy converges and the dynamic spectrum access strategy under the new interference mode is obtained.

[0045] The object of the invention of the present invention is achieved in this way:

[0046] The present invention discloses a dynamic spectrum adaptive access method for coping with dynamic switching of interference modes. A sensor is first configured for a dynamic spectrum access device to sense the switching of the interference mode of a jammer. Before sensing the switching of the interference mode, the dynamic spectrum access device first collects a sufficient amount of source experience samples and stores them in a source memory bank. When the sensor senses the switching of the interference mode, the device continues to collect a small amount of target experience samples and stores them in a target memory bank. Subsequently, the device leaves the spectrum environment and carries out an offline pre-training process to learn an initial dynamic spectrum access strategy for the new interference mode. Finally, the device re-accesses the spectrum environment and further optimizes based on the initial dynamic spectrum access strategy until the strategy converges, thereby obtaining an optimal dynamic spectrum access strategy suitable for the new interference mode.

[0047] At the same time, the dynamic spectrum adaptive access method for dynamically switching interference modes in the present invention also has the following beneficial effects:

[0048] (1) The present invention configures a sensor for the dynamic spectrum access device to sense the switching of the jammer's interference mode, which can provide a reaction basis for the device to quickly adapt to the new interference mode.

[0049] (2) The present invention can collect source experience samples during the data transmission process before the interference mode is switched, and after the interference mode is switched, only a small number of target experience samples need to be obtained. Therefore, the acquisition of experience samples only takes a very short time. The initial dynamic spectrum access strategy based on the experience samples combined with the offline pre-training process can achieve good initial performance in the new interference mode.

[0050] (3) The present invention introduces a special weight adjustment strategy, which can screen out suitable source experience samples from the source memory library and apply them to the learning process of the pre-training strategy, combined with the weight adjustment strategy for the target memory library, and can significantly improve the performance and convergence speed of the pre-training strategy.

[0051] (4) Compared with directly accessing a new interference environment to learn an online dynamic spectrum access strategy, the offline pre-training process consumes very little time to obtain a dynamic spectrum access strategy with the same performance. The present invention first quickly obtains a dynamic spectrum access strategy with good performance through an offline pre-training process, and then re-accesses the spectrum to optimize the strategy until convergence. This can greatly reduce the time consumption for obtaining the optimal dynamic spectrum access strategy under the new interference mode, thereby greatly improving the speed and performance of the device's adaptation to the new interference mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a related schematic diagram of dynamic spectrum access technology.

[0053] Figure 2 The present invention is a flow chart of a dynamic spectrum adaptive access method for dynamically switching interference modes.

[0054] Figure 3 It is a related schematic diagram of anti-interference communication under the change of interference pattern.

[0055] Figure 4 It is a schematic diagram of the change of interference pattern.

[0056] Figure 5 It is a schematic diagram of the weight adjustment of the experience sample.

[0057] Figure 6 It is a schematic diagram of adaptive indicators under the new interference mode.

[0058] Figure 7 This is a diagram showing the spectrum utilization results after the simulation interference mode is switched.

[0059] Figure 8 This is a graph showing the spectrum access conflict rate results after the simulation interference mode is switched. DETAILED DESCRIPTION

[0060] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0061] Example

[0062] In order to explain the technical solution of the present invention in detail, further elaboration is given below in conjunction with the accompanying drawings and specific embodiments of the specification.

[0063] Figure 2 The present invention is a flow chart of a dynamic spectrum adaptive access method for dynamically switching interference modes.

[0064] In this embodiment, if Figure 2 As shown, the present invention provides a dynamic spectrum adaptive access method for dynamically switching interference modes, comprising the following steps:

[0065] (1) Environment construction:

[0066] In this embodiment, consider Figure 3 The anti-interference communication scenario of the dynamic spectrum access device shown in the figure is implemented through the Python language, and the system simulation of this embodiment is implemented. The DQN algorithm network structure of the dynamic spectrum access user D is built based on the Tensorflow deep learning framework. Dynamic spectrum access user D uses the communication link to transmit data, and the jammer sends interference signals on the communication link to hinder the data transmission process of user D. Assume that there are K channels in the communication link, K=8, and the channel has two working states: 0 (idle) and 1 (occupied). When the interference signal exists in a certain channel, the channel is in state 1, otherwise the channel is in state 0. Assume that the jammer has two different interference modes, namely the sweeping interference mode and the comb interference mode, and the jammer will dynamically switch the interference mode in the current communication scenario. As Figure 4 As shown in the figure, the sweep interference mode and comb interference mode of the jammer and the interference mode switching scenario are shown, where the sweep interference mode corresponds to the interference mode before switching in the figure. After the interference mode is switched, the jammer will send interference signals in the communication link in the comb interference mode. In order to achieve the purpose of anti-interference communication, user D will learn the dynamic spectrum access strategy to use the communication link. Figure 1 This demonstrates the core idea of ​​the dynamic spectrum access strategy, which is to dynamically and opportunistically access and use idle channels. When the interference pattern changes, user D needs to relearn the dynamic spectrum access strategy for the new interference pattern.

[0067] In this embodiment, user D is a dynamic spectrum access device based on the DQN algorithm and has a sensor that can sense whether the interference pattern of the interference signal changes. The DQN network parameters of user D are randomly initialized, including an evaluated neural network and a target neural network.

[0068] (2) Obtaining the source memory bank and the target memory bank;

[0069] (2.1) Obtain the observation values ​​of all channels: User D observes all channels and obtains the state observation values ​​of all channels at the current time t S(t) = {s1(t), s2(t), …, s k (t),…,s K (t)}, where s k (t) represents the state observation value of the kth channel obtained by user D at time t, and K represents the number of channels;

[0070] (2.2) Action selection: The user obtains the Q value of all channel selection actions under the current channel state observation value S(t) through the evaluation neural network, and selects the next action a(t) according to the ε-greedy strategy. The user's channel selection action is any channel from channel 1 to channel K, and the exploration rate ε of ε-greedy is set to 1;

[0071] (2.3) Obtaining the reward value: After executing the channel selection action a(t), user D will obtain the reward value r(t+1) at time t+1. At the same time, the channel state will be transferred to the new state of the next step, and the observation value S(t+1) of all channels at time t+1 will be obtained;

[0072] (2.4) Store in the memory bank: store all channel observation values ​​S(t) at the current moment, channel selection action a(t), reward value r(t+1), and observation values ​​S(t+1) of all channels at the next moment in the form of experience sample x(t) = {S(t), a(t), r(t+1), S(t+1)} in the memory bank;

[0073] (2.5) Obtain source memory library: When the sensor does not sense the interference signal and its interference pattern changes, repeat steps (2.1) to (2.4) for a total of M times, M = 2048, and the memory library that stores M experience samples is called source memory library X source ={x(t)}={(S(t),a(t),r(t+1),S(t+1))}, t=1,2,…,M;

[0074] (2.6) Obtain the target memory library: At this time, the interference mode of the jammer is switched from the sweep interference mode to the comb interference mode. When the sensor senses that the interference mode of the interference signal has changed, repeat steps (2.1) to (2.4) for a total of N times, N = 512, and the memory library that stores N experience samples is called the target memory library X target ={x(t)}={(S(t),a(t),r(t+1),S(t+1))}, t=M+1,M+2,…,M+N;

[0075] (3) Offline pre-training initialization: First, the source memory library X source With target memory X target Merge into total memory X = {X souirce ,X target}={x(t)}={S(t),a(t),r(t+1),S(t+1)}, t=1,2,…,N+M, and then initialize a weight p for each empirical sample x(t) t , the weight initialization formula is as follows:

[0076]

[0077] Among them, κ≥1, represents the importance factor of the empirical sample, and here κ=1;

[0078] (4) Perform offline pre-training process:

[0079] (4.1) Sample extraction: User D extracts batch_size=20 experience samples from the total memory library X according to the sample weights. The extracted set of experience samples is used as X batch The sampling formula is as follows:

[0080]

[0081] Among them, P(l) represents the probability of extracting the experience sample x(l), and the extracted experience sample set X batch It is expressed as:

[0082] X batch ={x(l)}={S(l),a(l),r(l+1),S(l+1)},l∈{1,2,...,M+N},crad(X batch ) = batch_size

[0083] Among them, crad(X batch ) = batch_size represents X batch The total number of experience samples in is batch_size;

[0084] (4.2) Strategy update: based on X batch Perform a round of evaluation of the neural network parameters θ update, and based on the custom update step size Update the network parameters of the target neural network First, X batch For each empirical sample x(l), a normalized weight correction coefficient ω is calculated. l , the formula is described as follows:

[0085]

[0086] Among them, β represents a hyperparameter with a value range of 0 to 1, and here β = 0.9; Represents X batch All empirical samples in correspond to ((M+N)·P(l)) -β The maximum value of the value;

[0087] Calculate X batch The loss function value δ of each empirical sample x(l) l ;

[0088]

[0089] Where γ is the discount factor, which ranges from 0 to 1. Here, the value of γ is 0.95. target is the Q reality function obtained by the target neural network, are the parameters of the target neural network. Q eval is the Q estimation function obtained by evaluating the neural network, and θ is the parameter of the evaluation neural network. The table can make Q target The channel with the largest value selects action a.

[0090] Then X batch For each empirical sample x(l), a network weight adjustment is calculated and accumulated to obtain the total network weight adjustment Δ, which is calculated as follows:

[0091]

[0092] in It means to evaluate the gradient of the parameter θ of the neural network;

[0093] Finally, the neural network parameter θ is updated based on the learning rate η = 0.01, and the update formula is θ←θ+ηΔ. The total network weight adjustment Δ is reset to zero. The step size copies the evaluation neural network parameters θ to the target neural network, that is,

[0094] (4.3) Weight adjustment: adjust the weight of x(l) in the total memory library X and renormalize it;

[0095] Figure 5 The core idea of ​​adjusting the weights of x(l) in the total memory library X is demonstrated. By increasing the weights of experience samples with high training value to increase the probability of these samples being extracted, the speed and quality of the pre-training process can be effectively improved, thereby quickly obtaining an initial dynamic spectrum access strategy that is more in line with the new interference pattern.

[0096] When X batch The x(l) in the source memory is from X souirce When , the corresponding weight in X is updated as:

[0097] p l =1 / |δ l |

[0098] When X batch The x(l) in the target memory is from X target When , the corresponding weight in X is updated as:

[0099] p l =κ·|δ l |

[0100] After updating all weights, all experience samples in X are re-normalized. The normalization formula is as follows:

[0101]

[0102] in, represents the normalized weight of the t-th empirical sample in X;

[0103] (4.4) Repeat steps (4.1) to (4.3) until the evaluation neural network converges, thereby completing the offline pre-training process and obtaining the initial dynamic spectrum access strategy;

[0104] (5) The initial dynamic spectrum access strategy is optimized online by using the online strategy learning method of the DQN algorithm until the dynamic spectrum access strategy converges and the dynamic spectrum access strategy under the new interference mode is obtained.

[0105] Figure 6It is a schematic diagram of the adaptive indicators under the new interference mode, which is used to introduce the key indicators of the adaptive new interference mode of the present invention. It can be seen from the figure that the adaptive indicators mainly include three parts: initial performance gain, convergence speed gain, and progressive performance gain. The adaptive technology curve in the figure is the dynamic spectrum access indicator change curve corresponding to user D re-accessing the spectrum environment after the offline pre-training is completed. The non-adaptive technology curve is the dynamic spectrum access indicator change curve corresponding to user D directly accessing the spectrum environment without the offline pre-training process. The higher the initial performance gain, convergence speed performance gain, and progressive performance gain, the better the adaptive performance of the present invention in the scenario of dynamic switching of the interference mode.

[0106] Figure 7 This is a spectrum utilization result display diagram of this embodiment, which shows the change in spectrum utilization after user D re-accesses the new interference environment. Spectrum utilization refers to the proportion of spectrum usage, which is a key indicator of dynamic spectrum access technology, and can effectively reflect the ability of user D to utilize idle channels. The comparison method of this figure is a non-adaptive method, that is, the technology of the invention is not adopted, but a method of directly accessing the new interference environment for online strategy learning after sensing the switching of the interference mode. It can be seen from the figure that the technology based on the present invention has achieved an initial performance gain of 2.9% in spectrum utilization, a convergence speed gain of 335,000 training compensations, and a progressive performance gain of 0.8% compared with the technology of directly accessing the new interference environment for online strategy learning.

[0107] Figure 8 This is a result display diagram of the spectrum access conflict rate of this embodiment, which shows the change of the spectrum access conflict rate after user D re-accesses the new interference environment. The spectrum access conflict rate refers to the proportion of users successfully interfered by interference signals when accessing the channel for data transmission. The sum of the spectrum access success rate and the spectrum access conflict rate is 100%, so the anti-interference communication capability of user D can be known only from the spectrum access conflict rate. The comparison method in this figure is still a non-adaptive method, that is, the method of directly accessing the new interference environment for online strategy learning of the present invention is not adopted. It can be seen from the figure that the technology based on the present invention reduces the initial access conflict rate by 16.8% and improves the convergence speed of 335,000 training steps compared to the technology that does not adopt direct access to the new interference environment for online strategy learning, while the progressive access conflict rate is 0%. It can be seen that the present invention can effectively improve the adaptive performance of user D under the new interference mode.

[0108] Combined with the above Figure 7 and Figure 8 It can be seen from the results that the present invention D can significantly improve the adaptation speed of the new interference mode and the dynamic spectrum access performance after sensing the interference mode switching, and improve the stability in the interference mode switching scenario.

[0109] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A dynamic spectrum adaptive access method for dynamically switching interference modes, characterized in that: The following steps are involved: (1) Environment construction: Build a dynamic spectrum access user D based on the DQN algorithm and configure a sensor for user D to sense whether the interference pattern of the interference signal has changed; (2) Obtaining the source memory bank and the target memory bank; (2.1) Obtain the observation values ​​of all channels: User D observes all channels and obtains the state observation values ​​of all channels at the current time t S(t) = {s1(t), s2(t), …, s k (t),…,s K (t)}, where s k (t) represents the state observation value of the kth channel obtained by user D at time t, and K represents the number of channels; (2.2) Action selection: The user obtains the Q value of all channel selection actions under the current channel state observation value S(t) through the evaluation neural network according to the current channel state observation value S(t), and selects the next action a(t) according to the ε-greedy strategy; (2.3) Obtaining the reward value: After executing the channel selection action a(t), user D will obtain the reward value r(t+1) at time t+1. At the same time, the channel state will be transferred to the new state of the next step, and the observation value S(t+1) of all channels at time t+1 will be obtained; (2.4) Store in the memory bank: store all channel observation values ​​S(t) at the current moment, channel selection action a(t), reward value r(t+1), and observation values ​​S(t+1) of all channels at the next moment in the form of experience sample x(t) = {S(t), a(t), r(t+1), S(t+1)} in the memory bank; (2.5) Obtain source memory library: When the sensor does not sense the interference signal and its interference pattern changes, repeat steps (2.1) to (2.4) for a total of M times, and the memory library that stores M experience samples is called source memory library X source ={x(t)}={(S(t),a(t),r(t+1),S(t+1))}, t=1,2,…,M; (2.6) Obtain the target memory library: When the sensor senses that the interference pattern of the interference signal has changed, repeat steps (2.1) to (2.4) N times in total. The memory library that stores N experience samples is called the target memory library X. target ={x(t)}={(S(t),a(t),r(t+1),S(t+1))}, t=M+1,M+2,…,M+N; (3) Offline pre-training initialization: The source memory library X source With target memory X target Merge into total memory X = {X souirce ,X target }={x(t)}={S(t),a(t),r(t+1),S(t+1)}, t=1,2,…,N+M, and then initialize a weight p for each empirical sample x(t) t , the weight initialization formula is as follows: Among them, κ≥1, represents the importance factor of the empirical sample; (4) Perform offline pre-training process: (4.1) Sample extraction: User D extracts batch_size experience samples from the total memory library X according to the sample weights. The extracted set of experience samples is used as X batch The sampling formula is as follows: Among them, P(l) represents the probability of extracting the experience sample x(l), and the extracted experience sample set X batch It is expressed as: X batch ={x(l)}={S(l),a(l),r(l+1),S(l+1)},l∈{1,2,...,M+N},crad(X batch )=batch_size Among them, crad(X batch ) = batch_size represents X batch The total number of experience samples in is batch_size; (4.2) Strategy update: based on X batch Perform a round of evaluation of the neural network parameters θ update, and based on the custom update step size Update the network parameters of the target neural network First, X batch For each empirical sample x(l), a normalized weight correction coefficient ω is calculated. l ; Calculate X again batch The loss function value δ of each empirical sample x(l) l ; Then X batch For each empirical sample x(l), a network weight adjustment is calculated and accumulated to obtain the total network weight adjustment Δ, which is calculated as follows: Δ←Δ+ω l d l ·▽ θ Q eval (S(l),a(l);θ) Among them, θ Q eval (S(l), a(l); θ) means to evaluate the gradient of the parameter θ of the neural network; Finally, the neural network parameter θ is updated based on the learning rate η. The update formula is θ←θ+ηΔ. After the update is completed, the total network weight adjustment Δ is set to zero, and every The step size copies the evaluation neural network parameters θ to the target neural network, that is, (4.3) Weight adjustment: adjust the weight of x(l) in the total memory library X and renormalize it; When X batch The x(l) in the source memory X souirce When , the corresponding weight in X is updated as: p l =1 / |δ l | When X batch The x(l) in the target memory is from X target When , the corresponding weight in X is updated as: p l =κ·|δ l | After updating all weights, all experience samples in X are re-normalized. The normalization formula is as follows: in, represents the normalized weight of the t-th empirical sample in X; (4.4) Repeat steps (4.1) to (4.3) until the evaluation neural network converges, thereby completing the offline pre-training process and obtaining the initial dynamic spectrum access strategy; (5) The initial dynamic spectrum access strategy is optimized online by using the online strategy learning method of the DQN algorithm until the dynamic spectrum access strategy converges and the dynamic spectrum access strategy under the new interference mode is obtained.

2. A dynamic spectrum adaptive access method for coping with dynamic switching of interference modes according to claim 1, characterized in that: The weight correction coefficient ω l The calculation formula is: Among them, β represents a hyperparameter with a value range of 0 to 1; Represents X batch All empirical samples in correspond to ((M+N)·P(l)) -β The maximum value of the value.

3. A dynamic spectrum adaptive access method for coping with dynamic switching of interference modes according to claim 1, characterized in that: The loss function value δ l The calculation formula is: Among them, γ is the discount factor, which ranges from 0 to 1; Q target is the Q reality function obtained by the target neural network, are the parameters of the target neural network; Q eval is the Q estimation function obtained by evaluating the neural network, and θ is the parameter of the evaluation neural network; Indicates that energy is taken so that Q target The channel with the largest value selects action a.

Images