Dynamic spectrum access method based on target entropy and priority experience replay

Abstract

The application discloses a dynamic spectrum access method based on target entropy and priority experience replay, and belongs to the technical field of wireless communication and cognitive radio. The method comprises the following steps: a heterogeneous cognitive wireless network environment containing primary users, secondary users and orthogonal channels is constructed; the state, action and joint reward function of the cognitive user are defined; the policy network, evaluation network and experience replay buffer are initialized; experience samples are collected through interaction with the environment and stored; non-uniform sampling is carried out based on the priority of experience data, the deviation is corrected by using the importance sampling weight, and the network parameters are updated; the target entropy is dynamically updated, and the temperature coefficient is adaptively adjusted according to the difference between the target entropy and the current strategy entropy, so that the exploration intensity in different training stages is adaptively adjusted. The application can effectively improve the spectrum utilization rate, reduce the interference on the primary users and the conflict between users, and enhance the stability and convergence speed in the strategy learning process.

Description

Technical Field

[0001] This application relates to the fields of wireless communication and cognitive radio technology, and in particular to a dynamic spectrum access method based on target entropy and priority experience playback. Background Technology

[0002] Dynamic Spectrum Access (DSA) is an intelligent access technology that allows unlicensed devices (secondary users) to sense, identify, and opportunistically utilize idle spectrum resources in real time without interfering with licensed primary users. Its core objective is to improve spectrum utilization efficiency. DSA technology is key to improving spectrum resource utilization efficiency. In recent years, deep reinforcement learning, due to its powerful environmental awareness and decision-making capabilities, has been widely applied to dynamic spectrum access problems.

[0003] However, existing dynamic spectrum access methods based on deep reinforcement learning still have significant limitations. First, most of these methods are based on the homogeneous network assumption, which assumes that all secondary users have the same perception capabilities, access protocols, and service requirements.

[0004] However, real wireless networks are heterogeneous, containing cognitive users with varying capabilities as well as non-cognitive users using different fixed protocols (such as TDMA and ALOHA). Existing methods struggle to effectively coordinate spectrum contention and access in this complex environment.

[0005] Secondly, traditional methods often employ fixed exploration mechanisms (such as fixed exploration rates or target entropy), which cannot adaptively adjust the exploration intensity according to the training stage and the dynamics of the environment. This can lead to the agent being unable to explore sufficiently in the early stages of training and getting stuck in local optima, while in the later stages it may be unable to converge stably due to overexploration.

[0006] Finally, during training, the traditional uniform experience replay mechanism treats all interaction experiences equally, failing to prioritize learning key experiences that are more valuable for policy improvement (such as severe access conflicts or successful collaborative access), resulting in low sample utilization efficiency and slow policy learning speed.

[0007] The aforementioned problems collectively lead to technical bottlenecks in existing methods in heterogeneous dynamic spectrum access scenarios, such as policy oscillations, unstable convergence, slow convergence speed, and ultimately limited performance. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this application aims to provide a dynamic spectrum access method based on target entropy and priority experience replay, which can improve sample utilization efficiency and policy convergence speed, enhance training stability, and thus achieve efficient spectrum resource sharing in complex heterogeneous dynamic spectrum access environments.

[0009] To achieve the above objectives, this application adopts the following technical solution:

[0010] This application provides a dynamic spectrum access method based on target entropy and priority experience playback, including the following steps:

[0011] S1. Construct a heterogeneous cognitive wireless network environment containing multiple primary users, multiple secondary users, and multiple orthogonal wireless channels. The secondary users include non-cognitive users using different fixed access protocols and cognitive users based on deep reinforcement learning. The channels are divided into frequency bands corresponding to different access protocol types. The location of the cognitive users is updated using a random mobility model, and the propagation characteristics of the wireless channels are characterized using a channel model that includes path loss, multipath fading, and time correlation.

[0012] S2. Define the local observation information, action space, and a joint reward function for each cognitive user to comprehensively evaluate spectrum utilization efficiency, interference to the main user, inter-user conflicts, and communication quality.

[0013] S3. Initialize a policy network for each cognitive user, and initialize a centralized evaluation network and its target network. At the same time, initialize the experience replay buffer, temperature coefficient, target entropy and related update coefficients.

[0014] S4. Collect experience data through the interaction between the agent and the environment, and store the experience samples, including state, action, reward and next state, into the experience replay buffer.

[0015] S5. Policy optimization based on stored experience data. This optimization process includes: calculating the priority of experience samples based on their temporal difference error, and performing non-uniform sampling from the buffer according to the priority to prioritize the use of high-value experience; calculating importance sampling weights for the sampled samples to correct the bias introduced by non-uniform sampling, and updating the parameters of the evaluation network and policy network using the weighted sample data; dynamically updating the target entropy based on the average entropy of the current policy, and adjusting the temperature coefficient based on the difference between the updated target entropy and the current policy entropy, thereby achieving adaptive adjustment of the policy exploration intensity at different training stages.

[0016] S6. Repeat steps S4 and S5 until the policy network training is complete. The resulting policy network is used to guide cognitive users to make dynamic spectrum access decisions in the heterogeneous network environment.

[0017] As a preferred technical solution, in step S1, the non-cognitive users employing different fixed access protocols specifically include non-cognitive users employing time-division multiplexing protocols and non-cognitive users employing time-slot ALOHA protocols. Specifically, the non-cognitive users employing time-division multiplexing protocols access their corresponding time-division multiplexing frequency bands according to pre-allocated fixed periodic time slots and remain silent outside their time slots. The non-cognitive users employing time-slot ALOHA protocols attempt to transmit data at the beginning of each time slot with a preset probability within their corresponding random access frequency bands; if a collision occurs, they randomly back off and retry. These non-cognitive users lack intelligent decision-making capabilities based on environmental awareness, and their access behavior follows fixed media access control protocol rules, thus forming a heterogeneous secondary user group together with cognitive users who perform opportunistic spectrum access based on deep reinforcement learning.

[0018] As a preferred technical solution, in step S1, the channel is divided into three non-overlapping frequency bands, including: a random access band, used for probabilistic contention-based access by non-cognitive users employing the time-slotted ALOHA protocol; a time-division multiplexing band, used for non-cognitive users employing the time-division multiplexing protocol to occupy pre-allocated fixed time slots; and a primary user-dedicated band, exclusively for primary user use, whose occupancy status is simulated as a dynamically changing licensed spectrum; the primary user's occupancy behavior in the dedicated band is described using a two-state Markov chain, where one state indicates that the channel is idle and allows cognitive users to access opportunistically, and the other state indicates that the channel is occupied and prohibits cognitive users from accessing.

[0019] As a preferred technical solution, in step S2, the local observation information is the partially observable information about the surrounding wireless environment obtained by each cognitive user through its own spectrum sensing device in each decision time slot; the partially observable information includes at least: the idle or occupied state of each sub-channel in a series of consecutive channel blocks that the cognitive user is currently capable of sensing, the channel quality characterized by the channel gain or signal-to-noise ratio corresponding to the channel block, the set of available actions consisting of all currently selectable access actions determined by its sensing capability and the total number of channels in the system, and the minimum required bandwidth determined by its current service to be transmitted.

[0020] As a preferred technical solution, in step S2, the action space is specifically defined as: a decision set in which each cognitive user selects a continuous sub-channel from all orthogonal channels of the system, the number of which is equal to the cognitive user's sensing capability, as a whole channel block for sensing and access; the size of the action space is determined by the difference between the total number of channels in the system and the cognitive user's sensing capability plus one; each action corresponds to a specific continuous channel block starting index; the cognitive user selects an action from the action space to execute according to the probability distribution output by the policy network, that is, selects a target channel block; after executing the action, the system will verify whether the number of idle sub-channels in the target channel block meets the cognitive user's minimum required bandwidth, and detect whether there is a conflict with the channel blocks selected by the main user, other protocol users, or other cognitive users, thereby ultimately determining whether the access is successful, idle, or triggers a conflict penalty.

[0021] As a preferred technical solution, in step S3, the centralized evaluation network adopts a dual-Q network structure, which includes two sub-networks with the same structure but independent parameters. During training, the evaluation network receives the joint actions of all cognitive users and the global shared state of the environment as input, and outputs a long-term value estimate of the current state-action pair. The evaluation network works together with its target network, and the parameters of the target network slowly track the parameter changes of the evaluation network through a soft update method.

[0022] As a preferred technical solution, in step S5, the calculation of the priority based on the temporal difference error of the experience sample is specifically implemented as follows: For each experience sample in the experience replay buffer, the two independent sub-networks contained in the centralized evaluation network are used to calculate the two temporal difference error values ​​corresponding to the experience sample; the average of the absolute values ​​of the two temporal difference error values ​​is taken as the basic error metric for measuring the value of the experience sample for policy learning; finally, a positive minimum constant to prevent division by zero error is added to the basic error metric to obtain the final priority value of the experience sample; the higher the priority of the sample, the greater the probability of it being selected in subsequent sampling processes.

[0023] As a preferred technical solution, in step S5, updating the parameters of the evaluation network and the policy network using weighted sample data includes the following sub-steps: First, based on a batch of samples sampled from the experience replay buffer and their corresponding importance sampling weights, calculate the target Q value for each sample. This target Q value is composed of the immediate reward, discount factor, the target network's value estimate for the next state-action pair, and the policy entropy regularization term in the sample. Second, calculate the loss of the evaluation network using the weighted Huber loss function, where the loss of each sample is weighted by its importance sampling weight to correct for data distribution bias caused by priority sampling. Then, update the parameters of the evaluation network using the backpropagation algorithm. Finally, update the parameters of the policy network based on the maximum entropy reinforcement learning objective. This maximum entropy reinforcement learning objective maximizes long-term cumulative rewards while incorporating a policy entropy term weighted by a temperature coefficient to encourage the policy to maintain a certain degree of randomness in exploration. The temperature coefficient is adaptively adjusted according to the difference between the current policy entropy and the dynamically updated target entropy.

[0024] As a preferred technical solution, in step S5, the target entropy is dynamically updated based on the average entropy of the current policy through a smooth update formula: the target entropy value of the previous training stage or time step is weighted and summed with the currently calculated average entropy value of the policy to obtain the target entropy value of the next stage; wherein, the average entropy value of the policy is the average value of the information entropy of the policy action distribution calculated based on a batch of state samples under the current policy network; the weighted summation uses a smoothing coefficient between zero and one.

[0025] As a preferred technical solution, in step S3, initializing a centralized evaluation network and its target network means simultaneously initializing two evaluation networks with identical structures. One network serves as the master network for calculating and updating parameters, while the other serves as the target network for providing a stable value estimation target. During the strategy optimization process in step S5, the parameters of the target network are updated through a soft update mechanism. That is, each time the target network is updated, its parameters are set to a weighted average of the current master evaluation network parameters and the target network's own parameters, where the weight of the master network parameters is a soft update coefficient much smaller than one.

[0026] Compared with the prior art, the beneficial effects of this application are as follows:

[0027] This application constructs a highly realistic heterogeneous multi-protocol dynamic spectrum access simulation environment, which accurately simulates the complex scenario where primary users, non-cognitive users using different fixed protocols, and cognitive users with varying perception capabilities and business needs coexist. This makes the trained strategies more relevant to actual applications and improves the practicality and adaptability of the solution.

[0028] By defining differentiated local observation spaces and action spaces for each cognitive user, as well as a joint reward function that comprehensively considers spectrum utilization, interference to the main user, conflicts between users, and conflicts between heterogeneous protocols, the agent is effectively guided to learn efficient and low-interference access strategies under multiple complex constraints.

[0029] By using a preheating experience pool and a mechanism of continuous interaction, stability was ensured in the early stages of training and a wealth of sample data was accumulated.

[0030] This application introduces a priority experience replay mechanism based on time difference error and an importance sampling bias correction mechanism, which enables the training process to automatically identify and prioritize the use of high-value experience samples, greatly improving sample learning efficiency and thus accelerating policy convergence.

[0031] Meanwhile, by dynamically updating the target entropy based on the average entropy of the current policy, the exploration intensity is adaptively adjusted. In the early stage of training, it encourages full exploration to escape local optima, and reduces randomness to improve performance when the policy tends to stabilize. This significantly enhances the learning stability and robustness of the algorithm in non-stationary environments.

[0032] In summary, these technical means work together to solve the problems of low exploration efficiency and insufficient experience utilization of traditional methods in heterogeneous environments. Ultimately, this application enables higher spectrum utilization efficiency, higher access success rate, lower conflict rates, and a more stable and efficient training convergence process in complex dynamic spectrum access scenarios. Attached Figure Description

[0033] Figure 1 This is a diagram of the HAED algorithm framework for the dynamic spectrum access method based on target entropy and priority experience replay of the present invention.

[0034] Figure 2 This is a system model and signal-interference relationship diagram of the heterogeneous cognitive wireless network of the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in specific embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0036] Please see Figure 1 This paper demonstrates the HAED algorithm framework of the dynamic spectrum access method based on target entropy and priority experience replay provided by this invention. The overall method follows a cyclical process of "environment interaction - experience storage - priority sampling - network update - entropy adaptive adjustment", including the following steps:

[0037] S1. Construct a heterogeneous multi-protocol dynamic spectrum access environment.

[0038] This step aims to build a simulation training environment that closely resembles a real-world scenario and includes a variety of heterogeneous elements.

[0039] like Figure 2 As shown, the system model includes M primary users (PUs) with priority spectrum usage rights, K secondary users (SUs), and L orthogonal wireless channels. The secondary users are further divided into two categories: N cognitive secondary users (cognitive SUs) that make intelligent decisions based on deep reinforcement learning, and several non-cognitive secondary users using a fixed access protocol (in this embodiment, users using a time-division multiplexing protocol and a slotted ALOHA protocol are used as examples). For simplicity, ALOHA in the following text refers to the slotted ALOHA protocol. All secondary users are in a queue backlog state and continuously transmit data at each time step.

[0040] The The orthogonal channels are divided into three non-overlapping frequency bands according to the access protocol type: random access band. Provides time-slot ALOHA users with probabilistic contention for access; time-division multiplexing frequency band. Provided for TDMA users to access within a fixed allocated time slot; dedicated frequency band for primary users. This is exclusively for primary users. The primary user's usage behavior within the dedicated frequency band is described using a two-state Markov chain, where the states are... This indicates that the channel is idle and allows cognitive SU opportunistic access; state This indicates that the channel is occupied by the PU and access by the cognitive SU is prohibited.

[0041] To simulate the two-dimensional movement behavior of cognitive users in a dynamic environment, this invention adopts a random movement model based on speed and direction.

[0042] Assume all secondary users are in a queue backlog and continuously send data at each time step. They represent the first The SU and the first The position coordinates of the transmitter and receiver of each PU ( and These represent the transmitting end and the receiving end, respectively.

[0043] No. A cognitive user at any moment The two-dimensional position is Its position update formula in the next moment is:

[0044] ;

[0045] in, To understand the user's movement speed; Indicates the time step; It indicates its direction of movement, and is synchronously updated at each time step according to the random direction.

[0046] To characterize the propagation characteristics of wireless channels in complex real-world environments, the path loss process is modeled based on scenario B1 of the WINNERII channel model. This scenario is applicable to typical complex wireless propagation environments where base stations are located below rooftops, users are densely distributed, and both line-of-sight and non-line-of-sight propagation exist simultaneously. The path loss model is shown below:

[0047] ;

[0048] in, This represents the path loss constant at the reference distance. The distance between the transmitter and receiver. It is the carrier frequency. and These are the distance attenuation factor and the frequency dependence factor, respectively. This represents the shadowing fading term. The channel gain corresponding to the path loss is... .

[0049] Because SU-SU and PU-SU links exhibit instantaneous channel characteristics in multipath propagation environments, this invention employs a Rician fading model to model the channel gain in the presence of a direct path. This model effectively reflects the multipath fading effect when both LOS and scattering paths coexist. The mathematical expression is as follows:

[0050] ;

[0051] in, It is the Rician factor, representing the power ratio of the direct path to the scattered path; The phase of the direct path; It is complex Gaussian noise, representing the scattering path component; The average received power is determined by path loss. The final channel gain is... ,in .

[0052] To further characterize the channel dynamics caused by node movement or environmental changes, this invention, based on Rician fading modeling, introduces an exponentially correlated channel model based on the Doppler effect to simulate the correlation of channel gain evolution over time. Specifically, it is expressed as follows:

[0053] ;

[0054] in, This represents the time-dependent factor, reflecting the stationarity of the channel state between adjacent moments; The channel coefficients from the previous time step. These are independently generated Rician channel samples. This model can be used to characterize the slow fading characteristics of a channel over a short timescale.

[0055] Interference will occur when a PU and a SU, or multiple SUs, simultaneously occupy the same radio channel. For the first... A cognitive user, who chose Each channel meets its bandwidth requirements. At the same time, there exists Each main user occupied [amount] There are channels, and also The first channel conflicts due to the joint selection by several cognitive users. The signal-to-noise ratio obtained by each secondary user is defined as:

[0056] ;

[0057] in, For the first Secondary users selected the frequency band sub-channel. Gain, For the first Each main user in the sub-channel Gain, Representing cognitive users With the remainder The gain generated by each conflicting channel of a cognitive user Representing cognitive users noise spectral density, To understand users The transmission power, To interfere with other cognitive users of the current cognitive user The transmit power. The transmission rate is estimated using the standard Shannon capacity model approximation.

[0058] S2. Define the agent's state, actions, and reward functions.

[0059] This step defines the Markov decision process elements required for each cognitive sub-user's decision-making, including the state space, action space, and reward function.

[0060] State space: At each time step, each cognitive user has a view of all... The channel performs spectrum sensing to detect the channel state for each time slot. These states are time-varying. It can be represented as:

[0061] ;

[0062] Considering the inherent limitations of actual spectrum sensing, the channel state observations obtained by cognitive users may contain errors; let the first... On the channel 1 The perception results of individual cognitive users are , No. On the channel 1 The perceptual error probability of each SU is The channel state transition probability is: Due to hardware limitations of the cognitive user's sensing devices, they cannot perceive all aspects of the environment. The nth channel; assuming the nth... The perception capability of each SU is The required bandwidth is ,and Each SU senses The first channel block, and the second Each SU-observed channel block is evaluated based on whether the number of idle sub-channels meets the minimum bandwidth requirement. There are two scenarios: transmission requirements are met (1) and transmission requirements are not met (0). One SU in the time slot right Observations of each channel block It is expressed as follows:

[0063] .

[0064] Due to the heterogeneity of the wireless environment, the observation space differs among users with different perceptions. This heterogeneity mainly stems from the differences in available spectrum resources, differences in spectrum perception capabilities, and individual biases in their perception of spectrum availability.

[0065] Specifically, users' perception capabilities are affected by various factors such as their geographical location, hardware conditions, and local interference, resulting in different observation results of the spectrum state at the same time, which further exacerbates the heterogeneity of the system.

[0066] Action Space: In this context, the cognitive user needs to understand the current time slot. The idle channels in the sensing channel block are aggregated, and their placement in the next time slot is decided based on the current observation results. The perceived channel block. Specifically, each cognitive user (SU) receives from... Selecting consecutive channels Each sub-channel performs spectrum sensing to indicate that there are a total of Each sensing action is received. The action space of the agent is represented as follows: .

[0067] However, due to differences in the selection of sensing targets, users' observations of the number and location of idle channels also differ. This variability in action space reflects the heterogeneity of individuals in a multi-user system in terms of sensing capabilities and spectrum access strategies. Each secondary user generates spectrum access decisions based on a deep reinforcement learning policy network, selects target aggregation channel blocks according to the real-time environmental state, and performs spectrum conflict detection and bandwidth guarantee verification on the selected aggregation channel blocks to determine whether to access or leave them idle.

[0068] Reward Function: This invention maximizes the system performance of heterogeneous cognitive users in wireless communication environments by constructing a joint reward function, thereby comprehensively evaluating the adaptability and effectiveness of spectrum access strategies in complex dynamic scenarios. This reward function not only measures the spectrum utilization efficiency of secondary users, but also comprehensively considers key factors such as interference with primary users, access conflicts between SUs, and energy consumption costs.

[0069] Specifically, if the channel block selected by the current cognitive user does not overlap with the channel blocks of other already connected users in physical location, and meets its bandwidth requirements... If the connection is successful, a conflict warning will be triggered; otherwise, it will be considered a successful connection. After the action is executed, the system will provide immediate reward feedback, which is influenced by the states of the PU and other SUs. Based on this, this embodiment defines the following immediate reward function:

[0070] a. No. A cognitive user chooses not to access the channel, and receives a reward. ;

[0071] b, the For each cognitive user, if they select the corresponding channel block, the basic reward is... If the channel block selected by the cognitive user meets the usage requirements and there are no conflicts, , , and These represent the reward and penalty mechanisms when the selection of the corresponding channel block involves the frequency band interval of this protocol:

[0072] (1) PU interference penalty term. Wherein, if the agent... Selected channel If the current PU is occupied, a penalty will be imposed:

[0073] ;

[0074] It is the penalty coefficient for PU interference. It is the proportion of primary user authorized channel access. Indicates channel Is the space available?

[0075] (2) ALOHA Access Rewards. Based on the number of ALOHA access conflicts. The following rewards will be given:

[0076] ;

[0077] It is an indicator function, which is 1 when there is no conflict. This indicates the proportion of ALOHA channel access.

[0078] (3) TDMA Access Reward. If an agent accesses a TDMA channel and the time slot is already occupied, it is penalized; otherwise, it is rewarded:

[0079] ;

[0080] Indicates whether the TDMA channel is occupied ( (Indicates that the space is occupied) This represents the TDMA channel access ratio. , and Retrieved based on the aggregated channel index of the selected action.

[0081] c. If the first The cognitive user and the first There is a conflict in the channel blocks selected by a cognitive user. Sub-channels and others When multiple cognitive users access overlapping channel blocks, the overall reward is:

[0082] ;

[0083] d. If the first When a cognitive user's selected channel block is occupied by a primary user or a secondary user of another protocol, preventing transmission, the reward is set as follows:

[0084] .

[0085] In summary, time slots The reward can be represented as In this embodiment, the movement behavior of cognitive users and the access of primary users cause dynamic changes in the channel state. Simultaneously, the signal-to-noise ratio (SNR) is significantly affected by the distance differences between users. Against this backdrop, to improve the utilization efficiency of spectrum resources, the system design aims to maximize joint rewards. .

[0086] This embodiment focuses on the strategies and methods employed by each SU (User Unit) to achieve the highest access success rate under limited spectrum resources. These strategies must satisfy constraints such as limited spectrum resources and fixed user power while ensuring system throughput, thereby achieving efficient resource utilization. The core challenge lies in optimizing data transmission performance while meeting these constraints, ensuring efficient resource use. Furthermore, the proposed algorithm aims to effectively mitigate access conflicts among multiple users, thereby promoting improved cooperative access efficiency. The ultimate goal of the system is to learn an optimal cooperative spectrum access strategy to maximize long-term cumulative discounted returns.

[0087] S3. Initialize the HAED training module.

[0088] The agent employs a modular design, comprising core sub-modules: a policy network, an evaluation network, and an experience replay buffer. The policy network is implemented based on the HASAC algorithm and trained in conjunction with a priority experience replay mechanism. It possesses capabilities for action sampling, logarithmic probability calculation, and probability distribution output, and can be used in different stages such as policy exploration, policy optimization, and policy evaluation. The evaluation network, based on a dual-Q network structure, performs value prediction on shared observation information and joint action inputs.

[0089] Furthermore, the module introduces an automatic temperature parameter adjustment mechanism, which adaptively regulates the policy entropy by minimizing the temperature loss function, thereby dynamically balancing the exploratory and exploitative aspects of the policy. In addition, the module supports value normalization and Huber loss calculation, enhancing its robustness and adaptability in multi-agent reinforcement learning for tasks involving partial observability and cooperative control. The experience replay buffer constructs an off-policy experience replay structure based on global state information provided by the environment, used to store key experience data during the interaction between the multi-agent and the environment, including... By introducing an n-step temporal differential reward calculation mechanism, the module can more effectively capture long-term reward signals, enhancing policy training stability and sample utilization efficiency.

[0090] Specifically, an independent policy network is initialized for each cognitive SU. It is used to access the action probability distribution based on the local observation output spectrum; and to initialize the centralized double-Q evaluation network. Its target network is used to estimate state-action values ​​based on globally shared observations and joint actions; initialization of temperature coefficients. Target entropy Target entropy sliding update parameters Soft update coefficients and priority experience replay buffers.

[0091] The experience replay buffer is used to store multi-agent interaction samples, which include global state, local observations of each agent, available actions, joint actions, immediate rewards, next state, next local observation, termination flag, and auxiliary statistical information.

[0092] In one specific embodiment, the system can be configured with L=32 orthogonal channels, 3 cognitive units (SUs), and the bandwidth requirement of each cognitive SU is dynamically sampled from {2, 4, 6}. The random access frequency band corresponds to ALOHA contention access, the time-division multiplexing frequency band corresponds to TDMA periodic access, and the primary user dedicated frequency band is reserved for PU priority use. During training, 20 environments can run in parallel. The experience replay buffer capacity is set to 100,000, the warm-up steps are set to 1,000, the learning rate of both the policy network and the evaluation network is set to 0.0003, the batch size is set to 200, and the target network soft update coefficient is set to 0.03. These parameters can be adjusted according to network scale and service requirements.

[0093] S4, preheating the experience pool, where the agent interacts with the environment and stores experience.

[0094] The purpose of the warm-up phase is to fill the experience replay buffer by executing random actions (rather than policy generation actions) in the early stages of training to prevent the policy network from becoming unstable due to lack of experience.

[0095] Specifically, before each round of interaction begins, the agent acquires local observations. Global shared observation of the entire system and the set of currently available actions. In each warm-up step, the agent samples actions according to a random policy and interacts with the environment to obtain local observations for the next time step. Shared observation Instant rewards Termination mark And auxiliary statistical information. This constitutes a single empirical data point: ,in, , ,and These empirical data are sequentially stored in an experience replay buffer to build an initial experience base for subsequent policy training. Globally shared observations. During the intensive training phase, the input is used to evaluate the network, improving the estimation accuracy of the value function and the global collaborative modeling capability, while local observations... This is then used for independent decision-making in the policy network of each agent.

[0096] Simultaneously, during formal training, the agent selects the corresponding spectrum access action through the policy network based on the current state information and local observation results, and interacts with the dynamic spectrum access environment. The environment returns corresponding rewards, the next state, and environmental feedback information based on the current spectrum occupancy, user access behavior, and channel state. Subsequently, the interaction experience data, including the current state, action, reward, next state, and termination flag, are compiled into a complete experience sample and stored in the experience replay buffer, providing experience data support for the subsequent training and optimization of the policy network and evaluation network.

[0097] Specifically, during environmental interaction and sample collection, each agent operates based on its local observation space. Through policy network Generate Actions It interacts with the environment model built in S1. Next is experience storage, which returns the values. Actions at the current moment ,state Local observation Available actions To form complete empirical data Stored in the experience buffer to support the subsequent reinforcement learning training process.

[0098] S5. Based on PER, perform empirical sampling and bias correction, update the policy network and evaluation network, and perform adaptive adjustment of target entropy.

[0099] Formal training adopts a three-stage cyclical architecture of "interaction-training-evaluation", which includes the following ordered steps:

[0100] Step 1: Time difference error assessment.

[0101] In the dynamic spectrum access problem, agents need to select appropriate channels based on environmental conditions to maximize long-term gains. However, due to the constantly changing factors such as spectrum occupancy, user competition, and channel conflicts, different experiences have significantly different values ​​for policy updates. Traditional uniform experience replay methods assume that all experiences have equal importance, an assumption that is often difficult to hold in complex DSA environments. Therefore, this invention introduces a priority experience replay mechanism based on temporal difference error, adjusting the sampling probability by dynamically evaluating the importance of experiences. In reinforcement learning, temporal difference error is used to measure the gap between the current value function estimate and the target value. In the framework of this invention, for the i-th experience in the experience buffer, the TD error of the two Critic networks is calculated respectively:

[0102] ;

[0103] ;

[0104] in and These represent two independent Critic networks, with target values... Defined as:

[0105] ;

[0106] in, This represents the immediate reward for the current time slot. As a discount factor, For the temperature parameter in maximum entropy reinforcement learning, This represents the action probability output by the policy network. To obtain a more stable priority estimate, this invention uses the average of the absolute values ​​of two TD errors as the basic metric for sample priority:

[0107] .

[0108] Step 2: PER priority sampling.

[0109] When the TD error corresponding to a certain experience is large, it indicates that the current policy's estimation of the value of that state-action pair is inaccurate, and this experience has higher reference value for policy improvement. Therefore, this invention uses TD error as a measure of sample priority.

[0110] Specifically, for the first in the experience buffer The priority of each experience is defined as follows:

[0111] ,

[0112] in, Indicates experience priority. For the corresponding TD error, To prevent small constants with zero priority.

[0113] Based on priority, the probability of a sample being sampled can be further defined:

[0114] ,

[0115] in Representing experience The probability of being sampled. This is a priority control parameter used to adjust the degree to which the priority affects the sampling probability. This represents the sum of the priorities of all samples. When... When sampling degenerates into uniform sampling; when As the value increases, the probability of high-priority samples being sampled increases significantly.

[0116] Through the above mechanism, the samples in the experience replay buffer will no longer be sampled with uniform probability, but will be dynamically adjusted according to their importance to the current policy learning.

[0117] Step 3: IS deviation correction.

[0118] While priority experience replay can significantly improve sample utilization efficiency, it also introduces new problems. Since the experience sampling probability is no longer uniform but depends on the priority distribution, a deviation arises between the distribution of training data and the distribution of real-world interaction data. This distributional deviation affects the unbiasedness of the value function estimation, potentially leading to policy learning bias. To address this issue, this invention introduces an importance sampling bias correction mechanism based on priority experience replay. This method reweights gradient updates by assigning a weight to each sample, thereby compensating for the distributional bias caused by priority sampling.

[0119] Specifically, for the first sampled from the experience buffer The importance sampling weight of each experience is defined as follows:

[0120] .

[0121] in, Indicates sample weights, The total number of experiences in the buffer. The sampling probability is the parameter. Control the strength of bias correction. In the early stages of training, the policy is not yet stable, and excessively strong weight correction may cause training oscillations. Therefore, it is common practice to... Set it to a small value. Gradually increase it as the training process progresses. This allows for gradual weight adjustments, thereby achieving near-unbiased estimation in the later stages of training. In actual training, to avoid excessively large weights leading to gradient instability, this invention normalizes the weights:

[0122] .

[0123] The normalized weights are used to update the Q-network parameters. To fully utilize the importance sampling weights during parameter updates, this invention introduces them into the loss function calculation process of the Critic network. In the traditional SAC algorithm, the value function is usually optimized using mean squared error or Huber loss. However, after introducing the priority experience replay mechanism, the sampling probabilities of different samples are no longer the same, so it is necessary to adjust the loss function using weights.

[0124] Specifically, let the current batch sample size be... The importance sampling weighted Huber loss function used in this invention is defined as follows:

[0125] ;

[0126] in This represents the current Critic network's value estimate for state-action pairs. For the TD target value, The importance sampling weights are normalized.

[0127] The Huber loss function is defined as:

[0128] ;

[0129] in The smoothing parameter for Huber loss.

[0130] Compared to the traditional mean squared error loss function, the Huber loss exhibits squared loss when the error is small and linear loss when the error is large, thus effectively suppressing the impact of outliers on training stability. By incorporating importance sampling weights, the contribution of different experiences to gradient updates is dynamically adjusted according to their sampling probabilities, thereby improving the learning efficiency of key samples while maintaining the stability of the training process.

[0131] Step 4: Strategy optimization.

[0132] Each cumulative After obtaining a set of empirical data, the evaluation network and policy network are updated. After randomly sampling a batch of data from the buffer, each agent calculates its action for the next state through the policy network. And the corresponding log probabilities. These values ​​are used to subsequently calculate the target value. The target value is calculated as follows:

[0133]

[0134] It is the reward for the current time step. It is a discount factor. It is a hyperparameter that controls the weight of the entropy term in the policy. When training the evaluation network, the difference between the predicted value and the target value is minimized through the loss function.

[0135] This invention employs weighted Huber loss:

[0136] ;

[0137] in It is the target Q value. For batch size, Weights are sampled for normalized importance. Finally, the gradient of the loss is calculated, and the parameters of the evaluation network are updated using the optimizer.

[0138]

[0139] These are parameters used to evaluate the network. It's the learning rate. It is the gradient of the loss function with respect to the network parameters.

[0140] The policy network is updated based on a maximum entropy objective, aiming to optimize the policy while maintaining sufficient exploratory nature.

[0141] ;

[0142] Each agent samples an action based on the current observation and calculates the log probability of that action. Subsequently, each agent is trained in random order, estimating the Q-value of the state-action pair using the current evaluation network during training. The policy loss function consists of two parts: the expected Q-value under the policy and an entropy regularization term. Entropy coefficient. It controls the balance between exploration and exploitation, and can automatically adjust based on target entropy to maintain sufficient randomness in the strategy, preventing the strategy from prematurely falling into suboptimal solutions. Temperature parameter. The loss function is:

[0143] .

[0144] This mechanism is based on the current policy entropy. With target entropy Dynamic adjustment of differences between them When the policy entropy is lower than the target value, the system will increase... This increases the importance of the entropy term in the objective function, prompting a more diversified strategy and enhancing exploration capabilities; conversely, it decreases the importance of the entropy term. The guiding strategy is becoming more certain.

[0145] Step 5: Adaptive adjustment of target entropy.

[0146] In HASAC, target entropy It is usually set to a fixed value, for example This value is determined solely by the action space dimension. While this setting is simple to implement, it ignores the fact that the policy distribution dynamically changes with learning during actual training, which can easily lead to overexploration or underexploration at different training stages, thus affecting the convergence efficiency and performance stability of the policy. To address this limitation, the HAED algorithm is proposed, introducing a priority experience replay mechanism and a target entropy sliding update mechanism based on the actual average entropy of the policy.

[0147] Specifically, the algorithm every Step based on the average entropy of the current strategy The target entropy is adjusted by sliding to enable it to perceive and respond to the dynamic evolution of the policy distribution during training. The update format is as follows:

[0148]

[0149] in The smoothing coefficient controls the degree of inertia in the target entropy update.

[0150] Furthermore, this adaptive update process can be represented as a closed-loop feedback mechanism: policy change entropy change, target entropy update, temperature coefficient adjustment, and policy re-optimization. Faced with the non-stationary and rapidly changing channel environment in wireless communication spectrum access, agents need continuous perception and rapid response capabilities. Introducing a sliding update target entropy mechanism helps enhance the system's adaptability to environmental changes, further promotes efficient use of spectrum resources and interference suppression, and improves the collaborative access effect of multiple agents.

[0151] After calculating the loss, the policy network parameters are updated through backpropagation to optimize the policy. Finally, the target network is updated through a soft update mechanism (i.e., setting the target evaluation network parameters to an exponential moving average of the current network parameters) so that its parameters slowly approach those of the target network, thereby ensuring the stability and convergence of the training process.

[0152] Through the above implementation methods, this invention can prioritize the utilization of high-value interaction experiences in dynamic spectrum access environments with multiple users, multiple protocols, and multiple interference sources, and adaptively adjust the exploration intensity according to changes in policy entropy. Compared with reinforcement learning methods that only use fixed target entropy or uniform experience replay, this invention can improve the utilization efficiency of key experiences, accelerate policy convergence speed, reduce the probability of collisions between primary users and other secondary users, and improve access success rate, system throughput, and the proportion of high SINR states.

[0153] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Those skilled in the art can make equivalent substitutions or adjustments to the network scale, channel allocation method, reward parameters, priority calculation method, target entropy sliding update parameters, and training hyperparameters without departing from the spirit and substance of the present invention, and such substitutions or adjustments should all fall within the protection scope of the present invention.

[0154] It should be noted that the terms "first," "second," and similar terms used in this application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, "an" or "a" and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. "A plurality" or "several" indicates at least two. Unless otherwise stated, terms such as "front," "back," "left," "right," "lower," and / or "upper" are for illustrative purposes only and are not limited to a location or spatial orientation. Terms such as "comprising" or "including" indicate that the elements or objects preceding "comprising" encompass the elements or objects listed following "comprising" or "including" and their equivalents, and do not exclude other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0155] The singular forms “a,” “the,” and “the” used in this application specification and appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0156] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A dynamic spectrum access method based on target entropy and priority experience replay, characterized in that, Includes the following steps: S1. Construct a heterogeneous cognitive wireless network environment containing multiple primary users, multiple secondary users, and multiple orthogonal wireless channels, wherein the secondary users include non-cognitive users using different fixed access protocols and cognitive users based on deep reinforcement learning, and the channels are divided into frequency bands corresponding to different access protocol types. The location of cognitive users is updated using a random movement model, and the propagation characteristics of the wireless channel are characterized by a channel model that includes path loss, multipath fading, and time correlation. S2. Define the local observation information, action space, and a joint reward function for each cognitive user to comprehensively evaluate spectrum utilization efficiency, interference to the main user, inter-user conflicts, and communication quality. S3. Initialize a policy network for each cognitive user, and initialize a centralized evaluation network and its target network. At the same time, initialize the experience replay buffer, temperature coefficient, target entropy and related update coefficients. S4. Collect experience data through the interaction between the agent and the environment, and store the experience samples, including state, action, reward and next state, into the experience replay buffer. S5. Optimize the strategy based on the stored experience data. The optimization process includes: calculating the priority of the experience samples based on their time difference error, and performing non-uniform sampling from the buffer according to the priority to give priority to high-value experience. Importance sampling weights are calculated for the sampled samples to correct the bias introduced by non-uniform sampling, and the parameters of the evaluation network and policy network are updated using the weighted sample data; the target entropy is dynamically updated by sliding according to the average entropy of the current policy, and the temperature coefficient is adjusted based on the difference between the updated target entropy and the current policy entropy, thereby realizing the adaptive adjustment of the policy exploration intensity at different training stages. S6. Repeat steps S4 and S5 until the policy network training is complete. The resulting policy network is used to guide cognitive users to make dynamic spectrum access decisions in the heterogeneous network environment.

2. The dynamic spectrum access method based on target entropy and priority experience replay as described in claim 1, characterized in that, In step S1, the non-cognitive users employing different fixed access protocols specifically include non-cognitive users employing time-division multiplexing protocols and non-cognitive users employing time-slot ALOHA protocols. Specifically, the non-cognitive users employing time-division multiplexing protocols access their corresponding time-division multiplexing frequency bands according to pre-allocated fixed periodic time slots and remain silent outside their time slots. The non-cognitive users employing time-slot ALOHA protocols attempt to transmit data at the beginning of each time slot with a preset probability within their corresponding random access frequency bands; if a collision occurs, they randomly back off and retry. These non-cognitive users lack intelligent decision-making capabilities based on environmental awareness, and their access behavior follows fixed media access control protocol rules, thus forming a heterogeneous secondary user group together with cognitive users who use opportunistic spectrum access based on deep reinforcement learning.

3. The dynamic spectrum access method based on target entropy and priority experience replay as described in claim 1, characterized in that, In step S1, the channel is divided into three non-overlapping frequency bands: a random access band for probabilistic contention-based access by non-aware users using the time-slotted ALOHA protocol; a time-division multiplexing band for non-aware users using the time-division multiplexing protocol to occupy pre-allocated fixed time slots; and a primary user-dedicated band for primary user use, whose occupancy status is simulated as a dynamically changing licensed spectrum. The primary user's occupancy behavior in the dedicated band is described by a two-state Markov chain, where one state indicates that the channel is idle and allows cognitive users to access opportunistically, and the other state indicates that the channel is occupied and prohibits cognitive users from accessing.

4. The dynamic spectrum access method based on target entropy and priority experience replay as described in claim 1, characterized in that, In step S2, the local observation information is the partially observable information about the surrounding wireless environment obtained by each cognitive user through its own spectrum sensing device in each decision time slot; The observable information includes at least: the idle or occupied status of each sub-channel in a series of consecutive channel blocks that the cognitive user is currently able to perceive; the channel quality represented by the channel gain or signal-to-noise ratio corresponding to the channel block; the set of available actions consisting of all currently selectable access actions determined by the user's perception capability and the total number of channels in the system; and the minimum required bandwidth determined by the service currently to be transmitted.

5. The dynamic spectrum access method based on target entropy and priority experience replay as described in claim 1, characterized in that, In step S2, the action space is specifically defined as: a decision set in which each cognitive user selects a continuous sub-channel from all orthogonal channels of the system, the number of which is equal to the cognitive user's sensing capability, as a whole channel block for sensing and access; the size of the action space is determined by the difference between the total number of channels in the system and the cognitive user's sensing capability plus one; each action corresponds to a specific starting index of a continuous channel block; the cognitive user selects an action from the action space to execute according to the probability distribution output by the policy network, that is, selects a target channel block; after the action is executed, the system will verify whether the number of idle sub-channels in the target channel block meets the minimum bandwidth requirement of the cognitive user, and detect whether there is a conflict with the channel blocks selected by the main user, other protocol users, or other cognitive users, thereby ultimately determining whether the access is successful, idle, or triggers a conflict penalty.

6. The dynamic spectrum access method based on target entropy and priority experience replay according to claim 1, characterized in that, In step S3, the centralized evaluation network adopts a dual-Q network structure, which includes two sub-networks with the same structure but independent parameters. During training, the evaluation network receives the joint actions of all cognitive users and the global shared state of the environment as input, and outputs a long-term value estimate of the current state-action pair. The evaluation network works together with its target network, and the parameters of the target network slowly track the parameter changes of the evaluation network through soft updates.

7. The dynamic spectrum access method based on target entropy and priority experience replay as described in claim 1, characterized in that, In step S5, the priority of the empirical sample is calculated based on the time difference error. Specifically, for each empirical sample in the empirical playback buffer, the two independent sub-networks contained in the centralized evaluation network are used to calculate the two time difference error values ​​corresponding to the empirical sample. The average of the absolute values ​​of the two time difference errors is taken as the basic error metric for measuring the value of the empirical sample for policy learning. Finally, a small positive constant to prevent division by zero error is added to the basic error metric to obtain the final priority value of the empirical sample. The higher the priority of the sample, the greater the probability of it being selected in subsequent sampling processes.

8. The dynamic spectrum access method based on target entropy and priority experience replay according to claim 1, characterized in that, In step S5, updating the parameters of the evaluation network and the policy network using weighted sample data includes the following sub-steps: First, based on a batch of samples sampled from the experience replay buffer and their corresponding importance sampling weights, calculate the target Q value for each sample. This target Q value is composed of the immediate reward, discount factor, the target network's value estimate for the next state-action pair, and the policy entropy regularization term in the sample. Second, calculate the loss of the evaluation network using the weighted Huber loss function, where the loss of each sample is weighted by its importance sampling weight to correct for data distribution bias caused by priority sampling. Then, update the parameters of the evaluation network using the backpropagation algorithm. Finally, update the parameters of the policy network based on the maximum entropy reinforcement learning objective. This maximum entropy reinforcement learning objective maximizes long-term cumulative rewards while incorporating a policy entropy term weighted by a temperature coefficient to encourage the policy to maintain a certain degree of randomness in exploration. The temperature coefficient is adaptively adjusted according to the difference between the current policy entropy and the dynamically updated target entropy.

9. The dynamic spectrum access method based on target entropy and priority experience replay according to claim 1, characterized in that, In step S5, the target entropy is dynamically updated based on the average entropy of the current policy through a smooth update formula: the target entropy value of the previous training stage or time step is weighted and summed with the currently calculated average entropy value of the policy to obtain the target entropy value of the next stage; wherein, the average entropy value of the policy is the average value of the information entropy of the policy action distribution calculated based on a batch of state samples under the current policy network; the weighted summation uses a smoothing coefficient between zero and one.

10. The dynamic spectrum access method based on target entropy and priority experience replay according to claim 1, characterized in that, In step S3, initializing a centralized evaluation network and its target network means simultaneously initializing two evaluation networks with identical structures. One network serves as the master network for calculating and updating parameters, while the other serves as the target network for providing a stable value estimation target. During the policy optimization process in step S5, the parameters of the target network are updated through a soft update mechanism. That is, each time the target network is updated, its parameters are set to a weighted average of the current master evaluation network parameters and the target network's own parameters, where the weight of the master network parameters is a soft update coefficient much smaller than one.

Images