Dynamic spectrum access method based on proximal strategy optimization of heterogeneous agents

Abstract

The present invention discloses a dynamic spectrum access method based on heterogeneous agent proximal strategy optimization, which belongs to the field of cognitive wireless network communication technology, including S1, establishing a calculation model for a dynamic spectrum access environment with heterogeneous state space and action space; S2, generating a dynamic spectrum access optimization strategy according to the calculation model of S1; S3, each user state space of the dynamic spectrum access environment and each cognitive user perceive the spectrum environment through a state perception module, and introduce a reward mechanism; S4, constructing an adaptive dynamic spectrum access agent based on HAPPO; the agent interacts with the environment for training to obtain a dynamic spectrum access strategy; S5, performing dynamic spectrum access according to the parameterized dynamic spectrum access strategy obtained in S4. The present invention adopts the above-mentioned dynamic spectrum access method based on heterogeneous agent proximal strategy optimization to achieve the best channel transmission rate and utilization rate in dynamic spectrum access under environmental heterogeneity, thereby reducing interference between users.

Description

Technical Field

[0001] The present invention relates to the field of cognitive wireless network communication technology, and in particular to a dynamic spectrum access method based on heterogeneous intelligent agent proximal strategy optimization. Background Art

[0002] Spectrum resources, as the core resources for the development of communications, are facing the problem of resource scarcity. Improving spectrum utilization is one of the important ways to solve this problem. In deep reinforcement learning, the agent learns better strategies from trial and error interactions with the environment to obtain the optimal cumulative reward. In recent years, RL has been widely used in the field of wireless communications. As an important branch of machine learning, reinforcement learning can intelligently detect uncertain environments and obtain the required information, especially in dynamic system processing. Dynamic spectrum access technology can use users' different business needs to select transmission channels, thereby improving spectrum utilization and system capacity in limited frequency bands. Cognitive Radio (CR) is an intelligent wireless communication technology that solves the problem of scarce spectrum resources by utilizing vacant spectrum in time, frequency and space, and has developed into cognitive radio networks (CRN). Dynamic spectrum access in cognitive wireless networks is an important technology in wireless communications. The dynamic spectrum access strategy aims to improve dynamic spectrum access throughput and reduce interference during communication. At present, deep reinforcement learning is used to optimize the performance of dynamic spectrum access, but it rarely considers the bandwidth requirements of cognitive users and the heterogeneity of agents in the environment, and cannot achieve global optimization. Summary of the invention

[0003] The purpose of the present invention is to provide a dynamic spectrum access method based on heterogeneous agent proximal strategy optimization to solve the problems mentioned in the above background technology.

[0004] To achieve the above object, the present invention provides a dynamic spectrum access method based on heterogeneous agent proximal strategy optimization, comprising the following steps:

[0005] S1. Establish a computational model for a dynamic spectrum access environment with heterogeneous state space and action space, and initialize parameters, including the number of secondary users N, the number of channels perceived by each secondary user G, the number of channels in the entire network system L, the perceived length of each secondary user, and the bandwidth requirements, and then build the spectrum environment; initialize the neural network training parameters in the deep reinforcement learning of heterogeneous agents;

[0006] S2, based on the calculation model of S1, generates a dynamic spectrum access optimization strategy, that is, using the deep reinforcement learning algorithm of heterogeneous agents to perform task allocation and access planning for cognitive users in the basic framework scenario, maximizing system throughput and minimizing collisions between users while ensuring that the main users in the network are not interfered with;

[0007] S3. Dynamic spectrum access environment: Each user state space and each cognitive user perceive the spectrum environment through the state perception module, and introduce a reward mechanism. When accessing, consider the bandwidth requirements of different users and the design of corresponding rewards for different state spaces.

[0008] S4. Based on the proximal strategy optimization of heterogeneous agents, adaptive dynamic spectrum access agents and aggregate agents are constructed to create a policy network (Actor network), a value network (Critic network) and their corresponding target networks; the agent interacts with the environment for training, and the environment enters a new state to feedback rewards to the agent, thus obtaining an inheritable parameterized dynamic spectrum access strategy;

[0009] S5. Perform dynamic spectrum access according to the parameterized dynamic spectrum access strategy obtained in S4.

[0010] Preferably, the specific steps of establishing the calculation model in S1 are as follows: according to the characteristic parameters in the dynamic spectrum access environment, the cognitive user state space, action space, throughput calculation and interference expression in the task environment are established into a mathematical model.

[0011] Preferably, the dynamic spectrum access includes a heterogeneous state space, action space, and different perception aggregations.

[0012] Preferably, the state space is specifically:

[0013] At each time step, the agent updates the agent action network to obtain the optimal strategy; each cognitive user SU performs spectrum sensing on all channels to detect the channel status of each time slot:

[0014] ;

[0015] Setting spectrum sensing is flawed, so May contain errors. On the channel The probability of perception error of a SU is , the channel state transition probability is

[0016] ;

[0017] Due to the hardware limitation of cognitive user SU sensing device, it cannot sense all sub-L channels in the environment; assuming The sensing capability of a SU is , represents the perceived length, and , then each SU perceives There are two states for a subchannel: occupied by SU or idle. When SU ​​observes a channel block, there are two states depending on whether the number of subchannels meets the bandwidth requirement: the channel block meets the transmission requirement or does not meet the transmission requirement. In time slot right The observation results of sub-channels It is expressed as follows:

[0018] ;

[0019] in, Indicates A cognitive user.

[0020] Preferably, the action space is specifically:

[0021] Getting a time slot After the observation results, SU chooses to Perceived channel block, the perception length of SU is The continuous channel block, its action space is expressed as

[0022] ;

[0023] Each SU selects consecutive The sub-channels are sensed and the aggregated frequency band is judged to meet the bandwidth requirements. requirements, which means that there is A sensing access action.

[0024] Preferably, the reward mechanism in S3 is specifically as follows:

[0025] After performing spectrum sensing and aggregation, each SU decides to access or idle. After taking an action, the system receives an immediate reward, which will be affected by other SUs and PUs. When the system is interfered by SU or PU, a penalty term is set. ; Considering a SUs and A multi-user wireless communication network consisting of 10 channels. The primary user does not need to consider whether the secondary user is working when communicating. They only need to transmit according to their own needs. Assuming that in an indoor environment, the secondary user will be interfered by the primary user PU in some sub-channels. Considering that the secondary user may need a higher bandwidth when transmitting voice and image data, it is assumed that users in the system can select idle channels for aggregation and access to meet the transmission requirements. The system transmission rate is specifically expressed as follows:

[0026]

[0027] in, represents the cognitive user bandwidth, represents the signal-to-noise ratio loss coefficient, is the signal-to-noise ratio, and the user’s reward is set as follows:

[0028] a. No. The user chooses not to access the channel, and the reward ;

[0029] b. For The selected channel block is used independently or shared with other secondary users. The total reward is , Indicates The bandwidth requirements of each user.

[0030] c. When When a secondary user accesses a channel that is already occupied by a primary user and cannot transmit, the reward is set to .

[0031] Preferably, the adaptive dynamic spectrum access agent constructed in S4 is specifically:

[0032] Assumptions Respectively transmitter, Receiver, Transmitter and The receiver's location coordinates, and Represent the transmitter and receiver respectively, Indicates Cognitive users, Indicates primary users; pass The link distance of the desired signal is calculated, while the interference signal propagation distance is calculated by When the primary user and the secondary user or several secondary users use the same wireless channel at the same time, the transmission will be interfered. The wireless signal will be attenuated when it propagates in the space between the transmitter and the receiver, which is called path loss. In addition, the value of channel gain is also affected by factors such as multipath fading. In order to consider the complexity of indoor channels, the commonly used indoor path loss model is adopted:

[0033] ;

[0034] At the same time, in actual indoor situations, the transmitter and receiver equipment are generally within the line of sight, and the direct power cannot be ignored. Therefore, it is more reasonable to use the Rician attenuation model for simulation experiments. The channel gain formula can be expressed as:

[0035]

[0036] in, Determined by the path loss, for Factor, indicating The ratio of the receiver signal power of the path to the scattering path, and =5, yes The phase of the signal arriving on the path takes a value from a uniform distribution between 0 and 1. represents a circularly symmetric complex Gaussian random variable; due to the limited spectrum in indoor transmission environments, assuming that each channel has the same bandwidth, the entire spectrum can be evenly distributed as channels; at the same time, the transmission power of each channel The same, the carrier frequency is unique and fixed; according to the above settings, the Secondary users in The channel gain on a channel is defined as: ; In order to evaluate the quality performance of the channel, the signal-to-noise ratio and transmission rate are set as the evaluation criteria of the channel quality. In the frequency band selected by the secondary user, the secondary user selected channels to meet its bandwidth requirements , there is The main users occupied channels, and possibly The channel conflicts due to the joint selection of several secondary users. The signal-to-noise ratio obtained by a secondary user is defined as:

[0037]

[0038] in, For the The sub-channel in the frequency band selected by the secondary user The gain, For the Primary users in subchannels The gain, Indicates secondary user With the remaining The gain generated by each conflicting channel of each secondary user is Represents cognitive user The noise spectral density is For cognitive users The transmission power, To interfere with other cognitive users of the current cognitive user of transmission power.

[0039] Preferably, in S4, the agent interacts with the environment for training, and the environment enters a new state to feedback rewards to the agent, thereby obtaining an inheritable parameterized dynamic spectrum access strategy, which is specifically:

[0040] Step 1: Each cognitive user Treat it as a heterogeneous agent and then initialize the policy network and the global value network , and initialize the experience replay pool ;

[0041] Step 2: Learn potential better strategies through training and obtain a set of strategies , according to the action strategy combined with the current state Get the action vector, which includes the selection of the aggregation frequency band and the selection of the aggregation length; perform the action Get current moment rewards and the next state vector , Will Save in , randomly sample from the experience replay pool to obtain small batches of samples;

[0042] Step 3: Use the GAE function to calculate the advantage function based on the global V value network , which is the advantage function value of the action corresponding to the current state. The global V value network introduces an advantage weighting coefficient to weight the TD error and reduce the impact of noise. Random arrangement of

[0043] Step 4: Experimental command , for each agent from , update the actor by the following formula Network parameters :

[0044]

[0045] in, Indicates batch size, Indicates the length of each batch, Indicates status and The joint advantage function of Indicates the current strategy, Indicates the strategy before the update;

[0046] And update the V value network by the following formula:

[0047]

[0048] Step 5: Repeat steps 2 to 4 until the training is completed.

[0049] Therefore, the present invention adopts the above-mentioned dynamic spectrum access method based on heterogeneous agent proximal strategy optimization, takes into account the heterogeneity of user state space and action space in the cognitive wireless network environment, and considers user bandwidth requirements when performing spectrum access, and introduces spectrum aggregation theory as part of model construction; compared with other existing deep reinforcement learning algorithms, the improved heterogeneous agent deep reinforcement learning algorithm is used to realize cognitive user collaborative dynamic spectrum access, and all cognitive user actions are evaluated through the Critic network to guide each user to complete the update; when the policy network converges, the collaborative spectrum access strategy can be completed in a distributed manner.

[0050] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flowchart of proximal strategy optimization based on heterogeneous agents provided in an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a dynamic spectrum access environment scenario of the present invention;

[0053] Figure 3 A schematic diagram showing a comparison of reward performance obtained by an algorithm according to an embodiment of the present invention;

[0054] Figure 4 It is a schematic diagram of comparison of interference avoidance performance of algorithms according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] Example

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0057] Heterogeneous-Agent Proximal Policy Optimisation (HAPPO) introduces the idea of ​​Proximal Policy Optimisation (PPO) algorithm, which stabilizes the training process and improves the sample efficiency of the algorithm by using proximal policy optimization. It enables reinforcement learning to solve large-scale state space problems. In a dynamic spectrum access environment with heterogeneous users, cognitive users need to integrate and coordinate information from different agents, each of which has a unique observation perspective, and the information is heterogeneous, with different channel states and aggregation conditions.

[0058] A dynamic spectrum access method based on heterogeneous agent proximal strategy optimization includes the following steps:

[0059] S1. Establish a computational model for a dynamic spectrum access environment with heterogeneous state space and action space, and initialize parameters, including the number of secondary users N, the number of perception channels G of each secondary user, the number of channels L of the entire network system, the perception length of each secondary user, and the bandwidth requirements, and then build the spectrum environment; initialize the neural network training parameters in the deep reinforcement learning of heterogeneous agents; establish the computational model specifically as follows: according to the characteristic parameters in the dynamic spectrum access environment, establish the cognitive user state space, action space, throughput calculation and interference expression in the task environment into a mathematical model.

[0060] Dynamic spectrum access includes heterogeneous state spaces, action spaces, and different perception aggregations.

[0061] The state space is specifically:

[0062] At each time step, the agent updates the agent action network to obtain the optimal strategy; each cognitive user SU performs spectrum sensing on all channels to detect the channel status of each time slot:

[0063] ;

[0064] Setting spectrum sensing is flawed, so May contain errors. On the channel The probability of perception error of a SU is , the channel state transition probability is

[0065] ;

[0066] Due to the hardware limitation of cognitive user SU sensing device, it cannot sense all sub-L channels in the environment; assuming The sensing capability of a SU is , represents the perceived length, and , then each SU perceives There are two states for a subchannel: occupied by SU or idle. When SU ​​observes a channel block, there are two states depending on whether the number of subchannels meets the bandwidth requirement: the channel block meets the transmission requirement or does not meet the transmission requirement. In time slot right The observation results of sub-channels It is expressed as follows:

[0067] ;

[0068] in, Indicates A cognitive user.

[0069] The action space is specifically:

[0070] Getting a time slot After the observation results, SU chooses to Perceived channel block, the perception length of SU is The continuous channel block, its action space is expressed as

[0071] ;

[0072] Each SU selects consecutive The sub-channels are sensed and the aggregated frequency band is judged to meet the bandwidth requirements. requirements, which means that there is A sensing access action.

[0073] S2, based on the calculation model of S1, generates a dynamic spectrum access optimization strategy, that is, using the deep reinforcement learning algorithm of heterogeneous agents to perform task allocation and access planning for cognitive users in the basic framework scenario, maximizing system throughput and minimizing collisions between users while ensuring that the main users in the network are not interfered with;

[0074] S3. Dynamic spectrum access environment Each user state space and each cognitive user perceives the spectrum environment through the state perception module, and introduces a reward mechanism. When accessing, the corresponding rewards are designed taking into account the bandwidth requirements of different users and different state spaces; the specific reward mechanism is:

[0075] After performing spectrum sensing and aggregation, each SU decides to access or idle. After taking an action, the system receives an immediate reward, which will be affected by other SUs and PUs. When the system is interfered by SU or PU, a penalty term is set. ; Considering a SUs and A multi-user wireless communication network composed of channels; SUs are heterogeneous and have different action and observation spaces. The primary user does not need to consider whether the slave users are working when communicating. They only need to transmit according to their own needs. Assuming that in an indoor environment, the secondary user will be interfered by the primary user PU in some sub-channels. Considering that the secondary user may need a higher bandwidth when transmitting voice and image data, it is assumed that users in the system can select idle channels for aggregation and access to meet the transmission requirements. The system transmission rate is specifically expressed as follows:

[0076] ;

[0077] in, represents the cognitive user bandwidth, represents the signal-to-noise ratio loss coefficient, is the signal-to-noise ratio, and the user’s reward is set as follows:

[0078] a. No. The user chooses not to access the channel, and the reward ;

[0079] b. For The selected channel block is used independently or shared with other secondary users. , The total reward is , Indicates The bandwidth requirements of each user.

[0080] c. When When a secondary user accesses a channel that is already occupied by a primary user and cannot transmit, the reward is set to .

[0081] S4. Based on the optimization of the proximal strategy of heterogeneous agents, an adaptive dynamic spectrum access agent and an aggregate agent are constructed to create a strategy network, a value network and their corresponding target networks. The agent interacts with the environment for training, and the environment enters a new state to feedback rewards to the agent, and obtains an inheritable parameterized dynamic spectrum access strategy. The adaptive dynamic spectrum access agent constructed is specifically:

[0082] Assumptions , , , Respectively transmitter, Receiver, Transmitter and The receiver's location coordinates, and Represent the transmitter and receiver respectively, Indicates Cognitive users, Indicates primary users; pass The link distance of the desired signal is calculated, while the interference signal propagation distance is calculated by When the primary user and the secondary user or several secondary users use the same wireless channel at the same time, the transmission will be interfered. The wireless signal will be attenuated when it propagates in the space between the transmitter and the receiver, which is called path loss. In addition, the value of channel gain is also affected by factors such as multipath fading. In order to consider the complexity of indoor channels, the commonly used indoor path loss model is adopted:

[0083] ;

[0084] At the same time, in actual indoor situations, the transmitter and receiver equipment are generally within the line of sight, and the direct power cannot be ignored. Therefore, it is more reasonable to use the Rician attenuation model for simulation experiments. The channel gain formula can be expressed as:

[0085] ;

[0086] in, Determined by the path loss, for Factor, indicating The ratio of the receiver signal power of the path to the scattering path, and , yes The phase of the signal arriving on the path takes a value from a uniform distribution between 0 and 1. represents a circularly symmetric complex Gaussian random variable; due to the limited spectrum in indoor transmission environments, assuming that each channel has the same bandwidth, the entire spectrum can be evenly distributed as channels; at the same time, the transmission power of each channel The same, the carrier frequency is unique and fixed; according to the above settings, the Secondary users in The channel gain on a channel is defined as: ; In order to evaluate the quality performance of the channel, the signal-to-noise ratio and transmission rate are set as the evaluation criteria of the channel quality. In the frequency band selected by the secondary user, the secondary user selected channels to meet its bandwidth requirements , there is The main users occupied channels, and possibly The channel conflicts due to the joint selection of several secondary users. The signal-to-noise ratio obtained by a secondary user is defined as:

[0087] ;

[0088] in, For the The sub-channel in the frequency band selected by the secondary user The gain, For the Primary users in subchannels The gain, Indicates secondary user With the remaining The gain generated by each conflicting channel of each secondary user is Represents cognitive user The noise spectral density is For cognitive users The transmission power, To interfere with other cognitive users of the current cognitive user of transmission power.

[0089] In S4, the agent interacts with the environment for training. When the environment enters a new state, it gives rewards to the agent and obtains an inheritable parameterized dynamic spectrum access strategy, which is as follows:

[0090] Step 1: Each cognitive user Treat it as a heterogeneous agent and then initialize the policy network and the global value network , and initialize the experience replay pool ;

[0091] Step 2: Learn potential better strategies through training and obtain a set of strategies , according to the action strategy combined with the current state Get the action vector, which includes the selection of the aggregation frequency band and the selection of the aggregation length; perform the action Get current moment rewards and the next state vector , Will Save in , randomly sample from the experience replay pool to obtain small batches of samples;

[0092] Step 3: Use the GAE function to calculate the advantage function based on the global V value network , which is the advantage function value of the action corresponding to the current state. The global V value network introduces an advantage weighting coefficient to weight the TD error and reduce the impact of noise. Random arrangement of

[0093] Step 4: Experimental command , for each agent from , updated by the following formula Network parameters :

[0094] ;

[0095] in, Indicates batch size, Indicates the length of each batch, Indicates status and The joint advantage function of Indicates the current strategy, Indicates the strategy before the update;

[0096] And update the V value network by the following formula:

[0097] ;

[0098] Step 5: Repeat steps 2 to 4 until the training is completed.

[0099] S5. Perform dynamic spectrum access according to the parameterized dynamic spectrum access strategy obtained in S4.

[0100] like Figure 1 As shown in FIG. 1 , a flowchart of heterogeneous agent proximal strategy optimization is provided in this embodiment, and the specific steps are as follows:

[0101] S1. Construct a simulation environment, establish a mathematical model based on the dynamic spectrum access environment of cognitive wireless networks, and initialize parameters, including the number of secondary users N, the number of channels perceived by each secondary user G, the number of channels in the entire network system L, the perceived length of each secondary user, and the bandwidth requirements, and then construct the spectrum environment; initialize the neural network training parameters in the deep reinforcement learning of heterogeneous agents; each cognitive user node perceives the channel to obtain the state information as the basis for action selection.

[0102] Consider an indoor environment where several primary users and secondary users are randomly distributed to establish a multi-user multi-channel wireless network, such as Figure 2 Assume that the system exists Orthogonal authorized channels can be allocated to each primary user for normal data communication. The primary user only needs to transmit according to its own needs. Assume that the secondary user is interfered by 802.11 in the Rician channel. Assume that there are M PU users in the indoor area. The secondary user is interfered by 802.11 in the Rician channel. There are M PU users (802.11b) and The system needs to have higher bandwidth requirements when transmitting voice and images. The secondary users in the system will sense the frequency band and look for idle and high-performance aggregate channels to access to meet the transmission requirements. When the system transmits, there will be an association between the desired link and the interference link. When the wireless signal propagates in space, it will attenuate, that is, path loss. The value of the channel gain will be affected by factors such as multipath fading. Considering the complexity of indoor channels, we use the commonly used path loss model: At the same time, in actual indoor situations, the transmitter and receiver equipment are generally within the line of sight, and the direct power cannot be ignored. Therefore, it is more reasonable to use the Rician attenuation model for simulation experiments. The channel gain formula is expressed as + ,in is determined by the path loss, and Factor Representation The ratio between the signal power at the receiver of the path and the scattered path ( ). is the phase of the signal arriving on the path, represents a circularly symmetric complex Gaussian random variable. Figure 2 In wireless communication systems, multiple devices (such as and ) share limited spectrum resources. To improve spectrum utilization efficiency, these devices can use spectrum aggregation technology, that is, by coordinating their respective transmitters and receivers, they can transmit data simultaneously at different times or frequencies. The figure shows the spectrum occupancy of each device in different time periods. The red blank box indicates the Select the aggregated channel area. The blue slash area indicates Select the aggregated channel. Different users choose different numbers and areas of aggregated sub-channels. Users have different perceived aggregation capabilities. Due to the limited spectrum in indoor transmission environments, we assume in our experiments that the sub-channel bandwidths are consistent and the complete spectrum can be equally divided into multiple sub-channels. Assume that the transmission power P of each is also consistent, and the carrier frequency is also fixed. According to the above settings, Indicates SU users in The gain of the subchannel. The signal-to-interference-noise ratio and transmission rate are used to evaluate the channel quality. Given that the system has multiple users, the frequency band selected by a certain SU may contain channels occupied by the main user and the channel accessed by the SU the day before yesterday. Secondary users assume that they perceive channels and aggregate the idle channels to meet their own bandwidth requirements , there are M main users occupying channels, and possibly The channels conflict due to the joint selection of multiple secondary users. The observation state and action space of cognitive users are heterogeneous due to the complexity of the actual wireless communication environment, that is, they cannot be completely consistent.

[0103] A dynamic spectrum access optimization strategy is generated based on the computational model of S1, which uses the deep reinforcement learning algorithm of heterogeneous agents to perform task allocation and access planning for cognitive users in the basic framework scenario, maximizing system throughput and minimizing collisions between users while ensuring that the main users in the network are not interfered with.

[0104] S2. Create heterogeneous agents based on heterogeneous agent proximal strategy optimization, create corresponding policy networks (Actor networks), value networks (Critic networks), and target networks corresponding to the Actor networks and Critic networks, set the state space, action space, and required reward function, as follows:

[0105] Step 1: Input prior knowledge into the simulation environment, including environmental status information. Each user relies on perceiving the environmental status, making decisions based on the Actor network, and selecting the communication frequency band for sending data next. , put it in The state of constantly perceiving heterogeneity with other users is defined as , ,in is the current spectrum status, The number of channels observed by the user, considering the user's bandwidth requirements, can be aggregated. Assume that the aggregation length , the observed state is input into the Actor network, and actions are selected based on the strategy , the action is defined as the selected access aggregation band. , the length of the system action space ， ， ， . is the selected access frequency band.

[0106] Step 2: Each user accesses the selected frequency band and sends data, observing whether the data is successfully received and the interference situation, so as to calculate the current action reward.

[0107] Step 3: Each user uploads the reward received in this time slot to the central Critic network. Critic can obtain the current feedback information of all users. Then update the Critic network, using To update the network parameters, . It is a discount factor, which indicates the degree of reference to future reward values. The larger it is, the more importance is attached to future rewards.

[0108] Step 4: Update the Actor network parameters using the following formula ,

[0109] ;

[0110] in, ,and Represents the old strategy. Repeat the above steps until the training is completed. Indicates batch size, Indicates the length of each batch, Indicates status and The joint advantage function of Indicates An intelligent agent, Indicates the current strategy, Indicates the strategy before the update. Indicates a given Indicates the current observation Take action in case of probability;

[0111] S3. The neural network is trained through improved heterogeneous agent proximal strategy optimization. The agent selects actions according to the current environment state. The environment enters a new state and feeds back rewards to the agent to obtain a dynamic spectrum access strategy. The trained dynamic spectrum access strategy is applied in the dynamic spectrum access environment: dynamic spectrum access is realized in the cognitive wireless network scenario. Users obtain potential better strategies through training and obtain a set of strategies. , according to the action strategy combined with the current state Get the action vector, which includes the selection of the aggregation frequency band and the selection of the aggregation length, and then execute the action Get current moment rewards and the next state vector ,Will Save in , randomly sample from the experience replay pool to obtain a small batch of samples, and then use the GAE function to calculate the advantage function based on the global V value network , draw the agent The Actor and Critic in the training module are both based on the multi-agent deep reinforcement learning algorithm structure, the Actor is used to select the access action, and the Critic is used to evaluate the reward value of the current dynamic spectrum access state. According to the dynamic spectrum access coordination based on heterogeneous agent deep reinforcement learning provided in this embodiment, the spectrum access step in the cognitive wireless network is executed.

[0112] In this embodiment, the number of cognitive users N=3, the number of channels in the system environment channels=32, and the perception length of each user G=8. Since the user needs to be aggregated within the perception range according to the bandwidth requirements after perception, the heterogeneity of each state space and action space will appear. In deep reinforcement learning, the neural network adopts a three-layer fully connected network, in which actorlrandcriticlr is set to 0.0001 and 0.001, beta=0.95, and max_grad_norm is 10 in the HAPPO learning parameters.

[0113] In this embodiment, Experience Memory (experience replay buffer) is an important component in multi-agent deep reinforcement learning, which is used to store and replay the experience data collected by the agent during the training process. These experience data include information such as the state, action, reward and next state of the agent; based on the proximal policy optimization algorithm of heterogeneous agents, the Actor network learns the action strategy of each agent, that is, selects the best action according to the current state, and the Critic network learns the value function of each agent, that is, predicts the long-term reward of each agent's current strategy; in a multi-agent environment, the advantage function is the difference between the state-action value function (Q-value) and the state value function (V-value). The advantage of using GAE is that it can reduce the variance of policy updates and improve sample efficiency; the sequential update mechanism is a method of coordinating the policy updates of different agents. Heterogeneous agents have different capabilities, observation ranges, and action spaces. The use of a sequential update mechanism can solve the instability and non-stationarity of policy learning in heterogeneous environments. is an importance sampling weight, a key component of the algorithm, used to update the agent’s policy network. is at the time step Except for the agent The joint strategy of all agents except Take action The probability of is in state Take action The advantage function estimate measures the average benefit of taking a certain action relative to the current strategy in a given state. Each operating environment corresponds to the specific process of step S2. The system environment model diagram is shown in Figure 2 This technical solution is used to simulate the dynamic spectrum access performance in a dynamic spectrum access environment with heterogeneity. The comparison algorithm uses the MAPPO algorithm, the heterogeneous agent proximal policy optimization HAPPO (Heterogeneous-Agent Proximal Policy Optimization) with shared parameters, and the HAA2C algorithm, as shown in Figure 2. Figure 3 and Figure 4 As shown in the figure, it is a performance comparison chart with several algorithms, including reward comparison and access avoidance comparison chart. HAPPO can greatly reduce interference between users and increase system throughput, helping cognitive users to better perform dynamic spectrum access. The methods compared with this method are: (1) Multi-Agent Proximal Policy Optimization (MAPPO): MAPPO (Multi-Agent Proximal Policy Optimization) is a multi-agent reinforcement learning algorithm that aims to solve the coordination control problem in multi-agent systems and can provide stable synchronous policy updates. (2) Heterogeneous Agent Proximal Policy Optimization with Shared Parameters HAPPO-DSA (share param): Compared with the proposed algorithm, this baseline adopts a parameter sharing mechanism. Experimental results show that parameter sharing will lead to suboptimal policy convergence due to the introduction of unreasonable policy constraints. (3) Heterogeneous Agent Proximal Policy Optimization with Non-Sequential Updates HAPPO-DSA (fixed order): Compared with the proposed algorithm, this baseline does not consider the sequential update mechanism, and the experimental results show relatively poor performance. (4) Multi-agent Advantage Actor-Critic (MAA2C) algorithm: It is an algorithm for multi-agent reinforcement learning. It is based on the Advantage Actor-Critic (A2C) algorithm and is extended to adapt to multi-agent environments.

[0114] Therefore, the present invention adopts the above-mentioned dynamic spectrum access method based on heterogeneous agent proximal strategy optimization. Aiming at the heterogeneity of each user in the dynamic spectrum access environment, the traditional AC algorithm is extended to the multi-agent deep reinforcement learning algorithm and the heterogeneous agent deep reinforcement learning is introduced. In the training stage, the Critic network can estimate the strategy value according to the global state and guide the Actor network to update, so as to maximize the reward, achieve the best access performance and minimize the interference between users. The HAPPO used has the advantages of fast convergence speed, less collision interference and good collaborative access effect; it enables cognitive users to improve the access performance as much as possible when the state space and action space are heterogeneous, solves the conflict problem of spectrum access when there is heterogeneity of cognitive users, and reduces the interference collision caused by heterogeneity, thereby improving the throughput and accuracy of dynamic spectrum access.

[0115] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A dynamic spectrum access method based on heterogeneous agent proximal strategy optimization, characterized in that: The following steps are involved: S1. Establish a computational model for a dynamic spectrum access environment with heterogeneous state space and action space, and initialize parameters, including the number of secondary users N, the number of channels perceived by each secondary user G, the number of channels in the entire network system L, the perceived length of each secondary user, and the bandwidth requirement, and then build the spectrum environment; S2, generating a dynamic spectrum access optimization strategy according to the calculation model of S1; S3, dynamic spectrum access environment, each user state space, each cognitive user perceives the spectrum environment through the state perception module, and introduces a reward mechanism; S4. Based on the optimization of the proximal strategy of heterogeneous agents, an adaptive dynamic spectrum access agent and an aggregate agent are constructed to create a policy network, a value network and their corresponding target networks. The agent interacts with the environment for training, and the environment enters a new state to feedback rewards to the agent, and obtains an inheritable parameterized dynamic spectrum access strategy, which is as follows: Step 1: Each cognitive user Treat it as a heterogeneous agent and then initialize the policy network and the global value network , and initialize the experience replay pool ; Step 2: Learn potential better strategies through training and obtain a set of strategies , according to the action strategy combined with the current state Get the action vector, which includes the selection of the aggregation frequency band and the selection of the aggregation length; perform the action Get current moment rewards and the next state vector , Will Save in , randomly sample from the experience replay pool to obtain small batches of samples; Step 3: Use the GAE function to calculate the advantage function based on the global V value network , draw the proxy Random arrangement of Step 4: Experimental command , for each agent from , updated by the following formula Network parameters : ； in, Indicates an agent; Indicates batch size, Indicates the length of each batch, Indicates status and The joint advantage function of Indicates An intelligent agent, Indicates the current strategy, Indicates the strategy before the update. Indicates a given Indicates the current observation Take action in case of probability; And update the V value network by the following formula: ； Step 5: Repeat steps 2 to 4 until the training is completed; S5. Perform dynamic spectrum access according to the parameterized dynamic spectrum access strategy obtained in S4.

2. The dynamic spectrum access method based on heterogeneous agent proximal strategy optimization according to claim 1 is characterized in that: The specific steps of establishing the computational model in S1 are as follows: based on the characteristic parameters in the dynamic spectrum access environment, the cognitive user state space, action space, throughput calculation and interference expression in the task environment are established into a mathematical model.

3. The dynamic spectrum access method based on heterogeneous agent proximal strategy optimization according to claim 1 is characterized in that: Dynamic spectrum access includes heterogeneous state spaces, action spaces, and different perception aggregations.

4. The dynamic spectrum access method based on heterogeneous agent proximal strategy optimization according to claim 3 is characterized in that: The state space is specifically: In each time slot , the agents will update the agent action network to obtain the optimal strategy; each cognitive user SU performs spectrum sensing on all channels to detect the channel status of each time slot: ； Indicates user In time slot The channel status of On the channel The probability of perception error of a SU is , the channel state transition probability is ； Due to the hardware limitation of the cognitive user SU sensing device, it cannot perceive all the sub-channels L in the environment; assuming The sensing capability of a SU is , represents the perceived length, and , then each SU perceives There are two states for a subchannel: occupied by SU or idle. When SU ​​observes a channel block, there are two states depending on whether the number of subchannels meets the bandwidth requirement: the channel block meets the transmission requirement or does not meet the transmission requirement. In time slot right The observation results of sub-channels It is expressed as follows: ； in, Indicates A cognitive user.

5. The dynamic spectrum access method based on heterogeneous agent proximal strategy optimization according to claim 3 is characterized in that: The action space is specifically: Getting a time slot After the observation results, SU chooses to Perceived channel block, the perception length of SU is The continuous channel block, its action space is expressed as ； Each SU selects consecutive The sub-channels are sensed and the aggregated frequency band is judged to meet the bandwidth requirements, which indicates that there is A sensing access action.

6. The dynamic spectrum access method based on heterogeneous agent proximal strategy optimization according to claim 1 is characterized in that: The specific reward mechanism in S3 is: After performing spectrum sensing and aggregation, each SU decides to access or idle. After taking an action, the system receives an immediate reward. When the system is interfered by SU or PU, a penalty term is set. Assuming that in an indoor environment, secondary users will be interfered by primary users PU in some sub-channels, and assuming that users in the system select idle channels for aggregation and access, the system transmission rate is specifically expressed as follows: ； in, represents the cognitive user bandwidth, represents the signal-to-noise ratio loss coefficient, is the signal-to-noise ratio, and the user’s reward is set as follows: a. No. The user chooses not to access the channel, and the reward ; b. For The selected channel block is used independently or shared with other secondary users. The total reward is , Indicates Bandwidth requirements of individual users; c. When When a secondary user accesses a channel that is already occupied by a primary user and cannot transmit, the reward is set to .

7. The dynamic spectrum access method based on heterogeneous agent proximal strategy optimization according to claim 1 is characterized in that: The adaptive dynamic spectrum access agent constructed in S4 is specifically: Assumptions Respectively transmitter, Receiver, Transmitter and The receiver's location coordinates, and Represent the transmitter and receiver respectively, Indicates Cognitive users, Indicates primary users; pass The link distance of the desired signal is calculated, while the interference signal propagation distance is calculated by As defined; when the primary user and the secondary user or several secondary users use the same wireless channel at the same time, transmission will be interfered. The wireless signal will be attenuated when propagating in the space between the transmitter and the receiver, which is called path loss; the indoor path loss model is used: ； The channel gain formula is expressed as: ； in, Determined by the path loss, for Factor, indicating The ratio of the receiver signal power of the path to the scattering path, and =5, yes The phase of the signal arriving on the path takes a value from a uniform distribution between 0 and 1. represents a circularly symmetric complex Gaussian random variable; due to the limited spectrum in indoor transmission environments, assuming that each channel has the same bandwidth, the entire spectrum is evenly distributed as channels; at the same time, the transmission power of each channel The same, the carrier frequency is unique and fixed; according to the above settings, the Secondary users in The channel gain on a channel is defined as: ; Set the signal-to-noise ratio and transmission rate as the evaluation criteria for channel quality. In the frequency band selected by the secondary user, the secondary user selected channels to meet its bandwidth requirements , there is The main users occupied channels, or The channel conflicts due to the joint selection of several secondary users. The signal-to-noise ratio obtained by a secondary user is defined as: ； in, For the The sub-channel in the frequency band selected by the secondary user The gain, For the Primary users in subchannels The gain, Indicates secondary user With the remaining The gain generated by each conflicting channel of each secondary user is Represents cognitive user The noise spectral density is For cognitive users The transmission power, To interfere with other cognitive users of the current cognitive user of transmission power.

Images