Emergency communication dynamic spectrum aggregation and access method based on maximum entropy reinforcement learning

Abstract

The application belongs to the technical field of wireless communication network, and provides an emergency communication dynamic spectrum aggregation and access method based on maximum entropy reinforcement learning, which comprises the following steps: at the beginning of each time slot, a primary user randomly accesses any channel of the communication frequency band of an emergency communication base station; a secondary user detects the state of all channels according to spectrum sensing, finds out an idle channel, aggregates the entire communication frequency band of the emergency communication base station into multiple frequency bands according to the aggregation length, and selects one of the frequency bands with multiple preset gain channels for access; the emergency communication base station calculates the reward of the secondary user based on the actor-critic network constructed based on the maximum entropy reinforcement learning strategy according to the frequency band accessed by the secondary user, and determines whether the secondary user performs data transmission according to the positive and negative values of the reward; wherein the primary user and the secondary user are divided according to the user attributes of the emergency communication terminal.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of wireless communication network, and particularly relates to an emergency communication dynamic spectrum aggregation and access method based on maximum entropy reinforcement learning. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Natural disasters such as heavy rain, earthquake, volcano, tsunami, etc. often cause catastrophic damage to power and communication networks. Unmanned aerial vehicles (UAVs) can quickly restore communication services due to their flexible configuration and maneuverability, which is essential for assessing disaster conditions and providing emergency assistance. The advantage of deploying UAV-assisted rescue cognitive networks (UAV-RCNs) is that UAVs are more adaptable to different dangerous environments than ground base stations and can provide communication services as temporary mobile base stations, which can more efficiently assist search, rescue and communication tasks. In a short time after a natural disaster, UAVs can provide flexible communication services for ground users through ground control stations or satellite communication links. However, there are many challenges in providing communication through UAV-RCNs. First, due to the size and weight restrictions, the energy storage of UAVs is limited, which limits the communication services that UAVs can provide. Second, the spectrum resources authorized for UAV communication are very limited. Finally, although text and voice messages are usually preferred in emergency communication, different users (such as rescue personnel and disaster victims) may have different requirements for other communication services, including pictures, videos, geographic locations, and even browser access. Therefore, how to make full use of limited spectrum and energy to provide high-quality and low-energy communication for different users is still a problem faced by UAV-RCNs.

[0004] Cognitive radio networks are considered a promising solution to the problem of spectrum scarcity for UAVs in disaster rescue networks. Cognitive radio network technology can be used to improve the spectrum efficiency in UAV-RCNs, where UAVs act as rescue base stations to provide communication for all users. The inventors have found that existing spectrum access techniques based on cognitive radio networks still have the following problems:

[0005] 1) Most existing research considers that users only access one channel, ignoring the demand of users for higher bandwidth;

[0006] 2) Most research only considers that users have complete correct perception of the spectrum, but in real environments, due to the influence of noise and interference, spectrum perception may be wrong;

[0007] 3) Since there are multiple users with different broadband needs in the system, the DRL algorithm with deterministic strategy is prone to getting trapped in local optima, causing multiple users to choose the same optimal frequency band, which increases the collision between users and reduces the achievable rate and spectrum utilization of users. Summary of the Invention

[0008] To address the technical problems mentioned above, this invention provides an emergency communication dynamic spectrum aggregation and access method based on maximum entropy reinforcement learning. This method enables secondary users to learn the dynamically changing unknown spectrum environment and access spectrum holes with good channel quality while avoiding conflicts among all users, thereby improving the access success rate.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A dynamic spectrum aggregation and access method for emergency communication based on maximum entropy reinforcement learning, comprising:

[0011] At the beginning of each time slot, the main user randomly accesses any channel of the emergency communication base station's communication frequency band;

[0012] Secondary users detect the status of all channels based on spectrum sensing, find idle channels, and aggregate the entire communication frequency band of the emergency communication base station into multiple frequency bands according to the aggregation length, and select one of the frequency bands with multiple preset gain channels for access.

[0013] The emergency communication base station calculates the reward for secondary users based on the frequency band accessed by the secondary users and constructs an actor-commentator network based on the maximum entropy reinforcement learning strategy. The positive or negative value of the reward determines whether the secondary user should transmit data.

[0014] The distinction between primary and secondary users is based on the user attributes of the emergency communication terminal.

[0015] In one implementation, the characteristics of the channel remain unchanged within each time slot until the secondary user finishes data transmission and the primary user initiates new access at the start of the next time slot.

[0016] In one implementation, if the secondary user's reward is positive, the secondary user transmits data in the selected channel.

[0017] As one implementation, if the secondary user's reward is negative, the secondary user fails to transmit data and stores the state, reward, and action in the experience pool to train the parameters of the actor-critic network.

[0018] As one implementation method, the nth secondary user in the first... Rewards per time slot The expression is:

[0019]

[0020] in, This represents the transmission rate for the nth secondary user. This indicates a penalty.

[0021] As one implementation method, the nth secondary user transmission rate for:

[0022]

[0023] In the formula, For the bandwidth requirements of the nth secondary user, This represents the signal-to-noise ratio obtained after the nth secondary user accesses the channel.

[0024] As one implementation method, a regression loss function is used to update the parameters of the critic network in the actor-critic network.

[0025] As one implementation method, regression loss function for:

[0026]

[0027] In the formula Represents the expectation function, Indicates from the experience pool Sample a batch of empirical units to update the critic network parameters , This represents the Q-value of the critic network. This represents the target Q-value of the target critic network. As a discount factor, The output value of the target network. The entropy term representing the maximum entropy; To determine the randomness strategy for the next time slot of secondary users The action to be taken in the next time slot; , , These represent the observation state, action, and reward, respectively; the subscript n in the formula represents the nth secondary user, and the superscript... t Indicates the first t Each time slot, superscript t +1 indicates the first t +1 time slot.

[0028] As one implementation, the parameters of the actor network in the actor-critic network are updated according to the policy gradient function.

[0029] As an implementation, the policy gradient function is:

[0030]

[0031] wherein, is a policy gradient function, represents an expected function, represents a randomness policy from an experience pool sampling a batch of experience units to update the parameters of the actor network ; represents a randomness policy under conditions; represents a randomness policy under conditions; is a critic network parameter; is an entropy term of maximum entropy; , respectively represent an observation state and an action; the subscript n in the formula represents the nth secondary user, and the superscript t represents the t time slot.

[0032] Compared with the prior art, the present application has the following advantages:

[0033] (1) The present application is based on a spectrum usage model of a cognitive network for deploying unmanned aerial vehicle-assisted rescue, and divides primary users and secondary users according to the user attributes of emergency communication terminals, and dynamically shares available spectrum by using cognitive radio technology; the present application provides spectrum aggregation and access technology to manage the spectrum access of secondary users with different bandwidth requirements, and the secondary users first need to sense and aggregate multiple idle channels with good channel quality, and then access the spectrum without conflict, thereby improving the spectrum utilization rate;

[0034] (2) The present application solves the problem of dynamic spectrum access in an unmanned aerial vehicle-assisted rescue cognitive network based on a multi-agent reinforcement learning algorithm, and avoids local optimization and better explores more possibilities by combining an entropy term and a soft update policy; wherein, a distributed training and execution algorithm framework aims to optimize the efficiency and power consumption of spectrum access, maximally reduces interference and interruption, and maximally improves achievable rates, so as to robustly realize the access strategy of multiple secondary users, better improve the system performance, and effectively reduce the negative effects of user heterogeneity and independence.

[0035] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0037] Figure 1 This is a schematic diagram of a multi-agent maximum entropy reinforcement learning algorithm with distributed training and execution, as described in an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of a cognitive network scenario for drone-assisted rescue according to an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of the dynamic spectrum aggregation and access model according to an embodiment of the present invention;

[0040] Figure 4 The number of SUs in the embodiments of the present invention A graph showing the relationship between long-term cumulative reward and the number of training sessions;

[0041] Figure 5 The number of SUs in the embodiments of the present invention A graph showing the relationship between average collision rate and the number of training iterations;

[0042] Figure 6 The number of SUs in the embodiments of the present invention The relationship between power consumption and the number of training iterations. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0045] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0046] Terminology Explanation:

[0047] UAV-RCNs refers to Unmanned Aerial Vehicle-Assisted Rescue Cognitive Networks;

[0048] PUs refers to primary users;

[0049] SUs, are referred to as secondary users;

[0050] RL, is referred to as reinforcement learning;

[0051] DRL, is referred to as deep reinforcement learning;

[0052] DQN, is referred to as deep Q-network.

[0053] In the deployment of unmanned aerial vehicle (UAV) assisted rescue cognitive network, rescue personnel can be regarded as primary users (PUs) who have the right to use the licensed frequency band, while the disaster victims can be regarded as secondary users (SUs). SUs can detect the spectrum holes that are not occupied by PUs by using spectrum sensing technology. Incorrect spectrum sensing may be detrimental to the spectrum access of SUs. In addition, SUs access the perceived spectrum holes through spectrum access technology to improve spectrum utilization and avoid conflict with PUs. Therefore, SUs need to adjust their data transmission by adapting to the dynamic spectrum environment. However, spectrum holes are usually fragmented and cannot meet the communication needs of SUs alone. Spectrum aggregation technology, as a means to fully utilize fragmented free spectrum holes, has attracted widespread attention. Spectrum aggregation can aggregate multiple small bandwidth scattered spectrum holes into a larger continuous spectrum segment to improve spectrum utilization. However, due to the limitations of hardware transceivers, the bandwidth range that SUs can aggregate is limited. Spectrum resources are dynamic, and it is difficult for SUs to achieve spectrum aggregation and access. Therefore, an effective aggregation and access scheme needs to be designed to improve the spectrum efficiency and system performance of UAV-RCNs. Reinforcement learning (RL) can effectively enable the UAV to obtain the optimal strategy when the state and action space is small. However, for complex and large-scale networks, traditional RL algorithms cannot quickly process huge state and action spaces.

[0054] The embodiment provides an emergency communication dynamic spectrum aggregation and access method based on maximum entropy reinforcement learning, which comprises:

[0055] The primary user randomly accesses any channel of the emergency communication base station communication frequency band at the beginning of each time slot;

[0056] The secondary user detects the state of all channels according to spectrum sensing, finds out the idle channel, and aggregates the entire communication frequency band of the emergency communication base station into multiple frequency bands according to the aggregation length, and selects one of the frequency bands with multiple preset gain channels for access;

[0057] The emergency communication base station calculates the reward of the secondary user based on the actor-critic network constructed based on the maximum entropy reinforcement learning strategy according to the frequency band accessed by the secondary user, and determines whether the secondary user performs data transmission according to the positive and negative values of the reward.

[0058] The distinction between primary and secondary users is based on the user attributes of the emergency communication terminal.

[0059] Specifically, consider as follows Figure 2 The UAV-RCNs scenario shown is assumed to have network components with varying bandwidth requirements. of Each PU and has bandwidth requirements of One SUs. Consider as follows Figure 3 The channel access model shown, at the beginning of each time slot, first... Each PU (Power Utility) randomly accesses any channel within a frequency band. Each channel may be in either occupied (1) or idle (0) state. The SU (Subsystem for Utilities) is unaware of the channel occupancy patterns of the PUs. Within each time slot, the channel characteristics remain unchanged until the SU finishes data transmission. At the start of the next time slot, the PUs initiate new access. Secondly, the SU needs to perceive the state of all channels based on the spectrum sensing accuracy to find idle channels and avoid collisions with the PUs. Then, based on the observed state of the SU, the SU aggregates the entire frequency band into multiple frequency bands according to the aggregation length and selects a spectrum with multiple high-gain channels for access. Finally, the receiver feeds back a reward to the SU based on the accessed frequency band. If the reward is positive, the SU transmits data in the selected channel. If the reward is negative, the SU fails to transmit and stores the state, reward, and action in the experience pool to encourage the network to train in a positive direction.

[0060] The dynamic spectrum aggregation and access method based on the multi-agent maximum entropy reinforcement learning algorithm of the present invention is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0061] Step 1: Initialize environment parameters, including the number of channels. Bandwidth length Transmission power and carrier frequency The number of PUs and bandwidth requirements The number of SUs Bandwidth requirements and polymerization length Randomly initialize the locations of drones, PUs, and SUs to construct the spectrum environment; initialize the Actor-Critic (AC) network parameters, including the training parameters of the actor and critic networks. and Training parameters of the target actor network and the target critic network and , network learning rate , soft update factor , discount factor , capacity of the experience pool and the sampling batch size, the number of training epochs, the actor network inputs the state of each SU's sensing channel, gets the actions of all SUs, inputs these actions and the state of the sensing channel into the critic network, and can get the critic network's evaluation of the action Q function;

[0062] Step 2, all SUs according to their respective spectrum sensing accuracy sensing the spectrum state of the th time slot , wherein represents that the th channel in the th time slot is occupied by the PU, represents that the th channel is idle; and the observation state of the th SU is , wherein , , wherein represents that the th SU senses that the th channel state is occupied in the th time slot, represents that the th SU senses that the th channel state is idle, wherein the sensing accuracy of the th SU is , is the detection probability of sensing the spectrum, is the false alarm probability;

[0063] Step 3, based on a random strategy , each SU selects an action in turn according to the access order , wherein is the frequency band selected by the SUs in the th time slot, is the aggregate length of the SUs, then is the number of frequency bands that users can aggregate; means that the SU gives up accessing the frequency band because the selected frequency band does not meet the bandwidth requirement, means that the SU selects the th frequency band from the frequency bands for access; is the channel selected by the user in the selected frequency band; is the channel gain of the selected channel; The actions selected by each SUs are ;

[0064] Step 4: All SUs perform actions in the spectrum environment. Then, the reward for all SUs fed back from the receiver is... ,in It is the first The SU in the first The reward for each time slot is expressed as follows:

[0065] (I)

[0066] in This indicates penalties for actions that do not meet requirements. The reward can be described by the following four cases, representing the transmission rate obtained by SUs after accessing the channel:

[0067] 1) No. When a SU accesses multiple channels that are not occupied by PUs or other SUs, the first Each SU is only affected by interference from other channels occupied by PUs in the selected frequency band. The reward is set to the first... The transmission rate obtained by each SU ;

[0068] 2) When the first When one SU accesses one or more channels used by other SUs, collisions will occur between SUs. The reward is the transmission rate. and penalty items The sum of these is intended to prevent collisions between SUs during the action selection process.

[0069] 3) When the first When a channel occupied by a SU accessing a PUs causes interference to the PUs, the reward is set to... As negative feedback;

[0070] 4) When the first The length of the frequency band selected by each SU is less than the bandwidth requirement. At that time, Defined as a negative reward to encourage SUs to choose frequency bands with more available channels;

[0071] Furthermore, the first SU transmission rate The expression is as follows:

[0072] (II)

[0073] In the formula, For the first bandwidth requirement of the SU, denotes the signal-to-noise ratio obtained by the SU after accessing the channel, denotes the signal-to-noise ratio obtained by the SU after accessing the channel,

[0074] (III)

[0075] wherein denotes the noise spectral density, denotes the transmission power of the i-th channel, denotes the transmission power of the i-th channel, denotes the channel gain of the i-th channel in the selected frequency band of the i-th SU, denotes the channel gain of the i-th channel in the selected frequency band of the i-th SU, denotes the channel gain of the i-th channel in the selected frequency band of the i-th PU, denotes the channel gain of the i-th channel in the selected frequency band of the i-th PU, denotes the channel gain of the i-th channel in the selected frequency band of the i-th PU, denotes the channel gain of the i-th channel in the selected frequency band of the i-th PU, denotes the channel gain of the i-th channel in the selected frequency band of the i-th PU,

[0076] (IV)

[0077] wherein is the channel gain of the i-th conflicting channel in the selected frequency band of the i-th SU with the j-th conflicting SU, is the channel gain of the i-th conflicting channel in the selected frequency band of the i-th SU with the j-th conflicting SU, is the channel gain of the i-th conflicting channel in the selected frequency band of the i-th SU with the j-th conflicting SU, is the channel gain of the i-th conflicting channel in the selected frequency band of the i-th SU with the j-th conflicting SU, can be calculated by a channel model, as shown in the following formula: Figure 2

[0078] (V)

[0079] wherein , and are two-dimensional coordinates of the UAV, PUs and SUs respectively, is the vertical height of the UAV, and thus the path loss can be obtained by using an air-to-ground fading model as follows:

[0080] (VI)

[0081] wherein is the distance between the UAV and each user, denotes the carrier frequency of the channel, denotes the speed of light, and are additional losses of the visual path and the non-visual path respectively, ​This represents the probability of a visible path being generated. These are parameters determined by the environment; based on the fading model formula, the channel gain is calculated using the Rician channel model, which can be expressed by the formula as follows:

[0082] (VII)

[0083] In the formula , The Rician factor represents the ratio of the visible path energy to the scattered energy. This represents the visible path components that follow a uniform distribution. Represents the non-visible path component of a complex Gaussian random variable;

[0084] Step 5: Update the parameters of the spectrum environment. The SUs senses the new spectrum environment to obtain the observation status for the next time slot. ;

[0085] Step 6: Observe the current time slot status of all users. Current selected action Rewards for the current time slot Observation status in the next time slot Based on experience units The information is stored in the experience pool in the form of [data / data].

[0086] Step 7: Once the experience pool has accumulated to a certain size, begin the network's learning process; randomly sample a small batch of experience units using a custom sampling batch number. Update the networks of all SUs sequentially;

[0087] Step 8: Input the observation status of SUs in the empirical cells into the critic network. and actions Q-value is obtained through the critic network. ;Observe the state in the next time slot of the target actor network input SUs. According to the next time slot randomness strategy of SUs Action to obtain the next time slot ;Observe the state in the next time slot of the target critical network input SUs and actions The target Q-value is obtained through the target critic network. ;

[0088] Step 9: Based on the Q value obtained in Step 8 and target Q value Calculate the regression loss function This is used to update the parameters of the critic network; the regression loss function expression is as follows:

[0089] (VIII)

[0090] wherein represents the expected function, represents the experience pool, a batch of experience units are sampled to update the critic network parameters , represents the Q value of the critic network, represents the target Q value of the target critic network, is a discount factor, is the output value of the target network, and the entropy term of the maximum entropy is defined as , is a control factor for controlling the randomness of the optimal policy;

[0091] Step 10, inputting the observation state of SUs in the experience unit into the actor network , obtaining the current action according to the current random policy of SUs ; inputting the observation state of SUs and the action into the critic network, obtaining the Q value of the neural network ;

[0092] Step 11, calculating the policy gradient according to the current action and the Q value obtained in step 10, which is used to update the parameters of the actor network; the expression of the policy gradient function is as follows:

[0093] (IX)

[0094] wherein a batch of experience units are sampled from the experience pool to update the actor network parameters ;

[0095] Step 12, calculating the parameters and of the target actor network and the target critic network according to the network parameters and of the actor network and the critic network obtained in steps 8 to 11, which are expressed by the following formulas:

[0096] (X)

[0097] wherein, is a soft update factor.

[0098] Step 13, repeating steps 2 to 12 to update the network and policy until the end of the custom training round, obtaining the best policy of all users, so that all users obtain the maximum long-term cumulative reward.

[0099] The number of SUs in the embodiment is set to , the relationship between the number of SUs and the long-term cumulative reward, the average collision rate and the power consumption obtained by comparing the method provided by the application with the DQN algorithm is shown in the experimental graphs of Figure 4 , Figure 5 and Figure 6 . Using the same number of training rounds, the long-term cumulative reward obtained by the method provided by the application is significantly improved compared with the DQN algorithm, and the average collision rate and power are greatly reduced, that is, the quality of the multiple channels accessed by the SUs is better, the number of successful times and the number of times of avoiding conflicts are more, and the system performance is more excellent.

[0100] The above only describes the preferred embodiments of the application and is not intended to limit the application. For those skilled in the art, the application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. An emergency communication dynamic spectrum aggregation and access method based on maximum entropy reinforcement learning, characterized in that, The method comprises the following steps: The primary user randomly accesses any channel of the emergency communication base station at the beginning of each time slot; The secondary user detects the state of all channels according to spectrum sensing, finds an idle channel, aggregates the entire communication frequency band of the emergency communication base station into multiple frequency bands according to the aggregation length, and selects a frequency band with multiple preset gain channels for access; The emergency communication base station calculates the reward of the secondary user based on the actor-critic network constructed based on the maximum entropy reinforcement learning strategy according to the frequency band accessed by the secondary user, and determines whether the secondary user performs data transmission according to the positive or negative value of the reward; The primary user and the secondary user are divided according to the user attributes of the emergency communication terminal; The nth secondary user is rewarded in the nth time slot The expression is:​ wherein, is the nth secondary user transmission rate; denotes the penalty term; n th secondary user transmission rate is: wherein the bandwidth requirement of the nth secondary user, Snr(n) represents the signal-to-noise ratio obtained by the nth secondary user after accessing the channel. A regression loss function is used to update the parameters of the critic network in the actor-critic network; Regression loss function is: In the formula Represents the expectation function, Indicates from the experience pool Sample a batch of empirical units to update the critic network parameters , This represents the Q-value of the critic network. This represents the target Q-value of the target critic network. As a discount factor, The output value of the target network. The entropy term representing the maximum entropy; To determine the randomness strategy for the next time slot of secondary users The action to be taken in the next time slot; , , These represent the observation state, action, and reward, respectively; the subscript n in the formula represents the nth secondary user, and the superscript... t Indicates the first t Each time slot, superscript t +1 indicates the first t +1 time slot.

2. The method of claim 1, wherein the method is based on maximum entropy reinforcement learning for emergency communication dynamic spectrum aggregation and access. In each time slot, the characteristics of the channel remain unchanged until the secondary user ends data transmission, and the primary user performs new access at the beginning of the next time slot.

3. The method of claim 1, wherein the method is based on maximum entropy reinforcement learning for emergency communication dynamic spectrum aggregation and access, and wherein the method further comprises: If the reward of the secondary user is positive, the secondary user performs data transmission in the selected channel.

4. The method of claim 1 or 3, wherein, If the reward of the secondary user is negative, the secondary user fails to transmit data, and the state, reward and action are stored in the experience pool to train the parameters of the actor-critic network.

5. The method of claim 1, wherein the method is based on maximum entropy reinforcement learning for emergency communication dynamic spectrum aggregation and access, and wherein the method further comprises: The parameters of the actor network in the actor-critic network are updated according to the policy gradient function.

6. The method of claim 5, wherein the method is based on maximum entropy reinforcement learning for emergency communication dynamic spectrum aggregation and access, and wherein the method further comprises: The policy gradient function is: in, Let be the policy gradient function. Represents the expectation function, Indicates from the experience pool Sample a batch of experience units to update the parameters of the actor network. ; Indicates in under conditions The randomness strategy; This represents the Q-value of the critic network. For the critic's network parameters; The entropy term representing the maximum entropy; , These represent observation state and action, respectively; the subscript n in the formula represents the nth secondary user, and the superscript... t Indicates the first t Each time slot.

Images