Cognitive unmanned aerial vehicle network spectrum sharing method based on multi-agent reinforcement learning

Abstract

The invention provides a cognitive unmanned aerial vehicle network spectrum sharing method based on multi-agent reinforcement learning. The method comprises the implementation steps that a cognitive unmanned aerial vehicle network model and a cognitive unmanned aerial vehicle network and multi-agent reinforcement learning interaction network model are constructed; obtaining a training data set and defining a reward function; and carrying out iterative training on the interactive network model. In the process of training the interactive network model, the multiple agents interact with the cognitive unmanned aerial vehicle network in real time through reinforcement learning, spectrum resource allocation can be adaptively adjusted according to the communication requirement of each cognitive unmanned aerial vehicle and the dynamic change of the environment, and the spectrum resource allocation efficiency is improved. And the loss of the local Q network in each time step is calculated according to the target Q value corresponding to each experience calculated by calculating the reward of interaction between each Q agent and each environment state through the throughput of each cluster member on each channel, so that the resource waste is reduced, the spectrum conflict is avoided, and the maximization of the network throughput is realized. Therefore, the spectrum utilization rate is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology and relates to a cognitive drone network spectrum sharing method, and specifically to a cognitive drone network spectrum sharing method based on multi-agent reinforcement learning. Background Art

[0002] The cognitive drone network is a network composed of a group of cognitive drones and ground stations. Cognitive drone network spectrum sharing is to dynamically adjust the spectrum usage strategy through spectrum sensing technology, share limited spectrum resources between the aerial drone network and the ground network, optimize spectrum utilization, reduce interference, improve the efficiency of communication and data transmission, and use spectrum blanks to achieve efficient communication. Its technical idea is to use cognitive radio technology to allow cognitive drones to dynamically sense and select idle spectrum, and flexibly adjust spectrum usage strategies according to the needs of the ground network, thereby achieving efficient spectrum sharing.

[0003] The difficulties of cognitive drone network spectrum sharing mainly lie in spectrum perception and management, dynamic spectrum allocation, and interference suppression. Accurately perceiving spectrum usage in air and ground networks and efficiently managing resources are prerequisites for improving throughput. In the face of real-time changes in network status and demand, it is crucial to dynamically and adaptively adjust spectrum allocation strategies. In addition, if the interference between air and ground networks is not properly controlled, it will weaken resource utilization efficiency and reduce the actual throughput that can be achieved.

[0004] In cognitive drone networks, higher and more efficient spectrum utilization often helps improve throughput. Excellent spectrum allocation strategies, intelligent decision-making methods that adapt to dynamic environmental changes, favorable channel conditions, appropriate drone location and altitude control, and flexible response to business needs can all improve spectrum utilization and promote throughput growth. On the contrary, fixed strategies, increased interference, or inflexible allocation strategies will reduce spectrum utilization and data transmission efficiency, resulting in limited room for throughput improvement.

[0005] Existing cognitive drone network spectrum sharing methods, such as the patent application with application publication number CN113852966A and titled "A 3D spectrum sharing method and device for a cognitive drone network", discloses a 3D spectrum sharing method for a cognitive drone network. This method obtains the achievable throughput of the cognitive drone network downlink by constructing a three-dimensional composite spatiotemporal spectrum perception model based on a line-of-sight probability model and the flexibility of the drone position, and then formulates a cognitive drone network throughput optimization model. By jointly optimizing the drone perception time, transmission power and flight altitude, the downlink throughput is maximized, thereby solving the problem of low spectrum utilization. However, because it only relies on a fixed optimization solution process, this method has the defect of wasting spectrum resources when facing rapidly changing environments and diversified demand scenarios, which affects the room for improving the utilization of the cognitive drone network spectrum. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose a cognitive drone network spectrum sharing method based on multi-agent reinforcement learning to solve the technical problem of low spectrum utilization in the prior art.

[0007] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0008] (1) Building a cognitive drone network model:

[0009] A cognitive UAV network model is constructed, which includes M cognitive UAVs and U ground stations communicating with each other. The M cognitive UAVs are divided into K clusters containing at least two cognitive UAVs. The cognitive UAV located at the center of each cluster is the cluster head, and the other cognitive UAVs are cluster members communicating with the cluster head. The cognitive UAV network model includes I communication between K cluster heads and J cluster members. H I between channels and U ground stations G There are a total of I channels composed of channels, I = I H +I G ,J=MK, where M≥2, U≥2, K≥1;

[0010] (2) Constructing a reinforcement learning interactive network model of cognitive drone network and multi-agents:

[0011] Construct a reinforcement learning interaction network model including a cognitive drone network model and M Q agents, the same number as the number of cognitive drones in the network model, where each Q agent includes a local Q network and a target Q network with the same structure;

[0012] (3) Obtain the training data set and define the reward function:

[0013] Take a training dataset consisting of a state space S consisting of Z environment states and an action space A consisting of L actions, and define the reward r for the m-th Q-agent to interact with each environment state at the t-th time step t , where Z = 3, L = 2, t∈[1,T], T represents the total number of time steps, T ≥ 1000;

[0014] (4) Iterative training of the reinforcement learning interaction network model:

[0015] The reinforcement learning interaction network model is iteratively trained through the training data set, and the output of the trained reinforcement learning interaction network model is used as the spectrum sharing result of the cognitive UAV network.

[0016] Compared with the prior art, the present invention has the following advantages

[0017] 1. In the process of training the reinforcement learning interactive network model, the present invention uses a reinforcement learning algorithm to interact with the cognitive drone network in real time. According to the specific communication needs of each cognitive drone and the dynamic changes of the environment, the spectrum resource allocation is adaptively adjusted, which reduces resource waste and effectively avoids spectrum conflicts, maximizes network throughput, and effectively improves spectrum utilization.

[0018] 2. The present invention calculates the reward of each Q agent interacting with each environmental state through the throughput of each cluster member on each channel, and optimizes the loss function of the local Q network by combining the dual network structure with the target Q value corresponding to each experience calculated by the reward. During the optimization process, the dual network structure improves the stability and training efficiency of the model by separating the learning of state value and action advantage. The introduced noise network further enhances the exploration ability, reduces the waste of spectrum resources and spectrum conflicts, and thus further improves the spectrum utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is an implementation flow chart of the present invention.

[0020] Figure 2 It is a simulation comparison diagram of the total throughput of cognitive drone networks of the present invention and the prior art. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0022] Reference Figure 1 , the present invention comprises the following steps:

[0023] Step 1) Build a cognitive drone network model:

[0024] A cognitive UAV network model is constructed, which includes M cognitive UAVs and U ground stations communicating with each other. The M cognitive UAVs are divided into K clusters containing at least two cognitive UAVs. The cognitive UAV located at the center of each cluster is the cluster head, and the other cognitive UAVs are cluster members communicating with the cluster head. The cognitive UAV network model includes I communication between K cluster heads and J cluster members. H I between channels and U ground stations G There are a total of I channels composed of channels, I = I H +I G ,J=MK, where M≥2, U≥2, K≥1;

[0025] In this embodiment, M=120, U=4, K=4, J=29, I H =116,I G =20, I=136.

[0026] Step 2) Construct a reinforcement learning interactive network model of cognitive drone network and multi-agents:

[0027] Construct a reinforcement learning interaction network model including a cognitive drone network model and M Q agents, the same number as the number of cognitive drones in the network model, where each Q agent includes a local Q network and a target Q network with the same structure;

[0028] In this embodiment, the structures of the local Q network and the target Q network are as follows: the input layer receives feature inputs from the training data set. Then, the input state is feature extracted through 5 layers of superimposed fully connected layers. All fully connected layers introduce parameter noise, and ReLU is used as the activation function after each layer. Among them, the first layer maps the input state to 256-dimensional features, the second layer upgrades the features to 512, the third and fourth layers maintain 512-dimensional features, and the fifth layer maps the features to 256 dimensions. After completing the feature extraction of the fifth layer, the network structure is divided into a value stream and an advantage stream. The value stream maps the 256-dimensional features to a single scalar state value, and the advantage stream outputs the corresponding advantage value for each optional action. The state value and advantage value constitute the final output result of the network.

[0029] Step 3) Get the training dataset and define the reward function:

[0030] Take a training dataset consisting of a state space S consisting of Z environment states and an action space A consisting of L actions, and define the reward r for the m-th Q-agent to interact with each environment state at the t-th time step t , where Z = 3, L = 2, t∈[1,T], T represents the total number of time steps, T ≥ 1000;

[0031] The state space S and action space A described in this step, where:

[0032] S={s z |z∈[1,2,3]}

[0033] s1=γ u [i G ],s2=H,s3=W

[0034] A={a l |l∈[1,2]}

[0035] a1=ρ[i G ]i G ,a2=P select

[0036] Among them, H, W, ρ[i G ]i G and P select Represents the task information, waiting time, selected channel and transmission power of each cluster member, i G represents the i-th channel between U ground stations, γ u [i G ] represents the signal-to-noise ratio of the u-th ground station.

[0037] Step 4) Iteratively train the reinforcement learning interaction network model:

[0038] The reinforcement learning interaction network model is iteratively trained through the training data set, and the output of the trained reinforcement learning interaction network model is used as the spectrum sharing result of the cognitive drone network;

[0039] The iterative training of the reinforcement learning interaction network model described in this step is implemented in the following steps:

[0040] Step 4a) Initialize the number of iterations to x, the maximum number of iterations to X, X ≥ 1500, and the network parameters of the local Q network and the target Q network in the xth iteration are θ x ,θ' x , and let x=0.

[0041] Step 4b) The local Q network in each Q agent passes through each action a at the tth time step l and environmental state s z The signal-to-noise ratio γ of the jth cluster member on the i-th channel obtained by interaction j [i] Calculate the corresponding throughput C j [i], and through C j [i] Calculate the relationship between each Q agent and s z Interaction Reward t , then r t 、a tl and tz, and the environmental state s at the t+1th time step (t+1)z , whether the sign of the end d constitutes the experience (s tz ,a tl ,r t ,s (t+1)z ,d), and obtain the experience set E=[E1,E2,...,E m ,...E M ], where E m represents the experience of the mth agent T time steps;

[0042] The throughput C of the jth cluster member on the i-th channel in step 4b) j [i], and the reward r for each Q-agent’s interaction with each environment state t , the calculation formulas are:

[0043] C j [i] = Blog(1+γ j [i])

[0044]

[0045] I u =(P j [i]g j,u [i G ])ρ[i G ]

[0046]

[0047] I j =(P j' [i']g j',j [i G ']+P u [i G ] u,j [i G ])ρ[i G ]

[0048]

[0049] Where B represents the bandwidth of the channel, γ j [i] represents the signal-to-noise ratio of the jth cluster member on the i-th channel, γ u [i G ] th represents the minimum signal-to-noise ratio required to establish reliable communication between the u-th ground station and other ground stations, γ j [i] th represents the minimum signal-to-noise ratio required to establish reliable communication between the jth cluster member and the cluster head of its cluster, φ represents a negative reward, φ>0, Pu [i] represents the transmission power of the u-th ground station, g u [i G ] indicates that the uth ground station is the transmitter and the ith ground station is the transmitter. G The channel gain on the channel, σ 2 is the noise power, I u represents the interference power to the u-th ground station, P j [i] represents the transmission power of the jth cluster member on the i-th channel, g j,u [i G ] indicates that the jth cluster member is responsible for the uth ground station at the ith G The interference channel gain of the channel, g j [i] represents the channel gain of the jth cluster member communicating with the cluster head of the cluster through the i-th channel, I j represents the interference power to the jth cluster member, P j' [i'] represents the transmission power of the j'th cluster member on the i'th channel, g j',j [i G ] indicates that the j'th cluster member has a G The interference channel gain of the channel, g u,j [i G ] indicates that the u-th ground station has a G The interference channel gain of the channel, ρ[i G ]=1 means that the communication between the jth cluster member and the cluster head of its cluster shares the ith cluster in the ground network. G channels, ρ[i G ]=0 means the i-th G channels are not multiplexed;

[0050] In this embodiment, B=6 MHz, φ=5, σ 2 =-94dBm;

[0051] By designing the reward function as the total throughput of all cluster members, the present invention allows the multi-agent reinforcement learning model to always aim at improving spectrum utilization during the decision-making process, which helps the reinforcement learning algorithm to intuitively pursue the maximization of spectrum utilization and simplifies the optimization process in complex environments. This design can not only effectively guide multi-agents to coordinate resource allocation, but also adapt to diverse communication needs, ultimately achieving efficient operation and global performance improvement of cognitive drone networks.

[0052] Step 4c) The local Q network in each Q agent uses the experience set E of each agent m Perform N random samplings to obtain N experiences The target Q network and the local Q network are respectively obtained through each experience (sn ,a n ,r n ,s n+1 ,d n ) corresponds to the target Q value y n , Actual Q value

[0053]

[0054] Each experience(s) described in step 4c) nz ,a nl ,r n ,s (n+1)z ,d n ) corresponds to the target Q value y n , Actual Q value The calculation formula is:

[0055]

[0056]

[0057] Where γ is the discount factor, represents the Q value predicted by the target Q network, a' represents a dummy variable, and Represent value flow and advantage flow respectively, and Respectively and , |A| represents the size of the action space, ε V and ε A Represents the noise parameter, NoisyLinear() represents the noise fully connected layer;

[0058] In this embodiment, N=375, |A|=2, γ=0.99, ε V =ε A =0.017;

[0059] NoisyLinear() introduces learnable noise parameters based on the traditional fully connected layer. During each forward propagation, these noise parameters will dynamically generate random disturbances, making the Q-value prediction moderately random under the same state, avoiding the agent from falling into the local optimum too early, especially in complex or high-dimensional environments, and being able to try multiple strategies more efficiently. As training progresses, the noise parameters will gradually learn appropriate values. When the environment requires more exploration, the noise remains at a high level, and when the network strategy tends to stabilize, the noise gradually decreases. This learnable exploration strategy avoids the trouble of manually adjusting the exploration parameters, which not only enhances the network's ability to explore unknown environments, but also optimizes the stability and efficiency of the training process.

[0060] Step 4d) The target Q network and the local Q network use the mean square error loss function, respectively, through each experience (s nz ,a nl ,r n ,s (n+1)z ,d n ) corresponds to the target Q value y n , Actual Q value Compute the loss of the local Q network at each time step Then pass Parameters of the local Q network Update it N times in total to get the local Q network parameter θ for the xth iteration x The update result θ x+1 , and then through θ x+1 The parameter θ' of the target Q network x Update to obtain the interactive network model of this iteration;

[0061] The loss of the local Q network at each time step described in step 4d) The calculation formula is:

[0062]

[0063] Step 4d) The parameters of the local Q network Update the parameters θ' of the target Q network x To update, the update formulas are:

[0064]

[0065] θ' x+1 ←τθ x+1 +(1-τ)θ' x

[0066] Among them, α is the learning rate, The gradient of , τ is the soft update frequency;

[0067] In this embodiment, α=5e-4 and τ=1e-3 are set.

[0068] Step 4e) Determine whether x=X. If so, obtain the trained interaction network model. Otherwise, set x=x+1 and execute step (4b).

[0069] 1. Experimental conditions and contents:

[0070] Pycharm software is used to compare and simulate the total throughput of the cognitive drone network of the present invention and the prior art. The results are as follows: Figure 2 shown.

[0071] 2. Experimental results analysis:

[0072] Reference Figure 2 , a simulation experiment was carried out to verify the present invention and compared it with the prior art, showing the performance of the two methods in the total throughput of the cognitive drone network. The total throughput of the prior art was always stable at around 275Mbps, and the throughput of the present invention increased rapidly in the first few iterations, reaching a stable state around 200 iterations, and maintained a throughput higher than that of the prior art.

[0073] The results show that the present invention has a higher stable throughput than the prior art, which indicates that the present invention has a higher spectrum utilization rate.

Claims

1. A cognitive drone network spectrum sharing method based on multi-agent reinforcement learning, characterized in that: The steps include: (1) Building a cognitive drone network model: A cognitive UAV network model is constructed, which includes M cognitive UAVs and U ground stations communicating with each other. The M cognitive UAVs are divided into K clusters containing at least two cognitive UAVs. The cognitive UAV located at the center of each cluster is the cluster head, and the other cognitive UAVs are cluster members communicating with the cluster head. The cognitive UAV network model includes I communication between K cluster heads and J cluster members. H I between channels and U ground stations G There are a total of I channels composed of channels, I = I H +I G ,J=MK, where M≥2, U≥2, K≥1; (2) Constructing a reinforcement learning interactive network model of cognitive drone network and multi-agents: Construct a reinforcement learning interaction network model including a cognitive drone network model and M Q agents, the same number as the number of cognitive drones in the network model, where each Q agent includes a local Q network and a target Q network with the same structure; (3) Obtain the training data set and define the reward function: Take a training dataset consisting of a state space S consisting of Z environment states and an action space A consisting of L actions, and define the reward r for the m-th Q-agent to interact with each environment state at the t-th time step t , where Z = 3, L = 2, t∈[1,T], T represents the total number of time steps, T ≥ 1000; (4) Iterative training of the reinforcement learning interaction network model: The reinforcement learning interaction network model is iteratively trained through the training data set, and the output of the trained reinforcement learning interaction network model is used as the spectrum sharing result of the cognitive UAV network.

2. The method according to claim 1, characterized in that The state space S and action space A described in step (3), where: S={s z |z∈[1,2,3]} s1=γ u [and G ],s2=H,s3=W A={a l |l∈[1,2]} a1=ρ[i G ]i G ,a2=P select Among them, H, W, ρ[i G ]i G and P select Represents the task information, waiting time, selected channel and transmission power of each cluster member, i G represents the i-th channel between U ground stations, γ u [i G ] represents the signal-to-noise ratio of the u-th ground station.

3. The method according to claim 2, characterized in that The iterative training of the reinforcement learning interaction network model described in step (4) is implemented by: (4a) The number of initial iterations is x, the maximum number of iterations is X, X ≥ 1500, and the network parameters of the local Q network and the target Q network in the xth iteration are θ x ,θ' x , and let x = 0; (4b) The local Q network in each Q agent passes through each action a at the tth time step l and environmental state s z The signal-to-noise ratio of the jth cluster member on the i-th channel obtained by interaction is γ j [i] Calculate the corresponding throughput C j [i], and through C j [i] Calculate the relationship between each Q agent and s z Interaction Reward t , then r t 、a tl and tz , and the environmental state s at the t+1th time step (t+1)z , whether the sign of the end d constitutes the experience (s tz ,a tl ,r t ,s (t+1)z ,d), and obtain the experience set E=[E1,E2,...,E m ,...E M ], where E m represents the experience of the mth agent T time steps; (4c) The local Q network in each Q agent has an experience set E for each agent. m Perform N random samplings to obtain N experiences The target Q network and the local Q network are respectively obtained through each experience (s n ,a n ,r n ,s n+1 ,d n ) corresponds to the target Q value y n , Actual Q value (4d) The target Q network and the local Q network use the mean square error loss function, respectively, through each experience (s nz ,a nl ,r n ,s (n+1)z ,d n ) corresponds to the target Q value y n , Actual Q value Compute the loss of the local Q network at each time step Then pass Parameters of the local Q network Update it N times in total to get the local Q network parameter θ for the xth iteration x The update result θ x+1 , and then through θ x+1 The parameter θ' of the target Q network x Update to obtain the interactive network model of this iteration; (4e) Determine whether x=X. If so, obtain the trained interaction network model. Otherwise, set x=x+1 and execute step (4b).

4. The method according to claim 3, characterized in that The throughput C of the jth cluster member on the i-th channel described in step (4b) is j [i], and the reward r for each Q-agent’s interaction with each environment state t , the calculation formulas are: C j [i]=Blog(1+γ j [and]) I u =(P j [i]g j,u [i G ])ρ[i G ] I j =(P j' [i']g j',j [i G ']+P u [i G ]g u,j [i G ])ρ[i G ] Where B represents the bandwidth of the channel, γ j [i] represents the signal-to-noise ratio of the jth cluster member on the i-th channel, γ u [i G ] th represents the minimum signal-to-noise ratio required to establish reliable communication between the u-th ground station and other ground stations, γ j [i] th represents the minimum signal-to-noise ratio required to establish reliable communication between the jth cluster member and the cluster head of its cluster, φ represents a negative reward, φ>0, P u [i] represents the transmission power of the u-th ground station, g u [i G ] indicates that the uth ground station is the transmitter and the ith ground station is the transmitter. G The channel gain on the channel, σ 2 is the noise power, I u represents the interference power to the u-th ground station, P j [i] represents the transmission power of the jth cluster member on the i-th channel, g j,u [i G ] indicates that the jth cluster member is responsible for the uth ground station at the ith G The interference channel gain of the channel, g j [i] represents the channel gain of the jth cluster member communicating with the cluster head of the cluster through the i-th channel, I j represents the interference power to the jth cluster member, P j' [i'] represents the transmission power of the j'th cluster member on the i'th channel, g j',j [i G ] indicates that the j'th cluster member has a G The interference channel gain of the channel, g u,j [i G ] indicates that the u-th ground station has a G The interference channel gain of the channel, ρ[i G ]=1 means that the communication between the jth cluster member and the cluster head of its cluster shares the ith cluster in the ground network. G channels, ρ[i G ]=0 means the i-th G channels are not multiplexed.

5. The method according to claim 3, characterized in that: Each experience (s) described in step (4c) nz ,a nl ,r n ,s (n+1)z ,d n ) corresponds to the target Q value y n , Actual Q value The calculation formula is: Where γ is the discount factor, represents the Q value predicted by the target Q network, a' represents a dummy variable, and Represent the value flow and advantage flow respectively, and Respectively and , |A| represents the size of the action space, ε V and ε A Represents the noise parameters, and NoisyLinear() represents the noise fully connected layer.

6. The method according to claim 3, characterized in that: The loss of the local Q network at each time step described in step (4d) The calculation formula is:

7. The method according to claim 3, characterized in that The parameters of the local Q network described in step (4d) are Update the parameters θ' of the target Q network x To update, the update formulas are: I will x+1 ←tth x+1 +(1-τ)θ' x Among them, α is the learning rate, for is the gradient of , and τ is the soft update frequency.

Images