An unmanned swarm spectrum management method based on multi-agent reinforcement learning

Through multi-agent reinforcement learning method, the Markov decision-making model for spectrum allocation of unmanned clusters is solved, and the problem of low efficiency of spectrum resource allocation in drone clusters is achieved, and more efficient spectrum resource utilization is achieved, suitable for dynamic environment and scale growth Drone network.

CN118301759BActive Publication Date: 2024-11-22CHINESE PEOPLES LIBERATION ARMY UNIT 32806
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410471525.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-19
Publication Date
2024-11-22
Estimated Expiration
2044-04-19

AI Technical Summary

Technical Problem

UAV clusters have inefficient problems in spectrum resource allocation, especially in high-speed operation and dynamic environments, where existing centralized methods are difficult to effectively handle computing tasks with scale growth.

Method used

Using a method based on multi-agent reinforcement learning, a Markov decision-making model is constructed for unmanned cluster spectrum allocation. Each drone is trained as a single agent through Markov decision theory and multi-agent reinforcement learning algorithm to achieve real-time Spectrum allocation, optimize spectrum resource utilization using reward function and time discount factor.

Benefits of technology

The allocation efficiency and utilization rate of unmanned cluster spectrum resources are improved, and information exchange is processed through distributed perspectives, the complexity of modeling and computing is reduced, and the overall utilization efficiency of spectrum resources is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118301759B_ABST
    Figure CN118301759B_ABST
Patent Text Reader

Abstract

The present invention provides an unmanned cluster spectrum management method based on multi-agent reinforcement learning, which belongs to the technical field of unmanned cluster spectrum resource allocation. The method comprises the following steps: step 10, for the unmanned cluster to perform communication tasks, using Markov decision theory and multi-agent reinforcement learning algorithm to construct a Markov decision model for unmanned cluster spectrum allocation; step 20, training the unmanned cluster spectrum allocation Markov decision model to obtain a multi-agent reinforcement learning network; step 30, using the multi-agent reinforcement learning network to realize real-time spectrum allocation when the unmanned cluster performs communication tasks. The present invention provides an unmanned cluster spectrum management method based on multi-agent reinforcement learning, which optimizes the spectrum resource allocation efficiency of the unmanned cluster.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned cluster spectrum resource allocation, and in particular, relates to an unmanned cluster spectrum management method based on multi-agent reinforcement learning. Background Art

[0002] Drones have attracted much attention in wireless communication networks due to their low cost and flexible deployment. Although drone swarm networks have their own dedicated frequency bands, they are far from meeting the communication needs of the entire drone swarm. For the spectrum resource allocation problem of drone swarms, machine learning is a promising tool that can provide autonomous and effective solutions in an intelligent way to enhance drone-supported communication networks. Most of the machine learning algorithms proposed so far focus on single drone scenarios or multiple drone scenarios, assuming the availability of complete network information for each drone. However, in practice, due to the high speed of drone operation, it is not trivial to obtain perfect knowledge of the dynamic environment, which poses a daunting challenge to the design of reliable wireless communications between drones. In addition, most of the existing research contributions focus on centralized methods, and the modeling and computational tasks become challenging as the network size continues to increase. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an unmanned cluster spectrum management method based on multi-agent reinforcement learning, optimize the spectrum resource allocation efficiency of the unmanned cluster, and improve the spectrum resource utilization rate.

[0004] To solve the above technical problems, the embodiments of the present invention adopt the following technical solutions:

[0005] The embodiment of the present invention provides an unmanned cluster spectrum management method based on multi-agent reinforcement learning, comprising the following steps:

[0006] Step 10, for the unmanned cluster to perform communication tasks, the Markov decision theory and multi-agent reinforcement learning algorithm are used to build a Markov decision model for spectrum allocation of the unmanned cluster;

[0007] Step 20, training the unmanned cluster spectrum allocation Markov decision model to obtain a multi-agent reinforcement learning network;

[0008] Step 30, using a multi-agent reinforcement learning network to achieve real-time spectrum allocation when the unmanned cluster performs communication tasks.

[0009] As a further improvement of an embodiment of the present invention, in the unmanned cluster spectrum allocation Markov decision model constructed in step 10, each drone is a single intelligent agent; for a single drone, the frequency band occupancy of the drone is used as the input state of the single intelligent agent, and the specific frequency band and specific width allocated when a data packet needs to be sent are used as the output action; the output actions of all intelligent agents constitute a joint action, and after the joint action acts on the environment, the environment gives each single intelligent agent a reward value through the constructed reward function.

[0010] As a further improvement of the embodiment of the present invention, in step 10, the reward function is:

[0011]

[0012] In the formula, R i,t represents the benefit of the agent corresponding to drone i at time t, γ represents the time discount factor, V represents the set of drones adjacent to drone i, 0<μ<1; ρ i,j represents the spatial discount factor between UAV i and UAV j, d i,j represents the distance from UAV i to UAV j; r j,t represents the reward obtained by the agent corresponding to drone j at time t;

[0013] At time t, the reward obtained by the agent corresponding to drone i at time t is calculated using formula (2):

[0014]

[0015] In the formula, g m,i represents the channel transmission loss caused by the distance between UAV m and UAV i, λ represents the current wavelength of the transmitted signal, d m,i represents the distance from UAV m to UAV i, σ m,i represents the background noise in the vicinity of UAV m to UAV i, and h represents a constant.

[0016] As a further improvement of the embodiment of the present invention, the step 20 specifically includes:

[0017] Step 201, each drone runs a reinforcement learning agent;

[0018] Step 202, initializing the environment, initializing the state and strategy of each agent;

[0019] Step 203: The drone generates and sends data packets irregularly, and selects the frequency band and frequency band width according to the strategy of the intelligent agent; the joint strategy composed of the strategies of the most recent A time series of all drones, as well as the state and reward value of the corresponding time series of each drone, are put into the experience pool of reinforcement learning as training data;

[0020] Step 204: If W time series of training data are collected, they are taken from the experience pool of reinforcement learning, and the Q value of each drone corresponding to the intelligent agent is updated, and one training is completed; otherwise, step 203 is executed;

[0021] Step 205: If M trainings are completed, the training ends; otherwise, execute step 202.

[0022] As a further improvement of the embodiment of the present invention, in step 204, the Q value is updated using formula (3):

[0023]

[0024] In the formula, represents the Q value of the agent corresponding to drone m at time t+1, represents the Q value of the agent corresponding to drone m at time t, α represents the learning rate, represents the reward value of the drone m corresponding to the agent at time t, γ represents the time discount factor, Represents the state s' of the agent corresponding to drone m at time t+1 m The possible strategy θ' m The Q value, Θ m Represents the strategy set consisting of all possible strategies of the corresponding agent of drone m.

[0025] As a further improvement of the embodiment of the present invention, all drones in the unmanned cluster periodically send location information to neighboring drones to update communication routing information in the unmanned cluster.

[0026] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0027] The present invention provides a spectrum management method for unmanned clusters based on multi-agent reinforcement learning, which represents the spectrum resource allocation problem in unmanned clusters as a decentralized Markov decision process. Each unmanned aerial vehicle is used as an intelligent agent and is trained using a multi-agent reinforcement learning method to achieve spectrum resource allocation for unmanned clusters, effectively improve the spectrum allocation efficiency between unmanned clusters, and improve spectrum resource utilization. Multi-agent reinforcement learning can provide a distributed perspective for intelligent resource management of communication networks supported by unmanned aerial vehicles, especially when these unmanned aerial vehicles only have separate local information. By introducing multi-agent reinforcement learning into spectrum resource allocation for unmanned aerial vehicles, local information exchange between unmanned aerial vehicles can be modeled and investigated, and modeling and calculation difficulties can be handled in a distributed manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0029] Figure 1 It is a diagram of the unmanned cluster multi-agent architecture constructed in the method of the present invention;

[0030] Figure 2 The present invention is a flowchart of training the Markov decision model for spectrum allocation of unmanned clusters in the method of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0032] The embodiment of the present invention provides an unmanned cluster spectrum management method based on multi-agent reinforcement learning, comprising the following steps:

[0033] Step 10, for the unmanned cluster to perform communication tasks, use Markov decision theory and multi-agent reinforcement learning algorithm to build a Markov decision model for spectrum allocation of unmanned clusters.

[0034] Step 20: training the unmanned cluster spectrum allocation Markov decision model to obtain a multi-agent reinforcement learning network.

[0035] Step 30, using a multi-agent reinforcement learning network to achieve real-time spectrum allocation when the unmanned cluster performs communication tasks.

[0036] Specifically, in step 10, construct Figure 1The unmanned swarm multi-agent architecture diagram is shown in Figure 1. In an unmanned swarm, drones generate communication tasks from time to time: a drone sends a data packet to another drone, which requires occupying a certain bandwidth in its own frequency band (for example, occupying 800-812MHz in the 800-840MHz frequency band). Each drone corresponds to a single agent. For the agent corresponding to drone m, the frequency band occupancy of drone m at time t is used as the input state s m (t), the reward obtained by drone m at time t is used as the input reward r m (t), output strategy θ m (t) is the specific frequency band and specific width allocated to drone m at time t, that is, drone m that generates a data packet sends information to drone n, which frequency band of drone m needs to be allocated for this communication. All drones are multi-agents, and their strategies together constitute a joint strategy Θ. The environment faced by the agent is the network status of the drone cluster, that is, the occupancy of each frequency band of each drone. When training the agent, the strategies of multiple drones are used as a joint strategy to interact with the environment. The environment gives a reward value to the agent corresponding to each drone for training the agent.

[0037] The Markov characteristic is used to simulate the environment, and the reward of the drone is based only on the current state and strategy. The Markov decision process of the agent corresponding to a single drone m includes: a set of discrete environmental states s m , the discrete possible strategies θ m and the reward function R m .

[0038] In this embodiment, the reward function is:

[0039]

[0040] In the formula, R i,t represents the benefit of the agent corresponding to drone i at time t, γ represents the time discount factor, and γ τ-t Discount the future reward value to the present. The reward value that appears in the farther future is discounted to the present at a smaller ratio. V represents the set of neighboring drones of drone i. 0<μ<1. ρ i,j represents the spatial discount factor between UAV i and UAV j, d i,j represents the distance from UAV i to UAV j. The spatial discount factor between UAV i and UAV j is inversely proportional to the distance between UAV i and UAV j.

[0041] r j,t represents the reward obtained by the agent corresponding to drone j at time t. Specifically, at time t, the reward obtained by the agent corresponding to drone i is calculated using formula (2):

[0042]

[0043] In the formula, g m,i represents the channel transmission loss caused by the distance between UAV m and UAV i, λ represents the current wavelength of the transmitted signal, represents the distance from UAV m to UAV i, σ m,i represents the background noise in the vicinity of UAV m to UAV i, and h represents a constant.

[0044] In formula (1), when μ→0, The value of is very small, that is, the influence of the corresponding agents of other drones except drone i on the corresponding agent of drone i is very small, so each agent performs local greedy control; when μ→1, The value of is close to 1, that is, the corresponding agents of other drones except drone i have a greater influence on the corresponding agent of drone i, that is, each drone considers the status of other drones with a large weight, so each agent performs global coordination and strategy.

[0045] The reward function constructed by the embodiment of the present invention uses the single-step reward values ​​of the future T time series multiplied by the time discount factor and then accumulated as the expectation of the current value of each drone, and uses the reward values ​​of neighboring drones multiplied by the space discount factor and then accumulated as the single-step reward value of each drone. The time discount factor reflects the impact of future rewards on the current value expectation, and the space discount factor reflects the impact of the rewards of neighboring drones on the rewards of this drone. When calculating the single-step reward value of this drone, the rewards of neighboring drones are more considered. The multi-agent reinforcement learning network trained by the embodiment of the present invention integrates spatiotemporal reinforcement learning, has spatiotemporal characteristics, improves the convergence speed and stability of the model, and improves the efficiency of the model.

[0046] In this application, a multi-agent reinforcement learning algorithm is used to solve the Markov decision process, where each agent operates in an unknown random environment without knowing the reward and transition function. The two basic concepts of the algorithm for solving the above Markov decision process are the state value function and the action value function (Q function). Specifically, the state value function is the expected reward for the agent to follow certain strategies in certain states. The Q function of drone m is the expected reward for the agent to follow certain strategies in certain states. m Start adopting strategy θ m and the expected reward of the subsequent policy Π, θ m It can be expressed as:

[0047]

[0048] The value corresponding to the Q function is the Q value.

[0049] In the above embodiment, the training process in step 20 is as follows: Figure 2 As shown, specifically including:

[0050] In step 201, each drone runs a reinforcement learning agent.

[0051] Step 202, initialize the environment, and initialize the state and strategy of each agent.

[0052] In step 203, the drone generates and sends data packets at irregular intervals, and selects the frequency band and frequency band width according to the strategy of the intelligent agent; the strategies of the most recent A (preferably A=30) time series of all drones form a joint strategy, and the states and reward values ​​of the corresponding time series of each drone are put into the experience pool of reinforcement learning as training data.

[0053] Step 204: If W (preferably W=1000) time series of training data are collected, they are taken from the experience pool of reinforcement learning, and the Q value of the corresponding intelligent agent of each drone is updated, and one training is completed; otherwise, step 203 is executed.

[0054] Preferably, the Q value is updated using formula (3):

[0055]

[0056] In the formula, represents the Q value of the agent corresponding to drone m at time t+1, represents the Q value of the agent corresponding to drone m at time t, α represents the learning rate, represents the reward value of the drone m corresponding to the agent at time t, γ represents the time discount factor, Represents the state s' of the agent corresponding to drone m at time t+1 m The possible strategy θ' m The Q value, Θ m Represents the strategy set consisting of all possible strategies of the corresponding agent of drone m.

[0057] Whenever the Q value of a drone’s corresponding agent is updated, it is updated at a learning rate of α, and the updated difference is the reward of the next step plus the discounted value of the expected Q value of the next step minus the current Q value.

[0058] Step 205: If M (preferably M=10000) training times are completed, the training ends; otherwise, execute step 202.

[0059] Preferably, all drones in the unmanned cluster periodically send location information to neighboring drones to update communication routing information in the unmanned cluster.

[0060] Specifically, drones discover neighboring drones by broadcasting HELLO packets. Packets are exchanged periodically, carrying information such as the drone's geographic location, queuing delay, learning rate, and related parameters of its corresponding agent. When a drone receives a HELLO packet, it will use the information in the packet to build and maintain its neighbor table. This mechanism is set up to be ready for network communication at any time, so it will continue to run even in the current communication session. The time interval for broadcasting HELLO packets is 100 milliseconds. If a drone does not receive a packet from a neighbor drone within a 300 millisecond period, the neighbor drone is deleted from its routing table and the related routing information is deleted.

[0061] A specific example is provided below.

[0062] Simulation of unmanned swarm: Simulate a multi-UAV network deployed in a disk area with a radius of 3 km. Assume that all UAVs fly at a fixed altitude of H = 100 meters. In the simulation, the noise power is assumed to be σ2 = -80 dBm and the subchannel bandwidth is 75 kHz. The carrier frequency of the UAV is f = 2 GHz. For the LoS channel model, the channel power gain at the reference distance d0 = 1 m is set to β0 = -60 dB and the path loss coefficient is set to α = 2. In the simulation, the maximum number of power levels is J = 3, and the maximum power of each UAV is Pm = P = 23 dBm, where the maximum power is the discrete power value divided into J equal parts. The energy loss per unit level is ωm = ω = 100. The speed of the UAV is set to 40 m / s, the total number of UAVs is 200, and the maximum channel throughput of each UAV is 3000 Mbit / s.

[0063] The method of the present invention and two comparative methods are respectively used to perform unmanned cluster spectrum management, and the spectrum resource utilization simulation effect is obtained.

[0064] Both comparison methods 1 and 2 are existing methods. Comparison method 1 uses the asynchronous advantage actor-critic algorithm (A3C) to maintain an estimate of a policy and value function. The policy and value function will be updated after each execution of a number of actions or when the terminal state is reached. Comparison method 2 is that each drone runs a single agent, and each single agent applies the SAC (Soft actor-critic) algorithm.

[0065] Compared with the other two comparison methods, the method according to the embodiment of the present invention can significantly increase the spectrum resource utilization rate as the number of training rounds increases, thereby achieving better spectrum resource utilization rate.

[0066] The basic principles, main features and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above specific embodiments, and the descriptions in the above specific embodiments and the specification are only for further illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed for protection.

Claims

1. An unmanned cluster spectrum management method based on multi-agent reinforcement learning, characterized in that: The following steps are involved: Step 10, for the unmanned cluster to perform communication tasks, the Markov decision theory and multi-agent reinforcement learning algorithm are used to build a Markov decision model for spectrum allocation of the unmanned cluster; Step 20, training the unmanned cluster spectrum allocation Markov decision model to obtain a multi-agent reinforcement learning network; Step 30, using a multi-agent reinforcement learning network to achieve real-time spectrum allocation when the unmanned cluster performs communication tasks; In the unmanned cluster spectrum allocation Markov decision model constructed in step 10, each drone is a single agent; for a single drone, the frequency band occupancy of the drone is used as the input state of the single agent, and the specific frequency band and specific width allocated when a data packet needs to be sent are used as the output action; the output actions of all agents constitute a joint action, and after the joint action acts on the environment, the environment gives each agent a reward value through the constructed reward function; In step 10, the reward function is: In the formula, R i,t represents the benefit of the agent corresponding to drone i at time t, γ represents the time discount factor, V represents the set of drones adjacent to drone i, 0<μ<1; ρ i,j represents the spatial discount factor between UAV i and UAV j, d i,j represents the distance from UAV i to UAV j; r j,t represents the reward obtained by the agent corresponding to drone j at time t; At time t, the reward obtained by the agent corresponding to drone i is calculated using formula (2): In the formula, g m,i represents the channel transmission loss caused by the distance between UAV m and UAV i, λ represents the current wavelength of the transmitted signal, d m,i represents the distance from UAV m to UAV i, σ m,i represents the background noise in the vicinity of UAV m to UAV i, and h represents a constant; The step 20 specifically includes: Step 201, each drone runs a reinforcement learning agent; Step 202, initializing the environment, initializing the state and strategy of each agent; Step 203: The drone generates and sends data packets irregularly, and selects the frequency band and frequency band width according to the strategy of the intelligent agent; the joint strategy composed of the strategies of the most recent A time series of all drones, as well as the state and reward value of the corresponding time series of each drone, are put into the experience pool of reinforcement learning as training data; Step 204: If W time series of training data are collected, they are taken from the experience pool of reinforcement learning, and the Q value of each drone corresponding to the intelligent agent is updated, and one training is completed; otherwise, step 203 is executed; Step 205: if M trainings are completed, the training ends; otherwise, execute step 202; In step 204, the Q value is updated using formula (3): In the formula, represents the Q value of the agent corresponding to drone m at time t+1, represents the Q value of the agent corresponding to drone m at time t, α represents the learning rate, represents the reward value of the drone m corresponding to the agent at time t, γ represents the time discount factor, Represents the state s' of the agent corresponding to drone m at time t+1 m The possible strategy θ' m The Q value, Θ m Represents the strategy set consisting of all possible strategies of the corresponding agent of drone m.

2. The unmanned cluster spectrum management method according to claim 1, characterized in that: All drones in the unmanned cluster periodically send location information to neighboring drones to update communication routing information in the unmanned cluster.