Multi-intelligent reinforcement learning area scheduling method for unmanned aerial vehicle communication base station
Through the multi-intelligent reinforcement learning area scheduling method, combined with Markov modeling and nuclear method counterfactual multi-intelligent policy gradient method, the scheduling problem of drone groups in unknown environments is solved, and efficient communication resource allocation and Maximize system utility.
Patent Information
- Application Number
- CN202411135551.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-22
AI Technical Summary
The prior art is difficult to achieve efficient drone cluster area scheduling in unknown dynamic business environments. Traditional algorithms face the curse of dimensionality of state and action space, consume high computing resources, and have limitations in actual control networks.
Adopting the multi-intelligent body reinforcement learning regional scheduling method, through the collaborative cooperation between ground base stations and drone base stations, using the counterfactual multi-intelligent body strategy gradient method of Markov modeling and nuclear method, update the scheduling strategy of the drone, and optimize the Spectral resource allocation of communication networks.
In the service area where unknown service distribution is distributed, efficient collaborative learning of drone groups is realized, system effectiveness is maximized, business throughput and scheduling distance is reduced, and system robustness and convergence speed is improved.
Smart Images

Figure CN119005620A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a regional scheduling method for a base station, and in particular to a multi-agent reinforcement learning regional scheduling method for a UAV communication base station. Background Art
[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.
[0003] Due to the limitations of spectrum resources and geographical location, relying solely on 4G and 5G ground communication networks is not enough to provide ubiquitous, high-quality, and highly reliable services at any time and place, especially to meet the service needs of a large number of IoT devices in some remote areas. In order to achieve wireless communications over a wider range, it is necessary to build an air-ground integrated wireless network.
[0004] Drones are highly maneuverable, dynamically deployed, and easy to configure, and they play a pivotal role in military and civilian fields. With the continuous reduction in cost and the miniaturization of equipment, small drones are now more accessible to the public. As a result, many new applications have emerged in the civil and commercial fields, with typical examples including weather monitoring, forest fire detection, traffic control, cargo transportation, emergency search and rescue, and communication relay. At present, large-scale drone swarms consisting of multiple small and medium-sized drones have gradually served all walks of life and have broad application prospects. Compared with a single drone, the ability and efficiency of multiple drones to complete tasks in a collaborative manner will be greatly improved.
[0005] UAV communications can help to relieve the burden on ground networks and provide service capabilities in crowded locations with highly dynamic data traffic loads. For example, ground communications can be achieved through the use of drones. As aerial access points, drones can provide network access to ground users. As mobile relays, drones can forward information between users. In addition, satellites / drones with remote sensing technology can support the reliable acquisition of monitoring data and assist ground networks in effective resource management and planning decisions.
[0006] Consider a scenario of air-ground collaborative wireless communication network. To reduce the service pressure of ground base stations, the drones in the drone swarm carry communication base stations with control and scheduling functions to assist the ground base stations in providing uplink communication services for some ground users (such as IoT devices). However, the services in the entire service area may change dynamically and are unknown to the drone swarm. It is necessary to study how to obtain efficient regional scheduling strategies for drone swarms in unknown dynamic service environments.
[0007] The mission scenarios considered by traditional UAV path planning technology based on optimization models are mostly fixed, but in actual application scenarios, the control system often lacks prior information about the environment. The dynamically changing mission requirements make the strategy derived by the model-driven method not the optimal path scheduling. The currently emerging reinforcement learning technology provides a general method to solve complex uncertainty decision-making problems. The Q-learning algorithm is a model-free reinforcement learning algorithm, which has been applied to UAV path planning / black broadcast reconnaissance by relevant research. However, the increase in the number of missions and UAVs will cause the traditional table-based algorithm to face huge state and action spaces, thus falling into the curse of dimensionality and other problems, making the algorithm unable to converge. The prediction models based on neural networks or hidden Markov models proposed to solve such problems require a lot of computing resources for training, and need to be retrained regularly. At the same time, a large number of parameters are required to obtain relevant communication environment information, which has certain limitations in actual control networks with business delay requirements. Although the latest proposed reinforcement learning online anti-interference algorithm based on kernel function approximation has a good performance in dynamically monitoring environmental information, analyzing relevant data in real time, and making the optimal decision for frequency resource allocation, there is still a lot of room for improvement in algorithm robustness and convergence speed.
[0008] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0009] Purpose of the invention: The technical problem to be solved by the present invention is to provide a multi-agent reinforcement learning regional scheduling method for UAV communication base stations in view of the shortcomings of the prior art.
[0010] In order to solve the above technical problems, the present invention discloses a multi-agent reinforcement learning regional scheduling method for UAV communication base stations, comprising:
[0011] Step 1: At each moment, the ground base station in each area sends a request for communication assistance to the UAV flying to the area;
[0012] Step 2: The drone base station set on the drone responds and then feeds back to the ground base station. The ground base station sends a control command to the drone to perform auxiliary communication services or standby according to the number of drone base stations that respond. The drone feeds back the control command results and its own status information to the ground control center at a regular interval.
[0013] Step 3: The ground control center uses a multi-agent reinforcement learning scheduling method based on the control command results fed back by all drones to obtain and update the regional scheduling strategy for each drone. The drone makes a decision based on the strategy when it receives a request for communication assistance at the next moment.
[0014] Step 4: Repeat steps 1 to 3 to complete the multi-agent reinforcement learning regional scheduling for drone communication base stations.
[0015] Furthermore, the multi-agent reinforcement learning scheduling method described in step 3 specifically includes:
[0016] Step 3-1, Markov modeling of the UAV base station scheduling problem;
[0017] Step 3-2, defining the optimal strategy for the base station scheduling problem;
[0018] Step 3-3, converting the optimal strategy described in step 3-2 into a multi-agent reinforcement learning problem;
[0019] Step 3-4, use the kernel-based counterfactual multi-agent policy gradient method to solve the multi-agent reinforcement learning problem described in step 3-3.
[0020] Furthermore, the Markov modeling of the drone base station scheduling problem described in step 3-1 specifically includes:
[0021] Define the environment state as the joint state s of the entire drone swarm at time t t ,as follows:
[0022] s t =[s t,1 ,…,s t,i ,…,s t,N ,R t,1 ,…,R t,i ,…,R t,N ] T
[0023] Among them, s t,i is the binary code of the area where the i-th UAV is located, R t,i is whether the assistance request sent by the base station is received, which is 1 if it is, otherwise 0. N is the total number of drones, and the joint action of all drones at time t is a t ,as follows:
[0024] a t =[a t,1 ,...,a t,i ,...,a t,N ] T
[0025] Among them, a t,i represents the action of the i-th drone, A set of binary codes representing flyable sub-areas;
[0026] The transition probability is P(s t+1 |s t ,a t ), the utility function of UAV i performing assisted communication service at time t is:
[0027] r t,i -λD t,i
[0028] Among them, r t,i The number of ground communication users served by UAV i, D t,i is the moving distance, and λ is the proportional coefficient.
[0029] Furthermore, the optimal strategy for defining the base station scheduling problem described in step 3-2 specifically includes:
[0030] Assume that the action strategy of the entire system, i.e., all drones, is π. The optimal strategy is to maximize the expectation of the cumulative discounted return at the frequency-using devices in the communication network as the target utility function V(s), i.e., the optimal Bellman equation, as follows:
[0031]
[0032] Among them, γ is the discount factor, E π represents the long-term expected return of adopting strategy π, and s is the joint state of the drone swarm.
[0033] Furthermore, the step 3-3 described in converting the optimal strategy described in step 3-2 into a multi-agent reinforcement learning problem specifically includes:
[0034] Step 3-3-1, set an action strategy π for each drone i i , jointly optimize the global goal, as follows:
[0035]
[0036] Step 3-3-2, each drone i follows the action strategy π i Autonomously make regional dispatch decisions, and the ground control center only provides updates on action strategies, i.e. regional dispatch strategies;
[0037] Step 3-3-3, treat the above process as a multi-agent reinforcement learning problem.
[0038] Furthermore, the counterfactual multi-agent policy gradient method based on the kernel method described in step 3-4 specifically includes:
[0039] Step 3-4-1, each drone initializes a policy function network, as follows:
[0040] The strategy function satisfies the Gibbs sampling distribution or the Boltzmann strategy, and the characteristic parameters of each network are a linear combination of a set of Gaussian radial basis functions, which satisfies:
[0041]
[0042] in, Indicates the probability that the current drone chooses action a in state s, is the weight parameter;
[0043]
[0044] in, is the weight vector of the preference function of the kth UAV, T is the interval time slot, is the basis function vector of the kth UAV;
[0045] Step 3-4-2: At each moment, each drone makes and executes a regional scheduling strategy based on its own strategy function network;
[0046] Step 3-4-3, the ground control center initializes a global state function network, as follows:
[0047] V g (s,a)=w vT f v (s)
[0048] Among them, V g (s,a) is the global state function, w v is the weight vector of the global state function, f v is the basis function vector of the global state function;
[0049] Step 3-4-4, after each UAV flies to the target area, it will transmit the assistance communication service information to the ground control center, which will summarize it. All UAVs will transmit their status to the ground control center every T time slots and store it; every cT time slots, the ground control center will update the strategy functions of all UAVs.
[0050] Furthermore, the policy functions of all drones described in step 3-4-4 are updated, specifically including:
[0051] Step 3-4-4-1, design advantage function A(s j ,a), as follows:
[0052]
[0053] Among them, V(s j ,a) represents the global state value function, V(sj ,a,a -i ) represents the policy function π(a|v j,i ;w f,q,i ) represents the policy function value corresponding to the missing action;
[0054] Step 3-4-4-2, weight w for drone i q,i Update as follows:
[0055]
[0056] Among them, β q is the learning rate, is the gradient of the policy function with respect to the weight;
[0057] Step 3-4-4-3, update the weight w of the global state value function network according to the global state information v , as follows:
[0058]
[0059] Among them, r j is the immediate return, γ is the discount rate, V(s j ;w v ) is the current global state value function, V(s j ′; w v ) is the state value function of the next sample, is the gradient of the current global state value function;
[0060] Step 3-4-4-4, the characteristic functions of the two functions are updated using an online sparse algorithm.
[0061] Furthermore, the characteristic functions of the two functions described in step 3-4-4-4 are updated using an online sparse algorithm, that is, an approximate linear independence detection algorithm is used to sparsify the feature dictionary of the strategy function and the global value function, and the algorithm determines whether to add the dictionary by judging the linear correlation between the linear combination of the characteristic functions of the feature vector and the feature vector to be judged.
[0062] Furthermore, the characteristic functions of the two functions described in step 3-4-4-4 are updated using an online sparse algorithm, specifically including:
[0063] Step 3-4-4-4-1, calculate the expected difference δ between the linear combination of the feature vector to be judged and the feature vector in the feature dictionary;
[0064] Step 3-4-4-4-2, define the expected difference threshold μ;
[0065] Step 3-4-4-4-3, determine the size of the difference δ obtained in step 3-4-4-4-1 and the threshold μ. If it is greater than μ, add the feature vector to the feature dictionary. If it is less than or equal to the threshold μ, do not add it to the feature dictionary.
[0066] Furthermore, the expected difference δ between the linear combination of the feature vector to be judged and the feature vector in the feature dictionary described in step 3-4-4-4-1 is as follows:
[0067]
[0068] Among them, α is the weight, is the kth eigenvector.
[0069] Beneficial effects:
[0070] The present invention can be used for the regional scheduling problem of UAV base stations. In a service area with unknown business distribution, multiple UAVs with communication base station functions work together with a ground control center to collaboratively learn the changing rules of ground business volume under the condition that the business volume in the service area is unknown. The control center makes the optimal scheduling strategy by weighing the business throughput and scheduling distance, thereby maximizing the utility of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.
[0072] Figure 1 It is a schematic diagram of the overall process of the present invention.
[0073] Figure 2 Schematic diagram of a system model according to an embodiment of the present invention.
[0074] Figure 3 A schematic diagram of system policy update according to an embodiment of the present invention.
[0075] Figure 4 The figure is a flow chart of an embodiment of the present invention.
[0076] Figure 5 Schematic diagram of the process flow of the counterfactual multi-agent reinforcement learning method based on the kernel method proposed in the present invention.
[0077] Figure 6 It is a flowchart of the online feature dictionary sparse method in the present invention. DETAILED DESCRIPTION
[0078] The present invention proposes a multi-agent reinforcement learning regional scheduling method for UAV communication base stations. The basic principle is as follows: at each moment, the ground base station in each area sends an application for assisting communication to the UAV flying to the area. The UAV base station responds and then feeds back to the ground base station. The base station sends a control command to the UAV to execute a service or standby command according to the number of UAV base stations that respond. The UAV feeds back the command result to the ground control center. The ground control center trains and updates the multi-agent reinforcement learning scheduling algorithm according to the command results fed back by all UAVs. The UAV obtains the scheduling regional strategy of each UAV at the next moment and executes it according to its own strategy. The main purpose of the present invention is to enable multiple UAVs with communication base station functions to cooperate with the ground control center, and to collaboratively learn the law of ground business volume changes under the condition that the business volume in the service area is unknown. The control center makes the optimal scheduling strategy by weighing the business throughput and scheduling distance to maximize the utility of the system.
[0079] The technical solution adopted by the present invention is as follows: Figure 1 As shown, the details are as follows:
[0080] A multi-agent reinforcement learning regional scheduling method for UAV communication base stations, comprising:
[0081] Step 1: At each moment, the ground base station in each area sends a request for communication assistance to the drone flying to that area;
[0082] Step 2: The drone base station responds and then feeds back to the ground base station. The base station sends a control command to the drone to execute a service or standby command based on the number of drone base stations that responded. The drone feeds back the command result and its own status information to the ground control center at a regular interval.
[0083] Step 3: The drone feeds back the command results to the ground control center. The ground control center regularly updates the multi-agent reinforcement learning scheduling algorithm based on the command results fed back by all drones, trains and updates the scheduling area strategy of each drone, and the drone makes decisions and executes them according to its own strategy.
[0084] To optimize the above technical solutions, the specific measures taken also include:
[0085] In step 3, we first conduct Markov modeling on the UAV base station scheduling problem under unknown business demand, and define the environment state as the joint state s of the entire UAV group at time t t , the joint action of all drones at each moment is a t , the transition probability is P(s t+1 |s t ,a t ), the utility function of drone i at time t is r t,i -λD t,i , where rt,i Number of terrestrial communication users served, D t,i is the moving distance, λ is the proportional coefficient so that r t,i and D t,i At the same order of magnitude. The action strategy of the entire system is π. In order to find the optimal strategy, it is necessary to maximize the expectation of the cumulative discounted return at the frequency-using equipment in the communication network as the target utility function, that is, the optimal Bellman equation:
[0086]
[0087] Among them, γ is the discount factor.
[0088] The optimal Bellman equation mentioned above, since the computational complexity of solving the strategy π increases exponentially with the increase in the number of drones, when the number of drones is large, it is not realistic to use a centralized solution to a global strategy. Set a strategy π for each drone i i By jointly optimizing the global goal, drones can make regional scheduling decisions autonomously, while the ground control center is only responsible for providing strategy updates. This can be seen as a multi-agent reinforcement learning problem.
[0089] In order to solve the multi-agent reinforcement learning problem, a counterfactual multi-agent policy gradient algorithm based on the kernel method is used to solve it. Figure 5 As shown, the algorithm steps include:
[0090] Step 3.1: Each drone initializes a policy function network, where the policy function satisfies the Gibbs sampling distribution or the Boltzmann strategy. The characteristic parameters of each network are a linear combination of a set of Gaussian radial basis functions, which satisfies the formula: Indicates the probability that the algorithm chooses action a in state s; is the parameterized numerical preference for each state-action pair, is the preference function weight vector of the kth UAV, is the basis function vector of the kth UAV;
[0091] in, is the weight parameter of H. In each time slot, each action a has a different probability
[0092] At each moment, the regional scheduling strategy is made and executed according to its own strategy function; the ground control center initializes a global state function network V g (s,a)=w vT f v (s), w vis the weight vector of the global state function, f v (s) is the basis function vector of the global state function.
[0093] Step 3.2: After each drone flies to the target area, it uploads information to the ground control center according to the specific service situation. The center summarizes it. All drones upload their local states to the ground control center every T time slot and store them in the cache. Every cT time slot, the ground control center starts to update the strategy functions of all drones, such as Figure 3 As shown in the figure, firstly, an advantage function is designed, which is the global state value function minus the sum of the products of all policy functions that lack a certain action value and the policy function value corresponding to the missing action. Then the weight of drone i is updated, that is, the weight of the previous sample plus the learning rate multiplied by the policy advantage function multiplied by the gradient of the policy function with respect to the weight; the weight of the global state value function network is updated according to the global state information, that is, the value function weight of the previous sample plus the sum of the product of the immediate reward and the discount rate multiplied by the state value function of the next sample minus the product of the difference of the current global state value function multiplied by the learning rate multiplied by the current global state value function. The characteristic functions of the two functions are updated using an online sparse algorithm.
[0094] In step 3.2, the approximate linear independent detection algorithm is used to sparse the feature dictionary of the strategy function and the global value function, such as Figure 6 As shown, the algorithm determines whether to add the dictionary by judging the linear combination of the characteristic function of the characteristic vector and the linear correlation of the characteristic vector to be judged, specifically:
[0095] Step 3.2.1: Find the expected difference between the linear combination of the feature vector to be judged and the feature vector in the feature dictionary;
[0096] Step 3.2.2: Define the expected difference threshold μ;
[0097] Step 3.2.3: Determine the difference between the difference obtained in step 3.2.1 and the threshold μ. If it is greater than μ, add the feature vector to the feature dictionary. If it is less than or equal to the threshold μ, do not add it to the feature dictionary.
[0098] Example:
[0099] In an actual air-ground coordinated wireless communication system, Figure 4 As shown in the figure, the drone swarm contains M drones, each of which is equipped with a base station to provide uplink (UL) communication services for ground users and remotely collaborate with the ground control station. In order to reduce the service pressure of the ground base station, the drone swarm is used to assist the ground base stations in different areas to provide continuous T s Uplink communication services of time slots, such as Figure 2As shown in Figure 1, the service area is divided into Y*Y equal parts, and each area has a fixed index number. Since the business distribution in each area, that is, the number of users that need to be served, changes dynamically and is unknown to the drone swarm and the ground base station, the ground control station needs to dynamically schedule the drone swarm across regions. Assuming that the unified scheduling time slot of the drone swarm lasts T u At time t-1, assume that the distance a drone moves from area a to area b is the shortest distance between the center coordinates of the two areas, that is, the coordinate (x a ,y a ) and (x b ,y b ), The ground base station is connected every fixed time slot T = T u +T s A request for communication assistance will be sent to the drones flying to the area. After receiving a reply, the drone will be allocated a number of users N not exceeding its maximum accessible number according to the current business situation. The drone is in a hovering state when providing relay services.
[0100] The ground base station is connected every fixed time slot T = T u +T s A request for communication assistance will be sent to the drones flying to the area. After receiving a reply, the drones will be assigned a number of users not exceeding the maximum number of users that can be accessed according to the current business situation. The drones that have not been assigned tasks will park nearby in the area and wait for the next dispatch. In particular, when the drone provides relay services, it is in a hovering state. After completing the task, the drone group decides whether to fly to a new area or continue to provide services in the original place. In particular, if the drone flies to the target area and has not received a request for assistance from the ground base station for the time being, it will temporarily land nearby to reduce energy consumption.
[0101] The drone swarm changes its service area to serve as many ground users as possible. The whole problem can be modeled as an MDP. Assume that at time slot t, the joint state of the entire drone swarm is s t =[s t,1 ,...,s t,n ,R t,1 ,...,R t,N ] T , where s t,i is the binary code of the area where the ith UAV is located, for example, (16)2=10000, R t,i Indicates whether the assistance request sent by the base station is received, "yes" is 1, "no" is 0, and the joint action of the target area position of all drones flying in the next time slot is a t =[a t,1 ,...,a t,N ] T , Represents a set of binary codes for flyable sub-areas. Transition probability P(s t+1 |s t ,a t ) is unknown to the drone group, because the dynamic business distribution of each sub-area is unknown to the entire system. The energy consumption during the flight process is the main resource consumption of the drone, so it is necessary to consider the trade-off between the number of service users and the flight scheduling distance. Assume that at time slot t, the number of service users of drone i is r t,i , the utility function of each drone i is r t,i -λD t , where λ is the normalization coefficient, so that r t,i With λD t In the same order of magnitude. The optimization goal of ground control is to find the global strategy π, a t ∈π makes the long-term utility of the overall drone swarm optimal
[0102]
[0103] Where γ is the discount factor.
[0104] The joint state action space mentioned above is huge. In the actual air-ground collaborative system, the frequent interaction between the drone swarm and the ground control center will also cause a lot of signaling overhead. It is not realistic to use a centralized solution to a global strategy. The solution to this problem is usually to set a learning strategy π for each drone i. i Jointly optimize the global goal, in other words, Drones can make regional scheduling decisions autonomously, while the ground control center is only responsible for providing policy updates, which can be regarded as a multi-agent reinforcement learning problem. However, a major challenge of multi-agent learning is the multi-agent credit allocation: in a cooperative environment, joint actions usually only produce global rewards, making it difficult for each agent to infer its contribution to the success of the team. In some settings, a separate reward function can be designed for each agent to reduce the severity of this problem. However, these rewards are usually not available in a cooperative environment and often fail to encourage individual agents to sacrifice for greater benefits. A Counterfactual Multi-Agent Policy Gradients (COMAPG) algorithm is used to solve the above problem. This is an algorithm based on the actor-critic architecture. This chapter still uses the Gaussian Kernel Function (GKF) to construct the learning network and sparse it online. Based on the COMAPG architecture, a kernel method-based counterfactual multi-agent reinforcement learning algorithm (KMCMARL) is proposed. The specific process is as follows:
[0105] Actor-critic network: Actor network is a network used for decision-making. Each drone has an independent decision-making network and makes scheduling decisions based on this network. At the current moment, it is difficult for the drone group to fully know each other's actions and states, that is, the global state s t It is difficult for each UAV i to obtain, and each UAV i uses the local observation state v t,i =[s t-n,i ,..,s t,i ,R t,i ] T Make a decision, that is, the local observation state of UAV i is v t,i When the action a is taken i,t The probability is π i (a i,t |v i,t ),have
[0106]
[0107] in,
[0108]
[0109]
[0110] d q,i,tThe number of eigenvectors representing the preference function; σ v , σ a is the bandwidth of the radial basis function, and both are constants; as well as are all eigenvectors. Similarly, the critic network, i.e. the joint state action value function, is s′ t =[a t ; a].
[0111] Update the parameters in each network: the entire drone swarm has a joint state value function V(s′ t ), all drones upload their local states to the ground control center every T time slot and store them in the cache pool. Every cT time slot, the control center uses the d s Sample state pairs (s j ′,a j ′,r j ′,s j ″) Update π in sequence i and V(s′ t ). Perform the following operations in parallel for the policy function of each drone i:
[0112]
[0113] β q w q,i The corresponding learning rate, then, the feature weight of the joint state value function is updated as
[0114]
[0115] Among them, β v w v,i Corresponding learning rate, after the weight is updated, execute the ALD algorithm to determine whether the state features in the cache pool need to be added to the dictionary, based on the strategy function feature dictionary For example, the joint state value function dictionary The update of can be obtained in a similar way. q,i,j ≥μ q , otherwise μ q is the preset threshold. The strategy iteration matrix is obtained respectively and the joint state value function iteration matrix [G v,j ] -1 , G q,i,1 =1,δ q,i,j =1-g q,i,j-1 (v q,i,j )T η q,i,j , η q,i,j =[G q,i,j-1 ] -1 g q,i,j-1 (v q,i,j ),in g(v,v′)=<φ(v),φ(v′)>,
[0116] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of a multi-agent reinforcement learning regional scheduling method for a drone communication base station provided by the present invention and some or all of the steps in each embodiment can be run. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0117] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on such an understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of computer programs, i.e., software products, which can be stored in a storage medium and include several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, an MCU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.
[0118] The present invention provides a method and idea of a multi-agent reinforcement learning regional scheduling method for UAV communication base stations. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.
Claims
1. A multi-agent reinforcement learning regional scheduling method for UAV communication base stations, characterized in that: include: Step 1: At each moment, the ground base station in each area sends a request for communication assistance to the UAV flying to the area; Step 2: The drone base station set on the drone responds and then feeds back to the ground base station. The ground base station sends a control command to the drone to perform auxiliary communication services or standby according to the number of drone base stations that respond. The drone feeds back the control command results and its own status information to the ground control center at a regular interval. Step 3: The ground control center uses a multi-agent reinforcement learning scheduling method based on the control command results fed back by all drones to obtain and update the regional scheduling strategy for each drone. The drone makes a decision based on the strategy when it receives a request for communication assistance at the next moment. Step 4: Repeat steps 1 to 3 to complete the multi-agent reinforcement learning regional scheduling for drone communication base stations.
2. According to claim 1, a multi-agent reinforcement learning regional scheduling method for drone communication base stations is characterized in that: The multi-agent reinforcement learning scheduling method described in step 3 specifically includes: Step 3-1, Markov modeling of the UAV base station scheduling problem; Step 3-2, defining the optimal strategy for the base station scheduling problem; Step 3-3, converting the optimal strategy described in step 3-2 into a multi-agent reinforcement learning problem; Step 3-4, use the kernel-based counterfactual multi-agent policy gradient method to solve the multi-agent reinforcement learning problem described in step 3-3.
3. According to the multi-agent reinforcement learning regional scheduling method for drone communication base stations as described in claim 2, it is characterized in that: The Markov modeling of the drone base station scheduling problem described in step 3-1 specifically includes: Define the environment state as the joint state s of the entire drone swarm at time t t ,as follows: s t =[s t,1 ,…,s t,i ,…,s t,N ,R t,1 ,…,R t,i ,…,R t,N ] T Among them, s t,i is the binary code of the area where the i-th UAV is located, R t,i is whether the assistance request sent by the base station is received, which is 1 if it is, otherwise 0. N is the total number of drones, and the joint action of all drones at time t is a t ,as follows: a t =[a t,1 ,...,a t,i ,...,a t,N ] T Among them, a t,i represents the action of the i-th drone, A set of binary codes representing flyable sub-areas; The transition probability is P(s t+1 |s t ,a t ), the utility function of UAV i performing assisted communication service at time t is: r t,i -λD t,i Among them, r t,i The number of ground communication users served by UAV i, D t,i is the moving distance, and λ is the proportional coefficient.
4. According to the multi-agent reinforcement learning regional scheduling method for drone communication base stations as described in claim 3, it is characterized in that: The optimal strategy for defining the base station scheduling problem described in step 3-2 specifically includes: Assume that the action strategy of the entire system, i.e., all drones, is π. The optimal strategy is to maximize the expectation of the cumulative discounted return at the frequency-using devices in the communication network as the target utility function V(s), i.e., the optimal Bellman equation, as follows: Among them, γ is the discount factor, E π represents the long-term expected return of adopting strategy π, and s is the joint state of the drone swarm.
5. According to the multi-agent reinforcement learning regional scheduling method for drone communication base stations as described in claim 4, it is characterized in that: Step 3-3 converts the optimal strategy described in step 3-2 into a multi-agent reinforcement learning problem, specifically including: Step 3-3-1, set an action strategy π for each drone i i , jointly optimize the global goal, as follows: Step 3-3-2, each drone i follows the action strategy π i Autonomously make regional dispatch decisions, and the ground control center only provides updates on action strategies, i.e. regional dispatch strategies; Step 3-3-3, treat the above process as a multi-agent reinforcement learning problem.
6. According to the multi-agent reinforcement learning regional scheduling method for drone communication base stations as described in claim 5, it is characterized in that: The counterfactual multi-agent policy gradient method based on the kernel method described in step 3-4 specifically includes: Step 3-4-1, each drone initializes a policy function network, as follows: The strategy function satisfies the Gibbs sampling distribution or the Boltzmann strategy, and the characteristic parameters of each network are a linear combination of a set of Gaussian radial basis functions, which satisfies: in, Indicates the probability that the current drone chooses action a in state s, is the weight parameter; in, is the weight vector of the preference function of the kth UAV, T is the interval time slot, is the basis function vector of the kth UAV; Step 3-4-2: At each moment, each drone makes and executes a regional scheduling strategy based on its own strategy function network; Step 3-4-3, the ground control center initializes a global state function network, as follows: V g (s,a)=w vT f v (s) Among them, V g (s,a) is the global state function, w v is the weight vector of the global state function, f v is the basis function vector of the global state function; Step 3-4-4, after each UAV flies to the target area, it will transmit the assistance communication service information to the ground control center, which will summarize it. All UAVs will transmit their status to the ground control center every T time slots and store it; every cT time slots, the ground control center will update the strategy functions of all UAVs.
7. According to the multi-agent reinforcement learning regional scheduling method for drone communication base stations as described in claim 6, it is characterized in that: The policy functions of all drones described in step 3-4-4 are updated, including: Step 3-4-4-1, design advantage function A(s j ,a), as follows: Among them, V(s j ,a) represents the global state value function, V(s j ,a,a -i ) represents the policy function π(a|v j,i ;w f,q,i ) represents the policy function value corresponding to the missing action; Step 3-4-4-2, weight w for drone i q,i Update as follows: Among them, β q is the learning rate, is the gradient of the policy function with respect to the weight; Step 3-4-4-3, update the weight w of the global state value function network according to the global state information v , as follows: Among them, r j is the immediate return, γ is the discount rate, V(s j ;w v ) is the current global state value function, V(s j ′ ;w v ) is the state value function of the next sample, is the gradient of the current global state value function; Step 3-4-4-4, the characteristic functions of the two functions are updated using an online sparse algorithm.
8. The multi-agent reinforcement learning regional scheduling method for drone communication base stations according to claim 7 is characterized in that: The characteristic functions of the two functions described in step 3-4-4-4 are updated using an online sparse algorithm, that is, an approximate linear independence detection algorithm is used to sparsify the feature dictionary of the strategy function and the global value function, and the algorithm determines whether to add the dictionary by judging the linear correlation between the linear combination of the characteristic functions of the feature vector and the feature vector to be judged.
9. The multi-agent reinforcement learning regional scheduling method for drone communication base stations according to claim 8 is characterized in that: The characteristic functions of the two functions described in step 3-4-4-4 are updated using an online sparse algorithm, specifically including: Step 3-4-4-4-1, calculate the expected difference δ between the linear combination of the feature vector to be judged and the feature vector in the feature dictionary; Step 3-4-4-4-2, define the expected difference threshold μ; Step 3-4-4-4-3, determine the size of the difference δ obtained in step 3-4-4-4-1 and the threshold μ. If it is greater than μ, add the feature vector to the feature dictionary. If it is less than or equal to the threshold μ, do not add it to the feature dictionary.
10. The multi-agent reinforcement learning regional scheduling method for drone communication base stations according to claim 9 is characterized in that: The expected difference δ between the linear combination of the feature vector to be judged and the feature vector in the feature dictionary described in step 3-4-4-4-1 is as follows: Among them, α is the weight, is the kth eigenvector.
