Unmanned aerial vehicle base station control system based on reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle base station control system based on reinforcement learning, and belongs to the technical field of artificial intelligence. The system comprises a first reinforcement learning module, an outer ring controller and a first approaching calculator, the first approaching calculator generates a sequence error signal according to the actually measured position of the nth unmanned aerial vehicle base station, and the first reinforcement learning module generates a first state at time t according to the sequence error signal; and the first reinforcement learning module generates a control strategy of the outer ring controller according to the first state, the weight of the unmanned aerial vehicle base station and the coverage angle. The optimal control parameters of the controller do not need to be preset, and autonomous learning can be achieved.

Description

Technical Field

[0001] This invention relates to a control system for a drone base station based on reinforcement learning, belonging to the field of artificial intelligence technology. Background Technology

[0002] Chinese invention patent application CN118642366A discloses a multi-UAV base station collaborative path planning method based on meta-reinforcement learning. The method includes the following steps: Step 1, establishing a multi-UAV assisted communication system model; Step 2, modeling the optimization problem as a Markov decision process; Step 3, designing a meta-critic algorithm based on an actor-critic architecture; Step 4, using an asynchronous parallel structure to allow multiple UAVs to interact with the environment and iteratively update network parameters. This invention enables collaborative coverage of ground users through UAV swarms while ensuring the minimum throughput required by ground users, improving the planning efficiency and service quality of UAV base stations.

[0003] However, the patent application does not disclose how the control parameters of the controller that controls the drone base station are autonomously learned. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a control system for a UAV base station based on reinforcement learning, which does not require prior learning of the controller's control parameters and can learn autonomously.

[0005] To achieve the aforementioned objective, a reinforcement learning-based unmanned aerial vehicle (UAV) base station control system is characterized by comprising a first reinforcement learning module, an outer loop controller, and a first approach calculator, wherein the first approach calculator is configured based on the nth UAV base station... Measured position and best position The first reinforcement learning module generates a sequence error signal and then generates a first state at time t based on the sequence error signal. The first reinforcement learning module is based on the first state. Unmanned aerial vehicle (UAV) base stations weight and coverage angle Control strategy for generating outer loop controller , This is the control parameter vector of the outer loop controller at time t.

[0006] Compared with existing technologies, the UAV base station control system based on reinforcement learning provided by this invention utilizes a reinforcement learning module to determine the measured location of the UAV base station and... The sequence error signal is generated, and the reinforcement learning module generates the first state based on the sequence error signal. The reinforcement learning module is based on the first state. The weight of the drone base station and coverage angle The control parameters for the outer loop controller can be generated without prior learning; they can be learned autonomously. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the composition of the marine Internet of Things of the present invention.

[0008] Figure 2 This is a block diagram of the control system for a UAV base station based on reinforcement learning provided by the present invention. Detailed Implementation

[0009] It should be noted that, below, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The advantages and features of the present invention, as well as the methods for achieving these advantages and features, will become clear from the accompanying drawings and the detailed embodiments described below.

[0010] However, the present invention is not limited to the embodiments disclosed below, and can be implemented in many different forms. This embodiment is only used to make the disclosure of the present invention more complete and to fully inform those skilled in the art of the present invention of the scope of the invention. The present invention is defined only by the scope of the claims.

[0011] While terms such as "first," "second," etc., are used to describe various elements, components, and / or parts, these elements, components, and / or parts are not limited by these terms. These terms are used only to distinguish one element, component, or part from other elements, components, or parts. Therefore, it is apparent that, within the technical spirit of this disclosure, the first element, first component, or first part mentioned below may also be a second element, second component, or second part, and the terminology used in this specification is for describing embodiments only and is not intended to limit this disclosure.

[0012] In this specification, unless otherwise specified in the text, the singular includes the plural. The use of "comprising" and / or "consisting of" in this specification does not exclude the presence or addition of one or more other structural elements, steps, actions, and / or components mentioned.

[0013] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Furthermore, terms defined in commonly used dictionaries shall not be interpreted ideally or excessively unless explicitly and specifically defined.

[0014] Figure 1This is a schematic diagram illustrating the composition of the marine Internet of Things (IoT) of this invention. (As shown...) Figure 1 As shown, the marine Internet of Things (IoT) includes a low Earth orbit (LEO) satellite, N unmanned aerial vehicle (UAV) base stations, and J marine IoT devices. The LEO satellite operates at a constant speed and fixed altitude within a predefined orbital plane. Simultaneously, the N UAV base stations are deployed as airborne base stations to facilitate communication within the marine environment. For simplicity, this invention assumes that the LEO satellite remains within its designated coverage area without switching during the observation period, thereby ensuring continuous service within the marine IoT. The marine IoT includes a set of UAV base stations. A collection of marine IoT devices distributed across ocean areas, communicating via low Earth orbit satellites and drone base stations. This invention defines the communication process over discrete time slots t and t={1,2,…,T}, with the system dynamically operating to optimize network performance at each moment. In this invention, the total duration is represented by T, during which the marine IoT consistently manages multiple communication and resource allocation tasks in each time slot within the originally allocated time frame T. Without loss of generality, this invention incorporates unmanned aerial vehicle (UAV) base stations. Position in 3D space is defined as: , In the formula, Indicates drone base station Latitude and longitude coordinates, This indicates its flight altitude. Similarly, marine IoT devices... The position is represented as: , in, Corresponding to the latitude and longitude of the marine IoT devices, altitude is assumed to be zero because these devices are located at sea level. To ensure uninterrupted service and optimal connectivity, the drone base station dynamically adjusts its location in each time slot based on the real-time location of the marine IoT devices. This adaptive repositioning allows the drone base station to maintain a stable and high-quality communication link. Notably, the drone base station maintains a link with a terrestrial base station (TBS) or a low-Earth orbit satellite, depending on its location relative to the TBS coverage area. When the drone base station is operating within the TBS coverage area, it connects via a high-speed, low-latency link. However, once the drone base station moves beyond the coastal area and leaves the TBS range, it seamlessly switches to a satellite link by connecting to a low-Earth orbit satellite, thus ensuring continuous connectivity in remote or maritime areas.

[0015] For downlink communication, this invention employs a marine network of low-Earth orbit satellites and UAV base stations integrating MIOTD devices. The low-Earth orbit satellites utilize Interleaved Frequency Division Multiple Access (OFDMA) and Non-Orthogonal Multiple Access (NOMA) communication methods. OFDMA divides the available bandwidth into multiple orthogonal subcarriers, thereby allowing flexible and adaptive subchannel allocation based on real-time service requirements. Each subchannel is allocated equal bandwidth. Where B represents the total system bandwidth and W represents the total number of sub-channels. This partitioning ensures efficient spectrum utilization and reduces interference. Meanwhile, NOMA enhances spectral efficiency by enabling multiple MIOTDs to utilize the same frequency band via power domain multiplexing. In this setup, different MIOTDs receive different power levels based on their respective channel conditions.

[0016] This invention defines connectivity decision as the nth UAV base station The function of the location relative to the coverage area of ​​TBS and low Earth orbit satellites. The coverage radius and the location of TBS are expressed as follows: and .

[0017] At time t, the nth drone base station The distance between and TBS is expressed as: , Connectivity is represented as: , ,but, ,otherwise .

[0018] Meanwhile, at time t, the nth drone base station and the jth sea network device The connection status of the link between them is represented as The j-th marine internet device is within the coverage area of ​​the n-th drone base station. ,otherwise ,Right now

[0019] hour ,otherwise, .

[0020] In the formula, For the nth UAV base station The coverage angle.

[0021] In this invention, the method for constructing a marine Internet of Things based on a drone base station includes: S1: Group the J sea network devices using a load grouping model, where the load grouping model is as follows: , In the formula, The j-th marine network device belongs to the component published by the n-th Gaussian and belongs to the n-th unmanned aerial vehicle base station. The probability of coverage area, where U is the set of drones: ; It is the mixing coefficient; and It is the nth Gaussian distribution Two parameters; For a Gaussian distribution set, ; S2: Configured to allocate channels to N UAV base stations via the channel allocation module; S3: Calculate the total performance index using the total performance index calculator; S4: The nth drone base station It updates its control parameter vector based on the overall performance index, provides signal relay to its M maritime joint defense devices according to the assigned channel, and also reports its parameters to low Earth orbit satellites, n=1,2,…,N.

[0022] In this invention, the overall performance index is represented by a set. , Indicates total throughput; Indicates overall communication fairness; This indicates the total communication coverage.

[0023] In this invention, the nth UAV base station throughput for: , In the formula, To be assigned to the nth drone base station The mth marine network device of the relay The bandwidth of the connection; , , From low Earth orbit satellites to drone base stations The transmission power; This refers to the altitude of a low Earth orbit satellite. For the nth UAV base station The altitude at which it is located; Noise power; The average channel gain is at a reference distance of 1m; , , For the nth UAV base station The power; For drone base stations Location, For the nth UAV base station The m-th marine network device of the relay The location.

[0024] In this invention, the total throughput of the low-Earth orbit satellites is: .

[0025] In this invention, the nth UAV base station Fairness of communication for: , In the formula, M is the nth drone base station Number of IoT devices served; For the nth UAV base station and the mth sea network device At time t, connectivity is 1 if connected, otherwise 0. , For coefficients; To be assigned to the nth drone base station The mth marine network device of the relay The bandwidth of the connection.

[0026] In this invention, the overall communication fairness of low-Earth orbit satellites is: .

[0027] In this invention, the nth UAV base station Communication coverage for: , In the formula, For the nth UAV base station and the mth sea network device At time t, connectivity is 1 for connectivity and 0 otherwise.

[0028] In this invention, the total communication coverage of low-Earth orbit satellites is: .

[0029] In this invention, the optimal locations of N unmanned aerial vehicle (UAV) stations are obtained according to the following formula; ; In the formula, .

[0030] In this invention, the bandwidth allocated to N unmanned aerial vehicle (UAV) stations is obtained according to the following formula; ; In the formula, .

[0031] In this invention, the nth UAV base station is obtained according to the following formula. Control functions: , It is the nth drone base station The parameter vector of the control function; It is the nth drone base station The state vector, the state vector Including the first state and the second state The nth UAV base station is updated using gradient descent with a supervised function. parameter vector No prior settings are required.

[0032] Figure 2 This is a block diagram of the reinforcement learning-based control system provided by the present invention. Figure 2 As shown, the control system of the UAV base station based on reinforcement learning includes a first reinforcement learning module, an outer loop controller, and a first approach calculator. The first approach calculator calculates the signal based on the data from the nth UAV base station. Measured position and best position The first reinforcement learning module generates a sequence error signal and then generates a first state at time t based on the sequence error signal. The first reinforcement learning module is based on the first state. Unmanned aerial vehicle (UAV) base stations weight and coverage angle Control strategy for generating outer loop controller , This is the control parameter vector of the outer loop controller at time t.

[0033] In this invention, the output of the outer loop controller at time t is: Unmanned aerial vehicle (UAV) base stations The position at time t is L. n (t) and optimal position The error is The outer loop controller outputs the angular velocity of one of the motors driving the drone: , In the formula, .

[0034] Written in matrix form: , In the formula, , .

[0035] In this invention, for a multi-rotor drone, the outer ring controller outputs the angular velocities of the multiple motors that drive the drone.

[0036] Although the present invention is described with a control quantity of 3, it can also be K-1, where K is greater than or equal to 2.

[0037] In this invention, the outer loop controller controls the position of the nth UAV station. The control parameters of the outer loop controller are affected not only by the state but also by the UAV station. Coverage angle Impact and weight The impact.

[0038] In this invention, the neural network of the first reinforcement learning module includes a first input layer, a first hidden layer, and a first output layer. The first input layer includes K+1 neurons, which are respectively input to the first state. Various elements, drone base stations weight and coverage angle The first hidden layer comprises K+1 neurons; the first output layer comprises K neurons, and the first to K-1 neurons of the first output layer output K-1 control quantities of the outer loop controller respectively. , In the formula, , , , The constraint, control law, and center of the q-th function of the neural network in the first reinforcement learning module at time t are respectively defined. These are the weights between the q-th neuron in the hidden layer and the k-th neuron in the output layer of the neural network in the reinforcement learning module at time t; q=1,…,K+1; k=1,…,K-1. Indicates splicing; It is the nth drone base station The weight; For drone station The coverage angle.

[0039] In the first embodiment, the first reinforcement learning module according to Generate drone base station The first state-value function at time t The output of the Kth neuron in the output layer of the neural network of the first reinforcement learning module is: , In the formula, It is the weight between the q-th neuron in the hidden layer and the K-th neuron in the output layer of the first reinforcement learning model neural network.

[0040] In the first embodiment, the first reinforcement learning module further constructs a first supervision function according to the following formula: , In the formula, Let n be the first state value function of the nth UAV base station at time t; , drone base stations The first state-value function at time t and time t-1; These are the weighting coefficients; and They are drone stations The optimal position and the measured position at time t.

[0041] In this invention, the first reinforcement learning module updates the parameter vector according to the following formula. Parameters in: The first reinforcement learning module updates parameters according to the following formula: , , , , In the formula, , , , The learning coefficient; To strengthen the learning module, the weights between the q-th neuron in the hidden layer and the k-th neuron in the output layer at time t+1, where k=1,…,K-1; To reinforce the weights of the q-th neuron in the hidden layer of the neural network to the K-th neuron in the output layer at time t; Let q be the control law of the q-th function at time t+1; It is the center of the q-th function at time t+1.

[0042] In the first embodiment, ; , ; .

[0043] In the first embodiment, the first reinforcement learning module further determines the first supervision function. Is it the smallest? If not, , , , Repeat the above steps; if yes, output... , , , As the optimal parameter for calculation , and make Each of these parameters is assigned to one of the K-1 control parameters of the outer loop controller.

[0044] The control system for a UAV base station based on reinforcement learning provided by this invention further includes a second approach calculator, an inner-loop controller, and a second reinforcement learning module. The second approach calculator calculates the measured angular velocity of the nth UAV base station and the ideal angular velocity output by the inner-loop controller. Generate the second error signal of the sequence The second reinforcement learning module is based on The nth drone station Communication coverage angle Unmanned aerial vehicle (UAV) stations weight Control strategy for generating inner loop controller , This is the control parameter vector for the inner loop controller.

[0045] The output of the inner loop controller at time t is It is the AC voltage value of the motor that drives the drone; the base station that drives the drone. The measured angular velocity of the running motor at time t and the angular velocity output by the outer loop controller The bit error rate is Then we have: , In the formula, , The rotation angle of the motor that drives the unmanned aerial vehicle (UAV) station; Written in matrix form: , In the formula, , The inner loop controller controls the attitude of the unmanned aerial vehicle (UAV) station.

[0046] Although the attitude control quantity of this invention is described using 3 as an example, it can also be S-1, where S is greater than or equal to 2.

[0047] In this invention, for a multi-rotor drone, the inner loop control outputs the AC voltage values ​​of multiple motors that drive the drone's operation.

[0048] In this invention, the neural network of the second reinforcement learning module includes a second input layer, a second hidden layer, and a second output layer. The second input layer includes S+1 neurons, the second hidden layer includes S+1 neurons, and the second output layer includes S neurons. The first to (S-1)th neurons of the second output layer respectively output S-1 control quantities of the inner loop controller. , In the formula, , , , These are the constraints, control law, and center of the p-th function of the neural network in the second reinforcement learning module at time t; is the weight between the p-th neuron in the hidden layer and the s-th neuron in the output layer of the neural network of the second reinforcement learning module at time t; p=1,…,S+1; s=1,…,S-1; Indicates splicing.

[0049] In the first embodiment, the second reinforcement learning module according to Generate drone base station Second state value function The outputs of the S-th neuron in the second output layer of the neural network of the second reinforcement learning module are as follows: , In the formula, It represents the weights between the p-th neuron in the hidden layer and the S-th neuron in the output layer of the second reinforcement learning model neural network.

[0050] In the first embodiment, the second reinforcement learning module further constructs a second supervision function according to the following formula: , In the formula, Let be the set attitude vector of the nth UAV base station at time t. Let be the measured attitude vector of the nth UAV base station at time t; The second state value function at time t; The second state value function at time t-1; These are the weighting coefficients.

[0051] The second reinforcement learning module updates the parameter vector according to the following formula. Parameters in: , -, , , In the formula, , , , The learning coefficient; Let s be the weights between the p-th neuron in the hidden layer and the k-th neuron in the output layer of the neural network in the second reinforcement learning module at time t+1, where s=1,…,S-1; The strength is the weight between the p-th neuron in the hidden layer of the neural network of the second reinforcement learning module and the S-th neuron in the second output layer at time t; These are the control laws of the p-th function at time t+1; Let p be the center of the p-th function at time t+1.

[0052] In this invention, ; , , .

[0053] In the first embodiment, the second reinforcement learning module also determines the second supervision function. Is it the smallest? If not, , , , Repeat the above steps; if yes, output... ,、 , , As the optimal parameter for calculation , and make Assign values ​​to the S-1 control parameters of the inner loop controller respectively.

[0054] In this invention, , .

[0055] The present invention, through the above technical solution, eliminates the need to pre-set the control parameters of the unmanned aerial vehicle (UAV) station; the control parameters can be obtained through active learning.

[0056] The preferred embodiments of the present invention disclosed herein are merely for the purpose of illustrating the present invention. The preferred embodiments do not describe all the details exhaustively, nor do they limit the invention to specific implementation methods. Obviously, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A control system for a drone base station based on reinforcement learning, characterized in that, It includes a first reinforcement learning module, an outer loop controller, and a first approach calculator, the first approach calculator based on the nth unmanned aerial vehicle station. Measured position and best position The first reinforcement learning module generates a sequence error signal and then generates a first state at time t based on the sequence error signal. The first reinforcement learning module is based on the first state. Unmanned aerial vehicle (UAV) base stations weight and coverage angle Control strategy for generating outer loop controller , This is the control parameter vector of the outer loop controller at time t.

2. The control system for a UAV base station based on reinforcement learning according to claim 1, characterized in that, The neural network of the first reinforcement learning module includes a first input layer, a first hidden layer, and a first output layer. The first input layer includes K+1 neurons, which are respectively input to various quantities of the first state and the UAV base station. weight and coverage angle The first hidden layer comprises K+1 neurons; the first output layer comprises K neurons, and the first to K-1 neurons of the first output layer output K-1 control quantities of the outer loop controller respectively. ， In the formula, ; , , These are the constraints, control law, and center of the q-th function of the neural network in the first reinforcement learning module at time t; These are the weights between the q-th neuron in the hidden layer and the k-th neuron in the output layer of the neural network in the reinforcement learning module at time t; q=1,…,K+1; k=1,…,K-1. Indicates splicing; It is the nth drone base station The weight; For drone station The coverage angle.

3. The control system for a UAV base station based on reinforcement learning according to claim 2, characterized in that, The first reinforcement learning module is based on Generate drone base station Position state value function at time t The output of the Kth neuron in the output layer of the neural network of the first reinforcement learning module is: ，， In the formula, It is the weight between the q-th neuron in the hidden layer and the K-th neuron in the output layer of the first reinforcement learning model neural network.

4. The control system for a UAV base station based on reinforcement learning according to claim 3, characterized in that, The first reinforcement learning module also constructs the first supervision function according to the following formula: ， In the formula, Let n be the position state value function of the nth UAV base station at time t; , drone base stations Position state value function at time t and time t-1; These are the weighting coefficients; and They are drone stations The optimal position and the measured position at time t.

5. The control system for a UAV base station based on reinforcement learning according to claim 4, characterized in that, The first reinforcement learning module updates parameters according to the following formula: ， ， ， ， In the formula, , , , The learning coefficient; To strengthen the learning module, the weights between the q-th neuron in the hidden layer and the k-th neuron in the output layer at time t+1, where k=1,…,K-1; To reinforce the weights of the q-th neuron in the hidden layer of the neural network to the K-th neuron in the output layer at time t; Let q be the control law of the q-th function at time t+1; It is the center of the q-th function at time t+1.

6. The control system for a UAV base station based on reinforcement learning according to claim 5, characterized in that, Also includes: The second approximation calculator, the inner loop controller, and the second reinforcement learning module, the second approximation calculator based on the UAV base station The measured angular velocity and the output of the inner loop controller of the unmanned base station The ideal angular velocity of the output Generate the second error signal of the sequence ; The second reinforcement learning module is based on Generate the second state According to the second state Unmanned aerial vehicle (UAV) base stations weight and coverage angle Control strategy for generating inner loop controller , This is the control parameter vector for the inner loop controller.

7. The control system for a UAV base station based on reinforcement learning according to claim 6, characterized in that, The neural network of the second reinforcement learning module includes a second input layer, a second hidden layer, and a second output layer. The second input layer includes S+1 neurons, which are respectively input to the second state. Various quantities, drone base stations weight and coverage angle The second hidden layer consists of S+1 neurons, and the second output layer consists of S neurons. The first to the (S-1)th neurons of the second output layer output S-1 control variables of the inner loop controller, respectively. ， In the formula, , , , These are the constraints, control law, and center of the p-th function of the neural network in the second reinforcement learning module at time t; These are the weights between the p-th neuron in the hidden layer and the s-th neuron in the output layer of the neural network in the second reinforcement learning module at time t; p=1,…,S+1; s=1,…,S-1; Indicates splicing.

8. The control system for a UAV base station based on reinforcement learning according to claim 7, characterized in that, The second reinforcement learning module is based on Generate drone base station Attitude state value function The output of the S-th neuron in the second output layer of the neural network of the second reinforcement learning module is: ， In the formula, It represents the weights between the p-th neuron in the hidden layer and the S-th neuron in the output layer of the second reinforcement learning model neural network.

9. The control system for a UAV base station based on reinforcement learning according to claim 8, characterized in that, The second reinforcement learning module also constructs a second supervision function based on the following formula: ， In the formula, For drone base stations The attitude vector is set at time t. For drone base stations The measured attitude vector at time t; The attitude state value function at time t; The attitude state value function at time t-1; These are the weighting coefficients.

10. The control system for a UAV base station based on reinforcement learning according to claim 9, characterized in that, The second reinforcement learning module updates parameters according to the following formula: ， ， ， ， In the formula, , , , The learning coefficient; Let s be the weights between the p-th neuron in the hidden layer and the k-th neuron in the output layer of the neural network in the second reinforcement learning module at time t+1, where s=1,…,S-1; The strength is the weight between the p-th neuron in the hidden layer of the neural network of the second reinforcement learning module and the S-th neuron in the second output layer at time t; These are the control laws of the p-th function at time t+1; Let p be the center of the p-th function at time t+1.

Images