Unmanned aerial vehicle resource allocation method, system and device based on multi-agent reinforcement learning, and medium

Abstract

The invention discloses an unmanned aerial vehicle resource allocation method, system and device based on multi-agent reinforcement learning and a medium, and the method comprises the steps: building a multi-target optimization problem of multi-unmanned aerial vehicle resource allocation in a data service scene for the multi-mode unmanned aerial vehicle assisted communication and sensing integrated data service scene; designing an average peak information age, an accumulated data service amount and an unmanned aerial vehicle energy consumption evaluation index; expressing a multi-objective optimization problem as a Markov decision problem, and designing a state space, an action space and a corresponding instant reward function; a multi-agent TD3 algorithm, namely an MAHTD3 algorithm, of a mixed action space is obtained by designing a discrete actor network and a continuous actor network, network parameters of an unmanned aerial vehicle are trained and updated by using the MAHTD3 algorithm according to a Markov decision model, and the unmanned aerial vehicle obtained by training is applied to a data service scene for communication resource allocation; the system, the equipment and the medium are used for implementing the method. According to the method, the data service requirements of the ground users in a random and dynamic communication and sensing integrated scene can be better met.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle communication technology, and in particular relates to a method, system, device and medium for allocating unmanned aerial vehicle resources based on multi-agent reinforcement learning. Background Art

[0002] In recent years, the research on drones as aerial base stations has attracted widespread attention from countries around the world due to their advantages such as multi-functionality, high mobility, flexible deployment, low cost, and high probability of establishing line-of-sight links. On the one hand, small drones play an important role in achieving airspace coverage in the 6G sky-ground integrated system, and can be used as a communication platform for airspace assistance. For example, in high-density communication scenarios, drones can be used as temporary base stations or relay stations to support wireless communications and enhance user capacity. On the other hand, in the field of emergency communications, although the basic communication infrastructure can usually handle daily communication loads, drones can be used to support communications when emergencies or unconventional temporary scenarios occur. For example, in network reconstruction after major natural disasters, drones can quickly adjust their positions to provide rapid post-disaster wireless service recovery.

[0003] These characteristics of drones provide opportunities for establishing efficient, flexible and reliable drone-assisted communication networks when fixed ground infrastructure is insufficient or faulty. However, the resource allocation problem in current data service scenarios still faces many challenges. On the one hand, the suddenness and randomness of ground users' data demands pose a major challenge to the fairness of data services and the improvement of service quality for ground users. On the other hand, the assumption of prior information of ground users is too idealistic. Although some studies have considered the dynamic changes of ground users, they only obtain the downlink transmission strategy of drones when the ground user information (location distribution, etc.) is known a priori, without considering how to perceive the status of ground users. At the same time, due to the limited battery energy of drones, it is often necessary to minimize energy consumption when performing data service tasks.

[0004] The flight trajectory planning, mode selection, and transmit power allocation of UAVs are the basis for ensuring energy-efficient UAV-assisted wireless networks. The trajectory of the UAV should be precisely designed to meet the dynamic data service needs of ground users in real time, while also minimizing energy consumption due to the limited energy of the UAV. At the same time, the transmit power of the UAV should be well allocated to replicate the dynamics of the channel state information between the UAV and the ground users, and the co-channel interference between UAVs should also be carefully considered. In addition, the UAV also needs to have the ability to select the action mode at the appropriate moment to perceive the information of the ground users, thereby enhancing its ability to meet the data needs of the ground users. In summary, it is of urgent importance to develop more practical methods to effectively advance UAV-assisted wireless networks.

[0005] The patent application document with publication number CN202411330172.5 discloses a multi-UAV path planning and power allocation method based on an autoencoder. By introducing a deep reinforcement algorithm based on PPO for multi-UAV path planning and power allocation, the average data transmission rate of user devices in the user environment is maximized. However, due to the limitation of the actor network structure, it cannot handle mixed action spaces with discrete and continuous actions, so the user position in the scene needs to be known a priori. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a drone resource allocation method, system, device and medium based on multi-agent reinforcement learning. By designing discrete and continuous actor networks and combining them with a multi-agent centralized training distributed execution training framework, the drone base station can select action modes and allocate communication resources in dynamic and random data service scenarios. The present invention has good scalability and high system performance.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A method for allocating drone resources based on multi-agent reinforcement learning includes the following steps:

[0009] Step 1, for the data service scenario of synaesthesia integration assisted by multimodal drones, a multi-objective optimization problem of multi-drone resource allocation in a data service scenario is established, and drone data service evaluation indicators are designed, wherein the drone data service evaluation indicators include: average peak information age, cumulative data service volume, and drone energy consumption;

[0010] Step 2: Formulate the multi-objective optimization problem of multi-UAV resource allocation in the data service scenario established in step 1 as a Markov decision problem, and design the state space, action space and corresponding immediate reward function respectively;

[0011] Step 3, by designing discrete actor networks and continuous actor networks, a multi-agent TD3 algorithm for mixed action space, namely the MAHTD3 algorithm, is obtained. Based on the Markov decision model obtained in step 2, the MAHTD3 algorithm is used to train and update the network parameters of the drone, and the trained drone is applied to the data service scenario for communication resource allocation.

[0012] The specific method of step 1 includes:

[0013] Step 1.1, make specific assumptions about the data service scenario; deploy multiple drones as aerial base stations in a specified area to provide data service support for ground users; during the operation of the drone, choose to enable the radar perception mode to locate ground users, or communicate with multiple ground users based on the orthogonal frequency division multiplexing (OFDM) scheme; at the same time, the location of some ground users is constantly changing, and they will have different data needs at different times; in addition, the drone provides data services by sending multiple data packets to ground users; as an aerial base station, the drone chooses its own behavior mode, radar perception mode or data transmission mode while deciding the action trajectory, and realizes power control in different modes;

[0014] Step 1.2, establish a communication link channel model between the UAV and the ground user; the channel capacity of the communication link is:

[0015]

[0016] Where B is the spectrum subband bandwidth, represents the probabilistic LoS channel gain of the communication link between UAV m and ground user n, P tm represents the transmission power of the UAV on the communication link, n0 represents the power spectral density of additive Gaussian white noise, represents the co-frequency interference on the communication link between UAV m and ground user n, specifically:

[0017]

[0018] Where M and N represent the number of drones and ground users, respectively, and P tij represents the transmission power of the communication link between UAV i and ground user j, A binary value is used to indicate whether UAV i is serving ground user j, where 1 indicates service and 0 indicates no service; A binary value is used to indicate whether the ground user j and the ground user n are using the frequency band for multiplexing, 1 indicates multiplexing, and 0 indicates no multiplexing;

[0019] Step 1.3, design the evaluation indicators of drone data services: average peak information age, cumulative data service volume and drone energy consumption, including the following sub-steps:

[0020] Step 1.31, design an average peak information age evaluation index; at time t>u, the age of the update data packet with timestamp u is determined to be tu, and when the updated timestamp corresponds to the current time t and the age is zero, the update data packet is considered fresh; the average peak information age of ground user i in time slot j is expressed as:

[0021]

[0022] Among them, U ij represents the cumulative peak number of terrestrial user i in time slot j, A a represents the ath peak information age; finally, the average peak information age of each terrestrial user in each time slot is expressed as:

[0023]

[0024] Where K represents the total number of time slots;

[0025] Step 1.32, design the energy consumption evaluation index of the UAV. During the data service process, the energy consumption of the UAV comes from flight, transmission and radar perception. The power model of the UAV flying at speed V is defined as:

[0026]

[0027] Among them, P0 and P i represent blade profile power and hover induced power respectively, U tip represents the tip speed of the rotor blade, v0 is the average rotor induced speed during hovering, d0 and s represent the fuselage drag ratio and rotor solidity, ρ and A represent the air density and rotor disc area respectively; therefore, the flight energy consumption of UAV i in time slot j is expressed as:

[0028] E fij =P f (V ij )τ

[0029] Among them, V ij represents the speed of UAV i in time slot j, τ represents the duration of each time slot; the transmission energy consumption of UAV is the total signal power transmitted by UAV to the relevant ground users on each subcarrier; the transmission energy consumption of UAV i in time slot j is expressed as:

[0030]

[0031] in, represents the transmission signal power of subcarrier f of drone i in time slot j, α f The occupancy of the subcarrier is represented by a binary value, 1 means that the subcarrier f is used for data transmission, and 0 means that it is not occupied. r is the radar perception binary value, then the radar perception energy consumption of UAV i in time slot j is:

[0032] E rij =α r P r τ

[0033] Among them, P r represents the power sensed by the radar; therefore, the energy consumption of the drone is expressed as:

[0034]

[0035] Among them, α tij A binary value is used to indicate whether UAV i performs data transmission in time slot j;

[0036] Step 1.33, design the evaluation index of the cumulative data service volume; the cumulative service data of the drone is expressed as:

[0037]

[0038] Among them, z ij represents the cumulative data service volume of UAV i in time slot j;

[0039] Step 1.4, combining the scenario assumptions of step 1.1, the communication link channel model between the UAV and the ground user established in step 1.2, and the UAV data service evaluation index designed in step 1.3, obtains the multi-objective optimization problem of multi-UAV resource allocation in the data service scenario, which is specifically described as:

[0040]

[0041] Among them, (a), (b) and (c) ensure the maximum flight speed constraints of the UAV and its location constraints; (d) represents the formulaic description of the selection of UAV behavior mode; (e) ensures that each ground user is served by only one UAV; (f) ensures that the number of ground users served by each UAV cannot exceed the number of subcarriers; (g) ensures that the power consumption of each UAV does not exceed the upper limit of the transmission power.

[0042] The specific method of step 2 includes:

[0043] Step 2.1, design the state space, including the state changes of ground users and drones; use three-channel images to represent the location and data demand of ground users, the location of drones, and the locations of other drones;

[0044] Among them, the image of one channel represents the status of the ground user, and the image size is L s ×W s , the grayscale has 256 levels; assuming the location coordinates of ground user i are (l i ,w i ), whose data requirement is q i Therefore, the coordinates of ground user i in the grayscale image (x i ,y i ) is expressed as:

[0045]

[0046] Where L and W represent the length and width of the data service area respectively. After determining the pixel position of the ground user in the image, the grayscale value of all pixels in the image is calculated. The size of the grayscale value represents the location information and data demand of the ground user. Using the two-dimensional Gaussian distribution and the data demand q i Determine (x i ,y i ) as the center, and the sub-pixel accuracy is guaranteed after the continuous mapping is discretized; at the same time, the distribution value φ of each pixel in the 3×3 image a It is expressed as:

[0047]

[0048] Where σ represents the variance of the two-dimensional Gaussian distribution, σ = 0.8; x a and a Represents the x-axis and y-axis coordinates of the pixels in the 3×3 image; therefore, the gray value gray a It is expressed as:

[0049]

[0050] in, Represents the normalized value of a two-dimensional Gaussian distribution;

[0051] The other two channel images represent the coordinates of the drone’s own position and the coordinates of other drones’ positions respectively. The grayscale and coordinate representation methods are the same as above, and the grayscale values ​​of the 3×3 images of all drones are the same;

[0052] Step 2.2, design the action space. The action of the drone consists of discrete actions and continuous actions. Discrete actions are to select action modes, including radar perception mode or data transmission mode; continuous actions are continuous parameters in all action modes. When the drone selects radar perception mode, since the radar power is fixed, the continuous action only changes on the x-axis and y-axis of the drone position coordinates; in addition, when the data transmission mode is selected, the continuous action also includes the transmission power allocated on each subcarrier when the drone selects data transmission.

[0053] Step 2.3, the reward function is designed as:

[0054]

[0055] Among them, k Z , k AoI and k E They represent the cumulative data service volume coefficient, average peak information age and energy consumption penalty factor in time period t respectively.

[0056] The specific method of step 3 includes:

[0057] Step 3.1, design discrete actor networks and continuous actor networks;

[0058] The discrete actor network includes two convolutional layers, a fully connected layer, and a softmax output layer, which are used to select the action mode of the drone; the continuous actor network includes two convolutional layers, a fully connected layer, and an output layer, which are used to output continuous parameters under all action modes. The inputs of the discrete actor network and the continuous actor network are the same, which are three-channel images of local observations of the drone;

[0059] Step 3.2, design a multi-agent TD3 algorithm in a hybrid action space, namely the MAHTD3 algorithm, and use the MAHTD3 algorithm to train and update the network parameters of the drone based on the Markov decision model obtained in step 2;

[0060] On the basis of the multi-agent TD3 algorithm, a hybrid action space design including discrete actions and continuous actions is added to obtain a multi-agent TD3 algorithm with a hybrid action space, namely the MAHTD3 algorithm; the MAHTD3 algorithm is within the framework of centralized training distributed execution, in which drones take actions based on their local observations in a distributed manner, and collect the experience of all drones for centralized training; therefore, the critic network of each drone needs to input the actions and local observations of all drones, while the discrete and continuous actor networks only need to input the drone's own local observations; the drones use the Markov decision model obtained in step 2, and each drone outputs the action to be taken based on its own local observations. The environment feeds back instant rewards to the drone based on the drone's actions, and the environmental state changes at the same time; the drone uses the MAHTD3 algorithm to train and update the parameters of its discrete and continuous actor networks and critic networks based on the above process, where the training process and parameter update method of the network parameters are the same as those of the multi-agent TD3 algorithm;

[0061] Step 3.3, apply the drone trained by MAHTD3 algorithm to the data service scenario for communication resource allocation;

[0062] For the hybrid action space design, each UAV is equipped with two actor networks: a discrete actor network and a continuous actor network. The discrete actor network is used to select the action mode, namely the radar perception mode or the data transmission mode, while the continuous actor network outputs the continuous action parameters of each action mode. After observing the current state of the environment, the UAV uses the discrete actor network to select the radar perception mode or the data transmission mode, and then obtains the corresponding continuous action parameters from the continuous actor network. When the UAV selects the radar perception mode, the UAV turns on the radar to perceive the position of the ground user and adjusts its own position at the same time. When the UAV selects the data transmission mode, the UAV transmits data to the ground user on each channel with the transmission power obtained by the continuous actor network and adjusts its own position.

[0063] The present invention also provides a drone resource allocation system based on multi-agent reinforcement learning, comprising:

[0064] A multi-objective optimization problem establishment module is used to establish a multi-objective optimization problem for multi-drone resource allocation in a data service scenario for a multi-modal drone-assisted synaesthesia-integrated data service scenario, and to design drone data service evaluation indicators, which include: average peak information age, cumulative data service volume, and drone energy consumption;

[0065] The Markov decision problem formulation module is used to formulate the multi-objective optimization problem of multi-UAV resource allocation in the established data service scenario as a Markov decision problem, and to design the state space, action space and corresponding immediate reward function respectively;

[0066] The UAV communication resource allocation module is used to obtain the multi-agent TD3 algorithm in the mixed action space, namely the MAHTD3 algorithm, by designing discrete actor networks and continuous actor networks. According to the Markov decision model, the MAHTD3 algorithm is used to train and update the network parameters of the UAV, and the trained UAV is applied to the data service scenario for communication resource allocation.

[0067] The present invention also provides a drone resource allocation device based on multi-agent reinforcement learning, comprising:

[0068] Memory: a computer program storing the above-mentioned method for allocating resources of unmanned aerial vehicles based on multi-agent reinforcement learning, which is a computer-readable device;

[0069] Processor: used to implement the method of drone resource allocation based on multi-agent reinforcement learning when executing the computer program.

[0070] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the drone resource allocation method based on multi-agent reinforcement learning.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] 1. Step 1 of the present invention designs average peak information age, cumulative data service volume and drone energy consumption evaluation indicators. Compared with traditional evaluation indicators such as data transmission rate, the present invention can maximize the quality of user data services while minimizing drone energy consumption.

[0073] 2. Step 2 of the present invention obtains a three-channel state diagram of the UAV by introducing a two-dimensional Gaussian distribution to calculate the pixel grayscale, which can reflect the current status of the ground user and the UAV in a more detailed and intuitive manner, is more adaptable to the dynamic and random changes of ground users in data service scenarios, and has good scalability.

[0074] 3. Step 3 of the present invention designs discrete actor networks and continuous actor networks, and combines them with a multi-agent training architecture of centralized training and distributed execution, so that the MAHTD3 algorithm can be applied to drones carrying a variety of payloads, has good scalability, and is more in line with actual scene requirements.

[0075] In summary, the present invention establishes user-oriented evaluation indicators and designs discrete and continuous actor networks, so that the UAV can make real-time decisions on action modes and communication resource allocation strategies in scenarios where ground users change randomly and dynamically, thereby improving the data service quality for ground users as much as possible, and has high scalability and excellent system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a data service scenario diagram for synaesthesia integration assisted by the multimodal drone of the present invention.

[0077] Figure 2 This is a graph of the information age changes of ground users in the synaesthesia-integrated data service scenario assisted by the multimodal drone in the present invention.

[0078] Figure 3 : is the state space diagram of the drone in the present invention, wherein Figures (a), (b), and (c) are grayscale images of three channels respectively.

[0079] Figure 4 This is the operating framework diagram of the MAHTD3 algorithm proposed in the present invention.

[0080] Figure 5This is a network structure diagram of the discrete and continuous actor networks in the MAHTD3 algorithm proposed in the present invention.

[0081] Figure 6 This is a graph showing the convergence of rewards for all drones using the MAHTD3 algorithm proposed in this invention.

[0082] Figure 7 This is a graph showing the convergence of rewards for each drone of the MAHTD3 algorithm proposed in the present invention.

[0083] Figure 8 This is a diagram showing changes in the average peak information age of ground users under different methods of the present invention.

[0084] Fig. 9 This is a diagram showing changes in energy consumption of a drone under different methods of the present invention.

[0085] Fig.10 This is a diagram showing changes in the cumulative data service volume of the drone under different methods of the present invention.

[0086] Fig.11 This is a schematic diagram of the final UAV's motion trajectory and action mode selection under the MAHTD3 algorithm proposed in the present invention. DETAILED DESCRIPTION

[0087] In order to more clearly understand the technical features, purposes and effects of the present invention, the following detailed description of its technical solution is now given. Obviously, the implementation cases described are only some embodiments of the present invention, not all, and should not be regarded as limiting the scope of implementation of the present invention; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0088] The discrete and continuous actor networks proposed in the present invention can select different action modes (radar perception mode or data transmission mode) to select appropriate time slots to perceive the location of ground users, and can realize real-time data services when their locations are unknown a priori, which is more in line with actual scenarios. The multi-agent TD3 algorithm in a hybrid action space proposed in the present invention can be used for data service resource allocation in a synaesthesia integration scenario assisted by drones.

[0089] See also Figure 4 , a drone resource allocation method based on multi-agent reinforcement learning, comprising the following steps:

[0090] Step 1: Establish a multi-objective optimization problem for multi-UAV resource allocation in a data service scenario for a multi-modal UAV-assisted synaesthesia-integrated data service scenario, and design UAV data service evaluation indicators, which include: average peak information age, cumulative data service volume, and UAV energy consumption; specifically:

[0091] Step 1.1, make specific assumptions about the data service scenario; deploy multiple drones as aerial base stations in a specified area to provide data service support for ground users; during the operation of the drone, you can choose to enable the radar perception mode to locate ground users, or communicate with multiple ground users based on the orthogonal frequency division multiplexing (OFDM) scheme; at the same time, the location of some ground users is constantly changing, and they will have different data needs at different times; in addition, the drone will provide data services by sending multiple data packets to ground users; as an aerial base station, the drone needs to choose its own behavior mode, radar perception mode or data transmission mode while deciding the action trajectory, and realize power control in different modes, so as to maximize the data service quality of ground users while minimizing the energy consumption of the drone;

[0092] Step 1.2, establish a communication link channel model between the UAV and the ground user; the channel capacity of the communication link is:

[0093]

[0094] Where B is the spectrum subband bandwidth, represents the probabilistic LoS channel gain of the communication link between UAV m and ground user n, P tm represents the transmission power of the UAV on the communication link, n0 represents the power spectral density of additive Gaussian white noise, represents the co-frequency interference on the communication link between UAV m and ground user n, specifically:

[0095]

[0096] Where M and N represent the number of drones and ground users, respectively, and P tij represents the transmission power of the communication link between UAV i and ground user j, A binary value is used to indicate whether UAV i is serving ground user j, 1 indicates service and 0 indicates no service. A binary value is used to indicate whether the ground user j and the ground user n are using the frequency band for multiplexing, 1 indicates multiplexing, and 0 indicates no multiplexing;

[0097] Step 1.3, design the evaluation indicators of drone data services: average peak information age, cumulative data service volume and drone energy consumption, including the following sub-steps:

[0098] Step 1.31, design an average peak information age evaluation indicator; information age is an end-to-end indicator that can describe the delay in the status update system and application, and is used to describe the timeliness of the data service in the scenario. At time t>u, the age of the update data packet with timestamp u is considered to be tu. When the timestamp of the update corresponds to the current time t and the age is zero, the update data packet is considered fresh; Figure 2 As shown, for example, each update of the information age depends on the age of the data packet received by the ground user. The ground user receives an update data packet with a timestamp of t1 from the UAV at time t2, and the information age is updated to t2-t1; in particular, the time slot ends at time t4, at which time the transmission process of an update packet is interrupted, which can be regarded as a data packet loss. In the next time slot, due to the UAV's re-decision, the age of the received update packet also changes, such as t5. Finally, when all data packets are transmitted, the data needs of the ground user are met, and the information age becomes zero at time t6; therefore, the average peak information age of ground user i in time slot j is expressed as:

[0099]

[0100] Among them, U ij represents the cumulative peak number of terrestrial user i in time slot j, A a represents the ath peak information age; finally, the average peak information age of each terrestrial user in each time slot is expressed as:

[0101]

[0102] Where K represents the total number of time slots;

[0103] Step 1.32, design the energy consumption evaluation index of the UAV. During the data service process, the energy consumption of the UAV comes from flight, transmission and radar perception. The power model of the UAV flying at speed V is defined as:

[0104]

[0105] Among them, P0 and P i represent blade profile power and hover induced power respectively, U tip represents the tip speed of the rotor blade, v0 is the average rotor induced speed during hovering, d0 and s represent the fuselage drag ratio and rotor solidity, ρ and A represent the air density and rotor disc area respectively; therefore, the flight energy consumption of UAV i in time slot j is expressed as:

[0106] E fij =P f (V ij )τ

[0107] Among them, V ij represents the speed of UAV i in time slot j. τ represents the duration of each time slot; the transmission energy consumption of the UAV is the total signal power transmitted by the UAV to the relevant ground users on each subcarrier; the transmission energy consumption of UAV i in time slot j is expressed as:

[0108]

[0109] in, represents the transmission signal power of subcarrier f of drone i in time slot j, α f The occupancy of the subcarrier is represented by a binary value, 1 means that the subcarrier f is used for data transmission, and 0 means that it is not occupied. r is the radar perception binary value, then the radar perception energy consumption of UAV i in time slot j is:

[0110] E rij =α r P r τ

[0111] Among them, P r represents the power sensed by the radar; therefore, the energy consumption of the drone is expressed as:

[0112]

[0113] Among them, α tij A binary value is used to indicate whether UAV i transmits data in time slot j.

[0114] Step 1.33, design the evaluation index of the cumulative data service volume; the cumulative service data of the drone is expressed as:

[0115]

[0116] Among them, z ij represents the cumulative data service volume of UAV i in time slot j;

[0117] Step 1.4, combining the scenario assumptions of step 1.1, the communication link channel model between the UAV and the ground user established in step 1.2, and the UAV data service evaluation index designed in step 1.3, obtains the multi-objective optimization problem of multi-UAV resource allocation in the data service scenario, which is specifically described as:

[0118]

[0119] Among them, (a), (b) and (c) ensure the maximum flight speed constraints of the UAV and its location constraints; (d) represents the formulaic description of the selection of UAV behavior mode; (e) ensures that each ground user is served by only one UAV; (f) ensures that the number of ground users served by each UAV cannot exceed the number of subcarriers; (g) ensures that the power consumption of each UAV does not exceed the upper limit of the transmission power.

[0120] Step 2: Formulate the multi-objective optimization problem of multi-UAV resource allocation in the data service scenario established in step 1 as a Markov decision problem, and design the state space, action space and corresponding immediate reward function respectively; specifically:

[0121] Step 2.1, design the state space, including the state changes of ground users and UAVs; therefore, in order to better display the dynamic changes of the states of ground users and UAVs, such as Figure 3 As shown, three-channel images are used to represent the location and data demand of ground users, the location of the drone, and the locations of other drones respectively;

[0122] Among them, the image of one channel represents the status of the ground user, and the image size is L s ×W s , the grayscale has 256 levels; assuming the location coordinates of ground user i are (l i ,w i ), whose data requirement is q i Therefore, the coordinates of ground user i in the grayscale image (x i ,y i ) is expressed as:

[0123]

[0124] Among them, L and W represent the length and width of the data service area respectively. After determining the pixel position of the ground user in the image, the grayscale value of all pixels in the image is calculated. The size of the grayscale value represents the location information and data demand of the ground user. In order to facilitate the convolutional neural network to process the state diagram, the two-dimensional Gaussian distribution and the data demand q are used. i Determine (x i ,y i ) as the center, and the sub-pixel accuracy is guaranteed after the continuous mapping is discretized; at the same time, the distribution value φ of each pixel in the 3×3 image a It is expressed as:

[0125]

[0126] Where σ represents the variance of the two-dimensional Gaussian distribution, σ = 0.8; x a and aRepresents the x-axis and y-axis coordinates of the pixels in the 3×3 image; therefore, the gray value gray a It is expressed as:

[0127]

[0128] in, Represents the normalized value of a two-dimensional Gaussian distribution;

[0129] The other two channel images represent the coordinates of the drone’s own position and the coordinates of other drones’ positions respectively. The grayscale and coordinate representation methods are the same as above, and the grayscale values ​​of the 3×3 images of all drones are the same;

[0130] Step 2.2, design the action space. The action of the drone consists of discrete actions and continuous actions. Discrete actions are to select action modes, including radar perception mode or data transmission mode; continuous actions are continuous parameters in all action modes. When the drone selects radar perception mode, since the radar power is fixed, the continuous action only changes on the x-axis and y-axis of the drone position coordinates; in addition, when the data transmission mode is selected, the continuous action also includes the transmission power allocated on each subcarrier when the drone selects data transmission.

[0131] Step 2.3: Since multiple drones work together to improve the data service quality of all ground users, each drone must not only maximize the cumulative data service volume, but also minimize the average peak information age and energy consumption. The reward function is designed as:

[0132]

[0133] Among them, k Z , k AoI and k E They represent the cumulative data service volume coefficient, average peak information age and energy consumption penalty factor in time period t respectively.

[0134] Step 3, by designing discrete actor networks and continuous actor networks, a multi-agent TD3 algorithm for mixed action space, namely the MAHTD3 algorithm, is obtained. Based on the Markov decision model obtained in step 2, the MAHTD3 algorithm is used to train and update the network parameters of the drone, and the trained drone is applied to the data service scenario for communication resource allocation.

[0135] like Figure 5 As shown, step 3.1, design discrete actor networks and continuous actor networks;

[0136] The discrete actor network includes two convolutional layers, a fully connected layer and a softmax output layer, which are used to select the action mode of the drone; the continuous actor network includes two convolutional layers, a fully connected layer and an output layer, which are used to output continuous parameters under all action modes; the inputs of the discrete actor network and the continuous actor network are the same, which are three-channel images of local observations of the drone;

[0137] Step 3.2, design a multi-agent TD3 algorithm in a hybrid action space, namely the MAHTD3 algorithm, and use the MAHTD3 algorithm to train and update the network parameters of the drone based on the Markov decision model obtained in step 2;

[0138] On the basis of the multi-agent TD3 algorithm, a hybrid action space design including discrete actions and continuous actions is added to obtain a multi-agent TD3 algorithm with a hybrid action space, namely the MAHTD3 algorithm; the MAHTD3 algorithm is within the framework of centralized training distributed execution, in which drones take actions based on their local observations in a distributed manner, and collect the experience of all drones for centralized training; therefore, the critic network of each drone needs to input the actions and local observations of all drones, while the discrete and continuous actor networks only need to input the drone's own local observations; the drones use the Markov decision model obtained in step 2, and each drone outputs the action to be taken based on its own local observations. The environment feeds back instant rewards to the drone based on the drone's actions, and the environmental state changes at the same time; the drone uses the MAHTD3 algorithm to train and update the parameters of its discrete and continuous actor networks and critic networks based on the above process, where the training process and parameter update method of the network parameters are the same as those of the multi-agent TD3 algorithm;

[0139] Step 3.3, apply the drone trained by MAHTD3 algorithm to the data service scenario for communication resource allocation;

[0140] For the hybrid action space design, each UAV is equipped with two actor networks: a discrete actor network and a continuous actor network. The discrete actor network is used to select the action mode, namely the radar perception mode or the data transmission mode, while the continuous actor network outputs the continuous action parameters of each action mode. After observing the current state of the environment, the UAV uses the discrete actor network to select the radar perception mode or the data transmission mode, and then obtains the corresponding continuous action parameters from the continuous actor network. When the UAV selects the radar perception mode, the UAV turns on the radar to perceive the position of the ground user and adjusts its own position at the same time. When the UAV selects the data transmission mode, the UAV transmits data to the ground user on each channel with the transmission power obtained by the continuous actor network and adjusts its own position.

[0141] Implementation case: A MAHTD3 algorithm is proposed and applied to a multimodal drone-assisted synaesthesia integrated data service scenario to solve the drone trajectory optimization and communication resource allocation problems.

[0142] Research problem scenarios such as Figure 1 As shown in the algorithm framework diagram Figure 4 As shown, the specific implementation method is introduced in three parts: the first part is the description of the simulation scenario and the setting of parameters, the second part is the specific application process of the MAHTD3 algorithm, and the third part is the simulation results and effect verification.

[0143] Part 1: Simulation scenario simulation and parameter setting;

[0144] The simulation experiment was conducted in a square area of ​​500m x 500m, where three drones flew at a fixed altitude of 100m to provide data services to 25 randomly distributed ground users. The actor and critic networks of the drones in the MAHTD3 algorithm are designed with three convolutional layers, two hidden layers, and a residual network. A total of 3,000 training runs were conducted, each of which contained 40 steps, and a performance test was performed every 50 training runs to evaluate the effectiveness of the agent.

[0145] The initial positions of ground users are uniformly distributed, and in order to simulate the dynamic changes of user positions in actual scenarios, ground users will move back and forth along a certain trajectory. At the same time, in order to simulate the randomness of user data demand in actual scenarios, the data demand of each ground user in different time slots is set to follow the normal distribution X in the simulation experiment. q ~N(4,1 2 )(MB). In addition, according to the TCP / IP protocol, the UAV as a base station will provide data services by sending multiple data packets to ground users. The working time of the UAV is divided into K time slots, and the length of each time slot is τ. The other important parameter settings are shown in Table 1 below.

[0146] Table 1: Simulation experiment parameter settings

[0147]

[0148]

[0149] Part II: Practical application of the MAHTD3 algorithm in the data service scenario of multimodal drone-assisted synaesthesia integration, which is specifically divided into the following 7 steps;

[0150] Step 1: Initialize the UAV's actor and critic networks and their corresponding target network parameters, network update frequency, delay update frequency, experience pool, and simulation environment parameters;

[0151] Step 2: Determine whether the maximum number of iterations has been reached. If so, terminate the algorithm and output the optimal trajectory optimization and communication resource allocation strategy, the maximum total reward value, the maximum cumulative data service volume, the minimum drone energy consumption and the average peak information age. Otherwise, proceed to step 3.

[0152] Step 3: Determine whether the maximum number of steps has been reached, if not, proceed to step 4, if yes, proceed to step 2;

[0153] Step 4: The agent obtains the current environment state S, inputs the state into the agent's actor network, and obtains the corresponding action A. At the same time, the environment state changes according to the action. The agent obtains the reward r obtained by the action and the next environment state S', as well as whether the next state is the final state end;

[0154] Step 5: Store the experience [S, A, r, S′, end] into the experience pool and determine whether the size of the experience pool is greater than the number of training batch samples. If so, proceed to step 6; otherwise, proceed to step 4.

[0155] Step 6: Randomly select a batch of samples from the experience pool, calculate the loss function values ​​of the drone's actor network and critic network, and update the drone's critic network and actor network according to the delayed update strategy;

[0156] Step 7: Determine whether the update frequency of the target network has been reached. If so, copy the parameters of the main network to the target network according to the soft update strategy. If not, proceed to step 3.

[0157] Part 3: Simulation experiment results presentation and results analysis;

[0158] First, the following comparison algorithms are listed:

[0159] MADDPG: A multi-agent reinforcement learning algorithm based on a centralized training distributed learning framework, using the DDPG algorithm and the same hybrid action space design as the MAHTD3 algorithm. In the MADDPG algorithm, each agent also has two actor networks, a discrete actor network and a continuous actor network, which output action mode selection and continuous action parameters respectively.

[0160] HTD3: A standalone training algorithm for TD3 using a hybrid action space design.

[0161] Clustering algorithm: Based on the mean shift clustering algorithm, a multi-UAV dynamic programming algorithm that considers the data requirements of ground users. Based on the spatial distribution of ground users, the mean shift clustering algorithm is used to periodically organize adjacent ground users into clusters. Subsequently, the data requirements of each cluster are calculated, and the cluster center and its weight relative to each UAV are determined. The weight is determined by the distance between the UAV and the cluster center and the data requirement. As the distance decreases, the data requirement increases, resulting in an increase in weight. Subsequently, each UAV will move to the cluster center associated with the highest weight.

[0162] The simulation results are as follows Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 and Fig.11 shown.

[0163] from Figure 6 It can be concluded that the MAHTD3 algorithm proposed in the present invention has a smoother training process and better convergence results than the other two algorithms by combining the advantages of the centralized training distribution execution framework and the TD3 algorithm. On the one hand, compared with the HTD3 algorithm, the rewards of all drones after the MAHTD3 algorithm and the MADDPG algorithm converge are 120.4% and 22.4% higher, respectively. Obviously, the HTD3 algorithm using independent learning ignores the impact of other agents on the environment, and the training effect is not ideal. Moreover, the centralized training method of the MAHTD3 algorithm of the present invention can more accurately evaluate the current state-action value, thereby enhancing the strategy of the agent. On the other hand, compared with the MADDPG algorithm, the MAHTD3 algorithm proposed in the present invention increases the average reward by 79.6%. The MAHTD3 algorithm reduces the target Q value estimation error by utilizing dual networks, target strategy smoothing regularization, and delayed update strategy. Therefore, the MAHTD3 algorithm proposed in the present invention effectively alleviates the overestimation error caused by the maximum operator. This helps the agent converge to the optimal strategy faster while preventing local optimality.

[0164] Figure 7 The reward value changes of each drone in each round of training are shown. It can be seen that the reward of drone 2 after convergence is significantly better than that of drone 1 and drone 3. Due to the different geographical locations of drones and their different capabilities to serve ground users, the final performance is also very different. According to the centralized training distributed execution framework, each drone can share information with other drones and cooperate to provide data services, thereby achieving overall performance improvement.

[0165] Figure 8The changes in the average peak information age of ground users in each round of different algorithms are shown. The simulation results show that the MAHTD3 algorithm proposed in this invention obtains the smallest average peak information age, and the average peak information age of other reinforcement learning algorithms (MADDPG and HTD3) is much smaller than that of the clustering algorithm. Compared with the MADDPG algorithm, the HTD3 algorithm and the clustering algorithm, the average peak information age of the MAHTD3 algorithm is 25.1%, 43.6% and 59.4% smaller, respectively, which shows that the MAHTD3 algorithm proposed in this invention has superior performance in meeting the timeliness requirements of data services.

[0166] Fig. 9 The energy consumption reduction of different algorithms in each iteration is shown. The simulation results show that the energy consumption of the drone of the MAHTD3 algorithm and the MADDPG algorithm proposed in the present invention is similar, which are 9632.4J and 9628.3J respectively. However, compared with the HTD3 algorithm (9881.1J) and the clustering algorithm (9801.6J), the energy consumption of the MAHTD3 algorithm is reduced by 2.52% and 1.73%. The above results reflect that the MAHTD3 algorithm proposed in the present invention still has a good effect in reducing the energy consumption of drones.

[0167] Fig.10 The changes in the cumulative data service volume under different algorithms are shown. The simulation results show that the cumulative data service volume of the MAHTD3 algorithm proposed in the present invention is 19.1%, 37.1% and 20.2% higher than that of the MADDPG algorithm, the HTD3 algorithm and the cluster algorithm, respectively. The drone with the MAHTD3 algorithm has the largest cumulative data service volume, which proves to a certain extent that the MAHTD3 algorithm proposed in the present invention can better meet the timeliness of data services.

[0168] Fig.11 The final drone motion trajectory and action mode selection under the MAHTD3 algorithm are shown. The short lines in the figure are the motion trajectories of movable ground users. The flight trajectory of the drone is consistent with the distribution of ground users, and it tries to fly to ground users with greater data demand in each time period. At the same time, the drone selects the radar perception mode at the appropriate time to provide better data services for ground users.

[0169] Therefore, the present invention adopts the above-mentioned MAHTD3 algorithm and applies it to the UAV-assisted synaesthesia integrated data service scenario to solve the UAV trajectory optimization and communication resource allocation problems. The simulation results show that the method proposed in the present invention has better stability and convergence than the comparison algorithms such as MADDPG algorithm, HTD3 algorithm and clustering algorithm, and finally obtains more rewards and has better data service performance, which proves the effectiveness of the MAHTD3 algorithm proposed in the present invention in complex and dynamic UAV-assisted data service scenarios.

[0170] The present invention also provides a drone resource allocation system based on multi-agent reinforcement learning, comprising:

[0171] A multi-objective optimization problem establishment module is used to establish a multi-objective optimization problem for multi-drone resource allocation in a data service scenario for a synaesthesia-integrated data service scenario assisted by a multi-modal drone in step 1, and design drone data service evaluation indicators, wherein the drone data service evaluation indicators include: average peak information age, cumulative data service volume, and drone energy consumption;

[0172] The Markov decision problem formulation module is used to implement the multi-objective optimization problem of multi-UAV resource allocation in the data service scenario established in step 1 as a Markov decision problem in step 2, and to design the state space, action space and corresponding immediate reward function respectively;

[0173] The UAV communication resource allocation module is used to implement the multi-agent TD3 algorithm in the mixed action space obtained by designing discrete actor networks and continuous actor networks in step 3, namely the MAHTD3 algorithm, and use the MAHTD3 algorithm to train and update the network parameters of the UAV according to the Markov decision model obtained in step 2, and apply the trained UAV to the data service scenario for communication resource allocation.

[0174] The present invention also provides a drone resource allocation device based on multi-agent reinforcement learning, comprising:

[0175] Memory: a computer program storing the above-mentioned method for allocating resources of unmanned aerial vehicles based on multi-agent reinforcement learning, which is a computer-readable device;

[0176] Processor: used to implement the method of drone resource allocation based on multi-agent reinforcement learning when executing the computer program.

[0177] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the drone resource allocation method based on multi-agent reinforcement learning.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and do not constitute limitations. Although the present invention has been described in detail in conjunction with the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present invention can still be modified or replaced by equivalents, and these modifications or replacements should not depart from the spirit and scope of the technical solutions of the present invention.

Claims

1. A drone resource allocation method based on multi-agent reinforcement learning, characterized in that: The following steps are involved: Step 1, for the data service scenario of synaesthesia integration assisted by multimodal drones, a multi-objective optimization problem of multi-drone resource allocation in a data service scenario is established, and drone data service evaluation indicators are designed, wherein the drone data service evaluation indicators include: average peak information age, cumulative data service volume, and drone energy consumption; Step 2: Formulate the multi-objective optimization problem of multi-UAV resource allocation in the data service scenario established in step 1 as a Markov decision problem, and design the state space, action space and corresponding immediate reward function respectively; Step 3, by designing discrete actor networks and continuous actor networks, a multi-agent TD3 algorithm for mixed action space, namely the MAHTD3 algorithm, is obtained. Based on the Markov decision model obtained in step 2, the MAHTD3 algorithm is used to train and update the network parameters of the drone, and the trained drone is applied to the data service scenario for communication resource allocation.

2. The method for allocating drone resources based on multi-agent reinforcement learning according to claim 1, characterized in that: The specific method of step 1 includes: Step 1.1, make specific assumptions about the data service scenario; deploy multiple drones as aerial base stations in a specified area to provide data service support for ground users; during the operation of the drone, choose to enable the radar perception mode to locate ground users, or communicate with multiple ground users based on the orthogonal frequency division multiplexing (OFDM) scheme; at the same time, the location of some ground users is constantly changing, and they will have different data needs at different times; in addition, the drone provides data services by sending multiple data packets to ground users; as an aerial base station, the drone chooses its own behavior mode, radar perception mode or data transmission mode while deciding the action trajectory, and realizes power control in different modes; Step 1.2, establish a communication link channel model between the UAV and the ground user; the channel capacity of the communication link is: Where B is the spectrum subband bandwidth, represents the probabilistic LoS channel gain of the communication link between UAV m and ground user n, P tm represents the transmission power of the UAV on the communication link, n0 represents the power spectral density of additive Gaussian white noise, represents the co-frequency interference on the communication link between UAV m and ground user n, specifically: Where M and N represent the number of drones and ground users, respectively, and P tij represents the transmission power of the communication link between UAV i and ground user j, A binary value is used to indicate whether UAV i is serving ground user j, where 1 indicates service and 0 indicates no service; A binary value is used to indicate whether the ground user j and the ground user n are using the frequency band for multiplexing, 1 indicates multiplexing, and 0 indicates no multiplexing; Step 1.3, design the evaluation indicators of drone data services: average peak information age, cumulative data service volume and drone energy consumption, including the following sub-steps: Step 1.31, design an average peak information age evaluation index; at time t>u, the age of the update data packet with timestamp u is determined to be tu, and when the updated timestamp corresponds to the current time t and the age is zero, the update data packet is considered fresh; the average peak information age of ground user i in time slot j is expressed as: Among them, U ij represents the cumulative peak number of terrestrial user i in time slot j, A a represents the ath peak information age; finally, the average peak information age of each terrestrial user in each time slot is expressed as: Where K represents the total number of time slots; Step 1.32, design the energy consumption evaluation index of the UAV. During the data service process, the energy consumption of the UAV comes from flight, transmission and radar perception. The power model of the UAV flying at speed V is defined as: Among them, P0 and P i represent blade profile power and hover induced power respectively, U tip represents the tip speed of the rotor blade, v0 is the average rotor induced speed during hovering, d0 and s represent the fuselage drag ratio and rotor solidity, ρ and A represent the air density and rotor disc area respectively; therefore, the flight energy consumption of UAV i in time slot j is expressed as: AND fij =P f (V ij )τ Among them, V ij represents the speed of UAV i in time slot j, τ represents the duration of each time slot; the transmission energy consumption of UAV is the total signal power transmitted by UAV to the relevant ground users on each subcarrier; the transmission energy consumption of UAV i in time slot j is expressed as: in, represents the transmission signal power of subcarrier f of drone i in time slot j, α f The occupancy of the subcarrier is represented by a binary value, 1 means that the subcarrier f is used for data transmission, and 0 means that it is not occupied. r is the radar perception binary value, then the radar perception energy consumption of UAV i in time slot j is: E rij =a r P r t Among them, P r represents the power sensed by the radar; therefore, the energy consumption of the drone is expressed as: Among them, α tij A binary value is used to indicate whether UAV i performs data transmission in time slot j; Step 1.33, design the evaluation index of the cumulative data service volume; the cumulative service data of the drone is expressed as: Among them, z ij represents the cumulative data service volume of UAV i in time slot j; Step 1.4, combining the scenario assumptions of step 1.1, the communication link channel model between the UAV and the ground user established in step 1.2, and the UAV data service evaluation index designed in step 1.3, obtains the multi-objective optimization problem of multi-UAV resource allocation in the data service scenario, which is specifically described as: Among them, (a), (b) and (c) ensure the maximum flight speed constraints of the UAV and its location constraints; (d) represents the formulaic description of the selection of UAV behavior mode; (e) ensures that each ground user is served by only one UAV; (f) ensures that the number of ground users served by each UAV cannot exceed the number of subcarriers; (g) ensures that the power consumption of each UAV does not exceed the upper limit of the transmission power.

3. The method for allocating drone resources based on multi-agent reinforcement learning according to claim 1, characterized in that: The specific method of step 2 includes: Step 2.1, design the state space, including the state changes of ground users and drones; use three-channel images to represent the location and data demand of ground users, the location of drones, and the locations of other drones; Among them, the image of one channel represents the status of the ground user, and the image size is L s ×W s , the grayscale has 256 levels; assuming the location coordinates of ground user i are (l i ,w i ), whose data requirement is q i Therefore, the coordinates of ground user i in the grayscale image (x i ,y i ) is expressed as: Where L and W represent the length and width of the data service area respectively. After determining the pixel position of the ground user in the image, the grayscale value of all pixels in the image is calculated. The size of the grayscale value represents the location information and data demand of the ground user. Using the two-dimensional Gaussian distribution and the data demand q i Determine (x i ,y i ) as the center, and the sub-pixel accuracy is guaranteed after the continuous mapping is discretized; at the same time, the distribution value φ of each pixel in the 3×3 image a It is expressed as: Where σ represents the variance of the two-dimensional Gaussian distribution, σ = 0.8; x a and a Represents the x-axis and y-axis coordinates of the pixels in the 3×3 image; therefore, the gray value gray a It is expressed as: in, Represents the normalized value of a two-dimensional Gaussian distribution; The other two channel images represent the coordinates of the drone’s own position and the coordinates of other drones’ positions respectively. The grayscale and coordinate representation methods are the same as above, and the grayscale values ​​of the 3×3 images of all drones are the same; Step 2.2, design the action space. The action of the drone consists of discrete actions and continuous actions. Discrete actions are to select action modes, including radar perception mode or data transmission mode; continuous actions are continuous parameters in all action modes. When the drone selects radar perception mode, since the radar power is fixed, the continuous action only changes on the x-axis and y-axis of the drone position coordinates; in addition, when the data transmission mode is selected, the continuous action also includes the transmission power allocated on each subcarrier when the drone selects data transmission. Step 2.3, the reward function is designed as: Among them, k Z , k AoI and k E They represent the cumulative data service volume coefficient, average peak information age and energy consumption penalty factor in time period t respectively.

4. The method for allocating drone resources based on multi-agent reinforcement learning according to claim 1, characterized in that: The specific method of step 3 includes: Step 3.1, design discrete actor networks and continuous actor networks; The discrete actor network includes two convolutional layers, a fully connected layer and a softmax output layer, which are used to select the action mode of the drone; the continuous actor network includes two convolutional layers, a fully connected layer and an output layer, which are used to output continuous parameters under all action modes; the inputs of the discrete actor network and the continuous actor network are the same, which are three-channel images of local observations of the drone; Step 3.2, design a multi-agent TD3 algorithm in a hybrid action space, namely the MAHTD3 algorithm, and use the MAHTD3 algorithm to train and update the network parameters of the drone based on the Markov decision model obtained in step 2; On the basis of the multi-agent TD3 algorithm, a hybrid action space design including discrete actions and continuous actions is added to obtain a multi-agent TD3 algorithm with a hybrid action space, namely the MAHTD3 algorithm; the MAHTD3 algorithm is within the framework of centralized training distributed execution, in which drones take actions based on their local observations in a distributed manner, and collect the experience of all drones for centralized training; therefore, the critic network of each drone needs to input the actions and local observations of all drones, while the discrete and continuous actor networks only need to input the drone's own local observations; the drones use the Markov decision model obtained in step 2, and each drone outputs the action to be taken based on its own local observations. The environment feeds back instant rewards to the drone based on the drone's actions, and the environmental state changes at the same time; the drone uses the MAHTD3 algorithm to train and update the parameters of its discrete and continuous actor networks and critic networks based on the above process, where the training process and parameter update method of the network parameters are the same as those of the multi-agent TD3 algorithm; Step 3.3, apply the drone trained by MAHTD3 algorithm to the data service scenario for communication resource allocation; For the hybrid action space design, each UAV is equipped with two actor networks: a discrete actor network and a continuous actor network. The discrete actor network is used to select the action mode, namely the radar perception mode or the data transmission mode, while the continuous actor network outputs the continuous action parameters of each action mode. After observing the current state of the environment, the UAV uses the discrete actor network to select the radar perception mode or the data transmission mode, and then obtains the corresponding continuous action parameters from the continuous actor network. When the UAV selects the radar perception mode, the UAV turns on the radar to perceive the position of the ground user and adjusts its own position at the same time. When the UAV selects the data transmission mode, the UAV transmits data to the ground user on each channel with the transmission power obtained by the continuous actor network and adjusts its own position.

5. A drone resource allocation system based on multi-agent reinforcement learning based on the method according to any one of claims 1 to 4, characterized in that: include: A multi-objective optimization problem establishment module is used to establish a multi-objective optimization problem for multi-drone resource allocation in a data service scenario for a multi-modal drone-assisted synaesthesia-integrated data service scenario, and to design drone data service evaluation indicators, which include: average peak information age, cumulative data service volume, and drone energy consumption; The Markov decision problem formulation module is used to formulate the multi-objective optimization problem of multi-UAV resource allocation in the established data service scenario as a Markov decision problem, and to design the state space, action space and corresponding immediate reward function respectively; The UAV communication resource allocation module is used to obtain the multi-agent TD3 algorithm in the mixed action space, namely the MAHTD3 algorithm, by designing discrete actor networks and continuous actor networks. According to the Markov decision model, the MAHTD3 algorithm is used to train and update the network parameters of the UAV, and the trained UAV is applied to the data service scenario for communication resource allocation.

6. A drone resource allocation device based on multi-agent reinforcement learning, characterized in that: include: Memory: a computer program storing a method for allocating drone resources based on multi-agent reinforcement learning as described in any one of claims 1 to 4, which is a computer-readable device; Processor: used to implement a drone resource allocation method based on multi-agent reinforcement learning as described in any one of claims 1-4 when executing the computer program.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a method for allocating drone resources based on multi-agent reinforcement learning as described in any one of claims 1 to 4.

Images