Throughput optimization method, device and unmanned aerial vehicle control system

Abstract

The present application relates to a throughput optimization method, device and drone control system, the method comprising: obtaining the signal to noise ratio of the target drone; the target drone includes a drone in U2D mode; according to the signal to noise ratio of the drone in U2D mode, obtaining the throughput of the drone in U2D mode; under preset constraints, constructing a random model that optimizes the total throughput of all drones in U2D mode according to the throughput; iteratively obtaining a drone operation strategy based on the random model; the drone operation strategy includes a strategy for resource allocation and trajectory planning of the target drone to achieve the optimal total throughput. The present application can construct a random model that optimizes the total throughput according to the throughput of the drone in U2D mode, and iteratively optimizes the drone operation strategy that can achieve the optimal total throughput based on the random model.

Description

Technical Field

[0001] The present application relates to the field of wireless communication technology, and in particular to a throughput optimization method, device and unmanned aerial vehicle control system. Background Art

[0002] UAV cellular networks can be divided into two transmission modes: Unmanned Aerial Vehicle to Network (U2N) and Unmanned Aerial Vehicle to Device (U2D), and a cellular network with coexisting U2N and U2D links can form a large and complex UAV-assisted communication system. However, UAVs have flexible flight characteristics and their coordinates are time-varying, which makes it difficult to optimize the system throughput in UAV cellular networks. Summary of the invention

[0003] Based on this, it is necessary to provide a throughput optimization method, device and drone control system that can optimize the throughput of the drone control system in response to the above technical problems.

[0004] In a first aspect, in one embodiment, the present application provides a throughput optimization method, the method comprising: obtaining a signal to interference and noise ratio of a target UAV; the target UAV comprises a UAV in U2D mode; according to the signal to interference and noise ratio of the UAV in U2D mode, obtaining the throughput of the UAV in U2D mode; under preset constraints, constructing a random model that optimizes the total throughput of all UAVs in U2D mode according to the throughput; based on the random model, iteratively obtaining a UAV operation strategy, and based on the UAV operation strategy, instructing the UAV in U2D mode to perform resource allocation and trajectory planning so that the total throughput is optimized.

[0005] In one of the embodiments, obtaining the signal to interference noise ratio of the target UAV includes: obtaining the effective transmission power and interference signal power of the target UAV based on the position status and resource allocation status of the target UAV; and obtaining the signal to interference noise ratio of the target UAV in a set time slot based on the effective transmission power and interference signal power of the target UAV.

[0006] In one of the embodiments, the target UAV also includes a UAV in U2N mode; the UAV in U2D mode reuses the communication spectrum of the UAV in U2N mode; based on the effective transmission power and interference signal power of the target UAV, the signal to noise ratio of the target UAV in the set time slot is obtained, including: based on the effective transmission power of the UAV in U2N mode when communicating on the set channel and the interference signal power of the UAV in U2D mode from the spectrum of the reused set channel, the signal to noise ratio of the UAV in U2N mode in the set time slot is obtained; based on the effective transmission power of the UAV in U2D mode when reusing the spectrum of the set channel and the interference signal power of the U2D mode, the signal to noise ratio of the UAV in U2D mode in the set time slot is obtained; wherein the interference signal power of the UAV in U2D mode includes the interference signal power from other U2D modes and the interference signal power from the U2N mode UAV on the corresponding channel of the reused spectrum.

[0007] In one of the embodiments, the resource allocation state includes the channel allocation state and the transmission power state of the target UAV; based on the position state and the resource allocation state of the target UAV, the effective transmission power and the interference signal power of the target UAV are obtained, including: according to the current position state, channel allocation state and transmission power state of the target UAV, the Los connection probability, Los path loss and NLos path loss of the channel transmission between the target UAV and the transmission object are obtained; based on the Los connection probability, Los path loss and NLos path loss, the average path loss of the target UAV is obtained; according to the average path loss, channel allocation state and transmission power state of the target UAV, the effective transmission power and the interference signal power of the target UAV are obtained.

[0008] In one of the embodiments, obtaining the throughput of the UAV in the U2D mode includes: obtaining the throughput according to a first association relationship based on a signal to interference noise ratio of the UAV in the U2D mode in a set time slot.

[0009] In one of the embodiments, a random model is constructed based on the throughput to optimize the total throughput of all drones in U2D mode, including: obtaining the total throughput of all drones in U2D mode based on the throughput of drones in U2D mode; and constructing a random model of the total throughput of all drones in U2D mode based on preset constraints.

[0010] In one of the embodiments, the total throughput of all drones in U2D mode is obtained based on the throughput of drones in U2D mode, including: based on the throughput, the channel allocation status when the drones in U2D mode reuse the communication spectrum of the drones in U2N mode, the transmission power status of the drones in U2D mode, and the flight speed of the drones in U2D mode within the set time slot, the total throughput of all drones in U2D mode is obtained.

[0011] In one of the embodiments, the preset constraints include at least one of the following conditions: a first constraint, which is used to constrain the target UAV to reuse the spectrum of at most one channel in one time slot; a second constraint, which is used to constrain the range of the target UAV's transmission power state; a third constraint, which is used to constrain the maximum limit of the target UAV's flight distance in one time slot; a fourth constraint, which is used to constrain the UAV in U2D mode to meet a preset minimum signal to interference and noise ratio; a fifth constraint, which is used to constrain the UAV in U2N mode to meet a preset minimum signal to interference and noise ratio.

[0012] In one embodiment, the random model includes a Markov model; under preset constraints, a random model is constructed based on the throughput to optimize the total throughput of all drones in U2D mode, including: treating all drones in U2D mode as independent intelligent agents, and modeling the Markov model based on optimization task requirements and environmental conditions; based on the preset constraints, a penalty for the action decision of the intelligent agent is obtained; based on the throughput and the penalty, a reward for the action decision of the intelligent agent is obtained according to a second association relationship, and the reward is added to the Markov model.

[0013] In one of the embodiments, based on the random model, the drone operation strategy is iteratively obtained, including: initializing the actor network, critic network and target network of each agent; based on the actor network, critic network and target network, selecting the optimal action strategy of the agent in the current state in the experience buffer of each agent; updating the experience buffer of the agent according to the optimal action strategy; after the experience buffers of all agents are updated, updating the actor network, critic network and target network of each agent; if all agents can no longer optimize and obtain rewards by changing their own action strategies, then determining that the action strategies of all agents at this time are the drone operation strategies that optimize the total throughput.

[0014] In one embodiment, the action strategy includes the channel allocation state, transmission power state and flight speed of the agent within a set time slot; the optimal action strategy for the agent in the current state is selected in the experience buffer of each agent, including: based on the local observation of the agent, according to the third association relationship, the optimal action strategy for the agent in the current state is selected in the experience buffer.

[0015] In one embodiment, the experience buffer of the agent is updated according to the optimal action strategy, including: after the optimal action strategy of all agents in the current state is selected, according to the Markov model, the reward of the agent in the current state is obtained, and the agent is transferred from the current state to the next state; based on the experience of the agent obtained and stored in the experience buffer, the experience buffer of all agents is updated; wherein the experience includes local observations of all agents in the current state, rewards of the agents in the current state, local observations of all agents in the next state, and action strategies of the agents in the current state.

[0016] In one embodiment, the actor network, critic network and target network of each agent are updated, including: updating the critic network by minimizing the loss function based on the updated experience buffer, and updating the actor network by policy gradient; after the actor network and critic network of all agents are updated, updating the target network of the agent.

[0017] In the second aspect, in one embodiment, the present application also provides a throughput optimization device, which includes: a signal-to-interference-and-noise ratio calculation module, used to obtain the signal-to-interference-and-noise ratio of a target UAV; the target UAV includes a UAV in U2D mode; a throughput calculation module, used to obtain the throughput of the UAV in U2D mode according to the signal-to-interference-and-noise ratio of the UAV in U2D mode; a random model construction module, used to construct a random model that optimizes the total throughput of all UAVs in U2D mode according to the throughput under preset constraints; an iterative optimization module, used to iteratively obtain a UAV operation strategy based on the random model, and instruct the UAV in U2D mode to perform resource allocation and trajectory planning based on the UAV operation strategy so that the total throughput is optimized.

[0018] In the third aspect, in one embodiment, the present application also provides a drone control system, the system comprising: a target drone, a terminal device, a base station and a computer device for executing the method as described in any one of the first aspects above; wherein the target drone includes a drone in U2D mode and a drone in U2N mode; the base station is connected to the terminal device and the drone in U2N mode respectively; the drone in U2D mode is connected to the terminal device and reuses the communication spectrum of the drone in U2N mode; the computer device is connected to the drone in U2N mode and the drone in U2D mode respectively.

[0019] The throughput optimization method, device and drone control system can obtain the signal-to-interference-to-noise ratio of the target drone, and obtain the throughput of the drone in U2D mode based on the signal-to-interference-to-noise ratio of the drone in U2D mode; under preset constraints, a random model is constructed based on the throughput to optimize the total throughput of all drones in U2D mode; based on the random model, the drone operation strategy is iteratively obtained. The present application can construct a random model that optimizes the total throughput based on the throughput of drones in U2D mode, and based on the random model, iteratively obtain a drone operation strategy that can achieve the optimal total throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 FIG. 4 is a flow chart of a throughput optimization method in an embodiment.

[0022] Figure 2 The first schematic diagram is a process for obtaining the signal to interference noise ratio of a target UAV in one embodiment.

[0023] Figure 3 The second schematic diagram is a process for obtaining the signal to interference noise ratio of a target UAV in one embodiment.

[0024] Figure 4 The figure is a schematic diagram of a process of obtaining effective transmission power and interference signal power in an embodiment.

[0025] Figure 5 The figure is a schematic diagram of a process of constructing a random model in one embodiment.

[0026] Figure 6 The figure is a flow chart of constructing a Markov model in one embodiment.

[0027] Figure 7 A schematic diagram of a process for iteratively obtaining a drone operation strategy in an embodiment.

[0028] Figure 8 A schematic diagram of a process for updating an agent's experience buffer in one embodiment.

[0029] Fig. 9 FIG. 4 is a structural block diagram of a throughput optimization device in an embodiment.

[0030] Fig.10 Schematic diagram of a drone control system in one embodiment.

[0031] Fig.11 The figure is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0033] With the development of drone technology, drones can now be deployed as a new type of infrastructure to provide more flexible wireless connections and expand the service coverage of ground base stations.

[0034] Compared with traditional wireless network technology, drone cellular networks have more advantages in dealing with complex environments. Drone cellular networks integrate a large number of IoT devices and can be used as one of the potential solutions to increase data transmission rates. In addition, drone cellular networks are committed to being applied in complex civilian and military scenarios, such as environmental monitoring, search and rescue, and other tasks. For drone cellular Internet, each drone can perform its specific tasks and transmit data to ground equipment for real-time monitoring, processing, and analysis.

[0035] UAV cellular networks can be divided into two transmission modes, namely, Unmanned Aerial Vehicle to Network (U2N) and Unmanned Aerial Vehicle to Device (U2D). In U2N mode, drones are similar to mobile devices in traditional cellular networks, and their data transmission needs to go through base stations. In U2D mode, sensor data can be directly transmitted to edge computing devices.

[0036] In complex application scenarios, drone cellular networks usually require the coordinated operation of U2N and U2D modes. In this case, a cellular network with coexisting U2N and U2D links will form a large and complex drone-assisted communication system. Complex application scenarios make the resource allocation plan for drones particularly critical.

[0037] The resource allocation and trajectory planning schemes for multi-UAV communication networks in traditional technologies usually adopt fixed strategies or simple centralized optimization methods, which are difficult to adapt to dynamic and complex environmental changes in practical applications, resulting in unstable system performance and waste of communication network resources. Since UAVs also have flexible flight characteristics, the coordinates of UAVs are time-varying, which further increases the difficulty of UAV resource allocation, trajectory design, and task allocation in UAV cellular networks. In addition, traditional technologies often use local optimization methods, which are prone to fall into local optimal solutions, making it difficult to maximize system performance.

[0038] In response to the above problems, the present application proposes a throughput optimization method, device and UAV control system. The present application can combine the properties of the UAV itself and the network environment to select appropriate resource allocation and trajectory planning strategies for the UAV, thereby achieving rational utilization of system resources, ensuring the overall transmission efficiency of the system, and optimizing the system throughput.

[0039] In an exemplary embodiment, Figure 1 As shown, a throughput optimization method is provided, and the method includes the following steps S102 to S108.

[0040] Step S102, obtaining the signal to interference noise ratio of the target UAV; the target UAV includes a UAV in U2D mode.

[0041] Among them, the signal to noise ratio is an important technical indicator for measuring the reliability of the communication quality of the target UAV. For example, the signal to noise ratio of the target UAV can be calculated by obtaining the position state and resource allocation state of the target UAV in the set time slot. It can be understood that the UAV in U2D mode establishes a communication connection with the terminal on the ground through the U2D link of the system, and the terminal may include an electronic device for the user to control the UAV, such as a UAV remote control, a tablet computer, etc.

[0042] Specifically, based on the position status and resource allocation status of the target UAV, the signal to interference and noise ratio of the target UAV in the set time slot can be obtained.

[0043] Step S104: obtaining the throughput of the UAV in the U2D mode according to the signal to interference noise ratio of the UAV in the U2D mode.

[0044] It can be understood that throughput can be the amount of data successfully transmitted by a drone or system in a unit of time (the amount is measured in bits, bytes, etc.), or it can represent the maximum data rate that a drone or system can receive and forward without frame loss. For example, the throughput of a drone in U2D mode can refer to the throughput between the drone in U2D mode and the device receiving information (the terminal device on the ground).

[0045] Specifically, based on the signal to interference and noise ratio of the drone in U2D mode, the throughput of the drone in U2D mode at a set time slot can be calculated.

[0046] Step S106: Under preset constraints, a random model is constructed according to the throughput to optimize the total throughput of all UAVs in the U2D mode.

[0047] The preset constraint condition may be preset based on the transmission power state of the target UAV, the flight distance of the target UAV in a time slot, etc. It is understandable that the random model is based on a random process, wherein the random model for optimizing the total throughput may include a random process such as a Markov process.

[0048] Specifically, based on the preset constraints, according to the throughput of each UAV in U2D mode, the total throughput of all UAVs in U2D mode can be obtained, and a random model for optimizing the total throughput can be constructed accordingly.

[0049] Step S108, based on the random model, iteratively obtain the UAV operation strategy, and based on the UAV operation strategy, instruct the UAV in the U2D mode to perform resource allocation and trajectory planning so that the total throughput is optimized.

[0050] Specifically, according to the obtained random model for optimizing the total throughput, after multiple rounds of iterative calculations, the drone operation strategy that optimizes the total throughput can be obtained.

[0051] In the above-mentioned throughput optimization method, the present application obtains the signal to noise ratio of the target drone, and obtains the throughput of each drone in U2D mode in the system according to the signal to noise ratio of the drone in U2D mode; then, based on the preset constraints, a random model is constructed according to the throughput of each drone in U2D mode to optimize the total throughput of all drones in U2D mode; based on the random model, after multiple rounds of iterative calculations, a drone operation strategy that optimizes the total throughput can be obtained.

[0052] In one embodiment, if Figure 2 As shown, step S102, obtaining the signal to interference noise ratio of the target UAV, also includes steps S202 to S204.

[0053] Step S202: based on the position status and resource allocation status of the target UAV, obtain the effective transmission power and interference signal power of the target UAV.

[0054] Wherein, illustratively, the position state of the target UAV may include the flight position coordinates of the target UAV in the current state. The resource allocation state may characterize the channel setting and power setting of the target UAV in the current state. The effective transmission power of the target UAV may be the effective transmission power between the target UAV and the corresponding connected ground equipment (such as a terminal, etc.). Optionally, the interference signal of the target UAV may include interference signals from other UAVs in the system, path loss between the target UAV and the transmission object, etc.

[0055] Specifically, according to the location status of the target UAV and its resource allocation status, the effective transmission power and interference signal power of the target UAV can be calculated.

[0056] Step S204: Based on the effective transmission power of the target UAV and the interference signal power, the signal to interference noise ratio of the target UAV in the set time slot is obtained.

[0057] Optionally, the signal to interference noise ratio of the target UAV in a set time slot can be calculated based on the effective transmission power of the target UAV and the interference signal power in combination with Gaussian white noise.

[0058] Specifically, the signal to interference noise ratio of the target UAV in the set time slot is obtained according to the obtained effective transmission power and interference signal power of the target UAV.

[0059] In one embodiment, the target drone also includes a drone in U2N mode, and the drone in U2D mode reuses the communication spectrum of the drone in U2N mode. Figure 3 As shown, step S204, based on the effective transmission power of the target UAV and the interference signal power, obtain the signal to interference noise ratio of the target UAV in the set time slot, and also includes steps S302 to S304.

[0060] Step S302, based on the effective transmission power of the drone in U2N mode when communicating on the set channel and the interference signal power of the drone in U2D mode from the spectrum of the multiplexed set channel, the signal to interference noise ratio of the drone in U2N mode on the set time slot is obtained.

[0061] Among them, the U2D mode drone reuses the communication spectrum of the U2N mode drone, which means that within the same frequency band, the drone to the network (i.e., U2N mode) and the drone to the device (i.e., U2D mode) can jointly use the radio resources of the frequency band under certain conditions. Optionally, the U2N mode may include a mode in which a communication connection is established between the drone and the base station. By sharing the spectrum with the U2D link and the U2N link, the spectrum utilization efficiency of the system can be effectively improved.

[0062] It should be understood that the U2N mode drone is one of the factors that affect the communication of the U2D mode drone in the environment, and it mainly provides a reused channel for the U2D mode drone. It can be understood that all U2D mode drones constitute a multi-drone communication system that reuses the spectrum, and the optimization of the system throughput in this application refers to optimizing the total throughput of all U2D mode drones.

[0063] Specifically, based on the effective transmission power of the drone in U2N mode when communicating on the set channel and the interference signal power of the drone in U2D mode from the spectrum of the multiplexed set channel, the signal-to-interference-noise ratio of the drone in U2N mode on the set time slot can be calculated according to the following formulas 1 to 2.

[0064] (Formula 1)

[0065] (Formula 2)

[0066] in, For drones in U2N mode The signal-to-interference-to-noise ratio in a given time slot t is: For drones in U2N mode In the The effective transmission power when communicating on a channel is For drones from multiplexing U2N mode No. The effective power of the interference signal of the U2D mode drone in the spectrum of the channel, is the average power of Gaussian white noise, is the set of channels, For drones in U2N mode In the The signal-to-interference-noise ratio on a channel.

[0067] Step S304, based on the effective transmission power when the drone in U2D mode reuses the spectrum of the set channel and the interference signal power of the drone in U2D mode, the signal to interference noise ratio of the drone in U2D mode in the set time slot is obtained.

[0068] Among them, the interference signal power of the drone in U2D mode includes the interference signal power from other drones in U2D mode and the interference signal power from the drone in U2N mode on the corresponding channel of the multiplexed spectrum.

[0069] Specifically, according to the effective transmission power when the drone in U2D mode reuses the spectrum of the set channel and the interference signal power of the drone in U2D mode, the signal to interference noise ratio of the drone in U2D mode at the set time slot can be calculated according to the following formulas 3 to 4.

[0070] (Formula 3)

[0071] (Formula 4)

[0072] in, represents the average power of Gaussian white noise, For drones in U2D mode In setting time slot The signal-to-interference-noise ratio in For drones in U2D mode Reuse The effective transmission power when the spectrum of the channel is is the effective power of the interference signal from other UAVs in U2D mode, is the effective power of the interference signal from the U2N mode drone on the corresponding channel of the reused spectrum, For drones in U2D mode exist The signal-to-interference-to-noise ratio on the channel is is a collection of channels.

[0073] In one embodiment, the resource allocation state includes the channel allocation state and the transmission power state of the target drone. Figure 4 As shown, step 202, based on the position status and resource allocation status of the target UAV, obtains the effective transmission power and interference signal power of the target UAV, and also includes steps S402 to S406.

[0074] Step S402, according to the current position state, channel allocation state and transmission power state of the target UAV, obtain the Los connection probability, Los path loss and NLos path loss of the channel transmission between the target UAV and the transmission object.

[0075] Among them, exemplarily, when the target UAV is a UAV in U2D mode, the transmission object can be a ground device (such as a terminal, etc.) that establishes a communication connection with the target UAV; when the target UAV is a U2N mode, the transmission object can be a base station that establishes a communication connection with the target UAV.

[0076] It can be understood that LoS (Line-of-Sight) connection means that in wireless communication, a communication connection exists between two communication nodes and can be established through a direct line-of-sight path; NLoS (Non-Line-of-Sight) connection means that there is no direct line-of-sight path between the transmitter and the receiver of the communication signal, and the signal transmitted by the transmitter needs to reach the receiver through reflection, diffraction or scattering.

[0077] Optionally, the transmission power state of the target UAV may be a ratio of the transmission power of the current target UAV to the maximum transmittable power. For example, taking the calculation of the power state of the UAV in the U2D mode as an example, the transmission power state of the UAV in the U2D mode may be calculated by the following formula 5.

[0078] (Formula 5)

[0079] in, is the transmission power of the drone in the current U2D mode, is the maximum transmittable power of the drone in U2D mode, For drones in U2D mode The transmission power state.

[0080] Exemplarily, the channel allocation status of the target UAV includes the channel allocation status of the UAV in U2D mode and the channel allocation status of the UAV in U2N mode.

[0081] Optionally, when a drone in U2D mode reuses the communication spectrum of a drone in U2N mode, the channel allocation state of the drone in U2D mode can be expressed as the following formula 6.

[0082] (Formula 6)

[0083] Among them, illustratively, Indicates that the time slot is set , No. UAV reuse The first For example, each UAV in U2D mode can reuse at most one channel in each time slot, which is expressed as ,in is a collection of channels.

[0084] Optionally, the channel allocation state of the drone in U2N mode can be expressed as follows: Formula 7.

[0085] (Formula 7)

[0086] in, Indicates that the time slot is set , No. The spectrum of the kth channel used by the U2N mode drones. For example, each U2N mode drone can use at most one channel in each time slot, expressed as ,in is the set of channels. It can be understood that the constraint that the target UAV can use at most one channel in each time slot can fit the actual operation of the system and ensure the rationality of the experimental scenario.

[0087] Exemplarily, according to the following formulas 8 to 11, the Loss connection probability of the channel transmission between the target UAV and the transmission object can be obtained.

[0088] (Formula 8)

[0089] (Formula 9)

[0090] (Formula 10)

[0091] (Formula 11)

[0092] in, is the index identifier of the target drone, is the index identifier of the transfer object. Designing symbols for distinctions that have no physical meaning, Target UAV The received power (usually expressed in decibel milliwatts, which can be used to estimate the impact of different heights on the signal), Target UAV and transfer objects The distance of the device on the plane, Target UAV and transfer objects The distance between the devices in space (including the height), It is the abbreviation symbol of Unmanned Aerial Vehicle (UAV). It is the abbreviation of the device of the transmission object. To set the time slot On the drone In the spatial coordinate system The coordinates in the direction, Transmission object for drones The equipment is The coordinates in the direction, To set the time slot On target drone exist The coordinates in the direction, Transmission object for drones The y coordinate of the device, Target UAV flight altitude.

[0093] Furthermore, according to the following formula 12, the NLoS connection probability of the channel transmission between the target UAV and the transmission object can be obtained.

[0094] (Formula 12)

[0095] in, Target UAV and transfer objects The NLoS connection probability of the channel transmission between Target UAV and transfer objects The LoS connection probability of the channel transmission between them.

[0096] Exemplarily, according to the following formulas 13 to 14, the Loss path loss and the NLos path loss of the channel transmission between the target UAV and the transmission object can be obtained.

[0097] (Formula 13)

[0098] (Formula 14)

[0099] in, Target UAV With the transfer object The LoS path loss of the channel transmission between Target UAV With the transfer object The NLoS path loss of the channel transmission between Target UAV With the transfer object The distance between devices, is the carrier frequency of the communication signal (in gigahertz), Target UAV flight altitude.

[0100] Step S404, based on the Los connection probability, Los path loss and NLos path loss, the average path loss of the target UAV is obtained.

[0101] Specifically, the average path loss of the target UAV when the target UAV communicates with the transmission object can be calculated according to the following formula 15.

[0102] (Formula 15)

[0103] in, Target UAV With the transfer object The average path loss of the channel transmission between Target UAV and transfer objects The LoS connection probability of the channel transmission is Target UAV and transfer objects The LoS path loss of the channel transmission between Target UAV and transfer objects The NLoS path loss of the channel transmission between Target UAV and transfer objects The NLoS connection probability of the channel transmission between them.

[0104] Step S406, obtaining the effective transmission power and interference signal power of the target UAV according to the average path loss, channel allocation status and transmission power status of the target UAV.

[0105] Specifically, when the target UAV is a UAV in U2D mode, the effective transmission power when the UAV in U2D mode communicates with the transmission object can be calculated according to the following formula 16.

[0106] (Formula 16)

[0107] in, For drones in U2D mode Reuse The effective transmission power when the spectrum of the channel is To set the time slot Drone in U2D mode Reuse The spectrum of the channels, To set the time slot Drone in U2D mode The current transmission power, For drones in U2D mode The average path loss between the corresponding ground equipment (such as terminals, etc.).

[0108] Exemplarily, when the target UAV is a UAV in U2N mode, the effective transmission power when the UAV in U2N mode communicates with the transmission object can be calculated according to the following formula 17.

[0109] (Formula 17)

[0110] Among them, the drone in U2N mode is Communicate on channels, For drones in U2N mode In the The effective transmission power when communicating on a channel is To set the time slot Drone in U2N mode The used The spectrum of the channels, For drones in U2N mode The average path loss when communicating with the base station, For drones in U2N mode The current transmission power.

[0111] Furthermore, it can be understood that the interference signal received by the drone in U2D mode may come from other drones in U2D mode and drones in U2N mode on the corresponding channel of the multiplexed spectrum.

[0112] Exemplarily, when the target UAV is a UAV in U2D mode, the interference signal power of the interference signal received by the UAV in U2D mode can be calculated according to the following formulas 18 to 19.

[0113] (Formula 18)

[0114] (Formula 19)

[0115] in, is the effective power of the interference signal from other U2D mode drones, is the effective power of the interference signal from the U2N mode drone on the corresponding channel of the reused spectrum, To set the time slot Drone in U2D mode Reuse The spectrum of the channels, To set the time slot Drone in U2D mode The current transmission power, For drones With drones The average path loss of the channel transmission between A collection of drones in U2D mode. The index of the current drone in U2D mode. A collection of drones in U2N mode.

[0116] Further, it can be understood that the interference signal received by the drone in U2N mode may come from the drone in U2D mode that reuses its channel spectrum.

[0117] Exemplarily, when the target UAV is a UAV in U2N mode, the interference signal power of the interference signal received by the UAV in U2N mode can be calculated according to the following formula 20.

[0118] (Formula 20)

[0119] in, For drones from multiplexing U2N mode The effective power of the interference signal of the UAV in the U2D mode of the channel spectrum, In setting time slot Drone in U2D mode Reuse The spectrum of the channels, To set the time slot Drone in U2D mode The current transmission power, For drones in U2N mode The average path loss when communicating with the base station, A collection of drones in U2D mode.

[0120] In one of the embodiments, step 104, obtaining the throughput of the UAV in the U2D mode, includes: obtaining the throughput according to the first association relationship based on the signal to interference noise ratio of the UAV in the U2D mode in the set time slot.

[0121] Exemplarily, the first association relationship can be expressed as the following formula 21.

[0122] (Formula 21)

[0123] in, For drones in U2D mode In setting time slot The throughput in For drones in U2D mode In setting time slot The signal-to-interference-noise ratio in is the channel bandwidth.

[0124] In one embodiment, if Figure 5 Step 106, constructing a random model to optimize the total throughput of all UAVs in U2D mode according to the throughput, also includes steps S502 to S504.

[0125] Step S502: According to the throughput of the UAVs in the U2D mode, the total throughput of all UAVs in the U2D mode is obtained.

[0126] Specifically, the throughput of each U2D mode drone in the set time slot can be obtained through the signal-to-interference-noise ratio of the U2D mode drone in the set time slot; then, the total throughput of all U2D mode drones in the system can be obtained based on the channel allocation status, transmission power status, flight status, etc. of the U2D mode drones in the system.

[0127] Step S504: construct a random model of the total throughput of all UAVs in U2D mode according to preset constraints.

[0128] Among them, exemplarily, the preset constraints can be used to fit the actual application scenarios of the system through constraints, and to ensure that the signal transmission quality of the drone in U2D mode and the drone in U2N mode meets the basic signal-to-interference-noise ratio threshold condition.

[0129] Specifically, according to the total throughput of all UAVs in U2D mode in the system, combined with preset constraints, a random model that optimizes the total throughput can be constructed.

[0130] In one of the embodiments, step S502, based on the throughput of the drones in U2D mode, obtains the total throughput of all drones in U2D mode, and also includes the following steps: based on the throughput, the channel allocation status when the drones in U2D mode reuse the communication spectrum of the drones in U2N mode, the transmission power status of the drones in U2D mode, and the flight speed of the drones in U2D mode within the set time slot, obtains the total throughput of all drones in U2D mode.

[0131] Specifically, the following formula 22 can be used to calculate the value from 0 to The total throughput of all U2D mode drones during the time period.

[0132] (Formula 22)

[0133] in, It is the index of the drone in U2D mode. For drones in U2D mode In setting time slot The throughput within For drones in U2D mode Channel allocation status when reusing the UAV communication spectrum in U2N mode; For drones in U2D mode The transmission power state of For drones in U2D mode In setting time slot The flight speed within The number of elements in the collection of drones in U2D mode.

[0134] In one of the embodiments, the preset constraints include at least one of the following conditions: a first constraint, which is used to constrain the target UAV to reuse the spectrum of at most one channel in one time slot; a second constraint, which is used to constrain the range of the target UAV's transmission power state; a third constraint, which is used to constrain the maximum limit of the target UAV's flight distance in one time slot; a fourth constraint, which is used to constrain the UAV in U2D mode to meet a preset minimum signal to interference and noise ratio; a fifth constraint, which is used to constrain the UAV in U2N mode to meet a preset minimum signal to interference and noise ratio.

[0135] Among them, it can be understood that for drones in U2D mode , in the set time slot The flight speed within the set time slot can be expressed as Drone inside Direction and For example, the component of the flight distance in the U2D mode can be obtained according to the following formula 23: In setting time slot The flight speed within.

[0136] (Formula 23)

[0137] in, For drones in U2D mode In setting time slot The flight speed within Set time slots for drones Internal The component of the flight distance in the direction, To set the time slot Internal The component of the flight distance in direction.

[0138] Optionally, the first constraint condition can be expressed by the following formula 24.

[0139] (Formula 24)

[0140] in, Indicates that the time slot is set Drone in U2D mode Reuse The spectrum of the channels, is a collection of channels.

[0141] Optionally, the second constraint condition can be expressed by the following formula 25.

[0142] (Formula 25)

[0143] in, For drones in U2D mode In setting time slot The transmit power state within.

[0144] Optionally, the third constraint condition can be expressed by the following formula 26.

[0145] (Formula 26)

[0146] in, For drones Direction and Maximum flight speed in direction, Set time slots for drones Internal The component of the flight distance in the direction, To set the time slot Internal The component of the flight distance in direction.

[0147] Optionally, the fourth constraint condition can be expressed by the following formula 27.

[0148] (Formula 27)

[0149] in, For drones in U2D mode In setting time slot The signal-to-interference-noise ratio in It is the minimum signal to interference and noise ratio of UAV communication quality in U2D mode.

[0150] Optionally, the fifth constraint condition can be expressed by the following formula 28.

[0151] (Formula 28)

[0152] in, For drones in U2N mode In setting time slot The signal-to-interference-noise ratio in is the minimum signal-to-interference-noise ratio of drone communication quality in U2N mode.

[0153] Specifically, exemplarily, based on the above preset constraints, the optimization process of the total throughput can be expressed by the following formula 29.

[0154] (Formula 29)

[0155] in, represents the total throughput based on the variable ( ) changes to the maximum value, to Represents the preset constraints of the optimization process, the preset constraints include the first to fifth constraints described in the above embodiment, and the meanings of other symbols are the same as in the above embodiment and will not be repeated here.

[0156] In one embodiment, the stochastic model comprises a Markov model. Figure 6 As shown, step S106, under preset constraints, constructs a random model based on throughput to optimize the total throughput of all drones in U2D mode, and also includes steps S602 to S606.

[0157] In step S602, all UAVs in the U2D mode are treated as independent intelligent entities, and a Markov model is modeled based on the optimization task requirements and environmental conditions.

[0158] It is understood that the Markov model includes a Markov process. Exemplarily, the optimization task requirement may include optimizing the task requirement of the total throughput by autonomously selecting a communication channel, transmission power, and flight trajectory during flight by the target UAV; the environmental conditions include the number of target UAVs, the location of the base station, the location of the device of the transmission object, and the like.

[0159] Specifically, the optimization task requirements and environmental conditions can be selected according to the actual application requirements of the system; based on the optimization task requirements and environmental conditions, a corresponding Markov model can be constructed to perform multi-agent reinforcement learning.

[0160] Step S604, based on the preset constraints, obtain the penalty for the action decision of the agent.

[0161] Among them, it can be understood that in the framework of Markov process, reward is a measure of the immediate gain or loss obtained by the agent after taking an action in a specific state. Reward can guide the agent to learn the optimal strategy to take the best action in different states to maximize the cumulative reward. Penalty can be regarded as a negative reward, which can be used to guide the agent to avoid certain unfavorable behaviors or states. In the Markov process, penalty can be included in the reward function as a negative reward, that is, if the agent enters an undesirable state or takes an undesirable action, a negative reward value can be given to the agent so that it can gradually learn to avoid unfavorable situations during the learning process of the agent.

[0162] Specifically, the penalties for the agent when making action decisions can be constructed based on preset constraints.

[0163] Step S606, based on the throughput and the penalty, according to the second association relationship, a reward for the action decision of the agent is obtained, and the reward is added to the Markov model.

[0164] It should be understood that, when each UAV in the U2D mode is regarded as an independent intelligent agent, the resource allocation and trajectory planning problem of the entire system's UAVs can be modeled as a Markov process involving M intelligent agents. For example, for intelligent agent m, it sets the time slot The local observation within can be expressed by the following formulas 30 to 32.

[0165] (Formula 30)

[0166] (Formula 31)

[0167] (Formula 32)

[0168] in, For drones in U2D mode In setting time slot Local observation within is the position of the drone in U2D mode and , For drones A collection of relative distances to other drones in U2D mode, To set the time slot Drone inside The signal-to-interference-noise ratio, To set the time slot Drone inside The relative distance to the target communication device (such as a terminal, etc.) with which it is connected to the corresponding communication. It is the collection of all U2D mode drones in the system. In addition to the current drone A collection of other drones, For drones The coordinates of The distance component in the direction, For drones Corresponding target communication device exist The distance component in the direction, For drones The coordinates of The distance component in the direction, For drones Corresponding target communication device exist The distance component in the direction, is the mathematical identifier of the drone, For drones The mathematical identifier of the corresponding target communication device.

[0169] Optionally, the signal to noise ratio of the drone in a set time slot can be converted into a Boolean variable according to the following formula 33 to facilitate better observation.

[0170] (Formula 33)

[0171] in, To set the time slot Internal drone A Boolean variable indicating whether the signal to noise ratio meets the minimum limit. For drones In setting time slot The signal-to-interference-noise ratio in It is the minimum signal-to-interference-noise ratio to ensure the communication quality of UAV in U2D mode.

[0172] Further, based on the local observation of the agent in the set time slot, the observation of all agents can be obtained. Exemplarily, the observation of all agents in the set time slot can be obtained according to the following formula 34.

[0173] (Formula 34)

[0174] in, It is the status set of all drones in U2D mode in the whole system. For intelligent agents In setting time slot Local observation within It is the collection of all U2D mode drones in the drone system.

[0175] Furthermore, the action strategy selected by the agent within the set time slot can be expressed by the following formula 35.

[0176] (Formula 35)

[0177] in, Representing an Agent The action strategy selected, For intelligent agents The channel allocation status, For intelligent agents The transmission power state, For intelligent agents For example, the action space of the action strategy selected by the agent in the set time slot is .

[0178] Exemplarily, the penalty for the action decision of the agent may include a first signal to interference and noise ratio penalty and a second signal to interference and noise ratio penalty; wherein the first signal to interference and noise ratio penalty may represent the penalty caused by the signal to interference and noise ratio of the agent not reaching the minimum limit; and the second signal to interference and noise ratio penalty represents the sum of penalties received by the agent due to changes in the communication quality of the drone in the U2N mode.

[0179] Optionally, based on the fourth constraint condition, a first signal to interference plus noise ratio penalty is obtained. Exemplarily, the first signal to interference plus noise ratio penalty may be obtained based on the fourth constraint condition according to the following formula 36.

[0180] (Formula 36)

[0181] in, For drones in U2D mode In setting time slot The signal-to-interference-noise ratio in The minimum signal-to-interference-to-noise ratio that can ensure the communication quality of the drone in U2D mode. To indicate that the time slot is set Because of drones The penalty is incurred when the signal to interference and noise ratio does not reach the minimum limit.

[0182] Optionally, based on the fifth constraint, a second signal to noise ratio penalty is obtained. It can be understood that there is no information about the drones that reuse the spectrum in the local observation of the drone in the U2D mode, and the penalty related to the fifth constraint can be assumed to be shared by all agents in the system. Exemplarily, the second signal to noise ratio penalty can be obtained based on the fifth constraint according to the following formulas 37 to 38.

[0183] (Formula 37)

[0184] (Formula 38)

[0185] in, For drones in U2N mode In setting time slot The signal-to-interference-noise ratio in To ensure the minimum signal-to-interference-to-noise ratio for communication quality of drones in U2N mode, To set the time slot Because of drones The penalty caused by the signal to noise ratio not reaching the minimum limit is For intelligent agents The sum of the penalties received due to changes in the communication quality of the drone in U2N mode is the second SINR penalty.

[0186] Furthermore, based on the first signal to interference and noise ratio penalty and the second signal to interference and noise ratio penalty, a reward for the action decision of the agent can be obtained according to the second association relationship.

[0187] Optionally, the second association relationship is a reward function of the agent, which can be expressed as the following formula 39.

[0188] (Formula 39)

[0189] in, For drones in U2D mode In setting time slot The throughput within and is the weight coefficient of the first SINR penalty and the second SINR penalty, A drone (agent) in U2D mode Rewards, To indicate that the time slot is set Internal reason is drone The penalty caused by the signal to noise ratio not reaching the minimum limit (i.e. the first signal to noise ratio penalty), For intelligent agents The sum of the penalties received due to changes in the communication quality of the drone in U2N mode (i.e., the second SINR penalty).

[0190] It can be understood that, unlike the centralized optimization method in traditional technology, this application can enable the drone to autonomously select the optimal resource allocation and trajectory planning strategy according to its own mission requirements and environmental conditions through the distributed optimization mechanism of multi-agent reinforcement learning. Based on the adaptive optimization of distributed agents, this application can better cope with complex and changing communication environments, improve the flexibility and stability of drone communication networks, and solve the problems of high computational complexity and poor adaptability of traditional technologies in dynamic environments.

[0191] In one embodiment, if Figure 7 Step S108, iteratively obtains the UAV operation strategy based on the random model, and also includes steps S702 to S710.

[0192] Step S702, initialize the actor network, critic network and target network of each agent.

[0193] Specifically, for example, in the initialization phase, each agent initializes its own actor network for outputting actions , a critic network used to evaluate the quality of the current action strategy and a target network to stabilize the training process and ,in , , and are the network parameters related to strategy update during training. It can be understood that is the network parameter in the current actor network, by updating it to , update the current actor network to become the new actor network; is the network parameter in the current critic network, so that it is updated to , so that the current critic network is updated to a new critic network.

[0194] Step S704, based on the actor network, the critic network and the target network, select the optimal action strategy of the agent in the current state in the experience buffer of each agent.

[0195] Among them, the experience buffer can be data for storing the state transition related vectors of the agent. It can be understood that in actual scenarios, the experience buffer is generally stored in a centralized control node or server, or can be distributed and stored in a distributed system and shared in real time. Exemplarily, according to the distributed execution of the centralized training algorithm, all agents can share the same experience buffer. Exemplarily, the optimal action strategy can be used to select the action strategy through the current actor network at the current moment to maximize the expected reward of the agent in the future.

[0196] Specifically, based on the constructed actor network, critic network and target network, the optimal action strategy of the agent in the current state can be selected in the experience buffer.

[0197] Step S706, updating the agent's experience buffer according to the optimal action strategy.

[0198] Specifically, based on the action strategy chosen by each agent, the agent can obtain the corresponding experience in the new state; the corresponding experience obtained by the agent is stored in the experience buffer to update the experience buffer.

[0199] Step S708, after the experience buffers of all agents are updated, the actor network, critic network and target network of each agent are updated.

[0200] Specifically, based on the updated experience buffer, the actor network, critic network, and target network of each agent are also iteratively updated accordingly, thereby updating the action strategy of the agent.

[0201] Step S710: If all agents can no longer optimize and obtain rewards by changing their own action strategies, then the action strategies of all agents at this time are determined to be the drone operation strategies that optimize the total throughput.

[0202] Specifically, after multiple rounds of iterative updates of the agent's experience buffer, reinforcement learning network, etc., when all agents can no longer optimize the rewards they receive by changing their own action strategies, it means that the resource allocation and trajectory planning strategies of all agents have reached a convergence state, that is, the system's final optimal resource allocation and trajectory planning scheme is obtained.

[0203] It is understandable that the drone system optimization method of traditional technology is prone to fall into the local optimal solution, which makes it difficult to maximize the system performance. Compared with traditional technology, this application uses multiple rounds of iterative updates to enable the system to continuously adjust its resource allocation and trajectory planning scheme until the drone's action strategy converges and finally obtains the optimal solution for the global strategy. The iterative optimization method proposed in this application can effectively avoid the problem of easy entry into the local optimal solution in traditional technology, thereby improving the robustness and stability of system performance. In addition, after maximizing the total throughput, this application can reduce the system's signal interference and data packet loss, and significantly improve the communication quality.

[0204] In one embodiment, the action strategy includes the channel allocation state, transmission power state and flight speed of the agent in the set time slot. Step S704, based on the actor network, the critic network and the target network, selects the optimal action strategy of the agent in the current state in the experience buffer of each agent, including: based on the local observation of the agent, according to the third association relationship, selects the optimal action strategy of the agent in the current state in the experience buffer.

[0205] Specifically, at the beginning of a new time slot, each agent can obtain its own local observations; based on the local observations of the agent and combined with the third association, the optimal action strategy for agent resource allocation and trajectory planning in the current state is selected in the experience buffer.

[0206] Optionally, the third association relationship can be expressed by the following formula 40.

[0207] (Formula 40)

[0208] in, For intelligent agents In setting time slot Local observation within For intelligent agents The optimal action strategy in the current state, is the noise generated by exploring new strategies, The actor network obtains the process function of the new action through local observation. Are the network parameters in the current actor network.

[0209] In one embodiment, Figure 8 Step S706, updating the agent's experience buffer according to the optimal action strategy, includes steps S802 to S804.

[0210] Step S802, after all agents have selected the optimal action strategy in the current state, the reward of the agent in the current state is obtained according to the Markov model, and the agent is transferred from the current state to the next state.

[0211] Specifically, after the action strategies of all agents are selected, the reward of each agent can be calculated according to the Markov process, and each agent can be transferred from the current state to the next new state.

[0212] Step S804, based on the experience of the agent obtained and stored in the experience buffer, update the experience buffer of all agents; wherein the experience includes the local observation of all agents in the current state, the reward of the agent in the current state, the local observation of all agents in the next state, and the action strategy of the agent in the current state.

[0213] Specifically, the corresponding experience of the agent iterative update can be Stored in the agent's experience buffer ;in, is the observation of all agents in the current state, The current state in the time slot Inner Agent Rewards received, is the observation of all agents in the next state, for Moment Agent action strategy.

[0214] In one embodiment, step S708, after the experience buffers of all agents are updated, the actor network, critic network and target network of each agent are updated, including steps S902 to S904.

[0215] Step S902: Based on the updated experience buffer, the critic network is updated by minimizing the loss function, and the actor network is updated by the policy gradient.

[0216] Specifically, the critic network can be updated based on the experience extracted from the experience buffer in combination with minimizing the loss function, and the actor network can be updated in combination with the policy gradient.

[0217] Exemplarily, based on the updated experience buffer, the critic network can be updated by minimizing the loss function according to the following formula 41.

[0218] (Formula 41)

[0219] in, Refers to the agent The parameters of the neural network, is the amount of experience drawn from the experience buffer, To use Substitute the experience into the new critic network to update the reward expectation; is the first value calculated using the current critic network Experience reward, is the first The state of experience, is the first Action strategy in one experience.

[0220] Exemplarily, based on the updated experience buffer, the critic network can be updated by policy gradient according to the following formula 42.

[0221] (Formula 42)

[0222] in, is the amount of experience drawn from the experience buffer; ( ) is the gradient of the action with respect to the actor network parameters, which can explain how to influence the agent's choice by adjusting the parameters; It is the gradient of the Q value of the critic network relative to the action, which can reflect the impact of changing the action on the expected return in a specific state; The Q value output by the critic network indicates that Next agent takes action expected return; is the first The state of experience, is the first The action strategy in the experience, is the first value extracted from the experience buffer. Local observations in an experience.

[0223] Step S904, after the actor networks and critic networks of all agents are updated, the target network of the agent is updated.

[0224] Specifically, after the actor networks and critic networks of all agents are updated, the target network of each agent is updated.

[0225] Exemplarily, the target network may be updated according to the following formula 43.

[0226] (Formula 43)

[0227] in, are the network parameters of the updated target network, is a soft update parameter (usually a small value), are the network parameters of the target network before updating, and the left arrow represents the assignment operator.

[0228] This application introduces multi-agent reinforcement learning to achieve optimal adjustment of the dynamic state update, channel and power configuration of the drone, thereby optimizing the signal-to-interference-noise ratio of drone communications and optimizing the total throughput of drones in the U2D mode of the system, effectively improving the utilization rate and communication efficiency of limited communication resources and reducing the operating cost of the communication network.

[0229] Different from the centralized optimization method in traditional technology, this application uses the distributed optimization mechanism of multi-agent reinforcement learning to enable drones to autonomously select the optimal resource allocation and trajectory planning strategy based on their own mission requirements and environmental conditions. Based on the adaptive optimization of distributed agents, this application can better cope with complex and changing communication environments, improve the flexibility and stability of drone communication networks, and solve the problems of high computational complexity and poor adaptability of traditional technologies in dynamic environments.

[0230] The present application can also make the drones in the system continuously adjust their resource allocation and trajectory planning schemes through multiple rounds of iterative updates until the action strategy converges, and finally obtain the optimal solution for the global action strategy. Through the iterative optimization method of the present application, the problem of easily entering the local optimum in traditional technologies can be effectively avoided, and the robustness and stability of system performance can be effectively improved.

[0231] The present application can also be applied to including but not limited to the following application scenarios.

[0232] (1) Disaster relief and emergency communications: When natural disasters or emergencies (such as earthquakes, floods, or fires) occur, traditional communication infrastructure may be damaged or fail. The multi-UAV assisted communication network based on this application can be quickly deployed when natural disasters or emergencies occur to provide emergency communication services as a temporary communication platform. By optimizing the resource allocation and trajectory planning of drones, this application can improve the throughput and service quality of the communication network in a short period of time, thereby ensuring the stability and efficiency of the communication network in emergency situations.

[0233] (2) Smart cities and smart transportation: With the development of smart city construction, drones are widely used in smart transportation management, urban environment monitoring, logistics and distribution, etc. The throughput optimization method proposed in this application can achieve the optimal allocation of communication network resources in a dynamic and complex urban environment, thereby reducing communication delays, improving the collaboration efficiency of drones in the city, and facilitating the diversified applications of smart cities.

[0234] (3) Military and defense fields: In military reconnaissance, border patrols, battlefield communications and other scenarios, drone clusters usually need to work together to cope with complex and changing military environments. The optimization scheme of this application can not only improve the robustness and total throughput of the communication network, but also ensure the rapid and accurate transmission of information between drones, thereby improving the success rate and safety of military mission execution.

[0235] (4) Communication signal coverage in special scenarios: In remote areas or other special scenarios (such as ocean monitoring, aerial photography, etc.), traditional communication infrastructure is difficult to quickly cover reliable wireless communication signals. This application can effectively optimize the transmission data path of drones, thereby improving the coverage and data throughput of drone communication networks, and can provide communication networks in remote areas or other special scenarios with wider coverage, more stability and more reliability.

[0236] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0237] Based on the same inventive concept, the embodiment of the present application also provides a throughput optimization device for implementing the throughput optimization method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more throughput optimization device embodiments provided below can refer to the limitations on the throughput optimization method above, and will not be repeated here.

[0238] In one embodiment, Fig. 9As shown, the present application also provides a throughput optimization device 900, which includes: a signal interference and noise ratio calculation module 902, which is used to obtain the signal interference and noise ratio of the target drone; the target drone includes a drone in U2D mode; a throughput calculation module 904, which is used to obtain the throughput of the drone in U2D mode according to the signal interference and noise ratio of the drone in U2D mode; a random model construction module 906, which is used to construct a random model that optimizes the total throughput of all drones in U2D mode according to the throughput under preset constraints; an iterative optimization module 908, which is used to iteratively obtain the drone operation strategy based on the random model, and instruct the drone in U2D mode to perform resource allocation and trajectory planning based on the drone operation strategy so that the total throughput is optimized.

[0239] In one of the embodiments, the signal to interference and noise ratio calculation module 902 is further used to: obtain the effective transmission power and interference signal power of the target UAV based on the position status and resource allocation status of the target UAV; and obtain the signal to interference and noise ratio of the target UAV in a set time slot based on the effective transmission power and interference signal power of the target UAV.

[0240] In one of the embodiments, the target drone also includes a drone in U2N mode; the drone in U2D mode reuses the communication spectrum of the drone in U2N mode; the signal to interference and noise ratio calculation module 902 is also used to: obtain the signal to interference and noise ratio of the drone in U2N mode in the set time slot based on the effective transmission power of the drone in U2N mode when communicating on the set channel and the interference signal power of the drone in U2D mode from the spectrum of the reused set channel; obtain the signal to interference and noise ratio of the drone in U2D mode in the set time slot based on the effective transmission power of the drone in U2D mode when reusing the spectrum of the set channel and the interference signal power of the drone in U2D mode.

[0241] Among them, the interference signal power of the drone in U2D mode includes the interference signal power from other drones in U2D mode and the interference signal power from the drone in U2N mode on the corresponding channel of the multiplexed spectrum.

[0242] In one of the embodiments, the resource allocation state includes the channel allocation state and the transmission power state of the target UAV; the signal-to-interference-noise ratio calculation module 902 is also used to: obtain the Los connection probability, Los path loss and NLos path loss of the channel transmission between the target UAV and the transmission object according to the current position state, channel allocation state and transmission power state of the target UAV; obtain the average path loss of the target UAV based on the Los connection probability, Los path loss and NLos path loss; obtain the effective transmission power and interference signal power of the target UAV according to the average path loss, channel allocation state and transmission power state of the target UAV.

[0243] In one of the embodiments, the throughput calculation module 904 is further used to: obtain the throughput according to the first association relationship based on the signal to interference noise ratio of the drone in the U2D mode in the set time slot.

[0244] In one embodiment, the random model construction module 906 is further used to: obtain the total throughput of all drones in U2D mode based on the throughput of drones in U2D mode; and construct a random model of the total throughput of all drones in U2D mode based on preset constraints.

[0245] In one of the embodiments, the random model building module 906 is also used to obtain the total throughput of all drones in U2D mode based on the throughput, the channel allocation status when the drone in U2D mode reuses the communication spectrum of the drone in U2N mode, the transmission power status of the drone in U2D mode, and the flight speed of the drone in U2D mode within the set time slot.

[0246] In one of the embodiments, the preset constraints include at least one of the following conditions: a first constraint, which is used to constrain the target UAV to reuse the spectrum of at most one channel in one time slot; a second constraint, which is used to constrain the range of the target UAV's transmission power state; a third constraint, which is used to constrain the maximum limit of the target UAV's flight distance in one time slot; a fourth constraint, which is used to constrain the UAV in U2D mode to meet a preset minimum signal to interference and noise ratio; a fifth constraint, which is used to constrain the UAV in U2N mode to meet a preset minimum signal to interference and noise ratio.

[0247] In one embodiment, the random model includes a Markov model; the random model building module 906 is also used to: treat all drones in U2D mode as independent intelligent agents, and model the Markov model based on optimizing task requirements and environmental conditions; obtain the penalty for the action decision of the intelligent agent based on preset constraints; based on the throughput and the penalty, according to the second association relationship, obtain the reward for the action decision of the intelligent agent, and add the reward to the Markov model.

[0248] In one embodiment, the iterative optimization module 908 is also used to: initialize the actor network, critic network and target network of each agent; select the optimal action strategy of the agent in the current state in the experience buffer of each agent based on the actor network, critic network and target network; update the experience buffer of the agent according to the optimal action strategy; after the experience buffers of all agents are updated, update the actor network, critic network and target network of each agent; if all agents can no longer optimize and obtain rewards by changing their own action strategies, then determine that the action strategies of all agents at this time are the drone operation strategies that optimize the total throughput.

[0249] In one embodiment, the action strategy includes the channel allocation state, transmission power state and flight speed of the agent within a set time slot; the iterative optimization module 908 is also used to: based on the local observation of the agent, according to the third association relationship, select the optimal action strategy of the agent in the current state in the experience buffer.

[0250] In one embodiment, the iterative optimization module 908 is also used to: after all agents have selected the optimal action strategy in the current state, obtain the reward of the agent in the current state according to the Markov model, and transfer the agent from the current state to the next state; based on the experience of the agent obtained and stored in the experience buffer, update the experience buffer of all agents.

[0251] The experience includes the local observations of all agents in the current state, the rewards of the agents in the current state, the local observations of all agents in the next state, and the action strategies of the agents in the current state.

[0252] In one embodiment, the iterative optimization module 908 is further used to: update the critic network by minimizing the loss function based on the updated experience buffer, and update the actor network by policy gradient; after the actor network and critic network of all agents are updated, update the target network of the agent.

[0253] Each module in the above throughput optimization device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0254] In one embodiment, Fig.10As shown, the present application also provides a drone control system 100, the system comprising: a target drone, a terminal device 40, a base station 30 and a computer device (not shown in the figure) for executing the method described in the above method embodiment; wherein the target drone comprises a drone 10 in U2D mode and a drone 20 in U2N mode; the base station 30 is respectively connected to the terminal device 40 and the drone 20 in U2N mode; the drone in U2D mode is connected to the terminal device 40 and reuses the communication spectrum of the drone 20 in U2N mode; the computer device is respectively connected to the drone 20 in U2N mode and the drone 10 in U2D mode.

[0255] In an exemplary embodiment, the present application provides a computer device, which may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a throughput optimization method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0256] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0257] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0258] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0259] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0260] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0261] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0262] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A throughput optimization method, characterized in that: The method comprises: Acquire a signal to interference noise ratio of a target UAV; the target UAV includes a UAV in U2D mode; the target UAV also includes a UAV in U2N mode, and the UAV in U2D mode reuses the communication spectrum of the UAV in U2N mode; According to the signal to interference noise ratio of the UAV in the U2D mode, the throughput of the UAV in the U2D mode is obtained; Under preset constraints, constructing a random model that optimizes the total throughput of all UAVs in the U2D mode according to the throughput; Iteratively obtain a UAV operation strategy based on the random model, and instruct the UAV in the U2D mode to perform resource allocation and trajectory planning based on the UAV operation strategy so that the total throughput is optimized; The step of constructing a random model that optimizes the total throughput of all drones in the U2D mode according to the throughput includes: According to the throughput of the UAV in the U2D mode, obtaining the total throughput of all UAVs in the U2D mode; According to the preset constraints, the random model is constructed to optimize the total throughput of all drones in the U2D mode; wherein the total throughput is obtained based on the throughput, the channel allocation state when the drone in the U2D mode reuses the communication spectrum of the drone in the U2N mode, the transmission power state of the drone in the U2D mode, and the flight speed of the drone in the U2D mode within the set time slot.

2. The method according to claim 1, characterized in that The obtaining of the signal to interference noise ratio of the target UAV includes: Based on the position state and resource allocation state of the target UAV, the effective transmission power and interference signal power of the target UAV are obtained; Based on the effective transmission power and the interference signal power of the target UAV, the signal to interference noise ratio of the target UAV in the set time slot is obtained.

3. The method according to claim 2, characterized in that The acquiring the signal to interference noise ratio of the target UAV at the set time slot based on the effective transmission power and the interference signal power of the target UAV includes: Based on the effective transmission power of the UAV in the U2N mode when communicating on the set channel and the interference signal power of the UAV in the U2D mode from the spectrum multiplexing the set channel, the signal to interference noise ratio of the UAV in the U2N mode on the set time slot is obtained; Based on the effective transmission power when the drone in the U2D mode reuses the spectrum of the set channel and the interference signal power of the drone in the U2D mode, the signal to interference noise ratio of the drone in the U2D mode at the set time slot is obtained; Among them, the interference signal power of the drone in the U2D mode includes the interference signal power from other drones in the U2D mode and the interference signal power from the drone in the U2N mode on the corresponding channel of the multiplexed spectrum.

4. The method according to claim 3, characterized in that The resource allocation state includes the channel allocation state and the transmission power state of the target UAV; the effective transmission power and the interference signal power of the target UAV are obtained based on the position state and the resource allocation state of the target UAV, including: According to the current position state of the target UAV, the channel allocation state and the transmission power state, the Los connection probability, Los path loss and NLos path loss of the channel transmission between the target UAV and the transmission object are obtained; Based on the Los connection probability, the Los path loss and the NLos path loss, an average path loss of the target UAV is obtained; According to the average path loss, the channel allocation status and the transmission power status of the target UAV, the effective transmission power and the interference signal power of the target UAV are obtained.

5. The method according to claim 3, characterized in that: The obtaining of the throughput of the UAV in the U2D mode includes: Based on the signal to interference and noise ratio of the UAV in the U2D mode in the set time slot, the throughput is obtained according to the first association relationship.

6. The method according to claim 4, characterized in that The preset constraint condition includes at least one of the following conditions: A first constraint condition, wherein the first constraint condition is used to constrain the target UAV to reuse a frequency spectrum of at most one channel in one time slot; A second constraint condition, wherein the second constraint condition is used to constrain the range of the transmission power state of the target UAV; A third constraint condition, wherein the third constraint condition is used to constrain the maximum limit of the flight distance of the target UAV in a time slot; A fourth constraint condition, wherein the fourth constraint condition is used to constrain the UAV in the U2D mode to meet a preset minimum signal to interference noise ratio; The fifth constraint condition is used to constrain the UAV in the U2N mode to meet a preset minimum signal to interference noise ratio.

7. The method according to claim 4, characterized in that The random model includes a Markov model; the random model constructed according to the throughput to optimize the total throughput of all drones in the U2D mode under the preset constraint conditions also includes: All UAVs in U2D mode are treated as independent intelligent agents, and the Markov model is modeled based on the optimization task requirements and environmental conditions; Based on the preset constraint condition, a penalty for the action decision of the agent is obtained; Based on the throughput and the penalty, a reward for the action decision of the agent is obtained according to a second association relationship, and the reward is added to the Markov model.

8. The method according to claim 7, characterized in that The method of iteratively obtaining a UAV operation strategy based on the random model includes: Initialize the actor network, critic network and target network of each agent; Based on the actor network, the critic network and the target network, selecting the optimal action strategy of the agent in the current state in the experience buffer of each agent; According to the optimal action strategy, updating the experience buffer of the agent; After the experience buffers of all the agents are updated, updating the actor network, the critic network and the target network of each agent; If all the agents can no longer optimize the reward by changing their own action strategies, then it is determined that the action strategies of all the agents at this time are drone operation strategies that optimize the total throughput.

9. The method according to claim 8, characterized in that The action strategy includes the channel allocation state, the transmission power state and the flight speed of the agent in the set time slot; The step of selecting the optimal action strategy of the agent in the current state in the experience buffer of each agent comprises: Based on the local observation of the agent, according to the third association relationship, the optimal action strategy of the agent in the current state is selected in the experience buffer.

10. The method according to claim 9, characterized in that The updating of the experience buffer of the agent according to the optimal action strategy comprises: After the optimal action strategy of all the agents in the current state is selected, the reward of the agent in the current state is obtained according to the Markov model, and the agent is transferred from the current state to the next state; updating the experience buffers of all the agents based on the experience of the agents obtained and stored in the experience buffers; The experience includes the local observations of all the agents in the current state, the rewards of the agents in the current state, the local observations of all the agents in the next state, and the action strategies of the agents in the current state.

11. The method according to claim 9, characterized in that The updating of the actor network, the critic network and the target network of each agent includes: Based on the updated experience buffer, updating the critic network by minimizing the loss function, and updating the actor network by policy gradient; After the actor networks and the critic networks of all the agents are updated, the target network of the agent is updated.

12. A throughput optimization device, characterized in that: The device comprises: A signal to interference and noise ratio calculation module, used to obtain the signal to interference and noise ratio of the target UAV; the target UAV includes a UAV in U2D mode; the target UAV also includes a UAV in U2N mode, and the UAV in U2D mode reuses the communication spectrum of the UAV in U2N mode; A throughput calculation module, used to obtain the throughput of the UAV in the U2D mode according to the signal to interference noise ratio of the UAV in the U2D mode; A random model construction module is used to construct a random model that optimizes the total throughput of all drones in the U2D mode according to the throughput under preset constraints; the random model construction module is specifically used to obtain the total throughput of all drones in the U2D mode according to the throughput of the drones in the U2D mode; according to the preset constraints, construct the random model that optimizes the total throughput of all drones in the U2D mode; wherein the total throughput is obtained based on the throughput, the channel allocation state when the drone in the U2D mode reuses the communication spectrum of the drone in the U2N mode, the transmission power state of the drone in the U2D mode, and the flight speed of the drone in the U2D mode within a set time slot; The iterative optimization module is used to iteratively obtain the UAV operation strategy based on the random model, and instruct the UAV in the U2D mode to perform resource allocation and trajectory planning based on the UAV operation strategy so that the total throughput is optimized.

13. A drone control system, characterized in that: The system comprises: a target drone, a terminal device, a base station, and a computer device for executing the method according to any one of claims 1 to 11; The target drone includes a drone in U2D mode and a drone in U2N mode; the base station is connected to the terminal device and the drone in U2N mode respectively; the drone in U2D mode is connected to the terminal device and reuses the communication spectrum of the drone in U2N mode; The computer device is connected to the UAV in the U2N mode and the UAV in the U2D mode respectively.

Images