Air network resource allocation method, device, electronic equipment and storage medium
Through deep reinforcement learning algorithms, the air network resource allocation is optimized, and the air network's low communication coverage and resource utilization efficiency in post-disaster emergency scenarios are solved, and high-efficiency air network recovery and resource optimization are achieved.
Patent Information
- Application Number
- CN202211048190.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-30
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-08-30
AI Technical Summary
The existing aerial network has limited communication coverage and low resource utilization efficiency in post-disaster emergency scenarios. It is necessary to improve the communication coverage of aerial network recovery and achieve more reasonable resource allocation.
Deep reinforcement learning algorithm is used to solve the problem model of air network resource optimization, determine the target deployment location, transmission power and terminal channel allocation information of low-altitude platform equipment, and combine multiple types of low-altitude platform equipment (air base station, air radio frequency unit and millimeter wave enhancement without human-machine) to maximize air network energy efficiency.
It improves the communication coverage of emergency air network recovery, realizes local capacity enhancement and resource reuse, ensures air network performance and saves deployment costs.
Smart Images

Figure CN115567093B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of communication technology, and in particular to an air network resource allocation method, device, electronic equipment and storage medium. Background Art
[0002] With the increasing maturity of the fifth-generation communication technology, it has been popularized and applied in more and more fields.
[0003] At present, in post-disaster emergency scenarios, drone base stations can be used as air base stations to build air networks and form air communication networks. However, the communication coverage that can be restored by existing air networks is limited. At the same time, due to the shortage of resources in emergency scenarios, after the deployment of air network communication equipment, the recovery of air networks also needs to consider energy efficiency and resource utilization. The network energy efficiency of existing air networks also needs to be further optimized and upgraded.
[0004] Therefore, how to improve the communication coverage of air network restoration while more reasonably allocating resources to the air communication network and maximizing energy efficiency has become a technical problem that needs to be urgently solved in the industry. Summary of the invention
[0005] The present invention provides an air network resource allocation method, device, electronic equipment and storage medium, which are used to more reasonably allocate resources to the air communication network while improving the communication coverage of air network recovery.
[0006] The present invention provides an air network resource allocation method, comprising:
[0007] With the goal of maximizing the energy efficiency of the aerial network, a resource optimization problem model of the aerial network is established in combination with target constraints; the aerial network is built based on multiple types of low-altitude platform equipment;
[0008] A deep reinforcement learning algorithm is used to solve the resource optimization problem model to determine the target deployment position, target transmission power and channel allocation information of each terminal of the low-altitude platform equipment in the aerial network; the multiple types of low-altitude platform equipment include aerial base stations, aerial radio frequency units and millimeter-wave enhanced drones; the channel allocation information is the channel information between the terminal and each of the low-altitude platform devices.
[0009] According to a method for allocating air network resources provided by the present invention, the method takes maximizing air network energy efficiency as the goal and combines the target constraint conditions to establish an air network resource optimization problem model, including:
[0010] Based on the transmission rate of each terminal and the channel allocation information of each terminal, the network capacity of the air network is determined, and based on the network capacity of the air network and the total deployment cost of the air network, the network capacity under unit cost is determined; the total deployment cost of the air network is determined based on the hardware cost and energy consumption cost of each low-altitude platform equipment;
[0011] Taking the maximum network capacity under the unit cost as the optimization goal, an objective function is established; the air network energy efficiency includes the network capacity under the unit cost;
[0012] Determining the target constraint condition based on the transmission power of each low-altitude platform device in the aerial network and the channel allocation information of each terminal;
[0013] Based on the objective function and the objective constraint conditions, a resource optimization problem model of the air network is established.
[0014] According to an aerial network resource allocation method provided by the present invention, the deep reinforcement learning algorithm is used to solve the resource optimization problem model to determine the target deployment position, target transmission power and channel allocation information of each terminal of each low-altitude platform device in the aerial network, including:
[0015] Based on the state information and reward function of each of the low-altitude platform devices in the aerial network, the deep Q network model in the deep reinforcement learning algorithm is iteratively trained to obtain the optimal action value benefit value of the aerial network energy efficiency; the reward function is determined based on the network capacity of the aerial network and the total deployment cost of the aerial network;
[0016] Based on the optimal action value benefit value, the target deployment position, target transmission power and channel allocation information of each terminal of each low-altitude platform device in the aerial network are determined.
[0017] According to an air network resource allocation method provided by the present invention, based on the state information of each low-altitude platform device in the air network and the reward function, the deep Q network model in the deep reinforcement learning algorithm is iteratively trained to obtain the optimal action value benefit value of the air network energy efficiency, including:
[0018] Step 1, establishing a state space and an action space according to the state information of each low-altitude platform device in the aerial network;
[0019] Step 2, based on the state space, determining the initial state of the air network, and dividing the time of each training into multiple time intervals; the initial state is the state of the air network in the first time interval;
[0020] Step 3, determining a first reward value after the air network performs a first action in a first state within a current time interval and a second state in a next time interval; the first action is determined based on the action space;
[0021] Step 4, storing the first state, the first action, the first reward value and the second state as a data sample in a memory unit, and randomly extracting a data sample from the memory unit to update the network parameters of the deep Q network model; the first reward value is determined based on the reward function;
[0022] Step 5, traversing the multiple time intervals, executing steps 3 to 4, completing an iterative training of the deep Q network model, and obtaining the maximum action value benefit value of the air network energy efficiency after training;
[0023] Step 6, traverse the preset number of iterations, execute steps 3 to 5, train the deep Q network model for the preset number of iterations, and obtain the optimal action value benefit value of the air network energy efficiency after training; the optimal action value benefit value is the maximum action value benefit value of the air network energy efficiency obtained through the last iterative training.
[0024] According to a method for allocating resources in an aerial network provided by the present invention, based on the state information and reward function of each of the low-altitude platform devices in the aerial network, the deep Q network model in the deep reinforcement learning algorithm is iteratively trained to obtain the optimal action value benefit value of the aerial network energy efficiency, and further includes:
[0025] Divide each training session into multiple intervals;
[0026] Determine, based on the network capacity of the air network and the total deployment cost of the air network, a first increment of the network capacity in a current time interval relative to a previous time interval and a second increment of the total deployment cost in the current time interval relative to the previous time interval;
[0027] Based on the ratio of the first increment to the second increment, a reward function in the deep reinforcement learning algorithm is determined.
[0028] According to a method for allocating resources in an aerial network provided by the present invention, the state space and the action space are established according to the state information of each low-altitude platform device in the aerial network, including:
[0029] Establishing a state space according to the deployment position, transmission power and channel allocation information of each of the low-altitude platform devices;
[0030] An action space is established according to the movement type of each low-altitude platform device, the adjustment type of the transmission power of each low-altitude platform device, and the access status information between each low-altitude platform device and each terminal.
[0031] According to an aerial network resource allocation method provided by the present invention, the aerial base station is used to control the wireless coverage range of a cell; the aerial radio frequency unit is used to expand the wireless coverage range under the control of the aerial base station; and the millimeter wave enhanced drone is used to expand the capacity of the aerial base station and the aerial radio frequency unit.
[0032] The present invention also provides an air network resource allocation device, comprising:
[0033] Establishing a module for establishing a resource optimization problem model of an aerial network with the goal of maximizing the energy efficiency of the aerial network and combining the target constraints; the aerial network is constructed based on multiple types of low-altitude platform equipment;
[0034] An allocation module is used to solve the resource optimization problem model using a deep reinforcement learning algorithm to determine the target deployment position, target transmission power and channel allocation information of each low-altitude platform device in the aerial network; the multiple types of low-altitude platform devices include aerial base stations, aerial radio frequency units and millimeter-wave enhanced drones; the channel allocation information is the channel information between the terminal and each low-altitude platform device.
[0035] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned air network resource allocation methods is implemented.
[0036] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the air network resource allocation method described in any one of the above is implemented.
[0037] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the air network resource allocation method as described in any one of the above is implemented.
[0038] The air network resource allocation method, device, electronic device and storage medium provided by the present invention construct an air network by adopting a network deployment mechanism in which multiple types of low-altitude platform equipment jointly participate. The multiple types of low-altitude platform equipment include air base stations, air radio frequency units and millimeter wave enhanced unmanned aerial vehicles. Based on the large-scale coverage capability of the air base station, the network coverage expansion capability of the air radio frequency unit and the capacity expansion capability of the millimeter wave enhanced unmanned aerial vehicle, an emergency air network with high energy efficiency as the goal is formed, which can not only improve the communication coverage of the emergency air network restoration, but also realize local capacity enhancement and resource reuse. By maximizing the air network energy efficiency as the goal and combining the target constraint conditions, a resource optimization problem model of the air network is established. In view of the fact that it belongs to a non-convex optimization problem, according to the advantage of the deep reinforcement learning algorithm in solving non-convex optimization problems, the resource optimization problem model of the deep reinforcement learning algorithm is adopted for solving, which can effectively optimize and allocate resources for the air network, determine the target deployment position, target transmission power and channel allocation information of each low-altitude platform equipment in the air network, realize the energy efficiency optimization of the emergency air network, ensure the air network performance, and save the network deployment cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0040] Figure 1 It is a flow chart of the method for allocating air network resources provided by the present invention;
[0041] Figure 2 It is a structural schematic diagram of the air network resource allocation device provided by the present invention;
[0042] Figure 3 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0044] Combine the following Figure 1-Figure 3The invention describes the air network resource allocation method, device, electronic device and storage medium.
[0045] Figure 1 FIG. 1 is a flow chart of the method for allocating air network resources provided by the present invention, such as Figure 1 As shown, it includes: step 110 and step 120.
[0046] Step 110, with the goal of maximizing the energy efficiency of the aerial network, combined with the target constraints, a resource optimization problem model of the aerial network is established; the aerial network is constructed based on multiple types of low-altitude platform equipment;
[0047] Specifically, the aerial network described in the embodiment of the present invention may be applicable to post-disaster emergency response scenarios, and may also be applicable to other scenarios where an aerial network needs to be built for emergency network communications.
[0048] It should be noted that in an embodiment of the present invention, the aerial network may include a high altitude platform (HAP) and a low altitude platform (LAP). Compared with the high altitude platform (HAP), the low altitude platform (LAP) is easier to deploy and is consistent with the concept of cellular network applications. HAP has high regional coverage and long endurance, but it is expensive and prone to strong interference. In contrast, LAP is more flexible and efficient, and has a high cost-effectiveness for emergency deployment. LAP can be divided into rotorcraft, fixed-wing aircraft and airship according to the type of drone.
[0049] In an embodiment of the present invention, the aerial network is constructed based on multiple types of LAP devices, including aerial base stations (Aerial Base Station, AeBS), aerial remote radio heads (Aerial Remote Radio Head, AeRRH) and millimeter wave UAVs (mmW-UAVs). That is to say, in an embodiment of the present invention, the deployment architecture of the aerial network can include four network units: AeBS, AeRRH, mmW-UAV and user equipment (User Equipment, UE).
[0050] Optionally, in an embodiment of the present invention, AeBS is used to control the wireless coverage of a cell; AeRRH is used to expand the wireless coverage under the control of AeBS; and mmW-UAV is used to expand the capacity of an aerial base station and an aerial radio frequency unit.
[0051] Among them, in the embodiment of the present invention, AeBS can be an independent large low-altitude platform. The motorboat is used as a carrier to carry the base station module to complete rapid deployment. The wireless coverage of the cell can be controlled by appropriately setting the height and transmission power of AeBS. Although AeBS is inconvenient to move, it has a strong carrying capacity and can provide wide coverage.
[0052] In an embodiment of the present invention, AeRRH may be a baseband-free processing unit, which refers to a fixed-wing aircraft equipped with a radio remote unit and cannot work independently. It needs to transmit data under the control of AeBS and can be used to fill gaps at the edge of coverage, expand wireless coverage, and make up for the lack of AeBS coverage.
[0053] In the embodiment of the present invention, the mmW-UAV can be an independent small aerial platform. In the embodiment of the present invention, a fixed-wing aircraft equipped with a millimeter wave module can be selected as a simulation experiment object. It can be quickly deployed in an open scene where people gather to expand the capacity of AeBS and AeRRH.
[0054] User equipment UE includes ordinary mobile phone terminals and dedicated emergency equipment. They can communicate with each other through AeBS, AeRRH or mmW-UAV.
[0055] The communication links in emergency scenarios include: satellite backhaul link, AeBS inter-link, AeBS-RRH link, etc. These links provide reliable and flexible network connections for LAP and UE.
[0056] Assume that the communication quality of the backhaul link connecting the satellite to the AeBS is guaranteed, and all LAPs have sufficient signal strength and resources. Assume that all UAVs are equipped with two transmission modes, namely traditional frequency band transmission and millimeter wave frequency band transmission. Each UAV can switch transmission mode freely. When the UAV uses the traditional frequency band as an AeRRH to fill the coverage gap, its channel model is the same as that of the AeBS, except that the flight altitude and transmission power are limited by the carrier. Due to different transmission modes, the channel model of the HPN, the channel model of the AeBS and AeRRH, and the channel model of the mmW-UAV are different.
[0057] In this embodiment, the AeBS and AeRRH channel models are established. On the one hand, the path loss between the AeBS and the UE includes free space fading FSPL and large-scale fading. On the other hand, the coverage area of the AeBS is related to its power and height. At the same height, the higher the AeBS transmit power, the larger the coverage area. At the same transmit power, the higher the AeBS height, the larger the coverage area.
[0058] P(LoS) and P(NLos) are used to represent the line-of-sight transmission probability and non-line-of-sight transmission probability respectively.i Send to user UE j The path loss As shown in formula (1):
[0059]
[0060] Combining line of sight (LOS) and non-line of sight (NLOS) wireless transmission into one formula, the path loss expression is:
[0061]
[0062] Among them, free space fading FSPL ij =20logd ij +20logf-27.55, carrier frequency f = 2000MHz, and AeBS i To user UE j Distance: θ ij Represents the angle between the transmission path and the ground. The line-of-sight transmission probability and non-line-of-sight transmission probability are shown in formula (3) and formula (4).
[0063] P(LoS)=a(θ ij -θ o ) b ; (3)
[0064] P(NLoS)=1-a(θ ij -θ o ) b ; (4)
[0065] Among them, θ o is 15°, θ ij ∈[θ o ,90°], (a, b) = (0.76, 0.06), (PL LoS ,PL NLoS )=(0.1,21). Therefore, AeBS i When transmitting via the U2E link, UE j The received power is:
[0066]
[0067] In this embodiment, a channel model of mums-UAV is established. When UAV uses the millimeter wave frequency band, it is mainly responsible for hotspot coverage. The channel model is referenced. The mmW-UAV forms a beam with the antenna array of the target user for data transmission. In order to facilitate calculation, the fan antenna model is used to approximate the antenna gain G ADefine the main lobe gain as M and the side lobe gain as m. Then the antenna gain G A It can be expressed as formula (6).
[0068]
[0069] When the antenna of mmW-UAV is perfectly aligned with the UE antenna, the antenna gain is M*M; when the side lobes are aligned, the antenna gain is m*m; otherwise, it is M*m. The main lobe width δ of the beam can be 30°. M = 10dB, m = -10dB. Therefore, UAV i When transmitting via the millimeter wave band, UE j The received power is shown in formula (7).
[0070]
[0071] in, The calculation method and Same, different parameter values.
[0072] In the embodiment of the present invention, the deployment of mmW-UAV is to solve the problem of dense users, so the scene can be fixed as a dense urban environment. When the carrier frequency f = 28 GHz, (a, b) = (0.33, 0.23), (PL LoS ,PL NLoS )=(2,2.65).
[0073] The method of the embodiment of the present invention builds an aerial network by deploying multiple types of LAP equipment, including AeBS, AeRRH and mmW-UAV. Based on the large-scale coverage capability of AeBS, the network coverage expansion capability of AeRRH and the capacity expansion capability of mmW-UAV, an emergency aerial network with high energy efficiency as the goal is formed, which can not only improve the communication coverage of the emergency aerial network restoration, but also realize local capacity enhancement and resource reuse.
[0074] In post-disaster emergency networks, the joint deployment of multiple LAP devices must not only meet the communication needs of users in the network, but also avoid over-deployment, over-coverage and resource waste.
[0075] Furthermore, in an embodiment of the present invention, with the goal of maximizing the energy efficiency of the air network, a resource optimization problem model of the air network is established in combination with target constraints.
[0076] Based on the content of the above embodiment, as an optional embodiment, in step 110, with the goal of maximizing the energy efficiency of the air network, combined with the target constraint conditions, a resource optimization problem model of the air network is established, including:
[0077] Based on the transmission rate of each terminal and the channel allocation information of each terminal, the network capacity of the air network is determined, and based on the network capacity of the air network and the total deployment cost of the air network, the network capacity under unit cost is determined; the total deployment cost of the air network is determined based on the hardware cost and energy consumption cost of each low-altitude platform equipment;
[0078] The objective function is established with the maximum network capacity under unit cost as the optimization goal; the air network energy efficiency includes the network capacity under unit cost;
[0079] Determine the target constraint conditions based on the transmission power of each low-altitude platform device in the air network and the channel allocation information of each terminal;
[0080] Based on the objective function and objective constraints, a resource optimization problem model of the air network is established.
[0081] Specifically, the terminal described in the embodiment of the present invention refers to the user equipment UE side in the air network, and specifically may be all user terminal devices in the disaster-stricken area in a post-disaster emergency scenario.
[0082] The channel allocation information described in the embodiment of the present invention is the channel information between the terminal and each low-altitude platform device, that is, the allocation information of each terminal being allocated to access the corresponding LAP device channel.
[0083] In this embodiment, before determining the problem model, a network deployment cost indicator is first defined to measure and limit the deployment situation. According to the different types of low-altitude platform equipment, it is divided into: AeBS cost AeRRH Cost and mmW-UAV costs
[0084] Among them, AeBS cost It can be expressed as:
[0085]
[0086] As shown in formula (8), the deployment cost of AeBS is divided into two parts: B represents the deployment hardware cost of AeBS, Represents the energy consumption cost of AeBS. Among them, the fixed power of AeBSi is Pc B , the transmission power gain is represents the maximum transmit power gain of AeBS, and θ represents the hardware and power adaptation parameters.
[0087] The cost rate calculation formula of AeRRH and mmW-UVA is similar to that of AeBS, from which the cost of AeRRH can be determined. As shown in formula (9) and mmW-UAV cost As shown in formula (10).
[0088]
[0089]
[0090] By adjusting the costs of the three LAP devices, the deployment of the entire network can be controlled. The total deployment cost of the entire network, that is, the sum of the deployment costs of all LAP devices in the network, can be expressed as C Net , as shown in formula (11).
[0091]
[0092] Among them, c B , c R , c M The value of Θ can be configured according to the shortage of hardware and energy in the current emergency environment; N B Indicates the number of AeBS, N R Indicates the number of AeRRHs, N M Indicates the number of mmW-UAVs.
[0093] Furthermore, in an embodiment of the present invention, a problem model for the joint deployment of multiple LAPs in an emergency network is determined, and the goal can be set as: achieving the highest energy efficiency through the height, power allocation and channel allocation after the deployment of multiple types of aerial base stations, that is, obtaining the highest network capacity while controlling the network cost rate.
[0094] In an embodiment of the present invention, the network capacity of the air network is determined based on the transmission rate of each terminal and the channel allocation information of each terminal accessing the low-altitude platform device, and the network capacity under unit cost is determined based on the network capacity of the air network and the total deployment cost of the air network; taking the maximum network capacity under unit cost as the optimization goal, an objective function can be established, wherein the objective function can be expressed as:
[0095]
[0096] Among them, the matrices [B], [R], and [M] represent N B AeBS, N R AeRRH, N M The positions of the mmW-UAVs and their corresponding transmit powers. The 0-1 matrices [β], [γ], and [ω] represent the AeBS, AeRRH, mmW-UAV, and N U Channel allocation for each UE. N UThe transmission rate of a UE accessing AeBS, AeRRH and mmW-UAV.
[0097] Among them, the transmission rate of terminal i can be expressed as:
[0098]
[0099] Where, bandwidth w ij =10MHz,υ=0.1.
[0100] It should be noted that the signal to interference plus noise ratio (SINR) refers to the ratio of the strength of the received useful signal to the strength of the received interference signal. The SINR value of terminal i is expressed as
[0101]
[0102] In the formula, S i represents the received power at the terminal i, the interference signal N i It is the sum of interference from other co-frequency neighboring stations, that is, the sum of all signals minus the useful signal S i , noise σ 2 The possible value is -118dBm.
[0103] Furthermore, in an embodiment of the present invention, the target constraint condition is determined based on the transmission power of each low-altitude platform device in the air network and the channel allocation information of each terminal, such as formula (15).
[0104]
[0105] Among them, in formula (15) representing the target constraint condition, formula (a) represents the transmission rate satisfaction condition of each terminal, and at least one LAP channel meets the user terminal transmission rate requirement to restore all user communications in the emergency scenario; R 0 Represents the minimum transmission rate threshold. Formula (b) limits the channels allocated to each terminal, and there is only one channel to serve the terminal. And each channel can only be allocated to one terminal, as shown in formula (c), to avoid resource waste. The total number of system resources is limited. Formula (d) represents the total number of channel resources of AeBS, AeRRH, and mmW-UAV, which does not exceed the system resource upper limit η max Formula (e) limits the situation where each terminal obtains less than three AeBS effective coverage, that is, the situation of over-coverage; where, Indicates the minimum transmission rate of the terminal. Formula (f) indicates that the transmission power of all LAP devices cannot exceed their maximum transmission power.
[0106] Furthermore, based on the objective function and objective constraints, a resource optimization problem model for the air network can be established, such as formula (16).
[0107]
[0108] Optionally, in an embodiment of the present invention, when the deployment of AeBS, AeRRH and mmW-UAV in a two-dimensional plane and the number of devices are determined, the equipment cost is fixed, and the only variable related to cost consumption is the transmission power of AeBS, AeRRH and mmW-UAV. Therefore, the resource optimization problem model of the aerial network can be further optimized, as shown in formula (17).
[0109]
[0110] The method of the embodiment of the present invention, by considering the limited resources in emergency scenarios, constructs a resource optimization problem model for the air network based on the joint deployment of multiple LAP devices. On the one hand, it can meet the communication needs of users in the air network, and on the other hand, it can also ensure that the problems of over-deployment, over-coverage and waste of resources in emergency scenarios are avoided.
[0111] Step 120, using a deep reinforcement learning algorithm to solve the resource optimization problem model, determine the target deployment position, target transmission power and channel allocation information of each terminal of each low-altitude platform equipment in the aerial network; multiple types of low-altitude platform equipment include aerial base stations, aerial radio frequency units and millimeter wave enhanced drones.
[0112] Specifically, the target deployment position described in the embodiment of the present invention refers to the optimal deployment position for the deployment of each low-altitude platform device in the aerial network, wherein the deployment position may include two-dimensional position coordinates and altitude, and the two-dimensional position coordinates represent the deployment position coordinates of each LAP on a two-dimensional plane, which does not carry altitude information.
[0113] The target transmission power described in the embodiment of the present invention refers to the optimal transmission power parameter value allocated to each low-altitude platform device in the aerial network.
[0114] The channel allocation information of each terminal described in the embodiment of the present invention refers to the optimal allocation information of each terminal being allocated to access the corresponding low-altitude platform device channels.
[0115] In an embodiment of the present invention, by jointly deploying the target deployment positions of various low-altitude platform devices in the aerial network, allocating target transmission power, and channel allocation information of each terminal, the communication coverage of emergency aerial network recovery can be improved, ensuring maximum energy efficiency of the aerial network.
[0116] Therefore, in order to maximize the network energy efficiency, power and channel allocation are then performed to determine the deployment location and transmission power of AeBS, AeRRH, and mmW-UAV, as well as the selection of user access channels. This problem is a non-convex optimization problem, and the analytical solution cannot be directly calculated. In this embodiment, considering that the terminal position will be continuously updated within a small range, continuous trial learning is required during the deployment process. Therefore, the Deep Reinforcement Learning (DRL) algorithm can be used to solve the resource optimization problem model.
[0117] Based on the content of the above embodiment, as an optional embodiment, in step 120, a deep reinforcement learning algorithm is used to solve the resource optimization problem model to determine the target deployment position, target transmission power and channel allocation information of each low-altitude platform device in the air network, including:
[0118] Based on the status information and reward function of each low-altitude platform device in the aerial network, the deep Q network model in the deep reinforcement learning algorithm is iteratively trained to obtain the optimal action value benefit value of the aerial network energy efficiency; the reward function is determined based on the network capacity of the aerial network and the total deployment cost of the aerial network;
[0119] Based on the optimal action value benefit value, the target deployment position, target transmission power and channel allocation information of each low-altitude platform equipment in the aerial network are determined.
[0120] It should be noted that the DRL algorithm combines the perception ability of deep learning and the decision-making ability of reinforcement learning. Unlike supervised learning and unsupervised learning, the DRL algorithm has strong adaptive capabilities and real-time learning capabilities. The goal of reinforcement learning is to learn behavioral strategies in the interaction with the environment to obtain the maximum long-term reward and obtain the optimal solution by maximizing the cumulative reward.
[0121] In an embodiment of the present invention, the reward function is determined based on the network capacity of the air network and the total deployment cost of the air network.
[0122] Based on the content of the above embodiment, as an optional embodiment, based on the state information and reward function of each low-altitude platform device in the air network, the deep Q network model in the deep reinforcement learning algorithm is iteratively trained to obtain the optimal action value benefit value of the air network energy efficiency, and it also includes:
[0123] Divide each training session into multiple intervals;
[0124] Determine, based on the network capacity of the air network and the total deployment cost of the air network, a first increment of the network capacity in a current time interval relative to a previous time interval and a second increment of the total deployment cost in the current time interval relative to the previous time interval;
[0125] Based on the ratio of the first increment to the second increment, a reward function in the DRL algorithm is determined.
[0126] Specifically, the first increment described in the embodiment of the present invention refers to the increment of the network capacity of the current time interval relative to the network capacity of the previous time interval, which can be obtained by subtracting the network capacity of the previous time interval from the network capacity of the current time interval.
[0127] The second increment described in the embodiment of the present invention refers to the increment of the total network deployment cost of the current time interval relative to the total network deployment cost of the previous time interval, which can be specifically obtained by subtracting the total network deployment cost of the previous time interval from the total network deployment cost of the current time interval.
[0128] In the embodiment of the present invention, according to the resource optimization problem model, formula (12) shows that the network capacity R can be expressed as:
[0129]
[0130] In addition, the total deployment cost of the aerial network C Net It can be obtained by the above formula (11).
[0131] In an embodiment of the present invention, the reward function can be defined as the ratio of the capacity increment to the network cost increment. The numerator of the reward function is equivalent to the benefit after each low-altitude platform device performs an action, and the denominator is equivalent to the state cost in units of energy consumption. Therefore, the final result of maximizing the cumulative return is the maximization of the average energy efficiency.
[0132] Further, in an embodiment of the present invention, by dividing the time of each training of the DRL algorithm into multiple time intervals, the reward function in the DRL algorithm can be expressed as:
[0133]
[0134] In the formula, R t Indicates the network capacity of the current time interval; R t-1 Indicates the network capacity of the previous time interval; C t represents the total network deployment cost of the current time interval; C t-1 Indicates the total network deployment cost for the last time interval.
[0135] In addition, if the low-altitude platform equipment fails to reach the capacity or coverage required by the target constraint after executing the action, then r t =r t -r - , that is, a punishment rather than a reward, r - Indicates the size of the penalty value.
[0136] The method of the embodiment of the present invention defines the reward function in the DRL algorithm by considering the maximization of the energy efficiency of the air network, ensuring that the behavioral strategy that can maximize the energy efficiency of the air network is learned during the deep reinforcement learning process to obtain the maximum long-term reward, which is conducive to obtaining the optimal solution to the resource optimization problem model.
[0137] Furthermore, based on the state information and reward function of each low-altitude platform device in the air network, the state space and action space of the air network are constructed. By storing the state, action, and reward value of each low-altitude platform device in the memory unit, the Deep Q-Network (DQN) model in the deep reinforcement learning algorithm is repeatedly iterated to obtain the optimal action value benefit value of the air network energy efficiency and the optimal action of each low-altitude platform device. According to the optimal action of each low-altitude platform device, the target deployment position, target transmission power, and channel allocation information of each terminal of each low-altitude platform device in the air network can be determined.
[0138] The aerial network resource allocation method of the embodiment of the present invention constructs an aerial network by adopting a network deployment mechanism in which multiple types of low-altitude platform devices jointly participate. The multiple types of low-altitude platform devices include aerial base stations, aerial radio frequency units and millimeter-wave enhanced drones. Based on the large-scale coverage capability of the aerial base station, the network coverage expansion capability of the aerial radio frequency unit and the capacity expansion capability of the millimeter-wave enhanced drone, an emergency aerial network with high energy efficiency as the goal is formed, which can not only improve the communication coverage of the emergency aerial network restoration, but also realize local capacity enhancement and resource reuse; by maximizing the energy efficiency of the aerial network as the goal, combined with the target constraint conditions, a resource optimization problem model of the aerial network is established. In view of the fact that it belongs to a non-convex optimization problem, according to the advantages of the deep reinforcement learning algorithm in solving non-convex optimization problems, the resource optimization problem model of the deep reinforcement learning algorithm is adopted for solving, which can effectively optimize and allocate resources for the aerial network, determine the target deployment position, target transmission power and channel allocation information of each terminal of each low-altitude platform device in the aerial network, realize the energy efficiency optimization of the emergency aerial network, ensure the performance of the aerial network, and save the network deployment cost.
[0139] Based on the content of the above embodiment, as an optional embodiment, based on the state information and reward function of each low-altitude platform device in the air network, the deep Q network model in the deep reinforcement learning algorithm is iteratively trained to obtain the optimal action value benefit value of the air network energy efficiency, including:
[0140] Step 1: Establish state space and action space according to the state information of each low-altitude platform device in the aerial network;
[0141] Step 2: Based on the state space, determine the initial state of the air network and divide each training time into multiple time intervals; the initial state is the state of the air network in the first time interval;
[0142] Step 3, determining a first reward value after the air network performs a first action in a first state within the current time interval and a second state in the next time interval; the first action is determined based on the action space;
[0143] Step 4, storing the first state, the first action, the first reward value and the second state as a data sample in a memory unit, and randomly extracting a data sample from the memory unit to update the network parameters of the deep Q network model; the first reward value is determined based on the reward function;
[0144] Step 5, traverse multiple time intervals, execute steps 3 to 4, complete an iterative training of the deep Q network model, and obtain the maximum action value benefit value of the air network energy efficiency after training;
[0145] Step 6, traverse the preset number of iterations, execute steps 3 to 5, train the deep Q network model for the preset number of iterations, and obtain the optimal action value benefit value of the air network energy efficiency after training.
[0146] Specifically, the status information of each low-altitude platform device in the aerial network described in the present invention refers to the aerial height information, transmission power information and channel allocation information of each terminal of various low-altitude platform device nodes in the aerial network, such as AeBS nodes, AeRRH nodes and mmW-UAV nodes. It can also include the location information of each network node and the real-time location, direction and speed of each network node that can be obtained using GPS equipment.
[0147] In the embodiment of the present invention, the DQN model can be represented by {S, A, r}, where S represents the state space, A represents the action space, and r represents the reward determined by the reward function. At time t, the state S t , Action A t , reward r t .
[0148] Based on the content of the above embodiment, as an optional embodiment, in step 1, a state space and an action space are established according to the state information of each low-altitude platform device in the air network, including:
[0149] Establish state space based on the deployment location, transmission power and channel allocation information of each low-altitude platform equipment;
[0150] An action space is established according to the movement type of each low-altitude platform device, the adjustment type of the transmission power of each low-altitude platform device, and the access status information between each low-altitude platform device and each terminal.
[0151] Specifically, in an embodiment of the present invention, based on the status information of each low-altitude platform device in the aerial network, the deployment position, transmission power and allocation information of each terminal access channel of each low-altitude platform device are obtained, thereby establishing a state space.
[0152] S t It can contain the following parts: B AeBS, N R AeRRH, N M The deployment heights of the mmW-UAVs are expressed as The transmit power is expressed as And the signal allocation information of the terminal can be expressed as {β,γ,ω}.
[0153] It should be noted that, in the embodiment of the present invention, when each LAP device in the aerial network is not deployed in two dimensions, h t Indicates the location and deployment height of low-altitude platform devices in the aerial network. When the two-dimensional deployment of each LAP device is known, h t It can only indicate the deployment height of low-altitude platform equipment in the aerial network.
[0154] Furthermore, an action space is established according to the movement type of each low-altitude platform device, the adjustment type of the transmission power of each low-altitude platform device, and the access status information between each low-altitude platform device and each terminal.
[0155] In an embodiment of the present invention, the movement type of the low-altitude platform device can be determined by two situations. One is that when the LAP device has not been deployed in two dimensions, its movement type may include forward, backward, left, right, up, down and hovering. The second is that when the LAP device has been deployed in two dimensions, its movement type only includes up, down and hovering. The adjustment type of the transmission power of each low-altitude platform device may include increasing, decreasing and maintaining the transmission power. Among them, hovering and maintaining allow the aerial LAP device to maintain the current altitude and transmission power respectively. The access status information between each low-altitude platform device and each terminal can be represented by a 0-1 matrix, which characterizes the status of the LAP device to which the terminal is connected, including two actions: access and disconnection.
[0156] To simplify the description, specifically, when the LAP device is deployed in two dimensions, action a t In state S t The actions performed under the control of the LAP device are discretized and defined as 6 actions: altitude up, down and hovering, transmission power up, down and maintenance. L The action space can be expressed as {1,2,3,4,5,6}. When the action is selected, the action value is set to 1, otherwise it is set to 0. The total number of air base stations NL = N B +N R +N M , the action space of multi-aerial LAP deployment can be expressed as
[0157] The method of the embodiment of the present invention obtains the status information of each low-altitude platform device in the aerial network, determines the status and action of each low-altitude platform device, establishes a state space and an action space, and provides accurate input parameters for subsequent deep reinforcement learning.
[0158] Further, in an embodiment of the present invention, the value network Q function and error function of the DQN model are defined as follows:
[0159] The value network, which is described by a deep neural network (DNN), updates the Q function for the state-action pair. The Q function is the long-term expectation of the state and action in k time periods. The Q-learning learning mechanism is adopted to iteratively implement the optimization learning of the Q function according to formula (20).
[0160]
[0161] Among them, k Represents the learning rate. The larger the learning rate, the better the current Q k (st ,a t ) The greater the impact, the slower the convergence speed, and it can be set to 1. τ represents the discount factor, which represents the preference for rewards. The larger the discount factor, the greater the impact on future results. The value range can be between 0 and 1. s′ represents the time when executing action a t a′ represents the action in the action set that makes the Q function in the kth iteration executable in the s′ state.
[0162] Error function: If you want to achieve an approximation of the Q function Q k+1 (s t ,a t )≈Q k (s t ,a t ), the error function can be defined as shown in formula (21).
[0163]
[0164] The objective function is used to update the parameters of each iteration and obtain the optimal solution of the Q function. The update function of the Q network is shown in formula (22).
[0165]
[0166] Further, in step 2, by initializing the memory unit M-Unit and the Q network, the state is randomly initialized, that is, randomly selected from the state space, to determine the initial state of each LAP device in the air network. And the time of each training is divided into multiple time intervals. It can be understood that the initial state is the state of each LAP device in the air network in the first time interval.
[0167] In the embodiment of the present invention, each training time can be divided into multiple time intervals of equal length, and different time slots can be represented by index t∈{1,2,…,T}. The air network starts from the initial state and explores different states within T time periods, which represents a learning process.
[0168] In step 3, determine the first reward value after the aerial network performs the first action in the first state within the current time interval and the second state of the next time interval; the first action is determined based on the action space, which can be an action randomly selected from the action space, or it can be the optimal action in the action space learned through the deep Q network.
[0169] The first state described in the embodiment of the present invention refers to the state of each LAP device in the air network in the current time interval, which can be a state selected from the state space or a state updated based on the learning process of the previous time interval. The second state refers to the state of each LAP device in the air network in the next time interval obtained after the state in the current time interval is updated.
[0170] The first reward value described in the embodiment of the present invention is determined based on a reward function, which can be specifically a reward value calculated by the reward function after the air network selects to execute the first action in the first state.
[0171] In the embodiment of the present invention, in step 3, each LAP device enters the exploration process, and the ε-greedy exploration method is to randomly select the first action with probability ε, and select the first action of the Q network with probability 1-ε. The smaller ε is, the more emphasis is placed on using existing learning results to select actions; the larger ε is, the more emphasis is placed on randomly selecting actions away from existing learning results. The ε-greedy mechanism can prevent the algorithm from falling into a local optimal solution. ε is not a fixed value, but varies according to the number of cycles k. For example, ε can be set to 10 / k.
[0172] Furthermore, in step 4, the first state, the first action, the first reward value, and the second state can be stored in a memory unit as a data sample, and a data sample can be randomly extracted from the memory unit, and the network parameters of the deep Q network model can be updated using the Q network update formulas (20), (21), and (22).
[0173] In step 5, by traversing multiple time intervals and iteratively executing steps 3 to 4, an iterative training of the deep Q network model can be completed, and the maximum Q value of the air network energy efficiency after this training, that is, the maximum action value benefit value, can be obtained.
[0174] In step 6, by traversing the preset number of iterations, the process of steps 3 to 5 is cyclically executed, and the deep Q network model is continuously trained until the number of training reaches the preset number of iterations, and learning can be stopped. After the training is completed, the maximum action value benefit value of the air network energy efficiency obtained by the last iterative training is the optimal action value benefit value of the air network energy efficiency. Through the optimal action value benefit value, the optimal state and optimal action of each LAP device in the air network can be determined, and the target deployment position, target transmission power and channel allocation information of each terminal of each low-altitude platform device in the air network can be determined.
[0175] Optionally, in an embodiment of the present invention, a deep reinforcement learning algorithm is used to solve the resource optimization problem model, and the algorithm flow for determining the target deployment position, target transmission power and channel allocation information of each terminal of each low-altitude platform equipment in the aerial network is as shown below.
[0176]
[0177] The method of the embodiment of the present invention, in view of the fact that the resource optimization problem model of the constructed aerial network belongs to a non-convex optimization problem, utilizes the advantages of the deep reinforcement learning algorithm in solving non-convex optimization problems. By adopting the deep reinforcement learning algorithm to solve the resource optimization problem model, the resources of the aerial network can be effectively optimized and allocated according to the network information after the aerial network is deployed, thereby realizing the energy efficiency optimization of the emergency aerial network.
[0178] The application of the aerial network resource allocation method of the embodiment of the present invention can also be adaptively applied in the networking of 6G heterogeneous aerial emergency networks, which is conducive to realizing autonomous and energy-efficient optimized deployment of 6G emergency aerial networks for energy efficiency optimization.
[0179] The air network resource allocation device provided by the present invention is described below. The air network resource allocation device described below and the air network resource allocation method described above can be referred to each other.
[0180] Figure 2 is a schematic diagram of the structure of the air network resource allocation device provided by the present invention, such as Figure 2 As shown, including:
[0181] Establishing module 210, for establishing a resource optimization problem model of the air network with the goal of maximizing the energy efficiency of the air network and combining the target constraints; the air network is constructed based on multiple types of low-altitude platform equipment;
[0182] The allocation module 220 is used to solve the resource optimization problem model using a deep reinforcement learning algorithm to determine the target deployment position, target transmission power and channel allocation information of each low-altitude platform equipment in the aerial network; multiple types of low-altitude platform equipment include aerial base stations, aerial radio frequency units and millimeter-wave enhanced drones; channel allocation information is the channel information between the terminal and each low-altitude platform equipment.
[0183] The air network resource allocation device described in this embodiment can be used to execute the above-mentioned air network resource allocation method embodiment. Its principles and technical effects are similar and will not be repeated here.
[0184] The aerial network resource allocation device provided by the embodiment of the present invention constructs an aerial network by adopting a network deployment mechanism in which multiple types of low-altitude platform equipment jointly participate. The multiple types of low-altitude platform equipment include aerial base stations, aerial radio frequency units and millimeter-wave enhanced drones. Based on the large-scale coverage capability of the aerial base station, the network coverage expansion capability of the aerial radio frequency unit and the capacity expansion capability of the millimeter-wave enhanced drone, an emergency aerial network with high energy efficiency as the goal is formed, which can not only improve the communication coverage of the emergency aerial network restoration, but also realize local capacity enhancement and resource reuse; by maximizing the energy efficiency of the aerial network as the goal, combined with the target constraint conditions, a resource optimization problem model of the aerial network is established. In view of the fact that it belongs to a non-convex optimization problem, according to the advantages of the deep reinforcement learning algorithm in solving non-convex optimization problems, the resource optimization problem model of the deep reinforcement learning algorithm is adopted for solving, which can effectively optimize and allocate resources for the aerial network, determine the target deployment position, target transmission power and channel allocation information of each low-altitude platform equipment in the aerial network, realize the energy efficiency optimization of the emergency aerial network, ensure the performance of the aerial network, and save the network deployment cost.
[0185] Figure 3 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the air network resource allocation method provided by the above methods, the method comprising: taking maximizing the air network energy efficiency as the goal, combining the target constraint conditions, establishing the resource optimization problem model of the air network; the air network is constructed based on multiple types of low-altitude platform equipment; using a deep reinforcement learning algorithm to solve the resource optimization problem model, determine the target deployment position, target transmission power and channel allocation information of each terminal of each low-altitude platform equipment in the air network; the multiple types of low-altitude platform equipment include air base stations, air radio frequency units and millimeter wave enhanced drones; the channel allocation information is the channel information between the terminal and each of the low-altitude platform equipment.
[0186] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0187] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the air network resource allocation method provided by the above-mentioned methods, and the method includes: with the goal of maximizing the energy efficiency of the air network, combined with the target constraints, a resource optimization problem model of the air network is established; the air network is constructed based on multiple types of low-altitude platform equipment; a deep reinforcement learning algorithm is used to solve the resource optimization problem model to determine the target deployment position, target transmission power and channel allocation information of each terminal of each low-altitude platform equipment in the air network; the multiple types of low-altitude platform equipment include air base stations, air radio frequency units and millimeter wave enhanced drones; the channel allocation information is the channel information between the terminal and each of the low-altitude platform equipment.
[0188] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the air network resource allocation method provided by the above-mentioned methods, the method comprising: establishing an air network resource optimization problem model with the goal of maximizing the air network energy efficiency in combination with target constraints; the air network is constructed based on multiple types of low-altitude platform equipment; a deep reinforcement learning algorithm is used to solve the resource optimization problem model to determine the target deployment position, target transmission power and channel allocation information of each terminal of each low-altitude platform equipment in the air network; the multiple types of low-altitude platform equipment include air base stations, air radio frequency units and millimeter wave enhanced drones; the channel allocation information is the channel information between the terminal and each of the low-altitude platform equipment.
[0189] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0190] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for allocating air network resources, characterized in that: include: With the goal of maximizing the energy efficiency of the aerial network, a resource optimization problem model of the aerial network is established in combination with target constraints; the aerial network is built based on multiple types of low-altitude platform equipment; A deep reinforcement learning algorithm is used to solve the resource optimization problem model to determine the target deployment position, target transmission power and channel allocation information of each terminal of each low-altitude platform device in the aerial network; The multiple types of low-altitude platform equipment include aerial base stations, aerial radio frequency units and millimeter-wave enhanced drones; the channel allocation information is the channel information between the terminal and each of the low-altitude platform equipment.
2. The method for allocating air network resources according to claim 1, characterized in that: The above-mentioned goal is to maximize the energy efficiency of the air network, and in combination with the target constraints, a resource optimization problem model of the air network is established, including: Based on the transmission rate of each terminal and the channel allocation information of each terminal, the network capacity of the air network is determined, and based on the network capacity of the air network and the total deployment cost of the air network, the network capacity under unit cost is determined; the total deployment cost of the air network is determined based on the hardware cost and energy consumption cost of each low-altitude platform equipment; Taking the maximum network capacity under the unit cost as the optimization goal, an objective function is established; the air network energy efficiency includes the network capacity under the unit cost; Determining the target constraint condition based on the transmission power of each low-altitude platform device in the aerial network and the channel allocation information of each terminal; Based on the objective function and the objective constraint conditions, a resource optimization problem model of the air network is established.
3. The method for allocating air network resources according to claim 2, characterized in that: The deep reinforcement learning algorithm is used to solve the resource optimization problem model to determine the target deployment position, target transmission power and channel allocation information of each terminal of each low-altitude platform device in the air network, including: Based on the state information and reward function of each of the low-altitude platform devices in the aerial network, the deep Q network model in the deep reinforcement learning algorithm is iteratively trained to obtain the optimal action value benefit value of the aerial network energy efficiency; the reward function is determined based on the network capacity of the aerial network and the total deployment cost of the aerial network; Based on the optimal action value benefit value, the target deployment position, target transmission power and channel allocation information of each terminal of each low-altitude platform device in the aerial network are determined.
4. The method for allocating air network resources according to claim 3, characterized in that: Based on the state information of each of the low-altitude platform devices in the aerial network and the reward function, the deep Q network model in the deep reinforcement learning algorithm is iteratively trained to obtain the optimal action value benefit value of the aerial network energy efficiency, including: Step 1, establishing a state space and an action space according to the state information of each low-altitude platform device in the aerial network; Step 2, based on the state space, determining the initial state of the air network, and dividing the time of each training into multiple time intervals; the initial state is the state of the air network in the first time interval; Step 3, determining a first reward value after the air network performs a first action in a first state within a current time interval and a second state in a next time interval; the first action is determined based on the action space; Step 4, storing the first state, the first action, the first reward value and the second state as a data sample in a memory unit, and randomly extracting a data sample from the memory unit to update the network parameters of the deep Q network model; the first reward value is determined based on the reward function; Step 5, traversing the multiple time intervals, executing steps 3 to 4, completing an iterative training of the deep Q network model, and obtaining the maximum action value benefit value of the air network energy efficiency after training; Step 6, traverse the preset number of iterations, execute steps 3 to 5, train the deep Q network model for the preset number of iterations, and obtain the optimal action value benefit value of the air network energy efficiency after training; the optimal action value benefit value is the maximum action value benefit value of the air network energy efficiency obtained through the last iterative training.
5. The method for allocating air network resources according to claim 3, characterized in that: Based on the state information and reward function of each of the low-altitude platform devices in the aerial network, the deep Q network model in the deep reinforcement learning algorithm is iteratively trained to obtain the optimal action value benefit value of the aerial network energy efficiency, and further includes: Divide each training session into multiple intervals; Determine, based on the network capacity of the air network and the total deployment cost of the air network, a first increment of the network capacity in a current time interval relative to a previous time interval and a second increment of the total deployment cost in the current time interval relative to the previous time interval; Based on the ratio of the first increment to the second increment, a reward function in the deep reinforcement learning algorithm is determined.
6. The method for allocating air network resources according to claim 4, characterized in that: According to the state information of each low-altitude platform device in the aerial network, a state space and an action space are established, including: Establishing a state space according to the optional deployment position, optional transmission power and channel allocation information of each of the low-altitude platform devices; An action space is established according to the movement type of each low-altitude platform device, the adjustment type of the transmission power of each low-altitude platform device, and the access status information between each low-altitude platform device and each terminal.
7. The method for allocating air network resources according to any one of claims 1 to 6, characterized in that: The aerial base station is used to control the wireless coverage of the cell; the aerial radio frequency unit is used to expand the wireless coverage under the control of the aerial base station; The millimeter wave enhanced drone is used to expand the capacity of the aerial base station and the aerial radio frequency unit.
8. An air network resource allocation device, characterized in that: include: Establishing a module for establishing a resource optimization problem model of an aerial network with the goal of maximizing the energy efficiency of the aerial network and combining the target constraints; the aerial network is constructed based on multiple types of low-altitude platform equipment; An allocation module, used to solve the resource optimization problem model using a deep reinforcement learning algorithm to determine the target deployment position, target transmission power and channel allocation information of each terminal of each low-altitude platform device in the aerial network; The multiple types of low-altitude platform equipment include aerial base stations, aerial radio frequency units and millimeter-wave enhanced drones; the channel allocation information is the channel information between the terminal and each of the low-altitude platform equipment.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the air network resource allocation method according to any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for allocating air network resources as claimed in any one of claims 1 to 7 is implemented.
