Air-based ad hoc network method and system based on deep reinforcement learning, terminal and medium

Abstract

The invention belongs to the technical field of air-based ad hoc networks, and discloses an air-based ad hoc network method and system based on deep reinforcement learning, a terminal and a medium, and the method comprises the steps: obtaining a first air-based state which comprises the position information of an air node, the position information of a ground node, and channel information; the first air-based state is input into a pre-trained deep reinforcement learning model to obtain actions, and the actions comprise signal transmitting power distribution, a moving track of an air node and relay communication resource scheduling; and executing the action to enable the air-based ad hoc network to be transferred to the second air-based state. According to the method, the air node track, the communication scheduling and the transmitting power can be jointly optimized, and the data transmission throughput of the air-based ad hoc network is maximized.

Description

Technical Field

[0001] The present invention relates to the technical field of air-based self-organizing networks, and specifically to an air-based self-organizing network method, system, terminal and medium based on deep reinforcement learning. Background Art

[0002] Air-based network is a type of network in the communication network that contains at least one aerial node participating in the networking. Air-based ad hoc network is an air-based network that adopts a self-organizing method for networking. It is a special application of mobile ad hoc network in air-based network. Its aerial nodes automatically establish an interconnected communication network without relying on infrastructure. The nodes in the network use a single hop or multi-hop method to exchange data. Especially in the military field, compared with the single-hop communication method of the current mainstream air-to-air data link, multi-hop communication expands the communication range between nodes, enhances the network's anti-destruction ability, and improves information sharing capabilities.

[0003] The different network resources provided by the aerial nodes in the airborne ad hoc network and the aerial communication environment make the airborne ad hoc network special. On the one hand, the coverage of the airborne ad hoc network is often large, while the number of aerial nodes is relatively small, that is, the density of aerial nodes is often sparse, so the signal transmission attenuation is large and the coverage blind spot is large; on the other hand, there are fewer obstacles in the air, which causes co-channel interference between adjacent aerial nodes. Therefore, in order to reduce coverage blind spots, reduce co-channel interference and prevent signal attenuation, it is necessary to reasonably design the trajectory of aerial nodes, adjust the signal transmission power and allocate channels. The trajectory of aerial nodes, signal transmission power and channel allocation affect each other, and together affect the data transmission throughput of the airborne ad hoc network.

[0004] It can be seen that how to jointly optimize aerial node trajectories, communication scheduling and transmission power to improve the data transmission throughput of air-based ad hoc networks has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In order to overcome the defects of the above-mentioned prior art, the purpose of the present invention is to provide an air-based self-organizing network method, system, terminal and medium based on deep reinforcement learning to solve the technical problems in the prior art.

[0006] The present invention is achieved through the following technical solutions: In a first aspect, the present invention provides an air-based self-organizing network method based on deep reinforcement learning, comprising: Acquire a first air-based state, where the first air-based state includes position information of an air node, position information of a ground node, and channel information; Inputting the first air-based state into a pre-trained deep reinforcement learning model to obtain actions, wherein the actions include signal transmission power allocation, movement trajectory of air nodes, and relay communication resource scheduling; Executing the actions causes the air-based ad hoc network to transfer to a second air-based state.

[0007] Preferably, the first air-based status also includes communication demand information.

[0008] Preferably, the relay communication resource scheduling includes time resource scheduling, frequency resource scheduling and power resource scheduling.

[0009] Preferably, the deep reinforcement learning model is trained by the following method: Initialize deep reinforcement learning models; Initialize the environment and noise function; By interacting and learning with the environment, interaction samples are obtained, and the experience pool is filled with the interaction samples, wherein the interaction samples include the first state, the first action, the reward value of executing the first action, and the second state of each aerial node at the first moment, and the second moment is the moment after the first moment; When the number of samples in the experience pool reaches the trigger value, some samples are randomly selected from the experience pool according to the sample weights as selected samples, and the importance sampling weights, target Q values, temporal difference TD errors, and policy gradients corresponding to the selected samples are calculated. The sample priority is updated according to the TD error, the evaluation network is updated by minimizing the TD deviation, and the policy network is updated according to the policy gradient; When the number of training times and the number of updates reach a preset relationship, the parameters in the two target networks are soft-updated.

[0010] Preferably, the initializing deep reinforcement learning model includes: Initialize the parameters in the strategy network and the evaluation network to random values, and copy the parameters to the corresponding target network at the same time; Initialize the experience pool, set the maximum number of rounds and the maximum number of steps per round; Among them, one execution of the iterative step is recorded as step number 1, and the number of steps is recalculated at the beginning of each round; The air-based self-organizing network method based on the deep reinforcement learning algorithm also includes: When the number of interaction and learning steps of the current round reaches the maximum number of steps, and the number of current rounds has not reached the maximum number of rounds, enter the next round, and re-execute from the steps of initializing the environment and the noise function. When the number of interaction and learning steps of the current round reaches the maximum number of steps, and the number of current rounds reaches the maximum number of rounds, end the training; Among them, the iterative steps include: initializing the environment and the noise function; obtaining interaction samples by interacting and learning with the environment, and filling the experience pool with the interaction samples, wherein the interaction samples include the first state, the first action, the reward value for executing the first action and the second state of each aerial node at the first moment, and the second moment is the moment after the first moment; when the number of samples in the experience pool reaches the trigger value, a part of the samples are randomly selected from the experience pool according to the sample weights as selected samples, and the importance sampling weights, target Q values, temporal difference TD errors, and policy gradients corresponding to the selected samples are calculated, the sample priority is updated according to the TD error, the evaluation network is updated by minimizing the TD deviation, and the policy network is updated according to the policy gradient; when the number of training times and the number of updates reach a preset relationship, the parameters in the two target networks are soft-updated.

[0011] Preferably, the step of obtaining interaction samples by interacting and learning with the environment, and filling the experience pool with the interaction samples, wherein the interaction samples include a first state of each aerial node at a first moment, a first action, a reward value for executing the first action, and a second state at a second moment, wherein the second moment is a moment after the first moment, obtaining interaction samples by interacting and learning with the environment, and filling the experience pool with the interaction samples, comprises: Obtaining, through the strategy network, the first actions of all aerial nodes at the first moment according to the noise function at the first moment and the first state of each of the aerial nodes; According to the first action and the probability provided by the environment, a second state of each of the aerial nodes at a second moment and a reward value for executing the first action are obtained, wherein the second moment is a moment after the first moment; Store the interaction samples in the experience pool and set their priorities.

[0012] Preferably, the aerial node includes a drone.

[0013] In a second aspect, the present invention provides an air-based self-organizing network system based on deep reinforcement learning, comprising: An air-based state acquisition module, used to acquire a first air-based state, wherein the first air-based state includes position information of an air node, position information of a ground node, and channel information; An intelligent module, configured to input the first air-based state into a pre-trained deep reinforcement learning model to obtain an action, wherein the action includes signal transmission power allocation, movement trajectory of air nodes, and relay communication resource scheduling; An action module is used to execute the action to transfer the air-based self-organizing network to a second air-based state. Preferably, the first air-based status also includes communication demand information.

[0014] Preferably, the relay communication resource scheduling includes time resource scheduling, frequency resource scheduling and power resource scheduling.

[0015] Preferably, the deep reinforcement learning model is trained by the following method: Initialize deep reinforcement learning models; Initialize the environment and noise function; By interacting and learning with the environment, interaction samples are obtained, and the experience pool is filled with the interaction samples, wherein the interaction samples include the first state, the first action, the reward value of executing the first action, and the second state of each aerial node at the first moment, and the second moment is the moment after the first moment; When the number of samples in the experience pool reaches the trigger value, some samples are randomly selected from the experience pool according to the sample weights as selected samples, and the importance sampling weights, target Q values, temporal difference TD errors, and policy gradients corresponding to the selected samples are calculated. The sample priority is updated according to the TD error, the evaluation network is updated by minimizing the TD deviation, and the policy network is updated according to the policy gradient; When the number of training times and the number of updates reach a preset relationship, the parameters in the two target networks are soft-updated.

[0016] Preferably, the initializing deep reinforcement learning model includes: Initialize the parameters in the strategy network and the evaluation network to random values, and copy the parameters to the corresponding target network at the same time; Initialize the experience pool, set the maximum number of rounds and the maximum number of steps per round; Among them, one execution of the iterative step is recorded as step number 1, and the number of steps is recalculated at the beginning of each round; The air-based self-organizing network method based on the deep reinforcement learning algorithm also includes: When the number of interaction and learning steps of the current round reaches the maximum number of steps, and the number of current rounds has not reached the maximum number of rounds, enter the next round, and re-execute from the steps of initializing the environment and the noise function. When the number of interaction and learning steps of the current round reaches the maximum number of steps, and the number of current rounds reaches the maximum number of rounds, end the training; Among them, the iterative steps include: initializing the environment and the noise function; obtaining interaction samples by interacting and learning with the environment, and filling the experience pool with the interaction samples, wherein the interaction samples include the first state, the first action, the reward value for executing the first action and the second state of each aerial node at the first moment, and the second moment is the moment after the first moment; when the number of samples in the experience pool reaches the trigger value, a part of the samples are randomly selected from the experience pool according to the sample weights as selected samples, and the importance sampling weights, target Q values, temporal difference TD errors, and policy gradients corresponding to the selected samples are calculated, the sample priority is updated according to the TD error, the evaluation network is updated by minimizing the TD deviation, and the policy network is updated according to the policy gradient; when the number of training times and the number of updates reach a preset relationship, the parameters in the two target networks are soft-updated.

[0017] Preferably, the step of obtaining interaction samples by interacting and learning with the environment, and filling the experience pool with the interaction samples, wherein the interaction samples include a first state of each aerial node at a first moment, a first action, a reward value for executing the first action, and a second state at a second moment, wherein the second moment is a moment after the first moment, obtaining interaction samples by interacting and learning with the environment, and filling the experience pool with the interaction samples, comprises: Obtaining, through the strategy network, the first actions of all aerial nodes at the first moment according to the noise function at the first moment and the first state of each of the aerial nodes; According to the first action and the probability provided by the environment, a second state of each of the aerial nodes at a second moment and a reward value for executing the first action are obtained, wherein the second moment is a moment after the first moment; Store the interaction samples in the experience pool and set their priorities.

[0018] Preferably, the aerial node includes a drone. In a third aspect, the present invention provides a mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the air-based self-organizing network method based on a deep learning algorithm as described in any one of the above items are implemented. In a fourth aspect, the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the air-based self-organizing network system method based on the deep learning algorithm as described in any one of the above items.

[0019] Compared with the prior art, the present invention has the following beneficial technical effects: The present invention provides an air-based self-organizing network method, system, terminal and medium based on deep reinforcement learning. Based on deep reinforcement learning, by obtaining the first air-based state including the position information of the air node, the position information of the ground node and the channel information, the action including the signal transmission power allocation, the moving trajectory of the air node and the relay communication resource scheduling is obtained, thereby jointly optimizing the trajectory of the air node, the communication scheduling and the transmission power, and improving the air-based self-organizing network data transmission throughput. In addition, the present invention can also solve the problem of large-scale air-based self-organizing network deployment with low complexity, while the conventional algorithm will have difficulty in processing the amount of calculation brought by the large state action space, and cannot complete the large-scale air-based self-organizing network deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings: Figure 1 This is a schematic diagram of an air-based self-organizing network; Figure 2 It is a simultaneous uplink and downlink collaborative transmission model for air-based ad hoc networks; Figure 3 A flow chart of the air-based self-organizing network method based on deep reinforcement learning provided by the present invention; Figure 4 A flow chart of deep reinforcement learning model training in the air-based self-organizing network method based on deep reinforcement learning provided by the present invention; Figure 5 A comparison diagram of the relationship between channel capacity and iteration between the present invention and the prior art; Figure 6 A comparison diagram of the blind spot positions and heights covered by the present invention and the prior art; Figure 7 This is a block diagram of the airborne self-organizing network system based on deep reinforcement learning provided by the present invention. DETAILED DESCRIPTION

[0021] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0022] The following detailed description is an exemplary description, which is intended to provide further detailed description of the present invention. Unless otherwise specified, all technical terms used in the present invention have the same meaning as those generally understood by those skilled in the art to which the present invention belongs. The terms used in the present invention are only for describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present invention.

[0023] The following is an introduction to the existing air-based self-organizing networks.

[0024] Please refer to Figure 1 and Figure 2 , Figure 1 This is a schematic diagram of an air-based ad hoc network. Figure 2 It is a simultaneous uplink and downlink cooperative transmission model for air-based ad hoc networks. Figure 1 As shown in the figure, the air-based self-organizing network includes aerial nodes and ground nodes, wherein the ground node is a network node located on the ground, which can be mobile or fixed, and the aerial node is a network node flying in the air. The ground node transmits data through the aerial node relay to realize data transmission between air-based self-organizing networks. There are fewer obstacles in the air that hinder signal propagation, and the aerial nodes usually move at high speed. The speed of aerial node movement and the distance between nodes are incomparable to ground nodes. The above reasons directly lead to the instability of node links and drastic changes in the network topology. Ground nodes such as battlefield sensor networks have tens of thousands of nodes, but the number of aerial nodes that make up the air-based self-organizing network is limited, generally dozens to more than a hundred. Moreover, the distribution range of aerial nodes is wide, and the network coverage can reach hundreds of square kilometers, so the node density in the air-based self-organizing network is sparse. As Figure 2 As shown in the figure, when the air-based ad hoc network is transmitting data, in the downlink system, the ground node not only receives useful signals from the corresponding air node, but also receives interference signals from nearby adjacent ground nodes, while in the uplink system, the air node not only receives useful signals from the corresponding ground node, but also receives interference signals from adjacent air nodes. In order to reduce the blind spots of air nodes, their motion trajectories should be optimized according to the ground nodes. In order to prevent signal attenuation from affecting the transmitted data, the signal transmission power should be set according to the distance between the air node and the ground node. In order to reduce interference, different channels should be scheduled for air nodes with similar positions. The trajectory of the air node, the signal transmission power and the channel allocation affect each other, and together affect the data transmission throughput of the air-based ad hoc network.

[0025] It can be seen that how to jointly optimize aerial node trajectories, communication scheduling and transmission power to improve the data transmission throughput of air-based ad hoc networks has become a technical problem that technical personnel in this field urgently need to solve.

[0026] The present invention is further described in detail below in conjunction with the accompanying drawings: Example 1 In order to solve the above problems, the present application embodiment provides an air-based self-organizing network method based on deep reinforcement learning. Figure 3 , Figure 3 The present invention provides a flow chart of the air-based self-organizing network method based on deep reinforcement learning.

[0027] like Figure 3 As shown, in step S1, a first air-based state is obtained, and the first air-based state includes air node position information, ground node position information, and channel information.

[0028] The aerial node position information refers to the spatial position of the aerial node, and the ground node position information refers to the spatial position of the ground node. In some embodiments, for ease of calculation, the aerial node position information and the ground node position information can be represented by the same coordinate system. Channel information is information about the communication channel between the aerial node and the ground node, including the frequency of the channel.

[0029] In some embodiments, the aerial node is a drone.

[0030] In some embodiments, the ground node is an immovable base station, and in other embodiments, the ground node is a movable mobile terminal.

[0031] In some embodiments, the aerial node includes a plurality of nodes, and in other embodiments, the number of the aerial node may be 1.

[0032] To meet the needs of special application scenarios, in some embodiments, the first air-based state also includes communication requirement information, where the communication requirement information refers to preset requirements for data transmission, such as a successful delivery rate or low latency requirement for air data.

[0033] In step S2, the first air-based state is input into a pre-trained deep reinforcement learning model to obtain actions, which include signal transmission power allocation, moving trajectory of aerial nodes, and relay communication resource scheduling.

[0034] An action refers to a specific action among all possible actions that an airborne ad hoc network can perform, including the movement trajectory of an aerial node moving in a certain direction, the power of a signal sent by an aerial node or a ground node, and the resources allocated by the airborne ad hoc network to an aerial node and a ground node for communication. Specifically, the relay communication resource scheduling includes time resource scheduling and frequency resource scheduling.

[0035] In step S3, the actions are executed to transfer the air-based ad hoc network to a second air-based state.

[0036] By executing the action, the air-based self-organizing network is transformed into the second air-based state, completing this state transition, thereby completing the construction of the air-based self-organizing network and improving the high throughput. By repeating the above steps, the air-based self-organizing network can always maintain a high throughput.

[0037] Thus, the present invention provides an air-based self-organizing network method, system, terminal and medium based on deep reinforcement learning. Based on deep reinforcement learning, by obtaining the first air-based state including the position information of the air node, the position information of the ground node and the channel information, the action including the signal transmission power allocation, the moving trajectory of the air node and the relay communication resource scheduling is obtained, thereby jointly optimizing the air node trajectory, communication scheduling and transmission power, and improving the air-based self-organizing network data transmission throughput. In addition, the present invention can also solve the problem of large-scale air-based self-organizing network deployment with low complexity, while the conventional algorithm will have difficulty in processing the amount of calculation brought by the large state action space, and cannot complete the large-scale air-based self-organizing network deployment.

[0038] In some embodiments, the deep reinforcement learning model described in step S2 is trained, please refer to Figure 4 , Figure 4 A flow chart of deep reinforcement learning model training in the air-based self-organizing network method based on deep reinforcement learning provided by the present invention.

[0039] like Figure 4 As shown, the deep reinforcement learning model on which the present invention is based is trained by the following method: In step S21, the deep reinforcement learning model is initialized.

[0040] In some embodiments, initializing the deep reinforcement learning model includes: In step S211, the parameters in the strategy network and the evaluation network are initialized to random values, and the parameters are simultaneously copied to the corresponding target network.

[0041] In step S212, the experience pool is initialized, and the maximum number of rounds and the maximum number of steps per round are set.

[0042] Among them, one execution of steps S23 to step S251 shown in the figure is recorded as step number 1, and the number of steps is recalculated at the beginning of each round; In step S22, the environment and the noise function for motion exploration are initialized.

[0043] In step S23, interaction samples are obtained by interacting and learning with the environment, and the experience pool is filled with the interaction samples. The interaction samples include the first state, the first action, the reward value for executing the first action, and the second state of each aerial node at the first moment, and the second moment is the moment after the first moment.

[0044] In some embodiments, acquiring interaction samples by interacting and learning with the environment, and filling the experience pool with the interaction samples includes: In step S231, the first actions of all aerial nodes at the first moment are obtained through the policy network according to the noise function at the first moment and the first state of each aerial node.

[0045] In step S232, based on the first action and the probability provided by the environment, the second state of each aerial node at the second moment and the return reward value of executing the first action are obtained, and the second moment is a moment after the first moment.

[0046] In step S233, the interaction samples are stored in the experience pool and their priorities are set.

[0047] In step S24, it is determined whether the number of samples in the experience pool reaches a trigger value, if so, step S241 is executed, if not, step S25 is executed.

[0048] The trigger value is a preset integer value.

[0049] In step S241, some samples are randomly selected from the experience pool as selected samples according to the sample weights, and the importance sampling weights, target Q values, TD errors (temporal difference), and policy gradients corresponding to the selected samples are calculated. The sample priorities are updated according to the TD errors, the evaluation network is updated by minimizing the TD deviation, and the policy network is updated according to the policy gradient.

[0050] In step S25, it is determined whether the current training times and updating times reach a preset relationship, if so, step S251 is executed, if not, step S26 is executed.

[0051] In some embodiments, In step S251, parameters in the two target networks are soft updated.

[0052] In step S26, it is determined whether the current number of steps is the maximum number of steps. If so, step S27 is executed; otherwise, step S23 is executed.

[0053] In step S27, determine whether the current round number is the maximum round number, if so, end, if not, execute step S22.

[0054] In this way, the deep reinforcement learning model that can be used in air-based ad hoc networks in the present invention can be trained. The effect of the deep reinforcement learning model trained by the above method can be referred to Figure 5 , Figure 6 ,in Figure 5 This is a comparison chart of the relationship between the channel capacity and iteration of the present invention and the prior art. Figure 6 This is a comparison diagram of the blind spot points and heights covered by the present invention and the prior art.

[0055] like Figure 5 As shown in the figure, a comparison chart of the channel capacity performance of the proposed algorithm and the traditional algorithm is given. It can be seen from the figure that the performance of the proposed algorithm is better than that of the traditional algorithm. The simulation results show that the channel capacity of the proposed algorithm is increased from 12.3Mbit / s and 15.7Mbit / s of the comparison algorithm to 32.4Mbit / s, with performance improvements of 57.14% and 48.57% respectively, which can support the wide-area coverage of long-distance power drone inspections. Figure 6 As shown in the figure, the performance comparison of the coverage blind area ratio of the proposed algorithm and the traditional algorithm is given. It can be seen from the figure that the performance of the proposed algorithm is better than that of the traditional algorithm. The simulation results show that the coverage blind area ratio of the proposed algorithm is reduced from 3.75% and 2.46% of the comparison algorithm to 2.12%, and the performance improvement is 37.5% and 21.4% respectively, which effectively reduces the coverage blind area of ​​the drone.

[0056] In some embodiments, the deep learning algorithm in the embodiments of the present invention adopts the DDPG algorithm; the DDPG algorithm is a deep reinforcement learning algorithm used to solve continuous action space problems, and its theoretical basis is the Markov decision process, which is represented by a four-tuple (s, a, r, t).

[0057] (1) State space s: contains the UAV’s location information, channel information, the location of the ground nodes, and communication requirements; (2) Action space a: It contains all the decisions that the UAV can make at time t, including signal transmission power allocation, UAV movement trajectory planning and relay communication resource scheduling; (3) Reward function r: The reward value obtained when the drone, as an intelligent agent, takes action a in state s and moves to state s'; (4) State transition function t: The probability that the drone takes action a in state s and transitions to state s'; given by the communication environment.

[0058] Algorithm flow: When the algorithm starts executing, the parameters of the evaluation network and the policy network will be randomly initialized. After initialization, the parameters will be copied to the corresponding target network at the same time, and then the experience pool D will be initialized. The algorithm will be executed for a maximum of K rounds, and each round will have a maximum of i steps of interaction and learning with the environment. At the beginning of each round, the environment and the noise function used for action exploration will be initialized.

[0059] In each step of interaction, the algorithm first determines the action at of each drone at this moment based on the observed state quantity and noise function of each drone through the policy network. After obtaining the actions of all drones, the environment state at the next moment is confirmed, and the state observation quantity St of each drone in the environment and the reward value rt of executing the action at are obtained. Then the interaction samples are stored in the experience pool D and its priority is set.

[0060] After completing the above process, if the number of samples in the experience pool reaches a certain value, a certain number of samples are randomly drawn from the experience pool according to the sample weight, and the corresponding importance sampling weights and TD errors are calculated, and the sample priority is updated according to the TD error. Then the target Q value is calculated using the extracted samples, the evaluation network is updated by minimizing the TD deviation, and the policy network is updated according to the policy gradient calculated from the small batch samples. If the current number of training steps is a multiple of the update step, the parameters in the two target networks are soft-updated. Then enter the next step of interacting with the environment, repeating the process of interaction and learning until the end of the round.

[0061] Repeat the above process of running each round until the number of rounds reaches the maximum value and the training is completed.

[0062] Example 2 according to Figure 7 As shown, the present invention provides an air-based self-organizing network system based on a deep learning algorithm, comprising: The air-based state acquisition module 1 is used to acquire a first air-based state, wherein the first air-based state includes the position information of the air node, the position information of the ground node, and the channel information; Intelligent module 2, used for inputting the first air-based state into a pre-trained deep reinforcement learning model to obtain actions, wherein the actions include signal transmission power allocation, movement trajectory of air nodes, and relay communication resource scheduling; The action module 3 is used to execute the action to transfer the air-based ad hoc network to the second air-based state. Example 3 The present invention also provides a mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the air-based self-organizing network method based on a deep learning algorithm as described in any one of the above items are implemented. Example 4 The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the air-based self-organizing network system method based on the deep learning algorithm as described in any of the above items.

[0063] In summary, the present invention provides an air-based self-organizing network method, system, terminal and medium based on deep reinforcement learning. Based on deep reinforcement learning, by obtaining the first air-based state including the position information of the air node, the position information of the ground node, and the channel information, the actions including signal transmission power allocation, the moving trajectory of the air node, and the relay communication resource scheduling are obtained, thereby jointly optimizing the air node trajectory, communication scheduling and transmission power, and improving the air-based self-organizing network data transmission throughput. In addition, the present invention can also solve the problem of large-scale air-based self-organizing network deployment with low complexity, while conventional algorithms will have difficulty in handling the amount of calculation brought by the large state action space, and cannot complete large-scale air-based self-organizing network deployment.

[0064] It is known from common technical knowledge that the present invention can be implemented by other embodiments that do not deviate from its spirit or essential features. Therefore, the above disclosed embodiments are only illustrative in all respects and are not exclusive. All changes within the scope of the present invention or within the scope equivalent to the present invention are included in the present invention.

[0065] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0067] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A space-based self-organizing network method based on deep reinforcement learning, characterized in that: include: Acquire a first air-based state, where the first air-based state includes position information of an air node, position information of a ground node, and channel information; Inputting the first air-based state into a pre-trained deep reinforcement learning model to obtain actions, wherein the actions include signal transmission power allocation, movement trajectory of air nodes, and relay communication resource scheduling; Executing the actions causes the air-based ad hoc network to transfer to a second air-based state.

2. The air-based self-organizing network method based on deep reinforcement learning according to claim 1, characterized in that: The first air-based state also includes communication requirement information.

3. The air-based self-organizing network method based on deep reinforcement learning according to claim 1, characterized in that: The relay communication resource scheduling includes time resource scheduling, frequency resource scheduling and power resource scheduling.

4. The air-based self-organizing network method based on deep reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning model is trained by the following method: Initialize deep reinforcement learning models; Initialize the environment and noise function; By interacting and learning with the environment, interaction samples are obtained, and the experience pool is filled with the interaction samples, wherein the interaction samples include the first state, the first action, the reward value of executing the first action, and the second state of each aerial node at the first moment, and the second moment is the moment after the first moment; When the number of samples in the experience pool reaches the trigger value, some samples are randomly selected from the experience pool according to the sample weights as selected samples, and the importance sampling weights, target Q values, temporal difference TD errors, and policy gradients corresponding to the selected samples are calculated. The sample priority is updated according to the TD error, the evaluation network is updated by minimizing the TD deviation, and the policy network is updated according to the policy gradient; When the number of training times and the number of updates reach a preset relationship, the parameters in the two target networks are soft-updated.

5. The air-based self-organizing network method based on deep reinforcement learning as claimed in claim 4, characterized in that: The initialization of the deep reinforcement learning model includes: Initialize the parameters in the strategy network and the evaluation network to random values, and copy the parameters to the corresponding target network at the same time; Initialize the experience pool, set the maximum number of rounds and the maximum number of steps per round; Among them, one execution of the iterative step is recorded as step number 1, and the number of steps is recalculated at the beginning of each round; The air-based self-organizing network method based on the deep reinforcement learning algorithm also includes: When the number of interaction and learning steps of the current round reaches the maximum number of steps, and the number of current rounds has not reached the maximum number of rounds, enter the next round, and re-execute from the steps of initializing the environment and the noise function. When the number of interaction and learning steps of the current round reaches the maximum number of steps, and the number of current rounds reaches the maximum number of rounds, end the training; Among them, the iterative steps include: initializing the environment and the noise function; obtaining interaction samples by interacting and learning with the environment, and filling the experience pool with the interaction samples, wherein the interaction samples include the first state, the first action, the reward value for executing the first action and the second state of each aerial node at the first moment, and the second moment is the moment after the first moment; when the number of samples in the experience pool reaches the trigger value, a part of the samples are randomly selected from the experience pool according to the sample weights as selected samples, and the importance sampling weights, target Q values, temporal difference TD errors, and policy gradients corresponding to the selected samples are calculated, the sample priority is updated according to the TD error, the evaluation network is updated by minimizing the TD deviation, and the policy network is updated according to the policy gradient; when the number of training times and the number of updates reach a preset relationship, the parameters in the two target networks are soft-updated.

6. The air-based self-organizing network method based on deep reinforcement learning as claimed in claim 4, characterized in that: The step of obtaining interaction samples by interacting and learning with the environment, and filling the experience pool with the interaction samples, wherein the interaction samples include a first state, a first action, a reward value for executing the first action, and a second state of each aerial node at a first moment, wherein the second moment is a moment after the first moment, obtaining interaction samples by interacting and learning with the environment, and filling the experience pool with the interaction samples, includes: Obtaining, through the strategy network, the first actions of all aerial nodes at the first moment according to the noise function at the first moment and the first state of each of the aerial nodes; According to the first action and the probability provided by the environment, a second state of each of the aerial nodes at a second moment and a reward value for executing the first action are obtained, wherein the second moment is a moment after the first moment; Store the interaction samples in the experience pool and set their priorities.

7. The air-based self-organizing network method based on deep reinforcement learning according to any one of claims 1 to 6, characterized in that: The aerial node includes a drone.

8. A space-based self-organizing network system based on deep reinforcement learning, characterized in that: include: An air-based state acquisition module, used to acquire a first air-based state, wherein the first air-based state includes position information of an air node, position information of a ground node, and channel information; An intelligent module, used for inputting the first air-based state into a pre-trained deep reinforcement learning model to obtain an action, wherein the action includes signal transmission power allocation, movement trajectory of air nodes, and relay communication resource scheduling; An action module is used to execute the action to transfer the air-based self-organizing network to a second air-based state.

9. The air-based self-organizing network system based on deep reinforcement learning as claimed in claim 8, characterized in that: The first air-based state also includes communication requirement information.

10. The air-based self-organizing network system based on deep reinforcement learning according to claim 8, characterized in that: The relay communication resource scheduling includes time resource scheduling, frequency resource scheduling and power resource scheduling.

11. The air-based self-organizing network system based on deep reinforcement learning according to claim 8, characterized in that: The deep reinforcement learning model is trained by the following method: Initialize deep reinforcement learning models; Initialize the environment and noise function; By interacting and learning with the environment, interaction samples are obtained, and the experience pool is filled with the interaction samples, wherein the interaction samples include the first state, the first action, the reward value of executing the first action, and the second state of each aerial node at the first moment, and the second moment is the moment after the first moment; When the number of samples in the experience pool reaches the trigger value, some samples are randomly selected from the experience pool according to the sample weights as selected samples, and the importance sampling weights, target Q values, temporal difference TD errors, and policy gradients corresponding to the selected samples are calculated. The sample priority is updated according to the TD error, the evaluation network is updated by minimizing the TD deviation, and the policy network is updated according to the policy gradient; When the number of training times and the number of updates reach a preset relationship, the parameters in the two target networks are soft-updated.

12. The air-based self-organizing network system based on deep reinforcement learning according to claim 11, characterized in that: The initialization of the deep reinforcement learning model includes: Initialize the parameters in the strategy network and the evaluation network to random values, and copy the parameters to the corresponding target network at the same time; Initialize the experience pool, set the maximum number of rounds and the maximum number of steps per round; Among them, one execution of the iterative step is recorded as step number 1, and the number of steps is recalculated at the beginning of each round; The air-based self-organizing network method based on the deep reinforcement learning algorithm also includes: When the number of interaction and learning steps of the current round reaches the maximum number of steps, and the number of current rounds has not reached the maximum number of rounds, enter the next round, and re-execute from the steps of initializing the environment and the noise function. When the number of interaction and learning steps of the current round reaches the maximum number of steps, and the number of current rounds reaches the maximum number of rounds, end the training; Among them, the iterative steps include: initializing the environment and the noise function; obtaining interaction samples by interacting and learning with the environment, and filling the experience pool with the interaction samples, wherein the interaction samples include the first state, the first action, the reward value for executing the first action and the second state of each aerial node at the first moment, and the second moment is the moment after the first moment; when the number of samples in the experience pool reaches the trigger value, a part of the samples are randomly selected from the experience pool according to the sample weights as selected samples, and the importance sampling weights, target Q values, temporal difference TD errors, and policy gradients corresponding to the selected samples are calculated, the sample priority is updated according to the TD error, the evaluation network is updated by minimizing the TD deviation, and the policy network is updated according to the policy gradient; when the number of training times and the number of updates reach a preset relationship, the parameters in the two target networks are soft-updated.

13. The air-based self-organizing network system based on deep reinforcement learning according to claim 11, characterized in that: The step of obtaining interaction samples by interacting and learning with the environment, and filling the experience pool with the interaction samples, wherein the interaction samples include a first state, a first action, a reward value for executing the first action, and a second state of each aerial node at a first moment, wherein the second moment is a moment after the first moment, obtaining interaction samples by interacting and learning with the environment, and filling the experience pool with the interaction samples, includes: Obtaining, through the strategy network, the first actions of all aerial nodes at the first moment according to the noise function at the first moment and the first state of each of the aerial nodes; According to the first action and the probability provided by the environment, a second state of each of the aerial nodes at a second moment and a reward value for executing the first action are obtained, wherein the second moment is a moment after the first moment; Store the interaction samples in the experience pool and set their priorities.

14. The air-based self-organizing network system based on deep reinforcement learning according to any one of claims 8 to 13, characterized in that: The aerial node includes a drone.

15. A mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the air-based self-organizing network method based on deep reinforcement learning as described in any one of claims 1 to 7 are implemented.

16. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the air-based self-organizing network method based on deep reinforcement learning are implemented as described in any one of claims 1 to 7.

Images