Path Planning Method and Device for Multi-UAV-Assisted Communication in Deep Reinforcement Learning
Through the combination of deep reinforcement learning and millimeter wave radar, the problem of low path planning efficiency in a dynamic environment is solved, efficient path planning and signal coverage is achieved, training costs are reduced and the accuracy of ground user positioning is improved.
Patent Information
- Application Number
- CN202211453228.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-21
- Publication Date
- 2025-08-05
- Estimated Expiration
- 2042-11-21
AI Technical Summary
Traditional UAV path planning methods are inefficient in dynamic and unknown environments, making it difficult to plan effective collision-free paths, resulting in low intelligence and long solution time.
Using deep reinforcement learning methods, by building a virtual environment model, using millimeter-wave radar to detect obstacles and in combination with received signal strength indication (RSSI) to monitor the location of ground users in real time, train reinforcement learning neural networks to plan the optimal path.
It improves the efficiency of the path planning and signal coverage of the drone in complex environments, reduces training costs and improves the accuracy of positioning of ground users.
Smart Images
Figure CN115855055B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) flight control and wireless communication technology, and in particular to a path planning method and device for multi-UAV assisted communication using deep reinforcement learning. Background Art
[0002] With the development of 5G communication technology and the advancement of the Internet of Things (IoT), drone base stations are becoming widely used to assist in communication tasks. When fixed ground base stations fail, drone mobile base stations can provide temporary communication services to areas beyond the reach of ground signals. Drone aerial base stations offer high maneuverability, controllable mobility, flexible deployment, and support for high-speed data transmission, significantly improving the problem of signal transmission difficulties in complex environments. Given the unique characteristics of drone base stations and the complexity of mission environments, leveraging the drone's high mobility to plan an optimal collision-free path in complex flight paths with obstacles is a key challenge for drones in completing communication missions. Traditional path planning methods typically transform the drone's path planning problem into a path optimization problem, then solve an optimization model to determine the optimal flight path. These methods generally work well in static or known environments, but struggle to plan effective paths in dynamic or unknown environments.
[0003] Taking the above considerations into consideration, traditional UAV path planning methods often have problems such as low intelligence level, long solution time, and limited application scenarios, and cannot be applied to real scenarios with complex environments and changing scenarios. Summary of the Invention
[0004] In order to address the shortcomings of the existing technology and achieve the purpose of improving the efficiency of drone path planning, the present invention adopts the following technical solutions:
[0005] The path planning method for multi-UAV assisted communication based on deep reinforcement learning is implemented in a training phase that includes the following steps:
[0006] Step S101: Building a virtual environment model according to the actual task scenario;
[0007] Step S102: Detect the number and location information of obstacles around the drone; use millimeter wave radar to emit electromagnetic wave signals, and receive the reflected signals after the electromagnetic wave signals encounter obstacles. Compare the waveform of the transmitted signal with the waveform of the received signal to calculate the number and location coordinates of the obstacles.
[0008] Step S103: The location information of the ground user terminal is monitored in real time through the airborne base station of the UAV; the airborne base station of the UAV collects the received signal strength (RSSI) of the ground user terminal in real time. Since the power of the radio wave signal will attenuate with the increase of the propagation distance during the transmission of the radio wave in the medium, the distance between the UAV airborne base station and the ground user terminal is calculated based on the attenuation model constructed between the signal power and the distance according to the terminal's transmitted signal power and the received signal power of the UAV airborne base station, and the location information of the ground user is obtained.
[0009] Step S104: The obstacle information collected by the UAV and the location information of the ground user terminal are input into the reinforcement learning neural network. The obstacle information collected by the UAV during flight and the location information of the ground user are input into the reinforcement learning neural network as the state information S. The neural network obtains the next action A and the corresponding reward R based on the input state information. The neural network then completes the training of the neural network based on the obtained (S, A, R) sequence.
[0010] Step S105: The input multi-dimensional state information is classified and stored, and then model training is performed in the constructed virtual environment model to obtain the optimal path information through targeted training; the input state information S is composed of multi-dimensional state information, including: the position information of the UAV, the position information of the obstacle, the horizontal distance between the UAV and the obstacle, the distance between the UAV base station and the ground user terminal, the degree of approach of the UAV to the ground user terminal, and the number of ground user terminals that are effectively communicating with the UAV airborne base station. The input multi-dimensional state information is divided into three parts: position state, distance state and other states. These states are targetedly trained by the neural network, and the optimal flight path of the UAV is obtained through multiple rounds of iteration.
[0011] Furthermore, the virtual environment model in step S101, including the system environment model, obstacle model, and ground user model, describes the environmental state information faced by the UAV when performing the auxiliary communication task, and also provides a training venue for the intelligent UAV;
[0012] The system environment model refers to the number N of UAV mobile base stations that perform auxiliary communication tasks in the set scene, and the area S of each scene model. xy ,The entire mission area is divided into multiple sub-task areas according to the ,topography of the scenario model, and then the size of each sub-area is ,dynamically adjusted based on the performance of N UAVs;
[0013] The obstacle model is a set of obstacles K in the scene, which is randomly generated. The virtual scene dynamically adjusts the number of obstacles K in the environment according to the size of the actual task area and randomly places the obstacles at any position in the scene.
[0014] The ground user model refers to setting the number of ground users to M and constructing the ground user model in a random generation method. Since the position coordinates of ground users are not fixed and are often in a moving state, the ground users in the mission scene are set to appear randomly at any position in the scene model, and the user's position coordinates will change with time.
[0015] Furthermore, the number of task areas obtained by segmenting the system environment model in step S101 is N′={1, 2, 3, ... N′}. Since each task area will be used by a drone to collect the location information of ground users, the number of drone base stations is equal to the number of segmented areas, that is, N′=N. The area of N′ subtask areas is The area S of the scene model xy Expressed as:
[0016]
[0017] The area of N′ subtasks obtained by segmenting the scene model The size of is dynamically adjusted according to the performance of N drones. The performance of drones is comprehensively evaluated based on multiple indicators such as drone endurance, flight speed, and climb rate. The performance of N drones in the scene is {I 1 ,I 2 ,I 3 ,…,I N}, the area of the task area obtained by segmentation Expressed as:
[0018]
[0019] Where i = {0, 1, 2…N′}.
[0020] Furthermore, in step S101, the process of detecting obstacles by the drone is as follows:
[0021] The radar mounted on the aircraft monitors the position of surrounding obstacles in real time; according to the distance d between the aircraft and the obstacles, ci Send feedback signal to the drone:
[0022]
[0023] Among them, d ciIndicates the distance between the drone and the obstacle, D c Indicates the danger radius of the drone's collision risk. When the horizontal distance d between the drone and the obstacle is ci Greater than distance d d = 15 meters, the threat of obstacles to the drone is minimal, and it is assumed that there are no obstacles around the drone, that is, f c =1; when the horizontal distance d ci Greater than the danger radius D of the drone c Less than distance d d = 15 meters, the drone is considered to be approaching an obstacle, i.e., f c =0; when the distance is less than the danger radius D c When the UAV is in a state of collision with an obstacle, the mission is considered to have failed, i.e., f c =-1.
[0024] Furthermore, in step S101, the UAV base station collects the received signal strength R from the ground user terminal in real time during the flight, and calculates the distance d between the ground user terminal and the UAV base station. a Finally, the drone base station will search the result f R Sent to ground control station, search results:
[0025]
[0026] Among them, when the received signal strength R between the ground user terminal and the drone base station exceeds the set threshold, that is, the distance d between the drone base station and the ground user terminal a Less than or equal to the maximum search radius D of the drone base station to the ground terminal r When the ground control console is able to locate the location information of the ground user terminal.
[0027] Furthermore, in step S103, the received signal strength calculation formula is as follows:
[0028]
[0029] Among them, R represents the receiving signal strength of the UAV to the ground user terminal during the flight, P r Indicates the power of the received signal;
[0030] The received signal strength between the terminal of ground user m and the base station on the UAV is:
[0031] R m =A-10nlgd am
[0032] Among them, R mIt represents the receiving signal strength of the UAV to the terminal of ground user m during the flight, d am The distance between the airborne base station u of the drone and the terminal of the ground user m is:
[0033]
[0034] Among them, abs(R m ) represents the absolute value of the received signal strength of the UAV to the ground user terminal m during the flight, A represents the received signal power when the UAV onboard base station and the ground user terminal are 1 meter apart, and n represents the signal propagation factor.
[0035] Furthermore, in step S104, the state information S, action A, and reward R are constructed as follows:
[0036] State information S: The state information S of the drone during flight = (p o ,p c ,d c ,d a ,t r_a ,c), where p o Indicates the two-dimensional coordinates of the drone's location (x s_o ,y s_o ), p c Represents the two-dimensional position coordinates (x s_c ,y s_c ), d c Indicates the horizontal distance between the drone and the obstacle, d a represents the distance between the UAV airborne base station and the ground user terminal, t r_a Indicates the degree of approach of the UAV to the ground user terminal at the current moment compared to the previous moment. The formula is as follows:
[0037] t r_a =d a (t)-d a (t-1)
[0038] Where t represents the current time, t-1 represents the previous time, and c represents the number of ground user terminals that can effectively communicate with the UAV airborne base station at the same time;
[0039] Action A: Since the UAV's flight altitude is fixed at a distance H from the ground, the UAV's flight action can be regarded as a two-dimensional plane flight problem. The flight action A includes (a1, a2, a3, a4), where a0 represents one step forward, a1 represents one step left, a2 represents one step right, and a3 represents one step backward.
[0040] Reward R: The real-time reward r obtained by the drone in the process of completing the flight mission is determined by the state s of the drone at that moment. t and the next action to be performed a t Therefore, the reward obtained at time t is expressed as r t (s t ,a t ), the reward r obtained by the intelligent drone at each moment t (s t ,a t ) including rewards for approaching obstacles Collision Rewards Close to target reward Drone-mounted base station throughput reward And drone raw material loss rewards Right now
[0041]
[0042] ε1+ε2+ε3+ε4+ε5=1
[0043] Among them, ε1, ε2, ε3, and ε4 respectively represent the weights of the rewards obtained by completing the corresponding target tasks.
[0044] A path planning device for multi-UAV assisted communication using deep reinforcement learning is applied in the training phase and includes a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the path planning method for multi-UAV assisted communication using deep reinforcement learning.
[0045] Furthermore, the path planning stage includes the following steps:
[0046] Step S201: Deploy the trained reinforcement learning neural network to the system equipment of the drone airborne base station, and put the tested drone into a real scene;
[0047] Specifically, the trained path planning algorithm model is deployed to the drone's system equipment. Afterwards, the drone, with the model deployed, is placed in a constructed real-world scenario for physical testing. Through the actual interaction between the drone's mobile base station and the environment, the stability and robustness of the deployed algorithm model in a real-world mission environment are tested.
[0048] Step S202: The UAV collects the location information of the ground user terminal in real time during the flight, and uses the UAV airborne base station to send the data information to the ground control terminal;
[0049] Specifically, while performing auxiliary communication tasks, the drone collects real-time RSSI data from the drone's onboard base station to the ground user's mobile phone. This data is then sent to the ground control terminal via the base station to calculate the ground user's location coordinates.
[0050] Step S203: obtaining the spatial position coordinates of the ground user through further analysis and calculation;
[0051] Specifically, the ground control unit further analyzes and calculates the ground user terminal's location information transmitted by the drone base station, ultimately obtaining the ground user terminal's three-dimensional spatial coordinates. The calculated ground user terminal location coordinates are then transmitted to the drone's onboard base station via wireless signals for subsequent flight path planning.
[0052] Step S204: The UAV plans a mission path in the actual scenario based on the environmental information and the reinforcement learning neural network.
[0053] Specifically, during flight, the drone collects and receives real-time environmental information about its surroundings and the location of users on the ground. This data is then packaged and categorized and fed into an algorithmic model. The algorithmic model analyzes and measures the drone's current environmental state and determines the optimal action plan for the next step. Based on this environmental information and the algorithmic model's calculations, the drone base station plans the optimal dynamic flight path for the actual mission scenario.
[0054] A path planning device for multi-UAV assisted communication using deep reinforcement learning is applied to the path planning stage and includes a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the path planning method for multi-UAV assisted communication using deep reinforcement learning.
[0055] The advantages and beneficial effects of the present invention are:
[0056] 1. A path decision algorithm based on deep reinforcement learning is used to classify and store multi-dimensional state information, and the optimal path information is obtained through targeted training. This greatly improves the efficiency of drones in completing auxiliary communication tasks and also improves the signal coverage rate of drone mobile base stations in the mission area.
[0057] 2. Using a self-built virtual environment model to complete the training of intelligent drones effectively avoids the training loss of physical drones and greatly reduces the cost of model training.
[0058] 3. Introducing the received signal strength indicator (RSSI) into the deep reinforcement learning algorithm model to influence the rewards obtained by the intelligent drone, while reducing the difficulty of drone decision-making and improving the positioning accuracy of the ground user's position coordinates. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 4 is a flow chart of a UAV path planning method according to an embodiment of the present invention.
[0060] Figure 2 FIG. 4 is a flowchart of obstacle detection in an embodiment of the present invention.
[0061] Figure 3 This is a schematic diagram of the principle of using millimeter wave radar for ranging in an embodiment of the present invention. DETAILED DESCRIPTION
[0062] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.
[0063] like Figure 1 As shown in Figure 1, a deep reinforcement learning-based path planning method for multi-UAV assisted communication is primarily used when ground users cannot communicate properly after a ground fixed base station fails. This method uses a reinforcement learning algorithm to plan paths for UAVs and uses the received signal strength indicator to influence the reward received by the agent. This reduces the decision-making difficulty of the UAVs and greatly improves the efficiency of path planning. It includes the following steps:
[0064] Step S101: Building a virtual environment model according to the actual task scenario;
[0065] Step S102: Using millimeter-wave radar to detect the number and location information of obstacles around the drone;
[0066] Step S103: monitoring the location information of the ground user's mobile phone in real time through the drone's airborne base station;
[0067] Step S104: input the obstacle information and ground user location information collected by the drone into the reinforcement learning neural network architecture;
[0068] Step S105: Classify and store the input multi-dimensional state information, and then perform model training in the constructed virtual model to obtain the optimal path information through targeted training;
[0069] Step S201: Deploy the trained reinforcement learning algorithm model to the system equipment of the drone base station, and put the tested drone into the real scene;
[0070] Step S202: The UAV collects the location information of the ground user in real time during the flight and sends the data information to the ground control terminal using the airborne base station;
[0071] Step S203: obtaining the spatial position coordinates of the ground user through further analysis and calculation;
[0072] Step S204: The UAV plans a mission path in the actual scenario based on the environmental information and the algorithm model;
[0073] Building a virtual environment model based on actual mission scenarios involves modeling the actual environment after a ground-based fixed base station failure. The entire mission model is constructed from three aspects: a system environment model, an obstacle model, and a ground user model. This model describes the environmental conditions encountered by the drone when performing auxiliary communication tasks and also provides a training environment for the intelligent drone.
[0074] In this embodiment, the system environment model in the virtual environment refers to setting the number of drone mobile base stations performing auxiliary communication tasks in the scene to N, N = {1, 2, 3, ... N}, and the area of the scene model S xy =L x ×L y , L x Indicates the horizontal length of the scene, L y Indicates the vertical length of the scene. The entire mission area will be divided into multiple sub-task areas according to the terrain and topography of the scene model, and then the size of each sub-area will be dynamically adjusted based on the performance of N drones.
[0075] Specifically, the number of task areas obtained by segmentation is N ′ ,N ′ = {1, 2, 3, ... N'}. Since each mission area will be used by a UAV to collect the location information of ground users, the number of UAV base stations is equal to the number of divided areas, that is, N' = N. The area of N' sub-task areas is The area S of the scene model xy It can be expressed as:
[0076]
[0077] The area of N′ subtasks obtained by segmenting the scene model The size of will be dynamically adjusted according to the performance of N drones. The performance of drones will be comprehensively evaluated based on multiple indicators such as drone endurance, flight speed, and climb rate. The performance of N drones in the scene is {I 1 ,I 2 ,I 3 ,…,IN}, the area of the task area obtained by segmentation It can be expressed as:
[0078]
[0079] Where i = {0, 1, 2…N′}.
[0080] In this embodiment, the obstacle model in the virtual environment is constructed by setting the number of obstacles in the scene to K, where K = {1, 2, 3, ..., K}, and using a random generation method. The virtual scene dynamically adjusts the number of obstacles K based on the size of the actual task area and randomly places the obstacles at any position in the scene.
[0081] The flow chart of drone’s obstacle detection is as follows: Figure 2 As shown in the figure, the drone will monitor the location of surrounding obstacles in real time through the millimeter-wave radars installed around the drone during flight. The drone’s signal processing module will detect the location of obstacles based on the distance d between the drone and the obstacle. ci To transmit feedback signal f to the drone c , signal f c It can be expressed as:
[0082]
[0083] Among them, d ci Indicates the distance between the drone and the obstacle, D c Indicates the danger radius of the drone where there is a collision risk. When the horizontal distance between the drone and the obstacle is greater than 15m, the obstacle poses little threat to the drone. At this time, it can be considered that there are no obstacles around the drone, that is, f = 1; when the distance is greater than the danger radius of the drone but less than 15m, it can be considered that the drone is approaching the obstacle, that is, f = 0; when the distance is less than the danger radius D c When the UAV is moving, it is possible for it to collide with an obstacle at any time. In this case, the mission can be considered as failed, that is, f = -1.
[0084] In this embodiment, the ground user model in the virtual environment is constructed by setting the number of ground users to M and employing a random generation method. Because the location coordinates of ground users are not fixed and are often in a state of movement, ground users in the mission scenario are randomly positioned at any location within the scenario model, and their location coordinates change over time.
[0085] During the flight, the drone base station will collect the received signal strength R from the ground user's mobile phone in real time, and calculate the distance d between the ground user and the drone base station. a Finally, the drone base station will search the result fR Sent to ground control station. Search results f R as follows:
[0086]
[0087] Among them, D r This value represents the maximum search radius of the drone base station for ground users. When the received signal strength between the ground user's phone and the base station exceeds the set threshold, meaning the distance between the drone base station and the ground user's phone is less than the maximum search radius, the ground control console can accurately locate the ground user.
[0088] In this embodiment, the use of millimeter wave radar to detect the number and location of obstacles around the drone refers to the electromagnetic wave signal emitted by the transmitting antenna of the millimeter wave radar. When the electromagnetic wave signal encounters the obstacle in front, it will be reflected and then received by the radar receiving antenna. Figure 3 As shown, the signal processing system will further compare and calculate the waveform of the transmitted signal and the waveform of the received signal, and finally obtain the number of obstacles in front and the specific position coordinates of the obstacles.
[0089] In this embodiment, the real-time monitoring of the location information of a ground user's mobile phone by the drone's onboard base station refers to the drone's onboard base station collecting the received signal strength (RSSI) of the ground user's mobile phone in real time. Since radio wave power attenuates with increasing propagation distance during transmission through a medium, the distance between the drone base station and the ground user's mobile phone can be calculated based on the node's transmitted signal power and the base station's received signal power, using an attenuation model based on signal power and distance, thereby obtaining the ground user's location information.
[0090] The drone carries an airborne base station and flies over the mission area, collecting the received signal strength between the ground user's mobile phone and the base station in real time. The RSSI calculation formula is as follows
[0091]
[0092] Among them, R represents the signal strength received by the drone to the ground mobile phone during flight, P r Indicates the power of the received signal.
[0093] The received signal strength indicator (RSSI) is mainly used to measure the distance between the transmitter and the receiver. Its ranging method is mainly based on the principle that the power of the radio wave signal will attenuate as the propagation distance increases during the transmission of radio waves in the medium. The RSSI between the mobile terminal of user m and the drone base station can be expressed as:
[0094] R m =A-10nlgd am
[0095] Among them, R m It represents the receiving signal strength of the UAV to the mobile phone terminal of the ground user m during the flight, d am Represents the distance between the drone base station and user m. The distance between the aerial drone base station u and the ground user m is:
[0096]
[0097] Among them, abs(R m ) represents the absolute value of the received signal strength of the drone to the mobile phone terminal of ground user m during flight, A represents the received signal power when the drone base station and the ground user's mobile phone are 1 meter apart, and n represents the signal propagation factor.
[0098] In this embodiment, the obstacle information and ground user location information collected by the drone are input into the reinforcement learning neural network architecture, which means that the obstacle information and ground user location information collected by the drone during flight will be input into the reinforcement learning neural network architecture as state information S. The algorithm network will obtain the next action A to be performed and the corresponding reward R based on the input state information, and then complete the subsequent training work based on the obtained (S, A, R) sequence.
[0099] In this algorithm, the process of a drone base station performing a task in a simulated scenario is modeled. The entire model consists of three parts: state S, action A, and reward R. Each part is constructed as follows:
[0100] (1) State: The state of the drone during flight S = (p o ,p c ,d c ,d a ,t r ,c). Among them, p o Indicates the two-dimensional coordinates of the drone's location (x s_o ,y s_o ), p c Represents the two-dimensional position coordinates (x s_c ,y s_c ), d c Indicates the horizontal distance between the drone and the obstacle, d a represents the distance between the drone base station and the ground user, t r_a Indicates the degree to which the drone is moving closer to the ground user at the current moment compared to the previous moment. The formula is as follows:
[0101] tr_a =d a (t)-d a (t-1)
[0102] c represents the number of ground users that can effectively communicate with the airborne base station at the same time.
[0103] (2) Action: Since the drone's flight altitude is fixed at a distance H from the ground, its flight action can be considered as a two-dimensional plane flight problem. The flight action A is composed of (a1, a2, a3, a4). a0 represents one step forward, a1 represents one step left, a2 represents one step right, and a3 represents one step backward.
[0104] (3) Reward: The real-time reward r obtained by the drone in the process of completing the flight mission is determined by the state s of the drone at that moment. t and the next action to be performed a t Therefore, the reward obtained at time t can be expressed as r t (s t ,a t ). The reward r obtained by the agent at each moment t (s t ,a t ) is rewarded by approaching an obstacle Collision Rewards Close to target reward Drone Base Station Throughput Rewards And drone raw material loss rewards It consists of five parts, namely
[0105]
[0106] Among them, ε1, ε2, ε3, and ε4 represent the weights of the rewards obtained by completing the corresponding target tasks, and the sum of all reward weights satisfies
[0107] ε1+ε2+ε3+ε4+ε5=1
[0108] In this embodiment, the input multidimensional state information is classified and stored, and then model training is performed within a constructed virtual model to obtain optimal path information through targeted training. This means that the input state S is composed of multidimensional state information, including: the drone's position information, the obstacle's position information, the horizontal distance between the drone and the obstacle, the distance between the drone base station and ground users, the drone's approach to ground users, and the number of ground users in effective communication with the aerial base station. This input multidimensional state information is divided into three parts: position state, distance state, and other state. These states are then trained on a neural network for targeted training. Subsequently, through multiple rounds of iterative training, the optimal flight path for the drone is obtained.
[0109] In this embodiment, deploying a trained reinforcement learning algorithm model to the drone base station's system equipment and deploying the tested drone into a real-world scenario involves deploying a trained path planning algorithm model to the drone's system equipment. Afterward, the deployed drone is placed into a constructed real-world scenario for physical testing. Through the drone's mobile base station's real-world interaction with the environment, the deployed algorithm model's stability and robustness are tested in a real-world mission environment.
[0110] In this embodiment, the drone collects real-time location information of ground users during flight and transmits this data to the ground control terminal using an onboard base station. This means that while performing auxiliary communication tasks, the drone collects the received signal strength indicator (RSSI) of the ground user's mobile phone from the drone's onboard base station in real time. This data is then transmitted to the ground control terminal via the base station and used to calculate the ground user's location coordinates.
[0111] In this embodiment, further analysis and calculation to determine the ground user's spatial coordinates refers to the ground control unit performing further analysis and calculation on the ground user's location information transmitted from the drone base station, ultimately obtaining the ground user's three-dimensional spatial coordinates. The calculated user location coordinates are then transmitted to the drone's aerial base station via wireless signals for subsequent flight path planning.
[0112] In this embodiment, the drone plans its mission path in a real-world scenario based on environmental information and an algorithmic model. During flight, the drone collects and receives real-time environmental information about the surrounding space and the location of users on the ground. This state data is then packaged and categorized and fed into the algorithmic model. The algorithmic model analyzes and measures the drone's current environmental state and determines the optimal action plan for the next step. Based on this environmental information and the results of the algorithmic model, the drone base station plans the optimal dynamic flight path in the actual mission scenario.
[0113] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A path planning method for multi-UAV assisted communication based on deep reinforcement learning, characterized by The training phase includes the following steps: Step S101: Building a virtual environment model according to the actual task scenario; Step S102: Detect the number and location information of obstacles around the drone; Step S103: monitoring the location information of the ground user terminal in real time through the airborne base station of the UAV; The drone's airborne base station collects the received signal strength of the ground user terminal in real time. Based on the terminal's transmitted signal power and the drone's airborne base station's received signal power, the distance between the drone's airborne base station and the ground user terminal is calculated using an attenuation model built between signal power and distance, thereby obtaining the ground user's location information. Step S104: inputting obstacle information collected by the UAV and location information of the ground user terminal into a reinforcement learning neural network; The obstacle information and the location information of the ground user collected by the drone during flight are input into the reinforcement learning neural network as the state information S. The neural network obtains the next action A and the corresponding reward R based on the input state information. The neural network is then trained based on the obtained (S, A, R) sequence. Step S105: The input multi-dimensional state information is classified and stored, and then model training is performed in the constructed virtual environment model to obtain the optimal path information through targeted training; the input state information S is composed of multi-dimensional state information, and the input multi-dimensional state information is divided into three parts: position state, distance state and other states. These states are targetedly trained by the neural network, and the optimal flight path of the UAV is obtained through multiple rounds of iteration.
2. The path planning method for multi-UAV assisted communication based on deep reinforcement learning according to claim 1 is characterized by: The virtual environment model in step S101 includes a system environment model, an obstacle model, and a ground user model; The system environment model refers to the number N of UAV mobile base stations that perform auxiliary communication tasks in the set scene, and the area S of each scene model. xy ,The entire mission area is divided into multiple sub-task areas according to the ,topography of the scenario model, and then the size of each sub-area is ,dynamically adjusted based on the performance of N UAVs; The obstacle model is a set of obstacles K in the scene, which is randomly generated. The virtual scene dynamically adjusts the number of obstacles K in the environment according to the size of the actual task area and randomly places the obstacles at any position in the scene. The ground user model is constructed by setting the number of ground users to M and adopting a random generation method.
3. The path planning method for multi-UAV assisted communication based on deep reinforcement learning according to claim 2 is characterized by: The system environment model in step S101 is divided into task areas of N′={1, 2, 3, ... N′}. Since each task area will be used by a drone to collect the location information of ground users, the number of drone base stations is equal to the number of divided areas, that is, N′=N. The area of N′ subtask areas is The area S of the scene model xy Expressed as: The area of N′ subtasks obtained by segmenting the scene model The size of is dynamically adjusted according to the performance of N drones. The performance of drones is comprehensively evaluated based on multiple indicators such as flight time, flight speed, and climb rate. The performance of N drones in the scene is {I 1 ,I 2 ,I 3 ,…,I N }, the area of the task area obtained by segmentation Expressed as: Where i = {0, 1, 2…N′}.
4. The path planning method for multi-UAV assisted communication based on deep reinforcement learning according to claim 1, characterized in that: In step S101, the process of detecting obstacles by the drone is as follows: The radar mounted on the aircraft monitors the position of surrounding obstacles in real time; according to the distance d between the aircraft and the obstacles, ci Send feedback signal to the drone: Among them, d ci Indicates the distance between the drone and the obstacle, D c Indicates the danger radius of the drone's collision risk. When the horizontal distance d between the drone and the obstacle is ci Greater than distance d d = 15 meters, it is assumed that there are no obstacles around the aircraft, that is, f c =1; when the horizontal distance d ci Greater than the danger radius D of the drone c Less than distance d d = 15 meters, the drone is considered to be approaching an obstacle, i.e., f c =0; when the distance is less than the danger radius D c When the task is considered to have failed, that is, f c =-1.
5. The path planning method for multi-UAV assisted communication based on deep reinforcement learning according to claim 1, characterized in that: In step S101, the UAV base station collects the received signal strength R from the ground user terminal in real time during the flight, and calculates the distance d between the ground user terminal and the UAV base station. a Finally, the drone base station will search the result f R Sent to ground control station, search results: Among them, when the received signal strength R between the ground user terminal and the drone base station exceeds the set threshold, that is, the distance d between the drone base station and the ground user terminal a Less than or equal to the maximum search radius D of the drone base station to the ground terminal r When the ground control console is able to locate the location information of the ground user terminal.
6. The path planning method for multi-UAV assisted communication based on deep reinforcement learning according to claim 1, characterized in that: In step S103, the received signal strength calculation formula is as follows: Among them, R represents the receiving signal strength of the UAV to the ground user terminal during the flight, P r Indicates the power of the received signal; The received signal strength between the terminal of ground user m and the base station on the UAV is: R m =A-10nlgd am Among them, R m It represents the receiving signal strength of the UAV to the terminal of ground user m during the flight, d am The distance between the airborne base station u of the drone and the terminal of the ground user m is: Among them, abs(R m ) represents the absolute value of the received signal strength of the UAV to the ground user terminal m during the flight, A represents the received signal power when the UAV onboard base station and the ground user terminal are 1 meter apart, and n represents the signal propagation factor.
7. The path planning method for multi-UAV assisted communication based on deep reinforcement learning according to claim 1, characterized in that: In step S104, the state information S, action A, and reward R are constructed as follows: State information S: The state information S of the drone during flight = (p o ,p c ,d c ,d a ,t r_a ,c), where p o Indicates the two-dimensional coordinates of the drone's location (x s_o ,y s_o ), p c Represents the two-dimensional position coordinates (x s_c ,y s_c ), d c Indicates the horizontal distance between the drone and the obstacle, d a represents the distance between the UAV airborne base station and the ground user terminal, t r_a Indicates the degree of approach of the UAV to the ground user terminal at the current moment compared to the previous moment. The formula is as follows: t r_a =d a (t)-d a (t-1) Where t represents the current time, t-1 represents the previous time, and c represents the number of ground user terminals that can effectively communicate with the UAV airborne base station at the same time; Action A: The flight action of the drone is considered as a two-dimensional plane flight problem. The flight action A includes (a1, a2, a3, a4), where a0 represents one step forward, a1 represents one step left, a2 represents one step right, and a3 represents one step backward. Reward R: The real-time reward r obtained by the drone in the process of completing the flight mission is determined by the state s of the drone at that moment. t and the next action to be performed a t Therefore, the reward obtained at time t is expressed as r t (s t ,a t ), the reward r obtained by the intelligent drone at each moment t (s t ,a t ) including rewards for approaching obstacles Collision Rewards Close to target reward Drone-mounted base station throughput reward And drone raw material loss rewards Right now ε1+ε2+ε3+ε4+ε5=1 Among them, ε1, ε2, ε3, and ε4 respectively represent the weights of the rewards obtained by completing the corresponding target tasks.
8. The path planning method for multi-UAV assisted communication using deep reinforcement learning according to any one of claims 1 to 7, characterized in that The path planning phase includes the following steps: Step S201: Deploy the trained reinforcement learning neural network to the system equipment of the drone airborne base station, and put the tested drone into a real scene; Step S202: The UAV collects the location information of the ground user terminal in real time during the flight, and uses the UAV airborne base station to send the data information to the ground control terminal; Step S203: obtaining the spatial position coordinates of the ground user through further analysis and calculation; Step S204: The UAV plans a mission path in the actual scenario based on the environmental information and the reinforcement learning neural network.
9. A path planning device for multi-UAV assisted communication based on deep reinforcement learning, applied in the training phase, characterized by: The method comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the method is used to implement the path planning method for multi-UAV assisted communication based on deep reinforcement learning according to any one of claims 1 to 7.
10. A path planning device for multi-UAV assisted communication based on deep reinforcement learning, applied in the path planning stage, characterized by: The method comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the method is used to implement the path planning method for multi-UAV assisted communication based on deep reinforcement learning as described in claim 8.