Unmanned aerial vehicle adjustment method in forest fire scene

Abstract

The invention belongs to the technical field of mobile communication, and relates to an unmanned aerial vehicle adjustment method in a forest fire scene, which comprises the following steps: constructing a system model; the system model comprises an unmanned aerial vehicle model and a forest fire environment model; according to the unmanned aerial vehicle model and the forest fire environment model, constructing an optimization problem by taking minimization of energy consumption of the unmanned aerial vehicle as a target; the optimization problem is converted into a Markov decision process, the optimization problem is solved by adopting a deep reinforcement learning algorithm based on the Markov decision process, and the attitude of the unmanned aerial vehicle is obtained; according to the method, the unmanned aerial vehicle model and the forest fire environment model are constructed by comprehensively considering stroke, smoke and flame spreading in the forest fire scene, the optimization problem is constructed according to the unmanned aerial vehicle model and the forest fire environment model, and the target function of the optimization problem considers the total energy consumption of flight, hovering, data collection and transmission of the unmanned aerial vehicle. Therefore, the method adapts to the actual environment condition of forest fire, and the attitude of the unmanned aerial vehicle can be adjusted more effectively.

Description

Technical Field

[0001] The invention belongs to the technical field of mobile communications and relates to a method for adjusting a drone in a forest fire scene. Background Art

[0002] In recent years, drones have been increasingly used in civil and military fields due to their small size, simple structure and low cost, especially in dangerous scenarios such as disaster relief. Drones also play a big role in the detection and identification of forest fires. Forest resources play an indispensable role in human development and provide a good environment for human survival and development. At present, there are three methods for forest fire detection. One is the traditional observation tower. The advantage of this method is that the observation range is large and the effect is good, but there are blind spots in monitoring due to terrain restrictions. The second is video monitoring. This method has a large coverage and can monitor in real time, but it is expensive and has little effect after a fire occurs. The third is drone remote sensing technology. Drones have unique features such as 3D movement, small size, high flexibility, and low-altitude flight. They can be quickly and easily deployed, making them an effective tool for detecting and assessing wildfires, but drones have limited onboard energy and cannot work for a long time.

[0003] Considering the dynamic environmental constraints and uncertainties in forest fire scenarios, drones need to autonomously complete path planning, attitude adjustment, task collection, and other issues. Since drones have limited energy, they need to consider completing tasks before the energy is exhausted. Common drone attitude adjustment algorithms include PID control algorithm, Kalman filter algorithm (KF), model-based predictive control algorithm (MPC), reinforcement learning algorithm, etc.

[0004] However, the existing UAV adjustment methods in forest fire scenarios fail to fully consider the actual environment of the fire scene and the status of the UAV, and thus cannot adapt to the needs of the forest fire scene and complete the data collection and transmission tasks. Summary of the invention

[0005] In order to solve the above problems in the prior art, the present invention provides a method for adjusting a UAV in a forest fire scene, comprising:

[0006] S1. Constructing a system model; the system model includes: a drone model and a forest fire environment model;

[0007] S2, construct an optimization problem based on the UAV model and the forest fire environment model to minimize the energy consumption of the UAV;

[0008] S3. Convert the optimization problem into a Markov decision process, and use a deep reinforcement learning algorithm based on the Markov decision process to solve the optimization problem and obtain the acceleration and attitude of the drone.

[0009] Drone Model U k Includes: k = {q u (k),v u (k),a u (k),φ u (k),q target (k)}, k∈{1,2,…,K}; where k is the time slot of the UAV movement, K is the total number of time slots of the UAV movement, and q u (k) represents the position of the drone, v u (k) represents the speed of the drone, a u (k) is the acceleration of the UAV, Φ u (k) represents the attitude of the drone, q target (k) is the target position of the UAV.

[0010] The position of the drone u (k) = [x u (k),y u (k),z u (k)] T , the speed of the drone Drone posture The target position of the drone q target (k) = [x tree (k),y tree (k),z teee (k)] T ; where x u (k), y u (k), z u (k) are the positions of the drone on the x, y, and z axes, respectively. are the speeds of the drone on the x, y, and z axes, respectively. is the roll angle of the drone, θ u (k) is the pitch angle of the UAV, ψ u (k) is the yaw angle of the drone, x tree (k), y tree (k), z tree (k) are the target positions of the UAV on the x, y, and z axes, respectively.

[0011] The forest fire environment model includes: k ={f(k),w(k),s(k),q p (k),q tree(k)}, k∈{1,2,…,K}; where k is the time slot of the UAV movement, K is the total number of time slots of the UAV movement, f(k)={q f (k),R f (k)} is the flame model, q f (k) is the flame position, R f (k) is the fire spreading speed, w(k) = {q w (k),φ w (k),||v w (k)||} is the gust model, q w (k) is the wind position, φ w (k) is the wind direction, ||v w (k)|| is the wind speed, s(k)={q s (k),D s (k)} is the smoke model, q s (k) is the position of the smoke, D s (k) is the diffusion speed of smoke, q p (k) is the position of the firefighter, q tree (k) Location of trees.

[0012] The optimization problem constructed based on the UAV model and the forest fire environment model includes:

[0013] minE u (k)

[0014] st C1:v umin ≤||v u (k)||≤v umax

[0015] C2:d near ≤d(k)≤d far

[0016] C3:E u (k)≤E max

[0017]

[0018] C6:D s (k)T p <d ss

[0019] C7:R f (k)T p ≤d fs

[0020] Among them, C1 represents the speed limit of the drone, C2 represents the distance limit between the drone and the flame, C3 represents the energy limit of the drone, C4 represents the actual speed limit of the drone when the wind speed is medium or small, C5 represents the limit of the drone caused by smoke, C6 represents the distance limit between smoke and firefighters, and C7 represents the distance limit between flames and firefighters; E u (k) is the total energy consumed by the drone, v umin is the minimum speed of the drone, ||v u (k)|| is the speed of the drone when there is no wind, v umax is the maximum speed of the UAV, d(k) is the distance between the UAV and the flame, d near is the shortest distance between the drone and the flame, d far is the maximum distance between the drone and the flame, E max For maximum energy consumption, is the speed of the drone affected by wind speed, v wmid is the wind speed threshold of strong wind, x u (k), y u (k), z u (k) are the positions of the drone on the x, y, and z axes, respectively. f (k), y f (k), z f (k) are the positions of the flame on the x, y, and z axes, respectively. s (k), y s (k), z s (k) are the positions of the smoke on the x, y, and z axes, respectively, T p The time required for firefighters to complete the task at the initial location, d ss is the safe distance between smoke and firefighters, d fs A safe distance between the flame and firefighters.

[0021] Total energy consumed by the drone Among them, e u (i) is the total energy consumption of the UAV in time slot i, e com (i) is the computing energy consumption of the drone, e fly (i) is the flight energy consumption of the UAV, e hov (i) is the hovering energy consumption of the UAV, υ(i)∈{0,1} represents the state of the UAV, υ(i)=1 means that the UAV is in the flying state, υ(i)=0 means that the UAV is in the hovering state, ρ(i)∈{0,1} indicates whether the UAV calculates data, ρ(i)=1 means that the UAV calculates data, ρ(i)=0 means that the UAV does not calculate data, and i is the index of the time slot.

[0022] The speed of the drone affected by wind speed is:

[0023]

[0024] in, is the angle between the UAV flight path and the wind direction.

[0025] Converting the optimization problem into a Markov decision process involves:

[0026] Constructing a quadruple M=<S,A,r,γ> , where S is the state space, s k =(χ k ,ζ k ),s k ∈S,χ k =(q u (k),v u (k),Φ u (k),a u (k),e u (k)),ζ k =(q f (k),v f (k),φ f (k)); A is the action space, a k =(a u (k),Φ u (k)),a k ∈A; r is the reward function, γ is the discount rate, X k For drone status, k is the state of fire, s k is the state of time slot k, a k is the action in time slot k, e u (k) is the total energy consumption of the UAV in time slot k.

[0027] The reward value r for time slot k k =r energy,k +r crash,k +r target,k +r fire,k +r task,k , where r energy,k is the reward value of the drone’s energy consumption, r crash,k is the reward value of the collision rate between the drone and obstacles, the obstacles are trees and smoke, r target,k is the reward value of the distance between the drone and the target position, r fire,k is the reward value of the distance between the drone and the flame, r task,k The reward value for completing the task.

[0028] Based on the Markov decision process, the deep reinforcement learning algorithm is used to solve the optimization problem, including: constructing a DQN network, which includes: a Q network and a target Q network; obtaining the current state sk , the current state s k Input Q network and get action a k , perform action a k Get the state s at the next moment k+1 and reward r k+1 , will (s k ,a k ,r k+1 ,s k+1 ) is stored as an experience sample in the experience replay buffer D; a batch of experience samples are extracted from the experience replay buffer D to train the DQN network to obtain a trained DQN network; the acceleration and attitude of the drone are obtained according to the DQN network.

[0029] Beneficial effects:

[0030] 1. The present invention comprehensively considers the impact of wind, smoke, flame spread and flight altitude on the UAV's mission execution in forest fire scenes to construct an optimization problem. The objective function of the optimization problem considers the total energy consumption of the UAV's flight, hovering, collection and transmission of data to adapt to the actual forest fire environment, and the UAV can be adjusted more effectively; 2. UAVs need to face a changing environment, so the present invention expresses the optimization problem as a Markov decision process, and adopts a reinforcement learning algorithm based on a deep Q network to solve the optimization problem, so as to better resist external interference and adjust the UAV more effectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flowchart of a method for adjusting a drone in a forest fire scenario provided by an embodiment of the present invention;

[0032] Figure 2 A schematic diagram of the posture of a drone provided by an embodiment of the present invention;

[0033] Figure 3 A schematic diagram of the force conditions of a drone provided by an embodiment of the present invention;

[0034] Figure 4 A stereogram of the trajectory route of the drone provided by an embodiment of the present invention;

[0035] Figure 5 A top view of the trajectory of the drone provided by an embodiment of the present invention;

[0036] Figure 6 An average reward value graph of a DQN network provided by an embodiment of the present invention;

[0037] Figure 7 This is a graph showing the average success rate of the drone provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] like Figure 1 As shown, the present invention adopts a method for adjusting a UAV in a forest fire scenario, comprising:

[0040] S1. Constructing a system model; the system model includes: a drone model and a forest fire environment model;

[0041] S2, constructing the optimization problem based on the UAV model and the forest fire environment model;

[0042] S3. Convert the optimization problem into a Markov decision process, and use a deep reinforcement learning algorithm based on the Markov decision process to solve the optimization problem and obtain the acceleration and attitude of the drone.

[0043] The entire period of UAV movement is divided into K time slots, and the current time slot k∈{1,2,…,K}, then the UAV model includes: U k = {q u (k),v u (k),a u (k),Φ u (k),q target (k)}; where q u (k) = [x u (k),y u (k),z u (k)] T represents the position of the drone, x u (k), y u (k), z u (k) are the positions of the drone on the x, y, and z axes, respectively; Indicates the speed of the drone, are the speeds of the drone on the x, y, and z axes, respectively, u (k) is the acceleration of the UAV; Figure 2 As shown in the figure, the posture of the drone is related to the rotation of its fuselage around the x, y, and z axes. Indicates the attitude of the drone. is the roll angle of the drone, θ u (k) is the pitch angle of the UAV, ψ u (k) is the yaw angle of the UAV, q target (k) = [xtree (k),y tree (k),z teee (k)] T is the target position of the UAV, x tree (k), y tree (k), z tree (k) are the target positions of the UAV on the x, y, and z axes, respectively.

[0044] The forest fire environment model includes: k ={f(k),w(k),s(k),q p (k),q tree (k)}, k∈{1,2,…,K}; f(k) is the flame model, w(k) is the gust model, s(k) is the smoke model, q p (k) is the position of the firefighter, q tree (k) Location of trees.

[0045] The flame model includes: f(k) = {q f (k),R f (k)},q f (k) = [x f (k),y f (k),z f (k)] T is the flame position, i.e. the geometric center of the outermost flame in a forest fire, x f (k), y f (k), z f (k) are the positions of the flame on the x, y, and z axes, respectively. is the fire spread speed, where the initial speed of the fire is R f (0) = r f , K s is the correction coefficient of the fuel configuration pattern, and its size is related to the vegetation type and vegetation interval in the forest fire. w is the wind correction coefficient, and its magnitude is related to the current wind force. is the current terrain slope.

[0046] In forest fire scenes, gusts are the main wind, which have a great impact on the operation of drones; the gust model includes: w(k) = {q w (k),Φ w (k),||v w (k)||}, the center point of the wind is represented as the wind position, defined as q w (k) = [x w (k),y w (k),z w (k)]T , x w (k), y w (k), z w (k) are the positions of the gusts on the x, y, and z axes respectively, and the wind direction is expressed as θ w (k),ψ w (k) are the roll angle, pitch angle, and yaw angle of the gust, respectively. The average wind speed is defined as v ref (k), the average height is defined as h ref (k), the wind speed at any height z can be expressed by its average wind speed and height, defined as Among them, p is an empirical index, its value depends on the current air environment, and its value is usually p∈[0.4,0.6].

[0047] There is a lot of smoke in the forest fire scene, which will affect the information collection of the drone. The smoke model includes: s(k) = {q s (k),D s (k)}, the position of the smoke is denoted by q s (k) = [x s (k),y s (k),z s (k)], where z s (k) represents the smoke height, and the smoke concentration is represented by C s (k), the diffusion speed of smoke is expressed as α represents the environmental impact factor, which is related to the current air environment.

[0048] In a forest fire scenario, drones need to send fire scene information to on-site firefighters and monitor their locations to reduce the risk of injury to the workers; the location of the firefighters is defined as: p =[x p (k),y p (k),z p (k)],x p (k), y p (k), z p (k) are the positions of the firefighters on the x, y, and z axes, respectively.

[0049] According to the impact of various factors on drones in forest fires, the conditional models of the impact of these factors are established, including:

[0050] (1) Flame-based conditional constraint model

[0051] When the drone is far away from the fire, it cannot observe the situation at the fire scene. When the drone is close to the fire scene, the drone will be affected by the temperature of the fire scene and cannot work normally. Therefore, the drone needs to keep a certain distance from the fire scene. This distance is defined as the complete distance, which is expressed as:

[0052] d near ≤d(k)≤d far

[0053]

[0054] Among them, d near The shortest distance that the drone can avoid the influence of flame temperature, d far It represents the maximum distance at which the UAV can observe the situation at the fire scene, and d(k) represents the distance between the UAV and the flame.

[0055] (2) Gust-based conditional constraint model

[0056] When the drone is performing a mission, if the data collection line of sight is blocked by smoke and the scene cannot be photographed, or the distance is too far due to the spread of the fire, the drone's position needs to be adjusted by flying. If the current position can observe the fire scene and the distance is appropriate, it will remain in the hovering state. Whether in flight or hovering, wind often has the greatest impact on the drone. wmin is the threshold for no wind, assuming that when the wind speed ||v w (k)||≤v wmin It is defined as no wind when the wind has no effect on the drone; v wmid is the threshold value for strong winds, when the wind speed v wmin <||v w (k)||≤v wmid It is defined as light wind. The drone can offset the influence of wind by controlling the angle and speed. wmid <‖v w (k)‖ is defined as strong wind, and the drone needs to avoid it.

[0057] When the drone is flying, in the absence of wind, it chooses the shortest path to fly at a constant speed, with a speed of ||v u (k)||, the flight direction is φ u (k)=[0,0,ψ u (k)] T ; In the case of a tailwind, adjust the flight angle and speed of the drone, and weight it with the wind speed to allow the wind to assist the drone and increase the flight speed in, Indicates the angle between the flight path of the drone and the wind direction; in the case of headwind, if it is strong wind, choose to bypass the wind field or change the height of the drone z u (k) <h wlow or u (k)>h whigh ; If the wind is light, choose to drive against the wind, the speed of the drone

[0058] When the drone is in hovering, in the absence of wind, the drone remains stationary; in the case of light wind, the wind has little effect on the drone, and the drone can be kept in hovering by controlling its speed and direction, namely: u (k) = -v w (k), Φ u (k) = Φ w (k) indicates that the attitude of the drone is consistent with the wind direction, and the speed and direction of the drone are controlled to keep the drone in a hovering state. If it is a strong wind, the height of the drone is adjusted to move horizontally to leave the wind field.

[0059] (3) Smoke-based conditional constraint model

[0060] The main impact of smoke on drones is that they block their view of collecting fire data.

[0061]

[0062] It means that the drone's vision is not affected by smoke and the drone can still observe the fire scene. When the drone's vision is blocked, the drone's position needs to be changed.

[0063] (4) Conditional constraint model based on firefighters

[0064] In a forest fire scenario, the safety of firefighters is the most important thing. The drone needs to monitor the spread of the fire while also paying attention to the location information of the firefighters. Firefighters need to keep a certain distance from smoke and flames to ensure that they can evacuate in time while completing the firefighting mission. Assume that the time it takes for a firefighter to complete the mission at the initial location is T p The safe distances that smoke and flames need to maintain from firefighters are d ss and d fs , then D needs to be satisfied s (k)T p <d ss and R f (k)T p <d fs Otherwise, the drone needs to send information to the firefighters in time to ensure enough time to evacuate.

[0065] In one embodiment, when the fire spread speed is within a controllable range, the drone's monitoring target can be switched from the fire to the firefighters to ensure the safety of the firefighters, as shown in the following example:

[0066]

[0067] Where G(π,P) is a vector function, which represents multiple objective functions of the drone under path π and strategy P. f (π,P) means that the flame spread is the monitoring target, g 1 …g i It means that firefighters are monitored, and i represents the number of monitored firefighters.

[0068] Drone energy consumption is mainly divided into flight energy consumption, hovering energy consumption and computing energy consumption.

[0069] (1) Flight energy consumption

[0070] The flight power of a drone is related to the flight speed of the drone, the actual flight speed, the size of the blades, and the force applied. The actual flight speed of a drone is affected by the wind in the environment, and its size is expressed as:

[0071]

[0072] According to the 3D generalized propulsion energy consumption model (GPECM) of UAV flight, the flight power P of the UAV is fly (k) is expressed as:

[0073]

[0074] Among them, p b represents the blade profile power and is defined as:

[0075]

[0076] P i (k) represents the induced power, which is defined as:

[0077]

[0078] The force diagram of the drone is as follows Figure 3 As shown, T(k) represents the thrust of the drone, mg is the gravity, and D w (k) is the wind force, D a (k) is the air resistance. The wind force and air resistance are obtained by the sensors carried by the drone. The combined force is: F c =T(k)+D w (k)+D a (k)+mg, the thrust T(k) is expressed as:

[0079]

[0080] Among them, S FP is the equivalent flat area of ​​the UAV fuselage, c T is the area of ​​the drone disk, ρ is the air density, A u is the fan blade area of ​​the drone.

[0081] The flight time of the drone is defined as T fly According to the flight power of the UAV, the flight energy consumption of the UAV is obtained. fly (k) is:

[0082] e fly (k) = T fly P fly (k)

[0083] (2) Hovering energy consumption

[0084] When there is no wind, the wind speed is ||v w (k)||≤v wmin , set the parameter β(k) = 1, the hovering power of the drone is related to the weight of the drone itself m, the number of rotors n p (The selected drone is a quad-rotor drone) and radius r p and the current ambient air density ρ air Regarding the hovering power P of the drone hov (k) is defined as:

[0085]

[0086] When the drone is in a dynamic hovering state in a wind field, the hovering power at this time is obtained based on the flight power of the drone:

[0087]

[0088] At this time, according to the force analysis of the drone, its thrust is ||T(k)||=||F c -mg||.

[0089] The hovering time of the drone is defined as T hov , the parameter of whether there is wind is defined as β(k)∈{0,1}, β(k)=1 means no wind, β(k)=0 means wind. According to the hovering power and hovering time of the UAV, the hovering energy consumption of the UAV is obtained. hov (k) is:

[0090] e hov (k) = T hov [β(k)P hov (k)+(1-β(k))P′hov (k)]

[0091] (3) Calculate energy consumption

[0092] The computing energy consumption of drones includes the energy consumption of collecting information, calculating data, and sending data. Its size is related to the amount of data collected. The computing energy consumption is defined as e com (k).

[0093] The total energy consumed from the beginning to time k is Among them, e u (i) is the total energy consumption of the UAV in time slot i, e com (i) is the computing energy consumption of the drone, e fly (i) is the flight energy consumption of the UAV, e hov (i) is the hovering energy consumption of the drone, υ(i)∈{0,1} represents the state of the drone, υ(i)=1 means the drone is in flight state, υ(i)=0 means the drone is in hovering state, ρ(i)∈{0,1} represents whether the drone calculates data, ρ(i)=1 means the drone calculates data, ρ(i)=0 means the drone does not calculate data, i is the index of the time slot, thus the optimization problem is obtained:

[0094] minE u (k)

[0095] st C1:v umin ≤‖v u (k)||≤v umax

[0096] C2:d near ≤d(k)≤d far

[0097] C3:E u (k)≤E max

[0098]

[0099] C6:D s (k)T p <d ss

[0100] C7:R f (k)T p <d fs

[0101] Among them, C1 represents the speed limit of the drone, C2 represents the distance limit between the drone and the flame, C3 represents the energy limit of the drone, C4 represents the actual speed limit of the drone when the wind speed is medium or small, C5 represents the limit of the drone caused by smoke, C6 represents the distance limit between smoke and firefighters, and C7 represents the distance limit between flames and firefighters; umin is the minimum speed of the drone, ‖v u (k)‖ is the speed of the UAV when there is no wind, v umax is the maximum speed of the drone, E max For maximum energy consumption.

[0102] Converting the optimization problem into a Markov decision process involves:

[0103] Formulate the problem as a Markov decision process, represented by a set of four elements M =<S,A,R,γ> , where S represents the state space, A represents the action space, R represents the reward function, γ represents the discount rate and γ∈(0,1].

[0104] (1) State space

[0105] The state space S is composed of the drone states χ k and the target state ζ k To express, that is, s k =(χ k ,ζ k ), s k ∈S; χ k It represents the state information of the UAV at time k, which is represented by the current position, speed, attitude, acceleration and energy consumption of the UAV, and is defined as χ k =(q u (k),v u (k),Φ u (k),a u (k),e u (k)), e u (k) is the total energy consumption of the UAV in time slot k; ζ k represents the state information of the target at time k. The target information is represented by the flame position, spread direction and spread speed at the fire scene, and is defined as ζ k =(q f (k),v f (k),Φ f (k)).

[0106] (2) Action Space

[0107] The action space A represents the set of all actions that the drone can take, and the action at time k is defined as:

[0108] a k =(au (k),Φ u (k))

[0109] a k ∈A

[0110] Among them, a u (k) = v u ′(k) represents the acceleration of the drone at time k, Φ u (k) The posture of the drone at time k.

[0111] (3) Reward Function

[0112] In order for the drone to complete the goal, the reward value will be designed based on the drone energy consumption, drone collision rate, drone distance to the target, distance to the flame, and mission completion rate.

[0113] The reward value r of the drone’s energy consumption energy,k It is related to the flight and hovering state of the drone and the calculation of energy consumption, and is set to:

[0114]

[0115] The reward value r of the collision rate between the drone and the obstacle crash,k It is related to the distance between the drone and obstacles, including trees and smoke. The farther the drone is from the obstacle, the greater the reward value. crash,k Set to the distance between the drone's flight direction and the nearest obstacle, that is:

[0116] r crash,k =||q u -q block ||

[0117] Among them, q block (k) is the position of the obstacle closest to the UAV.

[0118] Determine the target point position of the drone. The closer the distance between the drone and the target point, the greater the reward value. The reward value r of the distance between the drone and the target point target,k Set to:

[0119]

[0120] Among them, q target (k) is the target position of the UAV.

[0121] The distance between the drone and the flame is related to the safety distance, so the reward value r of the distance between the drone and the flame is fire,k Set to:

[0122]

[0123] Task completion reward value r task,k It is related to whether the drone completes the mission, including drone crash, energy exhaustion, maximum step length, mission completion rate, etc. Each case is set to a large constant value, expressed as:

[0124]

[0125] Among them, crashing, energy exhaustion, and exceeding the maximum step length are critical situations, and the rewards are negative.

[0126] Optionally, the reward value r for the drone to successfully reach the target location success is 200, the reward value for a drone crash is r crash = -200, the reward value r for the drone to run out of energy noenergy = -100, the drone exceeds the maximum step length reward value r overstep is -100.

[0127] In summary, the total reward value is:

[0128] r k =r energy,k +r crash,k +r target,k +r fire,k +r task,k

[0129] The Deep Q-Network (DQN) reinforcement learning algorithm is used. The DQN network includes: a Q network and a target Q network. By finding a strategy π, for all state spaces s k and the action space a k All have long-term reward values ​​Q π (s k ,a k ) is maximized. DQN uses a deep neural network to approximate the Q function, namely Q π (s k ,a k ; θ), θ is the network parameter. The approximate value of the Q function is updated by the following steps:

[0130] Step 1, Initialization: Randomly initialize the network parameters θ.

[0131] Step 2: Agent interacts with the environment: Get the current state s k , the current state s k Input Q network and get action a k , perform action a k Get the state s at the next moment k+1 and reward r k+1 .

[0132] Step 3: Experience playback: The interaction data (s k ,a k ,r k+1 ,s k+1 ) is stored in the experience replay buffer D.

[0133] Step 4, target calculation: For each experience (s k ,a k ,r k+1 ,s k+1 ), calculate the target Y k :

[0134] Y k =r k+1 +γmaxQ(s k+1 ,a k |θ target )

[0135] Among them, θ target is a lagged copy of θ.

[0136] Step 5. Loss function: Define the loss function to measure the difference between the predicted value and the target value:

[0137]

[0138] Step 6: Parameter update: Update network parameters using gradient descent or other optimization algorithms:

[0139]

[0140] Among them, α represents the learning rate.

[0141] Step 7. Target network update: Regularly update θ target is the value of θ.

[0142] Step 8. Repeat: Repeat steps 2-7 until the stop condition is met.

[0143] In one embodiment, the experimental parameters are set according to the above model, as shown in Table 1:

[0144] Table 1 Model parameters

[0145] Training cycle Episodes = 20000 Discount Rate γ=0.9 Learning Rate α＝0.001 Greedy Probability 0.01 Maximum greedy times 10 Number of training drones 15 Map size 100×100×25

[0146] Figure 4 and Figure 5 This is a flight situation after the training is completed. The red dot indicates the target position. The target position in this example is (82, 68, 16). The black column indicates the trees in the environment, the white column indicates the smoke, and the trajectory diagram indicates the flight path of the drone. The experiment shows that the drone can successfully pass through the woods and smoke to reach the target position.

[0147] Figure 6 It is the average reward value. From this figure, we can see that the reward value of the DQN algorithm is relatively large, indicating that the DQN algorithm can well solve the problem studied in this invention.

[0148] Figure 7 It reflects the situation of the drone completing the task, collision, energy exhaustion and exceeding the step length. 15 drones were used for training. From this figure, it can be seen that the probability of the drone crashing (rate) is relatively large, the probability of the drone completing the goal (success) is about 40%, and the probability of energy exhaustion (none-energy) and exceeding the maximum step length (overstep) is almost 0. With the increase of the number of training episodes (episodes), the success rate of the drone (uav-success-rate) is also slowly increasing.

[0149] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for adjusting a drone in a forest fire scenario, characterized in that: include: S1. Build system model; The system model includes: a UAV model and a forest fire environment model; S2, construct an optimization problem based on the UAV model and the forest fire environment model with the goal of minimizing the energy consumption of the UAV; S3. Convert the optimization problem into a Markov decision process, and use a deep reinforcement learning algorithm based on the Markov decision process to solve the optimization problem and obtain the acceleration and attitude of the drone.

2. The method for adjusting a UAV in a forest fire scenario according to claim 1, characterized in that: Drone Model U k Includes: k = {q u (k),v u (k),a u (k),Φ u (k),q target (k)}, k∈{1,2,…,K}; where k is the time slot of the UAV movement, K is the total number of time slots of the UAV movement, and q u (k) represents the position of the drone, v u (k) represents the speed of the drone, a u (k) is the acceleration of the UAV, Φ u (k) represents the posture of the drone, q target (k) is the target position of the UAV.

3. The method for adjusting a UAV in a forest fire scenario according to claim 2, characterized in that: The position of the drone u (k) = [x u (k),y u (k),z u (k)] T , the speed of the drone Drone posture The target position of the drone q target (k) = [x tree (k),y tree (k),z teee (k)] T ; where x u (k), y u (k), z u (k) are the positions of the drone on the x, y, and z axes, respectively. are the speeds of the drone on the x, y, and z axes, respectively. is the rolling angle of the drone, θ u (k) is the pitch angle of the UAV, ψ u (k) is the yaw angle of the drone, x tree (k), y tree (k), z tree (k) are the target positions of the UAV on the x, y, and z axes, respectively.

4. The method for adjusting a UAV in a forest fire scenario according to claim 2, characterized in that: The forest fire environment model includes: k ={f(k),w(k),s(k),q p (k),q tree (k)}, k∈{1,2,…,K}; where k is the time slot of the UAV movement, K is the total number of time slots of the UAV movement, f(k)={q f (k),R f (k)} is the flame model, q f (k) is the flame position, R f (k) is the fire spreading speed, w(k) = {q w (k),Φ w (k),||v w (k)||} is the gust model, q w (k) is the wind position, Φ w (k) is the wind direction, ||v w (k)|| is the wind speed, s(k)={q s (k),D s (k)} is the smoke model, q s (k) is the position of the smoke, D s (k) is the diffusion speed of smoke, q p (k) is the position of the firefighter, q tree (k) Location of trees.

5. The method for adjusting a UAV in a forest fire scenario according to claim 4, characterized in that: The optimization problem constructed based on the UAV model and the forest fire environment model includes: Among them, C1 represents the speed limit of the drone, C2 represents the distance limit between the drone and the flame, C3 represents the energy limit of the drone, C4 represents the actual speed limit of the drone when the wind speed is medium or small, C5 represents the limit of the drone caused by smoke, C6 represents the distance limit between smoke and firefighters, and C7 represents the distance limit between flames and firefighters; E u (k) is the total energy consumed by the drone, v umin is the minimum speed of the drone, ||v u (k)|| is the speed of the drone when there is no wind, v umax is the maximum speed of the UAV, d(k) is the distance between the UAV and the flame, d near is the shortest distance between the drone and the flame, d far is the maximum distance between the drone and the flame, E max For maximum energy consumption, is the speed of the drone affected by wind speed, v wmid is the wind speed threshold of strong wind, x u (k), y u (k), z u (k) are the positions of the drone on the x, y, and z axes, respectively. f (k), y f (k), z f (k) are the positions of the flame on the x, y, and z axes, respectively. s (k), y s (k), z s (k) are the positions of the smoke on the x, y, and z axes, respectively, T p The time required for firefighters to complete the task at the initial location, d ss is the safe distance between smoke and firefighters, d fs A safe distance between the flame and firefighters.

6. The method for adjusting a UAV in a forest fire scenario according to claim 5, characterized in that: Total energy consumed by the drone Among them, e u (i) is the total energy consumption of the UAV in time slot i, e com (i) is the computing energy consumption of the drone, e fly (i) is the flight energy consumption of the UAV, e hov (i) is the hovering energy consumption of the UAV, υ(i)∈{0,1} represents the state of the UAV, υ(i)=1 means that the UAV is in the flying state, υ(i)=0 means that the UAV is in the hovering state, ρ(i)∈{0,1} indicates whether the UAV calculates data, ρ(i)=1 means that the UAV calculates data, ρ(i)=0 means that the UAV does not calculate data, and i is the index of the time slot.

7. The method for adjusting a UAV in a forest fire scenario according to claim 5, characterized in that: The speed of the drone affected by wind speed is: in, is the angle between the UAV flight path and the wind direction.

8. The method for adjusting a UAV in a forest fire scenario according to claim 5, characterized in that: Converting the optimization problem into a Markov decision process involves: Constructing a quadruple M=<S,A,r,γ> , where S is the state space, s k =(χ k ,ζ k ),s k ∈S,χ k =(q u (k),v u (k),Φ u (k),a u (k),e u (k)),ζ k =(q f (k),v f (k),φ f (k)); A is the action space, a k =(a u (k),Φ u (k)),a k ∈A; r is the reward function, γ is the discount rate, χ k For drone status, k is the state of fire, s k is the state of time slot k, a k is the action in time slot k, e u (k) is the total energy consumption of the UAV in time slot k.

9. The method for adjusting a UAV in a forest fire scenario according to claim 8, characterized in that: The reward value r for time slot k k =r energy,k +r crash,k +r target,k +r fire,k +r task,k , where r energy,k is the reward value of the drone’s energy consumption, r crash,k is the reward value of the collision rate between the drone and obstacles, the obstacles are trees and smoke, r target,k is the reward value of the distance between the drone and the target position, r fire,k is the reward value of the distance between the drone and the flame, r task,k The reward value for completing the task.

10. The method for adjusting a UAV in a forest fire scenario according to claim 8, characterized in that: Based on the Markov decision process, the deep reinforcement learning algorithm is used to solve the optimization problem, including: constructing a DQN network, which includes: a Q network and a target Q network; obtaining the current state s k , the current state s k Input Q network and get action a k , perform action a k Get the state s at the next moment k+1 and reward r k+1 , will (s k ,a k ,r k+1 ,s k+1 ) is stored as an experience sample in the experience replay buffer D; a batch of experience samples are extracted from the experience replay buffer D to train the DQN network to obtain a trained DQN network; The acceleration and attitude of the drone are obtained based on the DQN network.

Images