An Autonomous Image Navigation and Obstacle Avoidance Method for Unmanned Aerial Vehicles Based on Improved Reinforcement Learning

Abstract

This invention relates to an autonomous image navigation and obstacle avoidance method for unmanned aerial vehicles (UAVs) based on improved reinforcement learning. It proposes an image-based and experience pool data storage mechanism, namely the FRDDM-DQN method. In this invention, an agent that meets the requirements is trained using the FRDDM-DQN method; during task execution, the trained agent controls the UAV to achieve autonomous image navigation and obstacle avoidance. Beneficial effects: By introducing the Faster R-CNN model into the DQN algorithm and transforming the recognition results of the Faster R-CNN model, the ability of UAVs to perform autonomous image navigation and obstacle avoidance in complex environments is obtained. By introducing the experience pool data storage mechanism proposed in this invention into the DQN algorithm, the autonomous image navigation and obstacle avoidance capability of UAVs in complex environments is improved. The distributed training method reduces the retraining time when the application scenario changes.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of application of unmanned aerial vehicles, autonomous navigation and obstacle avoidance method, and relates to an unmanned aerial vehicle autonomous image navigation and obstacle avoidance method based on improved reinforcement learning, in particular to an unmanned aerial vehicle autonomous image navigation and obstacle avoidance method based on improved reinforcement learning. BACKGROUND

[0002] An unmanned aerial vehicle is an autonomous and semi-autonomous unmanned aerial vehicle with high mobility, good concealment and adaptability, which can replace humans to perform some dangerous military tasks. For example, unmanned aerial vehicles can replace humans to perform search and rescue on land and sea, and reconnaissance tasks. In some military scenarios, communication between the unmanned aerial vehicle and the ground station may be interfered, and the unmanned aerial vehicle cannot perform tasks in a manually controlled manner, so the unmanned aerial vehicle is required to have autonomous navigation and obstacle avoidance capabilities. In addition, in order to cope with military scenarios in which airborne radars cannot be used or fail, the unmanned aerial vehicle should have the ability to avoid obstacles through multiple sensors. Airborne photoelectric devices (such as airborne cameras) have been widely used in unmanned aerial vehicles at present due to their small size and light weight, especially in reconnaissance unmanned aerial vehicles and reconnaissance and strike integrated unmanned aerial vehicles. Therefore, the unmanned aerial vehicle should have the ability to autonomously navigate and avoid obstacles through images taken by the airborne camera.

[0003] Currently, some researches have achieved image-based navigation and obstacle avoidance. In the patent “Unmanned aerial vehicle visual image algorithm, obstacle avoidance steps and information fusion processing system” (Patent, publication number CN112286230A, publication date 2020.11.13), the navigation and obstacle avoidance of unmanned aerial vehicle are achieved through images collected by on-board camera. When encountering obstacles, the algorithm separates the tasks of obstacle avoidance and navigation, i.e., the obstacle avoidance algorithm calculates how to avoid obstacles, and after avoiding obstacles, it returns to the original navigation route to continue the task. This processing method may cause the action selected by the algorithm to avoid obstacles but not necessarily optimal for the navigation task, i.e., the selected action is not globally optimal. In “Deep Learning-based Monocular Obstacle Avoidance for Unmanned Aerial Vehicle Navigation in Tree Plantations: Faster Region-based Convolutional Neural Network Approach” (Journal of Intelligent & Robotic Systems (2021) 101:5), the Faster R-CNN model is used to extract obstacles in the image, and the obstacle avoidance of unmanned aerial vehicle is achieved. However, the obstacle avoidance strategy in this algorithm is based on human experience, and the limited strategy based on human experience is not necessarily the optimal strategy in all cases. Therefore, the current image-based navigation and obstacle avoidance algorithm has the problem that the action selection strategy is not globally optimal.

[0004] In addition, the autonomous navigation and obstacle avoidance problem can also be solved by a reinforcement learning-based method. This type of method does not require prior knowledge and can allow the agent to gradually find a globally optimal strategy suitable for the current rule (reward function) during training. In The autonomous navigation and obstacle avoidance for USVs with ANOA deep reinforcement learning method (Knowledge-Based Systems (2020) 5:196), autonomous navigation and obstacle avoidance of an unmanned ship in a simple simulation environment is achieved by a reinforcement learning algorithm, the DQN method with a convolutional neural network. In Robot obstacle avoidance method based on DoubleDQN network and deep reinforcement learning (patent, publication number CN109407676 A, publication date 2019.03.01), robot obstacle avoidance is achieved by a reinforcement learning algorithm, the DoubleDQN algorithm. Since the agent performs optimization by randomly sampling experiences from the experience pool, when the size of the application scenario of the above two algorithms changes, for example, the scenario in which the unmanned aerial vehicle performs tasks in the present invention is larger and the unmanned aerial vehicle travels at a slower speed compared to the size of the scenario, the proportion of various types of experiences in the experience pool will change, which will limit the further improvement of the training effect of the agent in the later training period, and may even cause the training of the agent to fail.

[0005] Therefore, it is of great significance to design a method that can achieve autonomous image navigation and obstacle avoidance of unmanned aerial vehicles in complex scenarios and has good application effect. SUMMARY

[0006] Technical problems to be solved

[0007] In order to avoid the shortcomings of the prior art, the present application proposes an unmanned aerial vehicle autonomous image navigation and obstacle avoidance method based on improved reinforcement learning. It is mainly used for unmanned aerial vehicles that perform reconnaissance, search and rescue and other low-altitude flight tasks, and realizes obstacle avoidance while performing tasks. Here, obstacle avoidance mainly refers to avoiding the unmanned aerial vehicle entering the no-fly zone generated by the ground obstacle. At present, the image-based autonomous navigation and obstacle avoidance algorithm has the problem that the output action is not globally optimal. The autonomous navigation and obstacle avoidance algorithm based on reinforcement learning also has the problem of poor training effect in the scenario of the present invention. Therefore, the present application proposes a method based on improved reinforcement learning, which can obtain an agent with stronger autonomous image navigation and obstacle avoidance capability, and the agent controls the unmanned aerial vehicle to perform tasks.

[0008] Technical solutions

[0009] The application discloses an unmanned aerial vehicle autonomous image navigation and obstacle avoidance method based on improved reinforcement learning, and is characterized in that the following steps are included.

[0010] Step 1: modeling of the unmanned aerial vehicle autonomous image navigation and obstacle avoidance problem;

[0011] 1. setting a kinematic model of the unmanned aerial vehicle;

[0012]

[0013] wherein P u =[x u (t),y u (t),z u (t)] is the position of the unmanned aerial vehicle, V is the speed of the unmanned aerial vehicle, χ(t) and γ(t) are respectively the heading angle and the climbing angle of the unmanned aerial vehicle, and [u γ ,u χ ] is the control quantity of the unmanned aerial vehicle;

[0014] 2. reaching definition;

[0015] the position of the destination is P g =[x g (t)y g (t)z g (t)] T , the radius of the destination influence area is R g , and the distance D g between the unmanned aerial vehicle and the destination is defined as

[0016]

[0017] when D g ≤R g , the unmanned aerial vehicle reaches the destination;

[0018] 3. collision definition;

[0019] the position of the obstacle is P obs =[x obs (t)y obs (t)z obs (t)] T , the radius of the no-fly zone generated by the obstacle is R obs , and the distance D obs between the unmanned aerial vehicle and the obstacle is defined as

[0020]

[0021] when D obs <R obs , the unmanned aerial vehicle enters the no-fly zone generated by the obstacle, and the unmanned aerial vehicle collides with the obstacle;

[0022] 4. Out-of-bound definition;

[0023] When the UAV is performing a task, its flyable range is

[0024] P range = {(x, y, z) | X min ≤ x ≤ X max , Y min ≤ y(t) ≤ Y max , H min ≤ z ≤ H max}

[0025] When , the UAV is out of bound;

[0026] Step 2: Extract obstacle position information from the image s o collected by the on-board camera;

[0027] 1. Identify the obstacles in the image s o by the Faster R-CNN model;

[0028]

[0029] where obs posImage is the identification result of the Faster R-CNN model, and subscript i represents the i-th obstacle identified by the Faster R-CNN model; x i,1 , y i,1 and x i,2 , y i,2 represent the coordinates of the upper left corner and the lower right corner of the obstacle, respectively;

[0030] 2. Process the identification result of the Faster R-CNN model;

[0031] obs′ pos = (x′ o , y′ o ) = (τ1×x oInImage , (-1×τ1×y oInImage ))

[0032] U′ pos = (x′ U , y′ U ) = (τ1×x image / 2, (-(τ1×y image +d c ))

[0033] where x image and y image are the dimensions of the image, τ1 is the scale of the image, and d cis the distance between the UAV and the field of view frame;

[0034] 3, the position information of the obstacle is

[0035]

[0036]

[0037] wherein θ′ o is the lead angle of the obstacle-UAV; D′ OtoU is the distance between the UAV and the obstacle; χ′ is the relative heading angle of the UAV in the line-of-sight frame;

[0038] Step 3: Experience storage mechanism when training the intelligent agent

[0039] 1, the intelligent agent

[0040] The structure of the intelligent agent decision network is 29x512x128x6, wherein 29 is the number of input nodes, and 6 is the number of output nodes;

[0041] 2, the input s′(t) of the intelligent agent

[0042] Assuming that the position of the UAV is P u =[x u (t), y u (t), z u (t)], the pre-designated destination position is P g =[x g (t), y g (t), z g (t)] T ; the distance D g between the UAV and the destination and the lead angle θ g_XOY of the UAV in the XOY plane are defined as:

[0043]

[0044]

[0045] The input of the intelligent agent is: s′(t)=[z u (t), H UtoG (t), D g (t), θ g_XOY (t), χ(t), s′ o (t)]

[0046] wherein H UtoG (t) is the height difference between the UAV and the destination, and s′ o =[D′ OtoU , θ′ o ].

[0047] 3. Define the reward function r U ;

[0048] Define the reward for the drone reaching its destination as

[0049] The reward for a drone collision is defined as

[0050] The reward for a drone going out of bounds is defined as

[0051] Therefore, the reward function r U For r U (s(t+1),a U ) = r arrived +r collision +r out

[0052] 4. Classification of experience

[0053] During the training process of the agent, the experience RM stored in the experience pool is...

[0054] RM = {RM(i)|RM(i) = (s)} i (a-),a U ,r U ,s i (a+)), i<RM Capacity}

[0055] Where the superscript i represents the current experience's ID in the experience pool; s i (a-) and s i (a+) represent the actions to be performed. U Before and execute action a U The subsequent state, RM Capacity It is the capacity of the experience pool;

[0056] In a single experience, state s i The task status represented by (t) is defined as follows:

[0057] in, It retrieves the task status represented by a specified state; e o ,e c ,e out ,e g These are parameters used to describe the task status: e o Used to describe whether the agent has detected an obstacle; e c Used to describe whether a collision has occurred in an intelligent agent; e out Used to describe whether an agent has gone out of bounds; e gfor describing whether the agent reaches the destination or not;

[0058] During the training of the UAV, the state s i (t) of the agent can be divided into the following categories: the state s safe in which the UAV does not detect an obstacle, does not collide, does not go out of bounds, and does not reach the destination; obs the state s collision in which the UAV collides with an obstacle; out the state s arrival in which the UAV goes out of bounds; and i the state s safe in which the UAV reaches the destination, that is:

[0059] s obs (t)∈{s collision ,s out ,s arrival}

[0060]

[0061]

[0062]

[0063]

[0064]

[0065] Therefore, for any experience RM(i) = (s i (a-), a U , s i (a+)), it is divided into the following categories:

[0066] (1) Result experience RE: divided into reaching experience RE arrival , collision experience RE collision , and out-of-bounds experience RE out , that is:

[0067] RE = {RE arrival , RE collision , RE out}, RE ∈ RM

[0068] RE arrival = {RM(i) {s i (a-) ∈ s safe , s i (a+) ∈ s arrival} ∪ {s i (a-) ∈ sobs ,s i (a+)∈s arrival}}

[0069] RE collision ={RM(i)|{s i (a-)∈s safe ,s i (a+)∈s collision}∪{s i (a-)∈s obs ,s i (a+)∈s collision}}

[0070] RE out ={RM(i)|{s i (a-)∈s safe ,s i (a+)∈s out}∪{s i (a-)∈s obs ,s i (a+)∈s out}}

[0071] (2) Dangerous experience (DE): This indicates that the agent has detected an obstacle, i.e.:

[0072] DE={RM(i)|{s i (a-)∈s obs ,s i (a+)∈s safe}∪{s i (a-)∈s safe ,s i (a+)∈s obs}∪{s i (a-)∈s obs ,s i (a+)∈s obs}}

[0073] (3) Safety Experience (SE): This refers to the intermediate state of the UAV as it flies towards its destination, away from obstacles.

[0074] SE={RM(i)|{s i (a-)∈s safe ,s i (a+)∈s safe}}

[0075] 5. Handling of experience

[0076] Set the storage ratio p for RE type experience, DE type experience, and SE type experience respectively. RE pDE , p SE ;

[0077] In the training process, the generated experience is classified according to the definition of experience type, and the experience is randomly screened according to the experience storage rate of the type to which it belongs, part of the experience is stored in the experience pool, and the rest of the experience is discarded; the number relationship of various types of experience in the experience pool RM' adjusted by the experience storage mechanism of the experience pool is:

[0078] |RM' | = p RE × |RE| + p DE × |DE| + p SE × |SE|

[0079] Where |·| is the number of specified experience in the experience pool;

[0080] Step 4: Train the agent according to the FRDDM-DQN algorithm

[0081] 1. Train the Faster-R CNN model to recognize the specified obstacles;

[0082] Initialize the Faster-R CNN model through the pre-trained model VGG16;

[0083] Set the initial learning rate, delay coefficient and delay weight of the Faster-R CNN model;

[0084] Collect images containing obstacles by the unmanned aerial vehicle, and label the positions of the obstacles in the images and the types of the obstacles;

[0085] Train the Faster-R CNN model through the images containing obstacles and the corresponding label information;

[0086] After training, obtain the Faster-R CNN model for recognizing obstacles;

[0087] 2. Train the agent based on the output of the Faster-R CNN model;

[0088] Step 2.1: Initialize the relevant parameters

[0089] Set the reward function r U , the experience storage rate p = p SE : p DE : p RE ;

[0090] Initialize the experience pool capacity RM Capacity , the decay coefficient γ, the maximum number of steps T e , the maximum effective training step T t , the network update frequency C;

[0091] Initialize exploration rate ε, exploration rate minimum value ε min , exploration rate reset period N, exploration rate reset value ε reset ;

[0092] Initialize learning rate α, segmented learning rate [α1, α2, α3, α4], and boundary (n1, n2, n3, n4) of segmented learning rate;

[0093] The decision network of the agent is divided into a prediction network and a target network; initialize the parameters θ and θ - of the prediction network Q and the target network ;

[0094] Step 2.2: Initialize the training scene;

[0095] Initialize the starting point and destination position of the unmanned aerial vehicle, and initialize the position of the obstacle;

[0096] Reset the single-episode executed step t e , and the effective training step t t to 0;

[0097] Obtain the initial state s'(t);

[0098] Step 2.3: select action a U according to state s'(t);

[0099] Take a random number p ∈ [0, 1], if p > ε, select action according to the prediction network Q; otherwise, select a random action;

[0100] Step 2.4: execute action a U , obtain reward r U and new state s'(t+1), and get the experience RM = (s'(t), a U , r U , s'(t+1)) generated at the current time step;

[0101] Step 2.5: process experience RM;

[0102] According to the experience pool storage mechanism, store experience RM in the experience pool or discard it;

[0103] Step 2.6: update learning rate:

[0104] The learning rate adopts segmented fixed learning rate, and the learning rate is updated according to the adjustment strategy;

[0105] Step 2.7: update exploration rate;

[0106] Update the exploration rate according to the exploration rate update strategy;

[0107] Step 2.8: Perform network optimization;

[0108] If network optimization is not performed, go to Step 2.9; otherwise, perform network optimization:

[0109] Randomly sample m groups of experiences from the experience pool;

[0110] If the experience is an end experience, let the target network predict the Q value y = r U ; if the experience is a non-end experience, let the target network predict the Q value as

[0111] Calculate the loss L(θ) = E(y - Q(s(t), a U (t), θ));

[0112] Optimize the parameters θ of the prediction network according to the loss value L(θ) by a gradient descent algorithm;

[0113] Every C valid steps, use the parameters of the prediction network to replace the target network, i.e. θ - = θ;

[0114] Step 2.9: Update the state s'(t) <- s'(t+1), t e <- t e +1;

[0115] Step 2.10: Judge the training state;

[0116] If the number of valid steps t t ≥ T t , the training is ended, and the agent at this time is saved; otherwise, continue to judge whether the UAV has reached the destination, or whether the UAV has collided, or whether the UAV has gone out of the boundary, or whether the maximum number of steps T e in the single act has been reached, if so, the current act is ended, and go to Step 2.2; otherwise, go to Step 2.3;

[0117] Step 5: Control the UAV to perform autonomous image navigation and obstacle avoidance by the agent saved in Step 4.

[0118] In Step 2, during the execution of the task, x image , y image , d c are fixed values; χ' is a constant.

[0119] In Step 3, the value and meaning of e g for describing whether the agent has reached the destination are as follows:

[0120]

[0121] The experience pool storage mechanism in step 3 is described as follows:

[0122]

[0123]

[0124] The adjustment strategy updates the learning rate in step 3 as follows:

[0125]

[0126]

[0127] The exploration rate update strategy in step 3 is as follows:

[0128]

[0129]

[0130] Beneficial effects

[0131] The present application provides an unmanned aerial vehicle autonomous image navigation and obstacle avoidance method based on improved reinforcement learning, and proposes an unmanned aerial vehicle autonomous image navigation and obstacle avoidance method based on image and experience pool storage mechanism, namely FRDDM-DQN method. In the present application, the agent meeting the requirements is trained by the FRDDM-DQN method; when performing the task, the unmanned aerial vehicle is controlled by the trained agent to realize autonomous image navigation and obstacle avoidance.

[0132] In the FRDDM-DQN method, first, the image collected by the airborne camera is processed by the Faster R-CNN model, that is, the obstacle information is extracted from the image, and the obstacle information is converted according to the kinematic characteristics of the unmanned aerial vehicle. Then, the obstacle information is added to the input state, and the agent is trained. During the training process, it is found that the number of various types of experience in the experience pool is greatly different, which affects the convergence of the agent. According to the characteristics of the experience generated when the unmanned aerial vehicle performs the task, an experience pool storage mechanism is proposed. During training, the mechanism sets a storage rate for each type of experience, that is, only part of each type of experience is stored, and the rest of the experience is discarded. Small batches of experience are randomly extracted from the optimized experience pool for network optimization. Finally, an agent with strong autonomous image navigation and obstacle avoidance ability in complex and unknown scenes is trained.

[0133] Beneficial effects:

[0134] 1. By introducing the Faster R-CNN model into the DQN algorithm and transforming the recognition results of the Faster R-CNN model, the autonomous image navigation and obstacle avoidance ability of the unmanned aerial vehicle in complex environments is obtained.

[0135] Since the Faster R-CNN model is added in the DQN algorithm, the Faster R-CNN model has strong image recognition capability, so that the DQN algorithm combined with the Faster R-CNN model can preliminarily realize the image-based autonomous navigation and obstacle avoidance of the unmanned aerial vehicle. In addition, since the output of the Faster R-CNN model is transformed, that is, the obstacle coordinate information output by the Faster R-CNN model is converted into angle and distance information according to the kinematic characteristics of the unmanned aerial vehicle, the agent trained by the algorithm proposed in the application can better control the unmanned aerial vehicle.

[0136] 2. The experience pool data storage mechanism proposed in the application is introduced into the DQN algorithm, which improves the autonomous image navigation and obstacle avoidance capability of the unmanned aerial vehicle in complex environments.

[0137] Since the experience pool data storage mechanism proposed in the application is added in the DQN algorithm, the mechanism classifies the experience generated during training and specifies the corresponding storage rate for each experience type, which makes the agent trained by the method proposed in the application have stronger autonomous image navigation and obstacle avoidance capability compared with the DQN algorithm.

[0138] 3. The method of partial training reduces the retraining time when the application scenario is changed.

[0139] Since the training process is divided into two parts and executed separately in the application, that is, the training of the Faster R-CNN model for image recognition and the FRDDM-DQN training based on the output value of the Faster R-CNN model are executed separately, so that when the application scenario is changed, the retraining time is reduced, that is, only the Faster R-CNN model needs to be retrained to recognize the specified obstacles, and the FRDDM-DQN does not need to be retrained. BRIEF DESCRIPTION OF DRAWINGS

[0140] Figure 1 : is a three-dimensional task scene diagram of an unmanned aerial vehicle.

[0141] Figure 2 : is a two-dimensional task scene diagram of an unmanned aerial vehicle.

[0142] Figure 3 : is a relative position relationship diagram of an unmanned aerial vehicle, an obstacle, and a camera field of view frame.

[0143] Figure 4 : is a flowchart of the FRDDM-DQN method proposed in the application.

[0144] Figure 5 : is the result of obstacle recognition by the Faster R-CNN model.

[0145] Figure 6 : is the arrival rate curve of the FRDDM-DQN method and the FR-DQN method in the training process.

[0146] Figure 7 : is the test result graph of the agent at different training stages in the training process of the FRDDM-DQN method and the FR-DQN method.

[0147] Figure 8 : is the trajectory graph of the agent trained based on the FRDDM-DQN method and the FR-DQN method in an environment containing multiple static obstacles.

[0148] Figure 9 : is the trajectory graph of the agent trained based on the FRDDM-DQN method and the FR-DQN method in an environment containing multiple dynamic obstacles. DETAILED DESCRIPTION

[0149] The present application will be further described in conjunction with embodiments, drawings:

[0150] Step 1: modeling of the problem of autonomous image navigation and obstacle avoidance of the unmanned aerial vehicle;

[0151] In order to realize the autonomous image navigation and obstacle avoidance function of the unmanned aerial vehicle, the problem should be defined first. Since the method proposed in the present application is a reinforcement learning algorithm, the core elements of the reinforcement learning algorithm, i.e. state, action and reward function, should also be defined;

[0152] Step 1-1: definition of the problem of autonomous image navigation and obstacle avoidance of the unmanned aerial vehicle, the scene of which is shown in Figure 1 、 Figure 2

[0153] In the present application, the task performed by the unmanned aerial vehicle is to quickly navigate from the starting point to the specified destination, and to avoid entering the no-fly zone generated by the ground obstacle during the execution of the task;

[0154] The position of the unmanned aerial vehicle is P u =[x u (t)y u (t)z u (t)] T , the speed is fixed as V, and the heading angle and the climbing angle of the unmanned aerial vehicle are χ(t) and γ(t) respectively. In order for the unmanned aerial vehicle to better complete the task, the unmanned aerial vehicle should navigate within (H min , H max ), wherein H min and H max represent the minimum and maximum values of the flight height of the unmanned aerial vehicle respectively;​

[0155] The position of the destination is P g = [x g (t)y g (t)z g (t)] T The distance D g of the UAV from the destination g_XOY is defined as:

[0156]

[0157]

[0158] The influence area of the destination is defined as a sphere with radius R g When D g ≤ R g , the UAV is considered to have reached the destination.

[0159] The position of the obstacle is P obs = [x obs (t)y obs (t)z obs (t)] T The radius of the no-fly zone generated by the obstacle is R obs The distance D obs of the UAV from the obstacle is defined as

[0160]

[0161] When D obs < R obs , i.e., the UAV enters the no-fly zone generated by the obstacle, we consider that the UAV has collided with the obstacle.

[0162] Step 1-2: Set the kinematic model of the UAV;

[0163] a U is the control quantity of the agent, i.e., a U = [u γ , u χ ]

[0164] Step 1-3: Set the state s(t) of the agent;

[0165] The state information available to the UAV includes the state s U of the UAV, the destination information s g , and the image information s o . Among them, the state of the UAV is obtained by GPS and gyroscope, s U = [x u(t), y u (t), z u (t), V, y(t), x(t)]; destination information s g is pre-specified before performing the task, s g = [x g (t), y g (t), z g (t)]; image information s o is collected by an on-board camera, which is mainly used to guide the UAV to avoid obstacles. Considering the kinematic characteristics of the UAV, the input state s(t) is defined as

[0166] s(t) = [z u (t), H UtoG (t), D g (t), q g_XOY (t), x(t), s o (t)]

[0167] where H UtoG (t) is the height difference between the UAV and the destination;

[0168] Step 1-4: Set the action a U (t) of the agent;

[0169] According to the input state s(t) defined in the previous step, the action is defined as a U (t) = [u γ , u χ ], where u χ and u γ are the control amounts of the heading angle and the climb angle, respectively;

[0170] Step 1-5: Set the reward function r U of the agent;

[0171] During the training process, the agent selects an action a U according to the current state s(t), reaches a new state s(t+1) after executing a U , and obtains a reward value from the environment. This reward value is the output of the reward function r U .

[0172] In order to make the agent know that it should sail to the destination, the reward in the reward function about whether the UAV has reached the destination is set as

[0173]

[0174] That is, when the UAV reaches the destination, the reward value is set to +1, and otherwise 0. During the UAV flight, when approaching the obstacle, in order to make the agent know that it should avoid collision, the reward about collision in the reward function is set to

[0175]

[0176] That is, when the UAV collides with the obstacle, the reward is set to -1, and otherwise 0. In addition, in order to ensure the task effect, the UAV should fly in the height range of (H min ,H max ); in order to improve the training efficiency of the agent and prevent the agent from detouring (detouring the obstacle area from the outermost side of the obstacle area) to avoid obstacles, a horizontal boundary is set in the XOY plane. In order to make the UAV know that it should fly within the boundary, the reward about out-of-bound in the reward function is set to

[0177]

[0178] That is, when the UAV is out of bounds, the reward value is set to -1, and otherwise 0. The method proposed in the application is a reinforcement learning algorithm, and the essence of the reinforcement learning algorithm is to let the agent find the optimal strategy in the process of interacting with the environment. Therefore, no other rewards about intermediate states are set. In summary, the reward function is set to

[0179] r U (s(t+1),a U )=r arrived +r collision +r out

[0180] Step 2: extracting obstacle information from image information collected by the onboard camera;

[0181] In the state s(t) defined in steps 1-3, s o is the image information collected by the onboard camera. In order to use s o to guide the agent to avoid obstacles, it is necessary to extract the obstacle information in the image through the Faster R-CNN model. In order to train more efficiently, the output information of the Faster R-CNN model also needs to be further processed.

[0182] Step 2-1: identifying the obstacle in the image through the Faster R-CNN model;

[0183] In the input state s(t) of the agent, the information about the obstacle is only the image information s oThe Faster R-CNN model identifies potential obstacle information in the image, and its output is...

[0184]

[0185] Where, the subscript i represents the i-th anchor point identified by the Faster R-CNN model; b i It is the bounding box (rectangle) of the i-th anchor point, x i,1 ,y i,1 and x i,2 ,y i,2 These represent the coordinates of the top-left and bottom-right corners of the anchor point bounding box, as shown in the figure; obs posImage These are the center coordinates of the anchor point bounding box, i.e., the relative coordinates of the obstacle's center in the image. posImage This is also the recognition result of the Faster R-CNN model.

[0186] Step 2-2: Process the recognition results of the Faster R-CNN model, such as... Figure 3 As shown;

[0187] Due to the recognition results of the Faster R-CNN model, obs posImage These are the relative coordinates of the obstacles in the image. The agent cannot directly use this information to avoid obstacles, therefore further processing is required. (The text then abruptly shifts to a seemingly unrelated topic about OBS.) posImage Convert to relative coordinates obs′ between the drone and the obstacle pos ,Right now

[0188] obs′ pos =(x′) o ,y′o)=(τ1×x oInImage ,(-1×τ1×y oInImage ))

[0189] Where, x image and y image τ1 is the image size, and τ2 is the image scale. Once the airborne camera is mounted and fixed to the drone, this value can be considered a fixed value during the drone's mission. Furthermore, since the positional relationship between the drone and the airborne camera's viewport is fixed, the drone's position in the relative positional relationship between the drone and the viewport can be considered fixed, with coordinates as follows:

[0190] U′ pos =(x′) U ,y′ U )=(τ1×x image / 2,(-(τ1×y image +d c )))

[0191] where d c is the distance between the UAV and the field of view frame. When the camera is fixed to the UAV, x image , y image , d c can be considered as fixed values during the execution of the task. Since the agent controls the UAV in the current task scenario, the input coordinate information is not conducive to the convergence of the agent. Therefore, the positional relationship between the UAV and the obstacles in the image is converted into the obstacle-UAV front angle θ′ o and the distance D′ OtoU between the UAV and the obstacle.

[0192]

[0193]

[0194] where χ′ is the relative heading angle of the UAV in the field of view frame, which is a constant when the onboard camera is fixed to the UAV.

[0195] Finally, the output of the Faster R-CNN model is

[0196] s′(t)=[z u (t),H UtoG (t),D g (t),θ g_XOY (t),χ(t),s′ o (t)]

[0197] where s′ o =[D′ OtoU ,θ′ o ].

[0198] Step 3: filtering the experience stored in the experience pool through the experience pool storage mechanism;

[0199] The new state s′(t) containing obstacle information obtained through step 2 is used to train the agent. In the training process, although the number of experiences in the experience pool is large, the number of experiences of various types is not evenly distributed. The experience pool storage mechanism proposed in the present application is to analyze each type of experience in the experience pool and set an appropriate storage rate for it, and the rest of the experiences are discarded, that is, only part of the experiences of each type are stored in the experience pool. Through this method, the number of experiences of each type in the experience pool can be rebalanced, which is conducive to improving the training effect of the agent.

[0200] Step 3-1: define a single experience;

[0201] In the training process of the agent, the experience RM stored in the experience pool is

[0202] RM = {RM(i)|RM(i) = (s)} i (a-),a U ,s i (a+)), i<RM Capacity}

[0203] Where the superscript i represents the current experience's ID in the experience pool; s i (a-) and s i (a+) represent the actions to be performed. U Before and execute action a U The subsequent state, RM Capacity This is the capacity of the experience pool. At this point, the state s in the i-th experience... i (a+) is the s of the (i+1)th experience. i+1 (a-).

[0204] In a single experience, state s i The task state represented by (t) can be defined as...

[0205] in, This represents retrieving the task status indicated by the specified state; e o ,e c ,e out ,e g This is a parameter used to describe the task status: e o Used to describe whether the agent has detected an obstacle; e c Used to describe whether a collision has occurred in an intelligent agent; e out Used to describe whether an agent has gone out of bounds; e g Used to describe whether an agent has reached its destination. Its values ​​and meanings are shown in the table below:

[0206]

[0207] During drone training, the agent's state s i (t) can be divided into the following categories: the state s where the drone did not detect an obstacle, did not collide, did not go out of bounds, and did not reach its destination. safe The drone detected an obstacle, did not collide, did not go out of bounds, and did not reach its destination. obs The state of a drone colliding with an obstacle. collision The state of the drone going out of bounds (s) out The drone's status upon reaching its destination. arrival .Right now:

[0208] s i (t)∈{s safe ,sobs ,s collision ,s out ,s arrival}

[0209]

[0210]

[0211]

[0212]

[0213]

[0214] Step 3-2: classify the experience RM(i) and set the experience storage ratio;

[0215] For any experience RM(i) = (s i (a-),a U ,s i (a+)), it can be classified as follows:

[0216] (1) Result experience RE, this type of experience is the experience generated at the last step of each scene. This type of experience can be further classified as reaching experience RE arrival , collision experience RE collision , and out-of-bounds experience RE out , i.e.

[0217] RE = {RE arrival , RE collision , RE out}, RE ∈ RM

[0218] RE arrival = {RM(i) | {s i (a-) ∈ s safe , s i (a+) ∈ s arrival} ∪ {s i (a-) ∈ s obs , s i (a+) ∈ s arrival}}

[0219] RE collision = {RM(i) | {s i (a-) ∈ s safe , s i (a+) ∈ s collision} ∪ {s i (a-) ∈ s obs , s i (a+) ∈ s collision}}

[0220] RE out ={RM(i)|{s i (a-)∈s safe ,s i (a+)∈s out}∪{s i (a-)∈s obs ,s i (a+)∈s out}}

[0221] The three experiences in the experience RE all have non-zero reward values, through which the agent can learn an effective strategy, that is, through the positive reward, the agent will know that it should sail to the destination; through the negative reward value, the agent will know that it should avoid collision or out-of-bounds. In addition, since the end experience is generated at the end of each episode, it results in fewer end experiences RE in the experience pool. Therefore, the experience of the RE type should be set to a higher storage rate p RE ;

[0222] (2) Dangerous experience DE, this type of experience represents that the agent has detected an obstacle. This type of experience indicates that the UAV has approached the obstacle, at which time the agent needs to decide whether to avoid the obstacle according to the current state, and if it needs to avoid the obstacle, to decide the avoidance action to be performed after integrating the destination information. The dangerous experience DE is defined as

[0223] DE={RM(i)|{s i (a-)∈s obs ,s i (a+)∈s safe}∪{s i (a-)∈s safe ,s i (a+)∈s obs}∪{s i (a-)∈s obs ,s i (a+)∈s obs}}wherein the experience {RM(i)|s i (a-)∈s obs ,s i (a+)∈s safe} represents that an obstacle is detected before the action is performed, and no obstacle is detected after the action is performed; the experience {RM(i)|s i (a-)∈s safe ,s i (a+)∈s obs} represents that no obstacle is detected before the action is performed, and an obstacle is detected after the action is performed; and the experience {RM(i)|si (a-)∈s obs ,s i (a+)∈s obs} represents that the obstacle is detected before and after the execution of the action;

[0224] Through the experience of RE type, the agent can know that it should avoid collision with the obstacle when approaching the obstacle. Through the experience of DE type, the agent can learn how to avoid collision when approaching the obstacle. During the training process, the experience of this type will be generated when the UAV approaches the obstacle. As the training goes on, the agent will behave more and more in accordance with the rules, and the situation of navigating between obstacles will also increase. Finally, the number of experiences of this type in the experience pool is also large. Therefore, a lower storage rate p DE is set for the experience of DE type;

[0225] (3) Safe experience SE, this type of experience is the intermediate state of the UAV away from the obstacle during the navigation to the destination, that is, SE = {RM(i) | {s i (a-)∈s safe ,s i (a+)∈s safe}}

[0226] In order for the agent to eventually reach the destination (obtain a positive reward), this type of experience can help the agent learn how to navigate to the destination. Since the scene of the UAV executing the task is large, the number of this type of experience is large. Therefore, a lower storage rate p SE is set for the safe experience SE;

[0227] Step 3-3: Process the experiences generated during the training process by setting the storage rate to obtain the optimized experience pool RM';

[0228] According to the above analysis, different experience storage rates are set for various types of experiences. During the training process, part of each type of experience is stored in the experience pool according to the different experience storage rates, and the rest of the experience is discarded. After the experience pool experience storage mechanism is adjusted, the number relationship of various types of experiences in the experience pool is:

[0229] |RM'| = p RE × |RE| + p DE × |DE| + p SE × |SE|

[0230] Where |·| is the number of specified experiences in the experience pool.

[0231] Step 4: Train the agent for realizing the UAV autonomous image navigation and obstacle avoidance by the FRDDM-DQN method;

[0232] In step 2, the image s for obstacle avoidance in the input state of the agent is obtained o The processing is performed to obtain a new state s'(t) that can help the agent to avoid obstacles; an experience pool experience storage mechanism is proposed in step 3, which specifies the storage rules of the generated experience in training. This step starts to train the agent by the optimized DQN algorithm, FRDDM-DQN method, and the structure of the FRDDM-DQN method is as shown in Figure 4 .

[0233] The training of the FRDDM-DQN method is divided into two parts: training the Faster-R CNN model to recognize the specified obstacles; and training the FRDDM-DQN method based on the output value of the Faster-R CNN model;

[0234] Step 4-1: Training the Faster-R CNN model to recognize the specified obstacles;

[0235] The Faster-R CNN model is initialized by the pre-trained model VGG16;

[0236] The initial learning rate, delay coefficient and delay weight of the Faster-R CNN model are set;

[0237] The images containing obstacles are collected by the unmanned aerial vehicle, and the positions of the obstacles and the types of the obstacles in the collected images are labeled;

[0238] The Faster-R CNN model is trained by the images containing obstacles and the corresponding labeled information;

[0239] After training, the Faster-R CNN model capable of recognizing obstacles is obtained, and the recognition result is as shown in Figure 5 .

[0240] Step 4-2: Training the FRDDM-DQN based on the output value of the Faster-R CNN model;

[0241] Step 4-2-1: Building the decision network of the agent in the FRDDM-DQN method and initializing the related parameters;

[0242] The decision network of the agent is built, and the reward function r U , the experience storage rate p = p SE : p DE : p RE in the experience pool experience storage mechanism are set;

[0243] The capacity RM Capacity of the experience pool, the learning rate a, the decay coefficient g, the exploration rate e, and the minimum exploration rate emin , exploration rate reset period N, exploration rate reset value ε reset , single episode maximum step number T e , maximum effective training step number T t , network update frequency C

[0244] In the training of the FRDDM-DQN method, the decision network of the agent is divided into a prediction network and a target network. The parameters θ and θ of the initialized prediction network Q and the target network - ;

[0245] Step 4-2-2: initialize the training scene;

[0246] Initialize the starting point and destination position of the unmanned aerial vehicle, and initialize the position of the obstacle;

[0247] Reset the single episode executed step number t e , effective training step number t t to 0;

[0248] Get the initial state s'(t);

[0249] Step 4-2-3: select an action a U according to the action selection strategy and the state s'(t);

[0250] Step 4-2-4: after executing the action a U , get the reward r U and the new state s'(t+1), and obtain the experience RM=(s'(t),a U ,r U ,s'(t+1)) generated at the current time step;

[0251] Step 4-2-5: select the experience according to the experience storage mechanism proposed in step 3;

[0252] If the experience is stored in the experience pool, the time step is an effective step, t t ←t t +1; if the experience is not stored in the experience pool, the experience is discarded, and the time step is not an effective step, t t ←t t ;

[0253] Step 4-2-6: change the learning rate α according to the learning rate adjustment strategy;

[0254] Step 4-2-7: change the exploration rate ε according to the exploration rate adjustment strategy;

[0255] Step 4-2-8: judge whether the network optimization is executed at the current time step;

[0256] If network optimization is not performed, go to step 4-2-10; if network optimization is performed:

[0257] Randomly sample m groups of experiences from the experience pool RM' and calculate the m groups of experiences:

[0258] If the experience is an end experience, let the Q value y predicted by the target network be r U ; if the experience is not an end experience, let the Q value y predicted by the target network be

[0259] Calculate the loss L(θ) = E(y - Q(s(t), a U (t), θ));

[0260] Optimize the parameters θ of the prediction network according to the loss value L(θ) by a gradient descent algorithm;

[0261] Every C valid steps, use the parameters of the prediction network to replace the target network, i.e. θ - = θ;

[0262] Step 4-2-9: update the state and step number: s'(t) <- s'(t+1), t e <- t e +1;

[0263] Step 4-2-10: judge the training state;

[0264] If the valid step number t t ≥ T t , go to step 4-2-11:; otherwise, continue to judge whether the current time step of the UAV reaches the destination, or whether the UAV collides, or whether the UAV is out of the boundary, if so, the current scene ends, go to step 4-2-2:; otherwise, continue to judge the relationship between the single-scene executed step number t e and the single-scene maximum step number T e , if t e ≥ T e , the current scene ends, go to step 4-2-2:; otherwise, go to step 4-2-3: continue training;

[0265] Step 4-2-11: training ends, save the current trained network. At this time, the network is the decision network of the trained intelligent agent, go to step 5;

[0266] Step 5: the intelligent agent trained in step 4-2-11 controls the UAV to perform autonomous image navigation and obstacle avoidance;

[0267] Test the trained intelligent agent by setting different scenes. Among them, the starting point and destination of the UAV are randomly generated, the initial position of the obstacle is randomly generated, and the movement direction of the dynamic obstacle is randomly generated; Embodiments:

[0269] Step 1: Modeling of the problem of autonomous image navigation and obstacle avoidance of UAVs;

[0270] Step 1-1: Definition of the problem of autonomous image navigation and obstacle avoidance of UAVs;

[0271] The position of the UAV is P u = [x u (t), y u (t), z u (t)] T , the heading angle and the climbing angle are χ(t) and γ(t) respectively, and the speed is fixed at v = 42 m / s;

[0272] The position of the destination is P g = [x g (t), y g (t), z g (t)] T , the radius of the destination influence area is R g = 2000 m, the destination-UAV pre-angle in the XOY plane is θ g_XOY , and the distance between the UAV and the destination is D g . When D g ≤ R g , it is considered that the UAV reaches the destination;

[0273] The position of the obstacle is P obs = [x obs (t), y obs (t), z obs (t)] T , the radius of the no-fly zone generated by the obstacle is R obs = 1500 m, and the distance between the UAV and the obstacle is D obs . When D obs < R obs , the UAV enters the no-fly zone generated by the obstacle, and it is considered that the UAV collides with the obstacle;

[0274] Step 1-2: Setting the kinematic model of the UAV;

[0275]

[0276] a U is the control quantity of the agent, i.e. a U = [u γ , u χ ]

[0277] Step 1-3: Setting the state s(t) of the agent;

[0278] The available state information for the drone includes: the drone's state s U Destination information g Image information s o The drone's status is obtained from GPS and gyroscope data. U =[x u (t),y u (t),z u [(t),V,γ(t),χ(t)];Destination information s g It is pre-specified before the task is executed, s g =[x g (t),y g (t),z g (t)];Image information s o The image was captured by an airborne camera and is primarily used to guide the drone in obstacle avoidance.

[0279] Considering the kinematic characteristics of the UAV, the input state s(t) is defined as

[0280] s(t)=[z u (t),H UtoG (t),D g (t),θ g_XOY (t),χ(t),s o (t)]

[0281] Among them, H UtoG (t) represents the altitude difference between the drone and the destination;

[0282] Steps 1-4: Define the agent's actions a U (t);

[0283] Based on the input state s(t) defined in the previous step, the action is defined as a. U (t)=[u γ ,u χ ], where u χ and u γ These are the control values ​​for the heading angle and climb angle, respectively;

[0284] Steps 1-5: Define the reward function r of the agent. U ;

[0285] To enable the agent to know that it should fly towards the destination, the reward in the reward function for whether the drone reaches the destination is set to...

[0286]

[0287] That is, the reward value is set to +1 when the drone reaches its destination, and 0 otherwise.

[0288] In the process of UAV navigation, when approaching the obstacle, in order to make the agent know that it should avoid collision, the reward about collision in the reward function is set to

[0289]

[0290] That is, when the UAV collides with the obstacle, the reward is set to -1, and 0 otherwise.

[0291] In order to make the UAV know that it should navigate within the boundary, the reward about out-of-bound in the reward function is set to

[0292]

[0293] That is, when the UAV is out of the boundary, the reward value is set to -1, and 0 otherwise.

[0294] The method proposed in the application is a reinforcement learning algorithm, and the essence of the reinforcement learning algorithm is to let the agent find the optimal strategy autonomously in the process of interacting with the environment. Therefore, no other rewards about intermediate states are set. In summary, the reward function is set to

[0295] r U (s(t+1),a U )=r arrived +r collision +r out

[0296] Step 2: training the agent for realizing autonomous image navigation and obstacle avoidance of the UAV by the FRDDM-DQN method;

[0297] Step 2-1: processing the image information s o in the input state s(t);

[0298] Step 2-1-1: training the Faster-R CNN model to recognize the obstacle specified in the image information s o ;

[0299] Initializing the Faster-R CNN model by the pre-trained model VGG16;

[0300] Setting the initial learning rate of the Faster-R CNN model to 0.001, the delay coefficient to 0.1, and the delay weight to 0.0005;

[0301] Collecting the image containing the obstacle by the UAV, and labeling the position of the obstacle and the type of the obstacle in the collected image;

[0302] The Faster-R CNN model is trained by using the image containing the obstacle and corresponding label information;

[0303] After the training, the Faster-R CNN model capable of recognizing the obstacle is obtained, and the output of the model is obs posImage The recognition result is shown in FIG. 4, and the gray points are the obstacles recognized by the Faster-R CNN model. The meaning of the text and numbers above the gray points is that the type of the current recognized object is an obstacle, and the probability of the object being an obstacle is; Figure 5

[0304] Step 2-1-2: converting the output obs posImage of the Faster-R CNN model;

[0305] obs posImage is converted into the relative coordinates obs′ pos between the UAV and the obstacle, that is,

[0306] obs′ pos = (x′ o , y′ o ) = (τ1×x oInImage , (-1×τ1×y oInImage ))

[0307] wherein τ1 is the scale of the image, and in the embodiment, τ1 = 2.5.

[0308] According to the positional relationship between the UAV and the field-of-view frame of the onboard camera, the relative position U′ pos of the UAV is calculated.

[0309] U′ pos = (x′ U , y′ U ) = (τ1×x image / 2, (-(τ1×y image +d c ))

[0310] wherein d c is the distance between the UAV and the field-of-view frame, and in the embodiment, d c = 624.

[0311] The positional relationship between the UAV and the obstacle in the image is converted into the obstacle-UAV front angle θ′ o and the distance D′ OtoU between the UAV and the obstacle.

[0312]

[0313]

[0314] wherein χ' is the relative heading angle of the UAV in the line-of-sight frame, in the present embodiment χ' = 90, i.e. the onboard camera is mounted facing directly forward of the UAV;

[0315] The Faster R-CNN model outputs s'(t) as s'(t) = [z'(t), H'(t), D'(t), θ'(t), χ'(t), s'(t)] o OtoU o

[0316] s'(t) = [z'(t), H'(t), D'(t), θ'(t), χ'(t), s'(t)] u UtoG g g_XOY o

[0317] Step 2-2: training of the FRDDM-DQN based on the output values of the Faster-R CNN model;

[0318] Step 2-2-1: building of the decision network of the agent in the FRDDM-DQN method and initialization of relevant parameters;

[0319] In the present embodiment, the structure of the decision network of the agent is 29 x 512 x 128 x 6, wherein 29 is the number of input nodes and 6 is the number of output nodes; the reward function is set as r(s(t+1), a) = r + r + r + r + r + r; the relevant parameters are set as follows: U U arrived collision out

[0320] Hyperparameters Values Number of samples m 64 Experience pool capacity RM Capacity ]]> 300,000 Decay coefficient γ 0.95 Segmented learning rate a = (a1, a2, a3, a4) 0.001,0.0005,0.0001,0.00005 [ [ segmental learning rate boundaries [n1, n2, n3, n4] ] ] [0,100000,200000,300000,350000] initial exploration rate ε0 1.0 minimum exploration rate ε min ]]> 0.001 Exploration rate reset period N 100000 Exploration rate reset value epsilon reset ]] 0.5 Maximum number of steps T for a single act e ]]> 600 maximum effective steps T t ]]> 350,000 Target network update frequency (C) 3000 Data storage rate p = p SE : p DE : p RE ]]> [0.16:0.05:1]

[0321] In the training of the FRDDM-DQN method, the decision network of the agent is divided into a prediction network and a target network. The parameters θ and θ of the initialized prediction network Q and target network -

[0322] Step 2-2-2: initialization of the training scenario;

[0323] Initialization of the starting point and destination position of the UAV, and initialization of the position of the obstacle;

[0324] Resetting the number of steps t executed in a single episode t and the number of effective training steps t to 0; e t

[0325] Obtaining the initial state s'(t);

[0326] ​​​​​​​​​​​​​​​​​Step 2-2-3: Select an action a according to the action selection policy and state s'(t) U In this embodiment, the action selection policy adopts an ε-greedy algorithm.

[0327] Step 2-2-4: Execute the action a U Afterwards, the reward r is obtained U and the new state s'(t+1), and the experience RM=(s'(t),a U , r U , s'(t+1)) generated at the current time step is obtained.

[0328] Step 2-2-5: The experience is filtered according to the experience storage mechanism proposed in the present application, and the filtering process is as follows:

[0329]

[0330] If the experience is stored in the experience pool (i.e., F storage is true), the time step is a valid step, t t ← t t +1; if the experience is not stored in the experience pool (i.e., F storage is false), the experience is discarded, and the time step is not a valid step, t t ← t t .

[0331] Step 2-2-6: Change the learning rate a according to the learning rate adjustment policy.

[0332] In this embodiment, the learning rate adopts a segmented fixed learning rate, and the adjustment policy is as follows:

[0333]

[0334]

[0335] Step 2-2-7: Change the exploration rate ε according to the exploration rate adjustment policy.

[0336] Since the concept of valid step is proposed in the present application, the change mode of the exploration rate ε in the ε-greedy algorithm is optimized, as shown below:

[0337]

[0338]

[0339] Step 2-2-8: Determine whether the network optimization is performed at the current time step.

[0340] If the network optimization is not performed, proceed to step 2-2-10; if the network optimization is performed:

[0341] Randomly sample m sets of experiences from the experience pool, and calculate the m sets of experiences:

[0342] If the experience is an end experience, let the target network predict the Q value y = r U ; if the experience is not an end experience, let the target network predict the Q value as

[0343] Calculate the loss L(θ) = E(y-Q(s(t),a U (t), θ));

[0344] Optimize the parameters θ of the prediction network according to the loss value L(θ) by a gradient descent algorithm;

[0345] Every C valid steps, use the parameters of the prediction network to cover the target network, that is, θ - = θ;

[0346] Step 2-2-9: Update state and step number: s'(t) <- s'(t+1), t e <- t e +1;

[0347] Step 2-2-10: Judge the training state;

[0348] If the valid step number t t ≥ T t , the training is ended, at this time the prediction network (or the target network) is the decision network of the trained agent, go to step 2-2-11:; Otherwise, continue to judge whether the UAV reaches the destination, or whether the UAV collides, or whether the UAV is out of the boundary, if so, the current scene is ended, go to step 2-2-2:; Otherwise, continue to judge the relationship between the single-scene executed step number t e and the single-scene maximum step number T e , if t e ≥ T e , the current scene is ended, go to step 2-2-2:; Otherwise, go to step 2-2-3: continue training;

[0349] Step 2-2-11: The training is ended, and the current trained network is saved. At this time, the network is the decision network of the trained agent.

[0350] In order to prove the superiority of the FRDDM-DQN method proposed in the training process, in the Figure 6The training curves of the FRDDM-DQN method are shown in FIG. 6. As a comparison, the training curves of the FR-DQN method in the same training environment are also shown. The FR-DQN method is a DQN method combined with a Faster R-CNN model. In FIG. 6, the performance of the agent trained by the FRDDM-DQN method does not improve significantly in the first 5000 episodes, while the performance of the agent trained by the FR-DQN method gradually improves. After 5000 episodes, the performance of the agent trained by the FRDDM-DQN method gradually improves, and the final arrival rate increases to 83%, while the performance of the agent trained by the FR-DQN method does not improve significantly, and the arrival rate fluctuates around 75%. Therefore, the agent trained by the FRDDM-DQN method has a higher final arrival rate. In addition, since the action selection strategy in the training process adopts an ε-greedy strategy, the selection of some actions of the agent is random, and the training curve cannot fully reflect the training effect of the agent. Therefore, during the training process, the decision network of the agent is saved once every 5000 episodes. Finally, each saved agent is tested for 500 episodes, and the test results are shown in FIG. 7. Similarly, a comparative experiment is conducted on the FR-DQN method. As shown in FIG. 8, in the first 15000 episodes of training, the FR-DQN method obtains better training results because it performs network optimization at each step, while the training effect of the FRDDM-DQN method improves slowly. After 15000 episodes, the arrival rate of the agent trained by the FR-DQN method fluctuates around 70%, while the arrival rate of the agent trained by the FRDDM-DQN method continues to rise, and the final arrival rate increases to 93%. Therefore, it is concluded that the FRDDM-DQN method proposed in the present application performs better in the training process. Figure 7 Figure 7 Figure 6 Figure 7

[0351] Step 3: Test the agent saved in step 2-2-11 in a three-dimensional environment containing multiple obstacles.

[0352] Step 3: Test the agent saved in step 2-2-11 in a three-dimensional environment containing multiple obstacles.

[0353] To verify the performance of the agent trained by the FRDDM-DQN method proposed in the present application, the agent is tested in a three-dimensional scene containing multiple static obstacles and dynamic obstacles as shown in FIG. 9 and FIG. 10. As a comparison, the agent trained by the FR-DQN method is also tested. As shown in FIG. 11 and FIG. 12, the agent trained by the FRDDM-DQN method has a higher final arrival rate than the agent trained by the FR-DQN method. Figure 8 Figure 9 Figure 8 Figure 9 ​​​​​​​In the diagram, subgraphs (a)-(f) represent scene diagrams at different times. The smaller spheres represent no-fly zones created by obstacles, with the black dot in the center of each sphere indicating the location of the obstacle; the larger spheres represent the drone's destination; the black lines represent the flight paths of the agent trained using the FRDDM-DQN method, and the gray lines represent the flight paths of the agent trained using the FR-DQN method. Figure 9 The black lines on the obstacles represent the tracks the obstacles travel.

[0354] Figure 8 In the initial stage of testing ( Figure 8 In (a) and (b), the agents trained by the FRDDM-DQN method and the FR-DQN method exhibited roughly the same performance. Figure 8 (c) Initially, when an obstacle is detected, the two exhibit different obstacle avoidance strategies: the agent trained using the FR-DQN method, upon detecting an obstacle, does not comprehensively consider how to avoid the obstacle and quickly reach the destination, choosing an obstacle avoidance strategy that avoids the obstacle but takes a detour (going around the route), ultimately reaching the destination at 672 seconds. Figure 8 (f)); After detecting an obstacle, the agent trained by the FRDDM-DQN method comprehensively considers how to avoid the obstacle and reach the destination quickly, and selects an obstacle avoidance strategy that can simultaneously satisfy both obstacle avoidance and rapid arrival at the destination. Finally, it reaches the destination in 366 seconds. Figure 8 (e)) takes far less time than agents trained using the FR-DQN method.

[0355] Figure 9 In the initial stage of training ( Figure 9 (a) and (b) show that agents trained using the FRDDM-DQN method and the FR-DQN method exhibit different strategies when navigating towards their destination. The agent trained using the FRDDM-DQN method navigates towards the destination, which is the fastest route when obstacle information is unknown. The agent trained using the FR-DQN method chooses a route closer to the destination, but not the fastest. When the no-fly zones created by two obstacles are close together, the agent trained using the FRDDM-DQN method can precisely control the UAV to navigate between the no-fly zones. Figure 9 (c)-(e)), and safely arrived at the destination at 321 seconds. Figure 9 (f)); The obstacle avoidance performance of the agent trained by the FR-DQN method is poor, at the 213th second ( Figure 9 (e) Collision occurs with obstacles. Therefore, in scenarios containing multiple static and dynamic obstacles, the agent trained by the FRDDM-DQN method proposed in this invention performs better.

[0356] The FRDDM-DQN method realizes autonomous image navigation and obstacle avoidance of the unmanned aerial vehicle in a complex environment.In the FRDDM-DQN method, firstly, the obstacles in the image collected by the airborne camera are identified through the Faster R-CNN model, and the identification result is transformed according to the kinematic characteristics of the unmanned aerial vehicle; secondly, in the training process, the number of various types of experience in the experience pool is adjusted through the experience pool storage mechanism in the application, and the imbalance problem of the proportion of various types of experience in the DQN algorithm is improved. In the training process, the intelligent agent trained by the FRDDM-DQN method finds a better strategy; in the test, the intelligent agent trained by the FRDDM-DQN method performs better. In conclusion, compared with the FR-DQN method, the FRDDM-DQN method improves the autonomous image navigation and obstacle avoidance ability of the unmanned aerial vehicle in a complex environment.

Claims

1. An unmanned aerial vehicle autonomous image navigation and obstacle avoidance method based on improved reinforcement learning, characterized in that The steps are as follows: Step 1: Modeling of unmanned aerial vehicle autonomous image navigation and obstacle avoidance problem; (1) Set the kinematic model of the unmanned aerial vehicle; Among them, P u =[x u (t),y u (t),z u [(t)] represents the position of the UAV, V represents the velocity of the UAV, χ(t) and γ(t) represent the heading angle and climb angle of the UAV, respectively, [u γ ,u χ [] is the control variable for the drone; (2) Reach definition; The location of the destination is P g = [x g (t) y g (t) z g (t)] T The radius of the destination influence area is R g The distance D g between the UAV and the destination is defined as When D g ≤ R g the drone reaches the destination; (3) Collision definition; The position of the obstacle is P obs = [x obs (t)y obs (t)z obs (t)] T The no-fly zone radius generated by the obstacle is R obs ; the distance D obs between the UAV and the obstacle is defined as When D obs When R obs When the UAV enters the no-fly zone generated by the obstacle, the UAV collides with the obstacle. (4) Out-of-bounds definition; When the unmanned aerial vehicle performs a task, its flyable range is P range = {(x, y, z) | X min ≤ x ≤ X max , Y min ≤ y(t) ≤ Y max , H min ≤ z ≤ H max} When the drone is out of bounds; Step 2: Extracting obstacle position information from images s captured by on-board cameras o from images s captured by on-board cameras (1) identifying obstacles in the image s by a Faster R-CNN model o ; wherein obs posImage is the recognition result of the FasterR-CNN model, the subscript i represents the ith obstacle recognized by the FasterR-CNN model; x i,1 , y i,1 and x i,2 , y i,2 respectively represent the coordinates of the upper left corner and the lower right corner of the obstacle. (2) Processing the recognition results of the FasterR-CNN model; obs′ pos = (x′ o , y′ o ) = (τ1×x oInImage , (-1×τ1×y oInImage )) U′ pos = (x′ U ,y′ U ) = (τ1×x image / 2,(-(τ1×y image +d c ))) where x image and y image are the dimensions of the image, τ1 is the scale of the image, d c is the distance between the UAV and the field of view frame; (3) The position information of the obstacle is where θ' is the obstacle-UAV lead angle; D' is the distance between the UAV and the obstacle; χ' is the relative heading angle of the UAV in the line-of-sight frame; and φ' is the relative bearing angle of the UAV in the line-of-sight frame. o OtoU where θ' is the obstacle-UAV lead angle; D' is the distance between the UAV and the obstacle; χ' is the relative heading angle of the UAV in the line-of-sight frame; and φ' is the relative bearing angle of the UAV in the​ Step 3: Experience storage mechanism when training the intelligent agent (1) The intelligent agent The structure of the intelligent agent decision network is 29×512×128×6, where 29 is the number of input nodes and 6 is the number of output nodes; (2) Intelligent agent input s′(t) Assume the position of the drone is P u = [x u (t), y u (t), z u (t)], the pre-specified destination position is P g = [x g (t), y g (t), z g (t)] T ; the distance D g between the drone and the destination, and the leading angle Θ g_XOY of the drone in the XOY plane is defined as: The input of the intelligent agent is s'(t) = [z u (t), H UtoG (t), D g (t), θ g_XOY (t), χ(t), s' o (t)] H UtoG (t) is the height difference between the UAV and the destination, s′ o = [D′ OtoU , θ′ o ] (3) define the reward function r U ; The reward for the unmanned aerial vehicle reaching the destination is defined as The reward for the unmanned aerial vehicle colliding is defined as The reward for the unmanned aerial vehicle out-of-bounds is defined as Thus, the reward function r U is r U (s(t+1),a U )=r arrived +r collision +r out (4) Classification of experience During the training of the intelligent agent, the experience RM stored in the experience pool is Where the superscript i represents the current experience's ID in the experience pool; s i (a-) and s i (a+) represent the actions to be performed. U Before and execute action a U The subsequent state, RM Capacity It is the capacity of the experience pool; In a single experience, the state s i The task conditions represented by (t) are defined as wherein, is a task condition that is acquired to represent a specified state;e o ,e c ,e out ,e g is a parameter for describing a task condition;e o for describing whether the agent detects an obstacle;e c for describing whether the agent collides;e out for describing whether the agent is out of bounds;e g for describing whether the agent reaches the destination; In the training process of the UAV, the state s of the agent i (t) can be divided into the following categories: the state s of the UAV not detecting an obstacle, not colliding, not out of bounds, and not reaching the destination safe ; the state s of the UAV detecting an obstacle, not colliding, not out of bounds, and not reaching the destination obs ; the state s of the UAV colliding with an obstacle collision ; the state s of the UAV out of bounds out ; the state s of the UAV reaching the destination arrival ; that is: s i (t) ∈ {s safe ,s obs ,s collision ,s out ,s arrival} Thus, for any empirical RM(i) = (s i (a-),a U ,s i (a+)) is classified into the following categories: The result experience RE is divided into an arrival experience RE arrival , a collision experience RE collision , and an out-of-bound experience RE out , i.e. RE = {RE arrival , RE collision , RE out}, RE G RM RE arrival = {RM(i) | {s i (a-)∈s safe ,s i (a+)∈s arrival}∪{s i (a-)∈s obs ,s i (a+)∈s arrival}} RE collision = {RM(i) | {s i (a-)∈s safe ,s i (a+)∈s collision}∪{s i (a-)∈s obs ,s i (a+)∈s collision}} RE out = {RM(i) | {s i (a-)∈s safe ,s i (a+)∈s out}∪{s i (a-)∈s obs ,s i (a+)∈s out}} ② Dangerous experience DE: represents that the intelligent agent has detected an obstacle, that is: DE = { RM(i) | {s i (a-)∈s obs ,s i (a+)∈s safe}∪{s i (a-)∈s safe ,s i (a+)∈s obs}∪{s i (a-)∈s obs ,s i (a+)∈s obs}} ③ Safe experience SE: is an intermediate state of the unmanned aerial vehicle sailing away from the obstacle, that is: SE = { RM(i) | {s i (a-) ∈ s safe s i (a+) ∈ s safe}} (5) Experience processing Set the deposit rate p to the experience of the RE type, the experience of the DE type, and the experience of the SE type, respectively RE , p DE , p SE ; During the training process, the generated experience is classified according to the definition of the experience type, and the experience is randomly selected according to the experience storage rate of the type to which it belongs, part of which is stored in the experience pool, and the rest is discarded; the number relationship of various types of experience in the experience pool RM′ adjusted by the experience storage mechanism of the experience pool is: | RM' | = p RE x | RE | + p DE x | DE | + p SE x | SE | Where |·| is the number of specified experience in the experience pool; Step 4: Training the intelligent agent according to the FRDDM-DQN algorithm (1) Training the Faster-R CNN model to recognize the specified obstacle; Initialize the Faster-RCNN model through the pre-trained model VGG16; Set the initial learning rate, delay coefficient, and delay weight of the Faster-RCNN model; Collect images containing obstacles by the unmanned aerial vehicle, and label the position of the obstacle in the image and the type of the obstacle; Train the Faster-RCNN model through the images containing obstacles and the corresponding labeling information; After training, the Faster-RCNN model for recognizing obstacles is obtained; (2) Training the intelligent agent based on the output of the Faster-R CNN model; Step 2.1: Initialize related parameters Set the reward function r U , the experience pool experience storage rate p = p SE : p DE : p RE ; Initialization of the experience pool capacity RM Capacity , decay coefficient γ, maximum number of steps T e , maximum number of effective training steps T t , network update frequency C; initializing an exploration rate ε, a minimum value of the exploration rate ε min , a reset period of the exploration rate N, a reset value of the exploration rate ε reset ; Initialize the learning rate α, the segmented learning rate [α1, α2, α3, α4], and the boundary of the segmented learning rate (n1, n2, n3, n4); The decision network of the agent is divided into a prediction network and a target network; the parameters θ and θ of the prediction network Q and the target network are initialized - ; Step 2.2: Initialize the training scene; Initialize the starting point and destination position of the unmanned aerial vehicle, and initialize the position of the obstacle; t = t + 1 e t = t + 1 t t = 0; Get the initial state s′(t); Step 2.3: Select action a according to state s'(t) U ; Take a random number p∈[0,1], if p>ε, select the action according to the prediction network Q; otherwise, select a random action; Step 2.4: Perform action a U After, get reward r U With new state s'(t+1), and get experience RM= (s'(t), a U , r U , s'(t+1)) for current time step Step 2.5: Process experience RM; According to the experience pool storage mechanism, store the experience RM in the experience pool or discard it; Step 2.6: Update the learning rate: The learning rate uses a segmented fixed learning rate, and the learning rate is updated according to the adjustment strategy; Step 2.7: Update the exploration rate; Update the exploration rate according to the exploration rate update strategy; Step 2.8: Perform network optimization; If the network optimization is not performed, go to step 2.9; otherwise, perform the network optimization: Randomly sample m groups of experiences from the experience pool; If the experience is an end experience, let the target network predict the Q-value y = r U ; if the experience is a non-end experience, let the target network predict the Q-value as Compute loss L(0) = E(y - Q(s(t), a U (t), 0)); Optimize the parameters theta of the prediction network according to the loss value L(theta) by the gradient descent algorithm; Every C effective steps the parameters of the prediction network are used to overwrite the target network, i.e. θ - = θ; Step 2.9: Update state s'(t) <- s'(t+1), t e e +1;​ Step 2.10: judge the training state; If the effective step number t t ≥ T t , the training is ended, and the agent at this time is saved; otherwise, it is judged whether the UAV reaches the destination, or the UAV collides, or the UAV is out of the boundary, or the maximum step number T e of the single scene is reached at the current time step; if yes, the current scene is ended, and the process goes to step 2.2; otherwise, the process goes to step 2.

3. Step 5: the intelligent agent controls the unmanned aerial vehicle to perform autonomous image navigation and obstacle avoidance through the intelligent agent saved in step 4.

2. The method of claim 1, wherein the method further comprises: In step 2: during execution of the task, x image , y image , d c is a fixed value; χ' is a constant.

3. The method of claim 1, wherein: The e used in step 3 to describe whether the agent has reached the destination g The values and meanings are as follows:

4. The method of claim 1, wherein: The experience pool storage mechanism is described as follows:

5. The method of claim 1, wherein: The adjustment strategy updates the learning rate as:

6. The method of claim 1, wherein: The exploration rate updating strategy is:

Images