A deep reinforcement learning accelerated training method for unmanned aerial vehicle image navigation

Abstract

The application relates to a kind of unmanned aerial vehicle image navigation-oriented deep reinforcement learning acceleration training methods, including the training of object detection model and the training of action selection strategy. Wherein, the training of action selection strategy is according to the simulation output value of object detection model and is carried out through deep reinforcement learning method. In the training of image-based deep reinforcement learning method, the simulation output value of image detection model is used instead of the output value of image detection model to accelerate training, and the method has the following advantages: reducing training time consumption, reducing GPU usage and memory occupancy ratio, and being relatively strong in universality.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to a deep reinforcement learning accelerated training method for unmanned aerial vehicle (UAV) image navigation. Background Art

[0002] With the advancement of technology, drones have become widely used in both military and civilian applications. In some military scenarios, given the complexity of battlefields, and to mitigate potential unavailability of drone communication systems and radar, drones require the ability to navigate using images from onboard cameras. In civilian applications, these cameras are also widely installed on drones due to their portability and affordability. Therefore, achieving image-based drone navigation is of great significance.

[0003] In recent years, with the advancement of computer vision technology, the recognition rate of object detection technology has significantly improved. To achieve more accurate image navigation for drones, object detection models can be used to identify obstacles in images. However, due to the complexity and large number of layers in object detection models, the calculation of their output values ​​is relatively time-consuming. Image-based deep reinforcement learning training methods are unsupervised algorithms, requiring extensive training to obtain a decision network that can achieve the desired goal. This, in turn, limits the application of object detection models in deep reinforcement learning methods.

[0004] In the "UAV Obstacle Detection and Avoidance Method and System Based on Images and Point Clouds" (Patent, Publication No.: CN114581831 A, Publication Date: June 3, 2022), obstacles in an image are identified using an object detection model, and then obstacle avoidance is performed according to a specified obstacle avoidance strategy based on the identification results. Although this method uses the Faster R-CNN model for image recognition and has a high object recognition rate, the obstacle avoidance strategy is manually specified. Manually specified obstacle avoidance strategies are suboptimal in some cases. In the "UAV Autonomous Obstacle Avoidance System and Method Based on Deep Reinforcement Learning" (Patent, Publication No.: CN 114326821 A, Publication Date: April 12, 2022), a depth image is obtained after processing the image using a fully convolutional neural network. Based on this depth image data, a deep reinforcement learning method is used to train the drone for obstacle avoidance. However, this method cannot distinguish between different objects appearing in the image, that is, it cannot distinguish between general objects and obstacles, which limits the obstacle avoidance effect.

[0005] Currently, due to the slow computational time required for high-precision object detection models and the large number of training steps required for deep reinforcement learning methods, current image navigation methods are unable to achieve optimal obstacle avoidance for drones with high-precision obstacle recognition rates. Therefore, it is crucial to design accelerated training methods for deep reinforcement learning based on image detection models. Summary of the Invention

[0006] Technical problems to be solved

[0007] To overcome the shortcomings of the prior art, the present invention proposes a method for accelerating deep reinforcement learning training for drone image navigation. This method is primarily used for drones to perform image navigation tasks. Currently, deep reinforcement learning methods for image navigation require an object detection model to identify specified objects. Object detection models with higher recognition rates require more computation, which results in longer training times for deep reinforcement learning methods and higher requirements for computer hardware. Therefore, the present invention proposes a method for accelerating deep reinforcement learning training for drone image navigation. This method can effectively reduce the training time of deep reinforcement learning methods and lower the computational complexity of training.

[0008] Technical Solution

[0009] A deep reinforcement learning accelerated training method for drone image navigation, characterized by including object detection model training and action selection strategy training, the training steps are as follows:

[0010] Training of object detection model:

[0011] Step 1: In the image s containing obstacles o Mark obstacle information where t o is the type of obstacle, Is the obstacle in s o The coordinates in ;

[0012] Step 2: Initialize the object detection model;

[0013] Step 3: Transform the image s containing obstacles into o and mark obstacle information Input into the object detection model, train the object detection model, and obtain an object detection model that can recognize obstacles. The recognition result is the position of the obstacle in the image obs posImage ;

[0014] Step 4: Output value obs according to the object detection model posImage The characteristics of the obstacle are determined by the actual coordinates p o Calculate the simulated output obs′ of the object detection model posImage ;

[0015] The relative coordinates of the obstacle in the field of view are:

[0016] obs′ posImage =(obs′ posImage_X ,obs′ posImage_Y )

[0017] in:

[0018] (obs x ,obs y ,0) is the actual position of the obstacle; (P2 x ,P2 y ) is the coordinate of the upper left corner of the field of view; Δχ is the rotation of the coordinate system;

[0019] When no obstacle enters the field of view, the simulated output of the object detection model is obs′ posImage =(0,0);

[0020] Training of action selection strategy:

[0021] Step 5: Define the state s(t), action a(t) and reward function R in the deep reinforcement learning method;

[0022] State s(t): s(t) = (obs′ posImage ,p u ,p g ), where the destination coordinates are p g , the coordinates of the drone are p u ;

[0023] The drone can perform an action a(t) described as turning left or right;

[0024] Reward function R: deduct 1 point when a collision or out-of-bounds occurs, and reward 1 point when reaching the finish line;

[0025] Step 6. Training initialization: Define the parameters of the prediction network in the action selection strategy π(·) as θ, and define the parameters of the target network as θ - ;

[0026] Step 7: Input the state s(t) into the action selection strategy π(θ) to get the selected action a U (t);

[0027] Step 8: Execute action a U After (t), the reward r is obtained according to the reward function R U , and will execute action a U The state after (t) is taken as the new state s(t+1);

[0028] Step 9: (s(t),a U (t), r, s(t+1)) are stored in the experience pool;

[0029] Step 10: randomly drawing experience from the experience pool, calculating the loss value, and optimizing the parameters of the selection strategy through the gradient descent algorithm according to the loss function L(θ) to obtain the action selection strategy π(θ) for realizing image navigation.

[0030] The loss value is calculated by the loss function L(θ) as follows:

[0031] L(θ) = E(y-Q(s(t),a U (t),θ))

[0032]

[0033] Wherein, γ is the discount rate, is the output of the target network, and Q(·) is the output of the prediction network.

[0034] The object detection model adopts an R-CNN series or a YOLO series object detection model.

[0035] The structure of the action selection strategy π(θ) is 29x512x128x6, wherein 29 is the number of input nodes, and 6 is the number of output nodes.

[0036] Advantages

[0037] The application provides an accelerated training method for deep reinforcement learning for image navigation of a UAV, which comprises training of an object detection model and training of an action selection strategy.

[0038] In the training of the image-based deep reinforcement learning method, the simulated output value of the image detection model is used instead of the output value of the image detection model to accelerate the training, and the method has the following advantages:

[0039] (1) Reducing training time

[0040] Since the image detection model has a large number of network layers and a complex network structure, the calculation of the real output value is relatively time-consuming. The time consumption of the simulation value calculation according to the application is less than that of the image detection model output calculation, which reduces the total time consumption of the image-based deep reinforcement learning method training.

[0041] (2) Reducing GPU usage and memory usage

[0042] Because image detection models are computed on the GPU, higher-accuracy models require more GPU and video memory, limiting the use cases of image-based deep reinforcement learning methods. The accelerated training method proposed in this paper moves deep reinforcement learning training to the CPU, effectively reducing the hardware requirements for training.

[0043] (3) Strong versatility

[0044] The accelerated training method proposed in the present invention uses the simulated output value of the image detection model. The calculation of the simulated output value is based on the coordinates of the object to be identified according to the characteristics of the image detection model output. This makes the calculation method of the simulated value in the accelerated training method suitable for simulating the output of most image recognition models, and therefore has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 :Faster R-CNN model recognition results

[0046] Figure 2 : Calculation of the center point of the field of view

[0047] Figure 3 : Calculation of the midpoint of the side of the field of view

[0048] Figure 4 : Calculation of the field of view vertex

[0049] Figure 5 : Flow chart of the method of the present invention DETAILED DESCRIPTION

[0050] The present invention will now be further described with reference to the embodiments and accompanying drawings:

[0051] This paper proposes a deep reinforcement learning accelerated training method for drone image navigation, which includes training of an object detection model and an action selection strategy. The action selection strategy is trained using deep reinforcement learning based on the simulated output values ​​of the object detection model.

[0052] Training an object detection model involves the following steps:

[0053] Step 1: Get an image s containing obstacles o ;

[0054] Step 2: Label image s o Obstacle information in where t o is the type of obstacle, Is the obstacle in s o The coordinates in ;

[0055] Step 3: Initialize the object detection model;

[0056] Step 4: Pass the image s containing obstacles o and marking information o Train the object detection model;

[0057] Step 5: After training is completed, an object detection model that can identify obstacles is obtained. The recognition result is the position of the obstacle in the image obs posImage .

[0058] Furthermore, according to the object detection model output value obs posImage The characteristics of the obstacle are determined by the actual coordinates p o Calculate the simulated output obs′ of the object detection model posImage .

[0059] Step 1: Get the drone's position p u =(x U ,y U ,z U ), the heading angle of the UAV χ, the actual position of the obstacle p o =(obs x ,obs y ,0);

[0060] Step 2: According to the drone position (x U ,y U ) Calculate the coordinates of the vertices P1, P2, P3, and P4 of the field of view frame;

[0061] Step 2-1: Pass the actual coordinate p of the obstacle o Calculate the center coordinates of the field of view frame C = (C x ,C y );

[0062]

[0063]

[0064] Among them, (x image ,y image ) is the size of the field of view, τ1 is the scale, d c is the distance between the field of view and the drone. When the camera is fixed to the drone, during the normal flight of the drone, (x image ,y image ),τ1,d c It is a fixed value.

[0065] Step 2-2: Calculate the midpoints A and B of the sides of the field of view frame based on the coordinates C of the center point of the field of view frame;

[0066]

[0067]

[0068]

[0069]

[0070] Step 2-3: Calculate the coordinates of the vertices P1, P2, P3, and P4 of the field of view frame based on the coordinates of points A and B;

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] Step 3: Determine whether there is an obstacle entering the field of view based on the coordinates P1, P2, P3, and P4;

[0080] F1 in =(P2 x -P1 x )*(obs y -P1 y )-(P2 y -P1 y )*(obs x -P1 x )

[0081] F2 in =(P3 x -P2 x )*(obs y -P2 y )-(P3 y -P2 y )*(obs x -P2 x )

[0082] F3 in =(P4 x -P3 x )*(obsy -P3 y )-(P4 y -P3 y )*(obs x -P3 x )

[0083] F4 in =(P1 x -P4 x )*(obs y -P4 y )-(P1 y -P4 y )*(obs x -P4 x )

[0084] Among them, when F1 in >0,F2 in >0,F3 in >0,F4 in >0 or F1 in <0,F2 in <0,F3 in <0,F4 in <0, there is an obstacle entering the field of view. Otherwise, there is no obstacle entering the field of view;

[0085] Step 4: Calculate the simulated output obs′ of the object detection model based on whether the obstacle enters the field of view posImage ;

[0086] When an obstacle enters the field of view, according to the actual position of the obstacle p o , the vertex P2 of the field of view, the heading angle χ of the drone, and the relative heading angle χ′ of the drone are used to calculate the simulated output obs′ of the object detection model. posImage .

[0087] Calculate the rotation Δχ of the coordinate system

[0088] Δχ=χ-χ′

[0089] The relative coordinate of the obstacle in the field of view is obs′ posImage =(obs′ posImage_X ,obs′ posImage_Y ),in

[0090] obs′ posImage_X =(obs x -P2 x )·cosΔχ+(obs y -P2 y )·sinΔχ

[0091] obs′posImage_Y = (obs x - P2 x ) · sin Δx - (obs y - P2 y ) · cos Δx

[0092] When there is no obstacle entering the field of view frame, obs' posImage = (0, 0).

[0093] Further, according to the simulation output obs' posImage of the object detection model, the action selection strategy π(θ) is trained by a deep reinforcement learning method, mainly including the following steps:

[0094] Step 1: define the state s(t), action a(t) and reward function R in the deep reinforcement learning method;

[0095] The state s(t) is defined as

[0096] s(t) = (obs' posImage , p u , p g )

[0097] Wherein, the destination coordinates are p g .

[0098] The action a(t) executable by the unmanned aerial vehicle is described as left turn or right turn;

[0099] The reward function R is defined as: deducting points when collision occurs, and rewarding when reaching the end point;

[0100] Step 2: training initialization. The parameters of the prediction network in the action selection strategy π(·) are defined as θ, and the parameters of the target network are defined as θ - .

[0101] Step 3: select the action a(t) according to the action selection strategy π(θ) and the state s(t);

[0102] Step 4: after executing the action a(t), the reward r is obtained according to the reward function R, and the new state s(t+1) is obtained from the environment;

[0103] Step 5: store (s(t), a(t), r, s(t+1)) in the experience pool;

[0104] Step 6: extract experience from the experience pool to optimize the action selection strategy π(θ);

[0105] The loss function L(θ) is defined as

[0106] L(θ) = E(y - Q(s(t), a U(t),θ))

[0107]

[0108] Where γ is the discount rate;

[0109] Optimize the parameters θ of the selection strategy according to the loss function L(θ) through the gradient descent algorithm;

[0110] After training is completed, the action selection strategy π(θ) that can realize image navigation is obtained

[0111] This example first trains an object detection model to identify obstacles using images containing obstacles. Then, based on the characteristics of the object detection model's output values, it calculates simulated output values. Finally, based on these simulated values, it trains an action selection strategy using deep reinforcement learning. The simulated values ​​of the object detection model are calculated based on the real-world coordinates of the obstacles.

[0112] In this embodiment, the object detection model uses the Faster R-CNN model.

[0113] Training an object detection model involves the following steps:

[0114] Step 1: Get an image s containing obstacles o The image can be captured and saved by the drone's onboard camera during the flight;

[0115] Step 2: Label image s o Obstacle information in where t o is the type of obstacle, Is the obstacle in s o The coordinates in ;

[0116] Step 3: Initialize the object detection model. In this embodiment, the Faster-R CNN model is initialized using the pre-trained VGG16 model. The initial learning rate, delay coefficient, and delay weight of the Faster-R CNN model are set to 0.001, 0.1, and 0.0005.

[0117] Step 4: Pass the image s containing obstacles o and marking information o Train the object detection model;

[0118] Step 5: After training is completed, an object detection model that can identify obstacles is obtained. The recognition result is the position of the obstacle in the image obs posImage ,like Figure 1During training, the obstacles to be identified are the gray objects in the figure, including the lower left corner of the "obstacle" character box, which is where the obstacle is located.

[0119] Furthermore, according to the object detection model output value obs posImage The characteristics of the obstacle are determined by the actual coordinates p o Calculate the simulated output obs′ of the object detection model posImage ;

[0120] Step 1: Get the drone's position p u =(x U ,y U ,z U ), the heading angle of the UAV χ, the actual position of the obstacle p o =(obs x ,obs y ,0). Among them, the position of the UAV is obtained by GPS, and the heading angle χ of the UAV can be obtained by the gyroscope. During training, the actual coordinates of the obstacle p o It is randomly generated;

[0121] Step 2: According to the drone position (x U ,y U ) Calculate the coordinates of the vertices P1, P2, P3, and P4 of the field of view frame;

[0122] Step 2-1: Pass the actual coordinate p of the obstacle o Calculate the center coordinates of the field of view frame C = (C x ,C y ),like Figure 2 As shown;

[0123]

[0124]

[0125] Among them, (x image ,y image ) is the size of the field of view, τ1 is the scale, d c is the distance between the field of view and the drone. When the camera is fixed to the drone, during the normal flight of the drone, (x image ,y image ),τ1,d c is a fixed value. In this embodiment, (x image ,y image )=(1000,600),d c =624m, τ1=2.5;

[0126] Step 2-2: Calculate the midpoints A and B of the sides of the field of view frame based on the coordinates C of the center point of the field of view frame, such as Figure 3 As shown;

[0127]

[0128]

[0129]

[0130]

[0131] Step 2-3: Calculate the coordinates of the view frame vertices P1, P2, P3, and P4 based on the coordinates of points A and B, as follows: Figure 4 As shown;

[0132]

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140] Step 3: Determine whether there is an obstacle entering the field of view based on the coordinates P1, P2, P3, and P4;

[0141] F1 in =(P2 x -P1 x )*(obs y -P1 y )-(P2 y -P1 y )*(obs x -P1 x )

[0142] F2 in =(P3 x -P2 x )*(obs y -P2 y )-(P3 y -P2 y )*(obs x -P2x )

[0143] F3 in =(P4 x -P3 x )*(obs y -P3 y )-(P4 y -P3 y )*(obs x -P3 x )

[0144] F4 in =(P1 x -P4 x )*(obs y -P4 y )-(P1 y -P4 y )*(obs x -P4 x )

[0145] Among them, when F1 in >0,F2 in >0,F3 in >0,F4 in >0 or F1 in <0,F2 in <0,F3 in <0,F4 in <0, there is an obstacle entering the field of view. Otherwise, there is no obstacle entering the field of view;

[0146] Step 4: When an obstacle enters the field of view, according to the actual position of the obstacle, o , the vertex P2 of the field of view, the heading angle χ of the drone, and the relative heading angle χ′ of the drone are used to calculate the simulated output obs′ of the object detection model. posImage In this embodiment, χ′=90°;

[0147] Calculate the rotation Δχ of the coordinate system

[0148] Δχ=χ-χ′

[0149] The relative coordinate of the obstacle in the field of view is obs′ posImage =(obs′ posImage_X ,obs′ posImage_Y ),in

[0150] obs′ posImage_X =(obs x -P2 x )·cosΔχ+(obs y -P2 y )·sinΔχ

[0151] obs′ posImage_Y =(obs x -P2 x )·sinΔχ-(obs y -P2 y )·cosΔχ

[0152] When no obstacles enter the field of view, obs′ posImage =(0,0);

[0153] Furthermore, according to the simulated output obs′ of the object detection model posImage , the action selection strategy π(θ) is trained by deep reinforcement learning method, such as Figure 5 As shown, it mainly includes the following steps:

[0154] Step 1: Define the state s(t), action a(t) and reward function R in the deep reinforcement learning method;

[0155] Define the state s(t) as

[0156] s(t)=(obs′ posImage ,p u ,p g )

[0157] The destination coordinates are p g .

[0158] The drone can perform an action a(t) described as turning left or right;

[0159] The reward function R is defined as: points are deducted when a collision occurs and rewards are given when the destination is reached. In this embodiment, 1 point is deducted when the drone collides with an obstacle, 1 point is rewarded when the drone reaches the destination, and 0 is rewarded in other cases.

[0160] Step 2: training initialization;

[0161] The parameters of the prediction network in the action selection strategy π(·) are defined as θ, and the parameters of the target network are defined as θ - ;

[0162] Step 3: Select action a(t) based on the action selection strategy π(θ) and the state s(t);

[0163] Step 4: After executing action a(t), obtain reward r according to reward function R and obtain the new state s(t+1) from the environment;

[0164] Step 5: Store (s(t), a(t), r, s(t+1)) into the experience pool;

[0165] Step 6: Extract experience from the experience pool to optimize the action selection strategy π(θ);

[0166] The loss function L(θ) is defined as

[0167] L(θ)=E(yQ(s(t),a U (t),θ))

[0168]

[0169] Where γ is the discount rate. In this embodiment, γ = 0.95;

[0170] Optimize the parameters θ of the selection strategy according to the loss function L(θ) through the gradient descent algorithm;

[0171] After training is completed, an action selection strategy π(θ) is obtained that can realize image navigation.

Claims

1. A deep reinforcement learning accelerated training method for UAV image navigation, characterized by Including the training of object detection model and action selection strategy, the training steps are as follows: Training of object detection model: Step 1: In the image s containing obstacles o Mark obstacle information where t o is the type of obstacle, Is the obstacle in s o The coordinates in ; Step 2: Initialize the object detection model; Step 3: Transform the image s containing obstacles into o and mark obstacle information Input into the object detection model, train the object detection model, and obtain an object detection model that can recognize obstacles. The recognition result is the position of the obstacle in the image obs posImage ; Step 4: Output value obs according to the object detection model posImage The characteristics of the obstacle are determined by the actual coordinates p o Calculate the simulated output obs′ of the object detection model posImage ; The relative coordinates of the obstacle in the field of view are: obs′ posImage =(obs′ posImage_X ,obs′ posImage_Y ) in: (obs x ,obs y ,0) is the actual position of the obstacle; (P2 x ,P2 y ) is the coordinate of the upper left corner of the field of view; Δχ is the rotation of the coordinate system; When no obstacle enters the field of view, the simulated output of the object detection model is obs′ posImage =(0,0); Training of action selection strategy: Step 5: Define the state s(t), action a(t) and reward function R in the deep reinforcement learning method; State s(t): s(t) = (obs′ posImage ,p u ,p g ), where the destination coordinates are p g , the coordinates of the drone are p u ; The drone can perform an action a(t) described as turning left or right; Reward function R: deduct 1 point when a collision or out-of-bounds occurs, and reward 1 point when reaching the finish line; Step 6. Training initialization: Define the parameters of the prediction network in the action selection strategy π(·) as θ, and define the parameters of the target network as θ - ; Step 7: Input the state s(t) into the action selection strategy π(θ) to get the selected action a U (t); Step 8: Execute action a U After (t), the reward r is obtained according to the reward function R U , and will execute action a U The state after (t) is taken as the new state s(t+1); Step 9: (s(t),a U (t), r, s(t+1)) are stored in the experience pool; Step 10: Randomly extract experience from the experience pool, calculate the loss value, and optimize the parameter θ of the selection strategy according to the loss function L(θ) through the gradient descent algorithm to obtain the action selection strategy π(θ) for image navigation; The loss value is calculated using the loss function L(θ): L(θ)=E(y-Q(s(t),a U (t),θ)) Where γ is the discount rate, is the output of the target network, and Q(·) is the output of the prediction network.

2. The deep reinforcement learning accelerated training method for drone image navigation according to claim 1 is characterized by: The object detection model adopts the R-CNN series and YOLO series object detection models.

3. The deep reinforcement learning accelerated training method for drone image navigation according to claim 1 is characterized by: The structure of the action selection strategy π(θ) is 29×512×128×6, where 29 is the number of input nodes and 6 is the number of output nodes.

Images