Monocular UAV Obstacle Avoidance Method, Device and Medium Based on Deep Deterministic Policy Gradient

Abstract

An embodiment of the present invention discloses a monocular UAV obstacle avoidance method, device and medium based on deep deterministic policy gradient; the method includes: converting an original RGB image collected by a monocular camera carried by a UAV into a depth image based on a conditional generative adversarial network CGAN; using the depth image and the current speed information of the UAV as state information to construct an actor network and a critic network in a deep deterministic policy gradient DDPG network; constructing a reward function according to the distance between the UAV and the target, the speed of the UAV and the collision information of the UAV; training the actor network and the critic network in the DDPG network according to the reward function to obtain the parameters of the actor network and the critic network; performing obstacle avoidance control on the monocular UAV according to the trained DDPG network.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of unmanned aerial vehicle (UAV) flight control, and particularly to a monocular UAV obstacle avoidance method, device and medium based on deep deterministic policy gradient. Background Technique

[0002] Due to the small size and high agility of small UAVs, they can successfully complete various tasks, such as search, rescue, environmental mapping, exploration of unknown areas, etc. UAVs play an increasingly important role in all walks of life. Since obstacle avoidance planning needs to be carried out in a complex indoor environment, an effective obstacle avoidance algorithm is particularly important to ensure that the UAV can safely execute various tasks.

[0003] Currently, the commonly used sensors in UAV obstacle avoidance tasks are ultrasonic sensors, radar sensors, depth cameras and monocular cameras, and these sensors have their own advantages and disadvantages. Compared with lidar and depth cameras, although the information provided by a monocular camera is less, due to its advantages of small size, light weight, low energy consumption, wide field of view, etc., it is more suitable for obstacle avoidance perception of small UAVs.

[0004] For a UAV equipped with a monocular camera (hereinafter may be simply referred to as a monocular UAV), since the depth information cannot be obtained, the depth information cannot be extracted from the images captured by the monocular camera for obstacle avoidance, and currently, there are problems with discontinuous obstacle avoidance speed in UAV obstacle avoidance schemes. Summary of the Invention

[0005] In view of this, the embodiments of the present invention expect to provide a monocular UAV obstacle avoidance method, device and medium based on deep deterministic policy gradient; it can predict depth pictures through the images captured by the monocular camera for action decision-making, and improve the stability and effectiveness of UAV flight control, so that the UAV can perform obstacle avoidance in a continuous action space, and the obstacle avoidance trajectory is smoother and more continuous.

[0006] The technical solution of the embodiments of the present invention is realized as follows:

[0007] In a first aspect, the embodiments of the present invention provide a monocular UAV obstacle avoidance method based on deep deterministic policy gradient, and the method includes:

[0008] Convert the original RGB image collected by the monocular camera carried by the UAV into a depth image based on a conditional generative adversarial network (CGAN, Conditional Generative Adversarial Network);

[0009] Construct an Actor network and a Critic network in a Deep Deterministic Policy Gradient (DDPG) network using the depth image and the current speed information of the UAV as state information;

[0010] Construct a reward function based on the distance between the UAV and the target, the speed of the UAV, and the collision information of the UAV;

[0011] Train the Actor network and the Critic network in the DDPG network according to the reward function to obtain the parameters of the Actor network and the Critic network;

[0012] Perform obstacle avoidance control on the monocular UAV according to the trained DDPG network.

[0013] In a second aspect, an embodiment of the present invention provides a monocular UAV obstacle avoidance device based on deep deterministic policy gradient. The device includes: an acquisition part, a conversion part, a first construction part, a second construction part, a training part, and an obstacle avoidance control part; where

[0014] The acquisition part is configured to collect an original RGB image through a monocular camera carried by the UAV;

[0015] The conversion part is configured to convert the original RGB image into a depth image based on a conditional generative adversarial network (CGAN);

[0016] The first construction part is configured to construct an Actor network and a Critic network in a deep deterministic policy gradient (DDPG) network using the depth image and the current speed information of the UAV as state information;

[0017] The second construction part is configured to construct a reward function based on the distance between the UAV and the target, the speed of the UAV, and the collision information of the UAV;

[0018] The training part is configured to train the Actor network and the Critic network in the DDPG network according to the reward function to obtain the parameters of the Actor network and the Critic network;

[0019] The obstacle avoidance control part is configured to perform obstacle avoidance control on the monocular UAV according to the trained DDPG network.

[0020] In a third aspect, an embodiment of the present invention provides a computing device, including: a communication interface, a memory, and a processor; each component is coupled together through a bus system; where

[0021] The communication interface is used for receiving and sending signals during the process of receiving and sending information with other external network elements.

[0022] The memory is used for storing a computer program that can run on the processor.

[0023] The processor is used for executing the steps of the monocular UAV obstacle avoidance method based on deep deterministic policy gradient described in the first aspect when running the computer program.

[0024] In a fourth aspect, an embodiment of the present invention provides a computer storage medium. The computer storage medium stores a monocular UAV obstacle avoidance program based on deep deterministic policy gradient. When the monocular UAV obstacle avoidance program based on deep deterministic policy gradient is executed by at least one processor, the steps of the monocular UAV obstacle avoidance method described in the first aspect are realized.

[0025] An embodiment of the present invention provides a monocular UAV obstacle avoidance method, device and medium based on deep deterministic policy gradient; predicting a depth picture through an image captured by a monocular camera for action decision-making; jointly introducing depth information and state information into a DDPG network for processing to improve the stability and effectiveness of UAV flight control; and constructing a reward function based on the distance between the UAV and the target, the speed of the UAV and the collision information of the UAV, so that the UAV can avoid obstacles in a continuous action space and the obstacle avoidance trajectory is smoother and more continuous. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic flow chart of a monocular UAV obstacle avoidance method based on deep deterministic policy gradient provided by an embodiment of the present invention;

[0027] Figure 2 It is a schematic diagram of the network architecture of CGAN provided by an embodiment of the present invention;

[0028] Figure 3 It is a schematic diagram of the deep reinforcement learning architecture provided by an embodiment of the present invention;

[0029] Figure 4 It is a schematic diagram of the generation of state information provided by an embodiment of the present invention;

[0030] Figure 5 It is a schematic diagram of the structure of the Actor network provided by an embodiment of the present invention;

[0031] Figure 6 It is a schematic diagram of the structure of the Critic network provided by an embodiment of the present invention;

[0032] Figure 7Schematic diagram of the overall processing flow architecture provided by an embodiment of the present invention;

[0033] Figure 8 Schematic diagram of the simulation environment provided by an embodiment of the present invention;

[0034] Figure 9 Schematic diagram of the obstacle avoidance training process of an unmanned aerial vehicle provided by an embodiment of the present invention;

[0035] Figure 10 Schematic diagram of the total score during the unmanned aerial vehicle training process provided by an embodiment of the present invention;

[0036] Figure 11 Schematic diagram of the average reward change curve during the unmanned aerial vehicle training process provided by an embodiment of the present invention;

[0037] Figure 12 Schematic diagram of the change curve of the Q value estimated by the network during the unmanned aerial vehicle training process provided by an embodiment of the present invention;

[0038] Figure 13 Schematic diagram of the loss value of the Actor network during the unmanned aerial vehicle training process provided by an embodiment of the present invention;

[0039] Figure 14 Schematic diagram of the loss value of the Critic network during the unmanned aerial vehicle training process provided by an embodiment of the present invention;

[0040] Figure 15 Schematic diagram of the speed change curve during the unmanned aerial vehicle training process provided by an embodiment of the present invention;

[0041] Figure 16 Comparison diagram of the obstacle avoidance routes of an unmanned aerial vehicle from an overhead view provided by an embodiment of the present invention;

[0042] Figure 17 Schematic diagram of the composition of a monocular unmanned aerial vehicle obstacle avoidance device based on deep deterministic policy gradient provided by an embodiment of the present invention;

[0043] Figure 18 Schematic diagram of the specific hardware structure of a computing device provided by an embodiment of the present invention. Detailed implementation manners

[0044] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention.

[0045] Refer to Figure 1 , which shows a monocular unmanned aerial vehicle obstacle avoidance method based on deep deterministic policy gradient provided by an embodiment of the present invention. The method may include:

[0046] S101: Convert the original RGB image captured by the monocular camera carried by the drone into a depth image based on the Conditional Generative Adversarial Network (CGAN).

[0047] S102: Construct the Actor network and the Critic network in the Deep Deterministic Policy Gradient (DDPG) network using the depth image and the current speed information of the drone as state information.

[0048] S103: Construct a reward function based on the distance between the drone and the target, the speed of the drone, and the collision information of the drone.

[0049] S104: Train the Actor network and the Critic network in the DDPG network according to the reward function to obtain the parameters of the Actor network and the Critic network.

[0050] S105: Perform obstacle avoidance control on the monocular drone according to the trained DDPG network.

[0051] Through the above technical solution, predict the depth picture from the image captured by the monocular camera for action decision-making; jointly introduce the depth information and state information into the DDPG network for processing to improve the stability and effectiveness of the drone flight control; and construct a reward function based on the distance between the drone and the target, the speed of the drone, and the collision information of the drone, enabling the drone to perform obstacle avoidance in the continuous action space, and the obstacle avoidance trajectory is smoother and more continuous.

[0052] For Figure 1 In some implementation manners of the shown technical solution, the conversion of the original RGB image captured by the monocular camera carried by the drone into a depth image based on the Conditional Generative Adversarial Network CGAN includes:

[0053] Construct two independent convolutional networks and set them as the generator and the discriminator respectively; wherein, the generator is used to generate a pseudo-depth image from the RGB image and input it into the discriminator; the discriminator is used to determine whether the input image is a true depth image.

[0054] Train the network parameters of the generator and the discriminator using the training sample set to minimize the loss function of the CGAN shown in the following formula:

[0055] L CGAN (θ D , θ G ) = Ex,y～Pdata [logD(x,y)]+E x～pdata(x),z～pz(z) [log(1-D(x,G(x,z)))]

[0056] where θ D represents the network parameters of the discriminator, and θ G represents the network parameters of the generator; the variable x represents the RGB image; z is the noise image; the variable y is the corresponding real depth image; G(x, z) is the pseudo-depth image generated by the generator; D(x, y) is the probability that the pseudo-depth image generated by the generator is judged by the discriminator as a real depth image; p data represents the data sample distribution, and p data (x) represents the real data sample distribution, and p z (z) represents the noise distribution; the symbol "~" means following a set probability distribution;

[0057] Input the original RGB image into the trained CGAN to obtain the depth image corresponding to the original RGB image.

[0058] Regarding the above implementation, it should be noted that the generator and the discriminator are two independent convolutional networks. The generator learns to generate realistic samples for each label in the training dataset, while the discriminator learns to distinguish between real sample-label pairs and fake sample-label pairs. In a sense, the goals of the two networks of the generator and the discriminator are opposite to each other, and this goal is reflected in the loss function shown in the above implementation.

[0059] Regarding the above implementation, in the specific implementation process, the embodiments of the present invention preferably use U-net as the model generator of CGAN. U-net can be regarded as an encoder-decoder model, which connects the corresponding feature maps of different layers between the encoder and the decoder. Among them, the discriminator is composed of an encoder, and the discriminator is trained to distinguish between true and false pictures. Specifically, the network architecture of CGAN adopted by the embodiments of the present invention is as Figure 2 shown. The generator takes the random noise image z and the RGB image as inputs and generates a "pseudo" depth picture G(x, z) that follows the actual image data distribution. The discriminator takes the RGB image x, the real depth picture y, and G(x, z) as inputs to determine whether the label generated by the generator is true. In the embodiments of the present invention, the specific network architectures of the generator and the discriminator of the CGAN network are consistent with this network structure design.

[0060] For Figure 1In some implementation manners of the described technical solution, constructing the Actor network and the Critic network in the Deep Deterministic Policy Gradient (DDPG) network by using the depth image and the current speed information of the unmanned aerial vehicle as state information includes:

[0061] Define the state information at the current moment as s t =[d t , v t ; where d t represents the depth image obtained by converting the RGB image collected by the monocular unmanned aerial vehicle at the current moment through the CGAN; the speed v at the current moment t =[v xt , v yt , v zt .

[0062] Construct the Actor network μ(s|θ μ ) to deterministically output an action policy based on the state information at the current moment; where μ represents the Actor network; θ μ represents the Actor network parameters; s represents the input state information.

[0063] Construct the Critic network Q(s, a|θ Q ) to evaluate the action policy output by the Actor network; where θ Q represents the Critic network parameters; s represents the state information input to the Actor network; a represents the action policy output by the Actor network.

[0064] Regarding the above implementation manner, it should be noted that deep reinforcement learning combines the "perception" ability of deep learning with the sequential decision-making ability of reinforcement learning, enabling the decision-making problem of an intelligent agent in a complex state space to be solved. The deep reinforcement learning architecture is as Figure 3 shown and generally consists of the following parts: an intelligent agent Agent, an interaction environment E, a state transition equation P (representing the probability that the intelligent agent transfers from one state to the next state), and a reward function R. Briefly speaking, the intelligent agent perceives the state of the environment where it is located, takes the state information as the input of the deep neural network, the neural network gives an action policy according to the state information, the intelligent agent executes the corresponding action and interacts with the environment, and the interaction environment gives the intelligent agent a reward r and the next state S' according to the reward function and the state transition rule. For the unmanned aerial vehicle during obstacle avoidance, the state of the unmanned aerial vehicle at the next moment only depends on the current scene and the state transition equation after the action it takes, and is independent of a series of historical states it was in before. Therefore, the obstacle avoidance problem of the unmanned aerial vehicle can be regarded as a Markov decision process and thus can be incorporated into the reinforcement learning framework for solution. Specifically, the current state St , the current action a t , the reward value r and the next state S t+1 As a tuple (s t , a t , r, s t+1 ) are collected and form a set (S, A, R, S'). The reinforcement learning algorithm optimizes the target policy π: s → a, so that the cumulative reward R obtained at time S t is maximized. The expression of R is: where γ ∈ (0, 1) is the decay factor, indicating the influence of the current reward on future rewards. The Q-value function is defined as Q π = E[R t |s t , a t . The training objective of reinforcement learning is to find the optimal policy π * to maximize the expectation of R, that is, to find the optimal policy π * such that the following equation holds:

[0065]

[0066] Based on the above description of the objective of the reinforcement learning algorithm, the embodiment of the present invention preferably uses DDPG as the algorithm model of deep reinforcement learning, and this model adopts the network structure of Actor-Critic. Combining the content described in the foregoing implementation manner, the Actor network (denoted as μ(s|θ μ )) deterministically maps the state S t to the action, that is, gives the deterministic action of the drone based on the current environmental state. The Critic network (denoted as Q(s, a|θ Q )) approximates and calculates the action extreme value function according to the current state action, where θ μ and θ Q are the network parameters of the Actor network and the Critic network respectively. In addition, in some examples, in order to reduce the overestimation of the value function, the Actor target network μ(s|θ μ' ) and the Critic target network Q(s|θ Q' ) are correspondingly designed in the DDPG model.

[0067] For the above implementation manner, in some examples, training the Actor network and the Critic network in the DDPG network according to the reward function to obtain the parameters of the Actor network and the Critic network includes:

[0068] Construct Actor target networks and Critic target networks corresponding to the Actor network and the Critic network respectively; wherein, the parameters of the Actor target network are θ μ′ , and the parameters of the Critic target network are θ Q′ ;

[0069] Execute an iterative training process based on set iterative conditions; each iterative training process is as follows:

[0070] Through the Actor network, based on the state information s at the current moment t Obtain the action policy a at the current moment t = μ(s t |θ μ ) + N(t); where the random noise N(t) linearly decays to 0 as the number of training iterations increases;

[0071] Execute the action policy through the drone and interact with the current environment to obtain the state information s at the next moment t+1 And the reward function value r i = R(s t+1 );

[0072] Store the data tuple (s t , a t , r i , s t+1 ) as a sample into the experience replay pool D for offline training of the DDPG network;

[0073] Randomly sample N samples (s i , a i , r i , s i+1 ) from the experience replay pool D;

[0074] According to the N samples, update the network parameters θ of the Actor network based on the direction of the sampling gradient ascent shown in Equation 1 μ :

[0075]

[0076] Where, E represents expectation, p β represents the state distribution of the deterministic policy, and Q(s, a|θ Q )| s＝st,a＝μ(st) represents the evaluation value of the Critic network under the state information and action policy at the current moment;

[0077] The Actor target network obtains the corresponding action policy a based on s in the sampling samples i+1 i+1 ​= μ'(s t+1 | θ μ' ), and output the said s i+1 and the said a i+1 to the Critic target network to calculate the evaluation value Q^ of the next state, where Q^ = Q(s t+1 , a t+1 | θ Q' );

[0078] According to the evaluation value Q(s t , a t | θ Q ) of the Critic network and the evaluation value Q^ of the next state of the Critic target network; update the network parameter θ Q of the Critic network by minimizing the loss function L(θ Q ):

[0079]

[0080] where Q^ = Q(s t+1 , a t+1 | θ Q' ) = r t + γQ'(s t+1 , μ'(s t+1 | θ μ' )| θ Q' ); st ∼ pβ means, at ∼ β means, rt ∼ E means, γ ∈ (0, 1) is the decay factor, representing the influence of the current moment reward on the future moment; μ'(s t+1 | θ μ' ) represents the action policy with added random noise corresponding to s i+1 in the sampling sample based on the Actor target network;

[0081] Update the network parameters of the Actor target network and the Critic target network according to Equation 3:

[0082]

[0083] where τ is the sliding average coefficient, and its value is less than 1.

[0084] Combined with the above implementation methods and their examples, specifically, for the random noise N(t), since the Actor network gives deterministic actions based on the current state information of the agent, when the state information is the same, the Actor network will only output the same action results, resulting in a reduction in the explored samples; to enable the Actor network to perform better exploration, add random noise N(t) to the policy μ to make the output action policy random, so that more exploration can be carried out.

[0085] Specifically, for the state information space and the action policy space, taking the example of using a monocular camera as a sensor to collect an RGB image with a resolution of 640*480, and through the CGAN network described in step S101, a depth image with a resolution of 84*84 can be obtained and preprocessed, in order to extract effective information from complex information, reduce complexity, and improve algorithm efficiency. As Figure 4 shown, convert the 84*84 depth image into an 84*84 grayscale image d, and superimpose the speed information v of the drone at this time t (this speed information includes the speed information of the drone in the x, y, and z directions) as the state information s at the current moment t =[d t , v t , where v t =[v x , v y , v z . For the action policy space, taking the speed v (m / s) as the main action, thus avoiding the problem of global positioning required by using position information, reducing the computational amount and algorithm complexity. Regarding the drone as a particle motion with three degrees of freedom, the action policy given by the Actor network is the speed magnitudes in three different directions. The action space of the drone at this time is shown in Table 1 below.

[0086] Table 1

[0087] Symbol Definition Action Description Value Range <![CDATA[v x > Drone speed along the x-axis (front and back) (-2，2) <![CDATA[v y > Drone speed along the y-axis (left and right) (-2，2) <![CDATA[v z > Drone speed along the z-axis (up and down) (-2，2)

[0088] For the Actor-Critic structure of the DDPG model, combined with the depth image exemplified above, the specific structures of the Actor network and the Critic network are respectively as Figure 5 and Figure 6 shown, and their specific parameter settings are as shown in Table 2 below.

[0089] Table 2

[0090] Parameter Type / Value Actor Optimizer Adam, α = 0.0001 Critic Optimizer Adam, α = 0.0005 γ 0.99 τ 0.005 D 100000 epsilon 0.98 gamma 0.99 batch size 64

[0091] Specifically, the Actor network takes in the state information of the drone at this time and outputs the actions of the drone based on this state. Compared with the conventional solution where the Actor network only inputs depth information for decision-making, the embodiments of the present invention use a multi-modal network to process decision-making information, thereby improving the network's control ability over the drone. According to the collaborative representation method, the speed information and the 84*84 image information are mapped to different feature subspaces for processing. Among them, 9 convolutional layers are used to extract image feature information, and fully connected layers are used to process speed information. After passing through the convolutional layers, the processed action information is merged with the action information that has passed through the fully connected processing, and the action information is output through three fully connected layers.

[0092] Secondly, in the Critic network, the processing process of the state information is the same as that of the Actor network, and their corresponding network structures are also the same. This process is called "observation preprocessing". Since the action input is one-dimensional data, it is input into a fully connected layer with 128 units for processing, so that the action and the processed state information have the same shape. This process is called "action preprocessing". Then, the "observation preprocessing result" and the "action preprocessing" result are combined using a merging layer and output the action evaluation value based on this state through 3 fully connected layers.

[0093] For the reward function, in some possible implementation manners, constructing the reward function according to the distance between the drone and the target, the speed of the drone, and the collision information of the drone includes:

[0094] Based on the action policy output by the Actor network according to the state information at the current moment, the state information s at the next moment is generated t+1 ;

[0095] When the state information at the next moment indicates that the drone has a collision, the reward function value r = R(s t+1 ) is the maximum penalty value, and the current training episode ends;

[0096] When the state information at the next moment indicates that the speed of the drone is lower than the set speed threshold, the reward function value is the minimum penalty value;

[0097] When the state information at the next moment indicates that the distance between the drone and the target is less than the set distance threshold, the reward function value is the maximum reward value;

[0098] Otherwise, the difference between the distance between the drone and the target at the current moment and the distance between the drone and the target at the previous moment is used as the reward function value.

[0099] Specifically, for the above implementation, the reward function inputs are {p, speed, collision}, where p is the position information used to calculate the straight-line distance dist between the drone and the target point at that moment; speed is the norm of the drone's current velocity in three directions; and collision is the collision flag, which indicates the drone's collision information provided by the simulation environment. For example, when collision is 1, it indicates that the drone has collided, while when it is 0, it indicates that the drone is in a safe flight state. The algorithm interaction process is shown in Table 3 below. When the collision flag is 1, it indicates that the drone has collided, a penalty of -20 is applied, and the training round ends. If the drone has not collided, the speed is checked to see if it is less than speed_limit. In this embodiment, speed_limit is set to 0.1 to prevent the DDPG network from freezing to avoid collision penalties. When dist < 3, it indicates that the drone has reached the target point, and a reward of 50 is given. Otherwise, the reward is the difference between the current distance between the drone and the target point and the previous distance, thus encouraging the drone to reach the designated target location while avoiding collisions.

[0100] Table 3

[0101]

[0102] In combination with the above implementation methods and their examples and explanatory contents, the overall processing flow of the technical solution provided by the embodiment of the present invention is shown in FIG. Figure 7 As shown in Figure 1, it can be divided into three parts: CGAN network, interactive environment and DDPG network. The interactive environment includes a drone equipped with a monocular camera. When the current moment is set to time t, the drone collects the RBG image of the current environment state, and inputs the RBG image into the trained CGAN network to generate a depth image and output it. The state information s at this time is obtained by processing the generated image. t The Actor network is based on the output state s at this time. t Given the drone's action strategy a t =μ(s t |θ μ ), and add noise to the action to get a t =μ(s t |θ μ )+N, change the action a t The data is sent to the drone in the form of data communication. The drone gets the action information and interacts with the environment to generate the next state value s t+1 And the action reward value r t =R(s t+1 At this point, the drone completes the environment exploration and sends the collected data (st , a t , r t , s t+1 ) are stored in the experience replay pool for subsequent network training.

[0103] The DDPG network training adopts an offline training method. First, samples are taken from the experience pool to obtain data (s i , a i , r i , s i+1 ), and then the network parameters of the Actor network are updated according to Equation 1 in the foregoing implementation manner to train it in the direction of increasing the Q value. The target Target-Actor network calculates the action value a t+1 based on the sampled data s t+1 , and transfers s t+1 , a t+1 to the target Target-Critic network to calculate the Q value Q^ = Q(s t+1 , a t+1 |θ Q' ) of the next moment state. At the same time, the Critic network calculates the current state Q value Q = Q(s t , a t |θ Q ). The Critic network is updated by minimizing the loss function, that is, Equation 2, and the target network is updated according to Equation 3.

[0104] For the foregoing technical solutions, their implementation manners and examples, the embodiments of the present invention elaborate on the technical effects through simulation experiments. The simulation conditions of the simulation experiment are as follows: The unmanned aerial vehicle uses a monocular camera to collect RGB pictures from the environment within a range of 180° in its forward direction and uses an inertial measurement unit (IMU, Inertial Measurement Unit) to measure its own linear velocity as the state observation value; Figure 8 is a schematic diagram of the simulation environment used, Figure 9The figure shows a schematic diagram of the obstacle avoidance training process for drones. When the drone begins training, the initial reward value is 0. If the drone collides, it will receive a -20 penalty. When the drone flies along the correct route and approaches the target location, it will receive a reward. Finally, when the drone safely reaches the target point, it will receive a +50 reward. The total score of the drone training is the sum of the rewards for each step during training. Therefore, after training, when the total score is greater than 50, it indicates that in the current training round, the drone can successfully avoid all obstacles and reach the designated target point. The higher the total score, the better the obstacle avoidance route selected by the drone. To verify the effectiveness of the algorithm, the DQN algorithm, DDPG, and DDPG with Prioritized Experience Replay (DDPG-PER) were trained and tested in the above simulation environment. DDPG-PER is an algorithm that introduces a prioritized experience replay mechanism, the purpose of which is to improve data utilization and accelerate training.

[0105] Figures 10 to 15 The total score, average reward, network estimated Q value, Actor network loss, Critic network loss, and average speed during the drone training process are shown as a function of training time. This training has a total of 6000 episodes.

[0106] from Figure 10 As can be seen from the graph, the DDPG algorithm provided by the embodiment of the present invention achieves a cumulative score greater than 0 after 500 training cycles, indicating that the drone has already mastered the ability to avoid obstacles. When the cumulative score reaches 50 after 1000 training cycles, it indicates that the drone has successfully avoided obstacles and reached the designated target location. Compared to DDPG, the DQN algorithm's overall score hovers around 0 throughout the training process, indicating that the drone has only learned simple obstacle avoidance capabilities but may be simply wandering in place or "roaming" in the simulation environment, unable to reach the designated location.

[0107] Figure 11 This is the average reward curve. The average reward is calculated as the total score divided by the number of steps the drone takes in that episode. It measures the effectiveness of the action decisions made by the algorithm in each training step. By comparison, the DDPG algorithm's average reward is significantly greater than that of the DQN algorithm. The introduction of the prioritized experience replay mechanism also improves its convergence and stability.

[0108] Figure 12Schematic diagram of the Q-value change curve estimated for the network. It can be seen from this that due to the repeated use of the maximum value theory, the DQN algorithm will lead to an overestimation of the Q-value, thus affecting the generation of correct decision-making actions. Since the DDPG algorithm decouples the two networks of action decision-making and value estimation, it can significantly reduce the overestimation of the value function. In addition, since the network structures of the DQN algorithm and the DDPG algorithm are different, their loss values are not comparable. Therefore Figure 13 and Figure 14 only compare the loss values Loss of DDPG and DDPG-PER. It can be seen from this that both algorithms can quickly decline and converge.

[0109] Figure 15 is the change curve of the UAV speed. This speed is the speed norm of the UAV in the x, y, and z directions. It can be seen from the figure that the speed of the UAV will finally converge to around 2.75 m / s.

[0110] Based on the above simulation experiments, the average reward value of the DDPG algorithm is significantly greater than that of the DQN algorithm. After introducing the prioritized experience replay mechanism, the convergence and stability of the algorithm are also improved. Since the DDPG algorithm decouples the two networks of action decision-making and value estimation, it can significantly reduce the overestimation of the value function.

[0111] Finally, the training results of the DQN and the DDPG algorithm model proposed in the embodiment of the present invention are tested 100 times in the test environment, and the results are shown in Table 4 below.

[0112] Table 4

[0113] Algorithm Type Average Cumulative Score Average Reward Value DQN -35.46 -2.77 DDPG 66.75 2.73

[0114] It can be seen from Table 4 that both the average cumulative score and the average reward value of DQN are much lower than those of the DDPG model proposed in the embodiment of the present invention. Specifically, the cumulative score of DQN being less than 0 points indicates that although the DQN algorithm performs well in avoiding obstacles during training, due to the weak robustness of the algorithm during testing, it is very likely that the UAV will fail to avoid obstacles due to a small perturbation. The average cumulative score of the DDPG algorithm model being greater than 50 in 100 tests indicates that in almost every test during the test process, the UAV can avoid obstacles and reach the target position. As Figure 16 shown is the comparison diagram of the UAV obstacle avoidance routes from above. Since the DDPG algorithm uses continuous speed control, its obstacle avoidance trajectory is smoother and the obstacle avoidance effect is better compared to other algorithms.

[0115] Based on the same inventive concept as the foregoing technical solution, see Figure 17, which shows a monocular UAV obstacle avoidance device 170 provided by an embodiment of the present invention. The device 170 includes: an acquisition part 1701, a conversion part 1702, a first construction part 1703, a second construction part 1704, a training part 1705, and an obstacle avoidance control part 1706; where

[0116] The acquisition part 1701 is configured to collect an original RGB image through a monocular camera carried by the UAV;

[0117] The conversion part 1702 is configured to convert the original RGB image into a depth image based on a conditional generative adversarial network CGAN;

[0118] The first construction part 1703 is configured to construct an actor network and a critic network in a deep deterministic policy gradient DDPG network by using the depth image and the current speed information of the UAV as state information;

[0119] The second construction part 1704 is configured to construct a reward function according to the distance between the UAV and the target, the speed of the UAV, and the collision information of the UAV;

[0120] The training part 1705 is configured to train the actor network and the critic network in the DDPG network according to the reward function, so as to obtain the parameters of the actor network and the critic network;

[0121] The obstacle avoidance control part 1706 is configured to perform obstacle avoidance control on the monocular UAV according to the trained DDPG network.

[0122] In some examples, the conversion part 1702 is configured to:

[0123] Construct two independent convolutional networks and set them as a generator and a discriminator respectively; where the generator is used to generate a pseudo-depth image from the RGB image and input it to the discriminator; the discriminator is used to determine whether the input image is a true depth image;

[0124] Train the network parameters of the generator and the discriminator through a training sample set so as to minimize the loss function of the CGAN shown by the following formula:

[0125] L CGAN (θ D ,θ G )=E x,y～Pdata [logD(x,y)]+E x～pdata(x),z～pz(z) [log(1-D(x,G(x,z)))]

[0126] Among them, θ D represents the network parameters of the discriminator, and θ G represents the network parameters of the generator; the variable x represents the RGB image; z is the noise image; the variable y is the corresponding true depth image; G(x, z) is the pseudo-depth image generated by the generator; D(x, y) is the probability that the pseudo-depth image generated by the generator is judged by the discriminator as a true depth image; p data represents the data sample distribution, and p data (x) represents the true data sample distribution, and p z (z) represents the noise distribution; the symbol "~" means following the set probability distribution;

[0127] Input the original RGB image into the trained CGAN to obtain the depth image corresponding to the original RGB image.

[0128] In some examples, the first construction part 1703 is configured to:

[0129] Define the state information at the current moment as s t =[d t , v t ; among them, d t represents the depth image obtained by converting the RGB image collected by the monocular drone at the current moment through the CGAN; the speed v at the current moment t =[v xt , v yt , v zt ;

[0130] Construct the Actor network μ(s|θ μ ) to represent the action policy deterministically output based on the state information at the current moment; among them, μ represents the Actor network; θ μ represents the Actor network parameters; s represents the input state information;

[0131] Construct the Critic network Q(s, a|θ Q ) to evaluate the action policy output by the Actor network; among them, θ Q represents the Critic network parameters; s represents the state information input to the Actor network; a represents the action policy output by the Actor network.

[0132] In some examples, the training part 1705 is configured to:

[0133] Construct Actor target networks and Critic target networks corresponding to the Actor network and the Critic network respectively; wherein, the parameters of the Actor target network are θ μ′ , and the parameters of the Critic target network are θ Q ′;

[0134] Execute an iterative training process based on the set iterative conditions; each iterative training process is as follows:

[0135] Based on the state information s at the current moment through the Actor network t Obtain the action policy a at the current moment t = μ(s t |θ μ ) + N(t); where the random noise N(t) linearly decays to 0 as the number of training iterations increases;

[0136] Execute the action policy through the drone and interact with the current environment to obtain the state information s at the next moment t+1 And the reward function value r i = R(s t+1 );

[0137] Store the data tuple (s t , a t , r i , s t+1 ) as a sample in the experience replay pool D for offline training of the DDPG network;

[0138] Randomly sample N samples (s i , a i , r i , s i+1 ) from the experience replay pool D;

[0139] Based on the N samples, update the network parameters θ of the Actor network according to the sampling gradient ascent direction shown in the following formula μ :

[0140]

[0141] where E represents expectation, p β represents the state distribution of the deterministic policy, and Q(s, a|θ Q )| s＝st,a＝μ(st) represents the evaluation value of the Critic network under the state information and action policy at the current moment;

[0142] The Actor target network obtains the corresponding action policy a based on s in the sampling samples i+1 i+1 ​= μ'(s t+1 |θ μ' ), and output the s i+1 and the a i+1 to the Critic target network to calculate the evaluation value Q^ of the next state, where Q^ = Q(s t+1 , a t+1 |θ Q' );

[0143] According to the evaluation value Q(s t , a t |θ Q ) of the Critic network and the evaluation value Q^ of the next state of the Critic target network; update the network parameter θ Q of the Critic network by minimizing the loss function L(θ Q ):

[0144]

[0145] where Q^ = Q(s t+1 , a t+1 |θ Q' ) = r t + γQ'(s t+1 , μ'(s t+1 |θ μ' )|θ Q' ); s t ~p β means, a t ~β means, r t ~E means, γ ∈ (0, 1) is the attenuation factor, representing the influence of the current moment reward on the future moment; μ'(s t+1 |θ μ' ) represents the action policy with added random noise corresponding to s i+1 in the sampling sample based on the Actor target network;

[0146] Update the network parameters of the Actor target network and the Critic target network according to the following formula:

[0147]

[0148] where τ is the sliding average coefficient, and its value is less than 1.

[0149] In some examples, the second construction part 1704 is configured to:

[0150] Generate the state information s t+1 of the next moment based on the action policy output by the Actor network according to the state information of the current moment;

[0151] When the state information at the next moment indicates that the UAV has a collision, the reward function value r = R(s t+1 ) is the maximum penalty value, and the current training episode ends;

[0152] When the state information at the next moment indicates that the speed of the UAV is lower than the set speed threshold, the reward function value is the minimum penalty value;

[0153] When the state information at the next moment indicates that the distance between the UAV and the target is less than the set distance threshold, the reward function value is the maximum reward value;

[0154] Otherwise, the difference between the distance between the UAV and the target at the current moment and the distance between the UAV and the target at the previous moment is used as the reward function value.

[0155] It can be understood that in this embodiment, "part" can be a part of a circuit, a part of a processor, a part of a program or software, etc. Of course, it can also be a unit, or a module, or non-modular.

[0156] In addition, the components in this embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware, or in the form of a software function module.

[0157] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0158] Therefore, this embodiment provides a computer storage medium storing a monocular UAV obstacle avoidance program based on deep deterministic policy gradient. When the monocular UAV obstacle avoidance program based on deep deterministic policy gradient is executed by at least one processor, the steps of the monocular UAV obstacle avoidance method in the above technical solution are implemented.

[0159] According to the above monocular UAV obstacle avoidance device 170 based on deep deterministic policy gradient and the computer storage medium, refer to Figure 18 , which shows the specific hardware structure of a computing device 180 that can implement the above monocular UAV obstacle avoidance device 170 based on deep deterministic policy gradient. The computing device 180 can be a wireless device, a mobile or cellular phone (including so-called smart phones), a personal digital assistant (PDA), a video game console (including a video display, a mobile video game device, a mobile video conferencing unit), a laptop computer, a desktop computer, a set-top box, a tablet computing device, an e-book reader, a fixed or mobile media player, etc. The computing device 180 includes: a communication interface 1801, a memory 1802, and a processor 1803; each component is coupled together through a bus system 1804. It can be understood that the bus system 1804 is used to realize the connection and communication between these components. In addition to including a data bus, the bus system 1804 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 18 all kinds of buses are labeled as the bus system 1804. Among them,

[0160] The communication interface 1801 is used for receiving and sending signals during the process of receiving and sending information with other external network elements;

[0161] The memory 1802 is used for storing a computer program that can run on the processor 1803;

[0162] The processor 1803 is used for executing the steps of the monocular UAV obstacle avoidance method in the above technical solution when running the computer program, which will not be elaborated here.

[0163] It can be understood that the memory 1802 in the embodiments of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM). The memory 1802 of the systems and methods described herein is intended to include but not be limited to these and any other suitable types of memory.

[0164] The processor 1803 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method can be completed by the integrated logic circuit of the hardware in the processor 1803 or the instructions in the form of software. The above processor 1803 may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 1802, and the processor 1803 reads the information in the memory 1802 and combines its hardware to complete the steps of the above method.

[0165] It can be understood that these embodiments described herein can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or a combination thereof.

[0166] For software implementation, the technologies described herein can be implemented by modules (such as procedures, functions, etc.) that execute the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented inside or outside the processor.

[0167] Understandably, the exemplary technical solutions of the above-mentioned monocular UAV obstacle avoidance device 170 and computing device 180 based on deep deterministic policy gradient belong to the same concept as the technical solutions of the aforementioned monocular UAV obstacle avoidance method based on deep deterministic policy gradient. Therefore, for the details not described in detail in the technical solutions of the above-mentioned monocular UAV obstacle avoidance device 170 and computing device 180 based on deep deterministic policy gradient, reference can be made to the description of the technical solutions of the aforementioned monocular UAV obstacle avoidance method based on deep deterministic policy gradient. The embodiments of the present invention will not repeat them here.

[0168] It should be noted that the technical solutions described in the embodiments of the present invention can be combined arbitrarily without conflict.

[0169] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, and all of them should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A monocular UAV obstacle avoidance method based on deep deterministic policy gradient, characterized by: The method comprises: The original RGB images captured by the monocular camera on the drone are converted into depth images based on the conditional generative adversarial network (CGAN). Using the depth image and the current speed information of the drone as state information, constructing an actor network and a critic network in a deep deterministic policy gradient (DDPG) network; Construct a reward function based on the distance between the drone and the target, the speed of the drone, and the collision information of the drone; Training the Actor network and the Critic network in the DDPG network according to the reward function, thereby obtaining parameters of the Actor network and the Critic network; Obstacle avoidance control of monocular drone is performed based on the trained DDPG network; The reward function is constructed based on the distance between the drone and the target, the speed of the drone, and the collision information of the drone, including: The state information at the next moment is generated based on the action strategy output by the Actor network according to the state information at the current moment ; When the state information at the next moment indicates that the drone has collided, the reward function value r = R ( s t+1 ) is the maximum penalty value, and the current training round ends; When the state information at the next moment indicates that the speed of the drone is lower than the set speed threshold, the reward function value is the minimum penalty value; When the state information at the next moment indicates that the distance between the UAV and the target is less than a set distance threshold, the reward function value is a maximum reward value; Otherwise, the difference between the distance between the drone and the target at the current moment and the distance between the drone and the target at the previous moment is used as the reward function value; The method uses the depth image and the current speed information of the drone as state information to construct the actor network and the critic network in the deep deterministic policy gradient DDPG network, including: Define the current state information as ;in, d t represents the depth image obtained by converting the RGB image collected by the monocular drone at the current moment through the CGAN; the speed at the current moment ; Build the Actor network To indicate the deterministic output of action strategies based on the current state information; Represents an Actor network; Represents Actor network parameters; Indicates the status information of the input; Constructing the Critic network To evaluate the action strategy output by the Actor network; Represents the critic network parameters; Represents the status information of the input Actor network; Represents the action strategy output by the Actor network.

2. The method according to claim 1, characterized in that The original RGB image captured by the monocular camera carried by the drone is converted into a depth image based on the conditional generative adversarial network (CGAN), including: Construct two independent convolutional networks and set them as a generator and a discriminator respectively; wherein the generator is used to generate a pseudo depth image from the RGB image and input it to the discriminator; the discriminator is used to determine whether the input image is a true depth image; The network parameters of the generator and the discriminator are trained using a training sample set to minimize the loss function of the CGAN shown in the following formula: in, represents the network parameters of the discriminator, Represents the network parameters of the generator; variables x Represents an RGB image; z is a noise image; variable y is the corresponding real depth image; The pseudo depth image generated by the generator; The probability that the pseudo depth image generated by the generator is identified as a real depth image by the discriminator; p data represents the data sample distribution, p data ( x ) represents the real data sample distribution, p z ( z ) represents the noise distribution; the symbol "~" means that it obeys the set probability distribution; The original RGB image is input into the trained CGAN to obtain a depth image corresponding to the original RGB image.

3. The method according to claim 1, characterized in that The step of training the Actor network and the Critic network in the DDPG network according to the reward function, thereby obtaining parameters of the Actor network and the Critic network, includes: Construct an Actor target network and a Critic target network corresponding to the Actor network and the Critic network respectively; wherein the Actor target network parameters are , the critic target network parameters are ; The iterative training process is performed based on the set iteration conditions; each iterative training process is: Based on the current state information of the Actor network Get the current action strategy ; Among them, the random noise N(t) decays linearly with the number of training iterations until it reaches 0; The UAV executes the action strategy and interacts with the current environment to obtain the state information of the next moment. And the reward function value r i = R ( s t+1 ); The data tuple Stored as a sample in the experience replay pool D For offline training of DDPG network; Randomly sample N samples from the experience replay pool D ; According to the N samples, the network parameters of the Actor network are updated based on the direction of the sampled gradient ascent shown in the following formula : Among them, E represents expectation, represents the state distribution of the deterministic policy, Represents the evaluation value of the critic network under the current state information and action strategy; The Actor target network is based on the sampled Get the corresponding action strategy , and the and the Output to the Critic target network to calculate the evaluation value of the next state ; According to the evaluation value of the Critic network And the evaluation value of the next state of the Critic target network ; By minimizing the loss function shown below To update the network parameters of the Critic network : in, ; express The state distribution obeys the deterministic policy, express Obey and The same probability distribution, express Obey and The same probability distribution, is the attenuation factor, which indicates the impact of the current moment reward on the future moment; Indicates that the Actor target network is based on the sampled samples The corresponding action strategy with random noise added; Update the network parameters of the Actor target network and the Critic target network according to the following formula: in, is the sliding average coefficient, which is less than 1.

4. A monocular drone obstacle avoidance device based on deep deterministic policy gradient, characterized in that: The device includes: an acquisition part, a conversion part, a first construction part, a second construction part, a training part and an obstacle avoidance control part; wherein, The acquisition part is configured to acquire the original RGB image through a monocular camera carried by the drone; The conversion part is configured to convert the original RGB image into a depth image based on a conditional generative adversarial network (CGAN); The first construction part is configured to use the depth image and the current speed information of the drone as state information to construct an actor network and a critic network in a deep deterministic policy gradient (DDPG) network; The second construction part is configured to construct a reward function according to the distance between the drone and the target, the speed of the drone, and the collision information of the drone; The training part is configured to train the Actor network and the Critic network in the DDPG network according to the reward function, thereby obtaining parameters of the Actor network and the Critic network; The obstacle avoidance control part is configured to perform obstacle avoidance control on the monocular drone based on the trained DDPG network; Wherein, the first construction part is configured as follows: Define the current state information as ;in, d t represents the depth image obtained by converting the RGB image collected by the monocular drone at the current moment through the CGAN; the speed at the current moment ; Build the Actor network To indicate the deterministic output of action strategies based on the current state information; Represents an Actor network; Represents Actor network parameters; Indicates the status information of the input; Constructing the Critic network To evaluate the action strategy output by the Actor network; Represents the critic network parameters; Represents the status information of the input Actor network; Represents the action strategy output by the Actor network; The second construction part is configured as follows: The state information at the next moment is generated based on the action strategy output by the Actor network according to the state information at the current moment ; When the state information at the next moment indicates that the drone has collided, the reward function value r = R ( s t+1 ) is the maximum penalty value, and the current training round ends; When the state information at the next moment indicates that the speed of the drone is lower than the set speed threshold, the reward function value is the minimum penalty value; When the state information at the next moment indicates that the distance between the UAV and the target is less than a set distance threshold, the reward function value is a maximum reward value; Otherwise, the difference between the distance between the drone and the target at the current moment and the distance between the drone and the target at the previous moment is used as the reward function value.

5. The device according to claim 4, characterized in that The conversion part is configured as follows: Construct two independent convolutional networks and set them as a generator and a discriminator respectively; wherein the generator is used to generate a pseudo depth image from the RGB image and input it to the discriminator; the discriminator is used to determine whether the input image is a true depth image; The network parameters of the generator and the discriminator are trained using a training sample set to minimize the loss function of the CGAN shown in the following formula: in, represents the network parameters of the discriminator, Represents the network parameters of the generator; variables x Represents an RGB image; z is a noise image; variable y is the corresponding real depth image; The pseudo depth image generated by the generator; The probability that the pseudo depth image generated by the generator is identified as a real depth image by the discriminator; p data represents the data sample distribution, p data ( x ) represents the real data sample distribution, p z ( z ) represents the noise distribution; the symbol "~" means that it obeys the set probability distribution; The original RGB image is input into the trained CGAN to obtain a depth image corresponding to the original RGB image.

6. The device according to claim 4, characterized in that The training part is configured as follows: Construct an Actor target network and a Critic target network corresponding to the Actor network and the Critic network respectively; wherein the Actor target network parameters are , the critic target network parameters are ; The iterative training process is performed based on the set iteration conditions; each iterative training process is: Based on the current state information of the Actor network Get the current action strategy ; Among them, the random noise N(t) decays linearly with the number of training iterations until it reaches 0; The UAV executes the action strategy and interacts with the current environment to obtain the state information of the next moment. And the reward function value r i = R ( s t+1 ); The data tuple Stored as a sample in the experience replay pool D For offline training of DDPG network; Randomly sample N samples from the experience replay pool D ; According to the N samples, the network parameters of the Actor network are updated based on the direction of the sampled gradient ascent shown in the following formula : Among them, E represents expectation, represents the state distribution of the deterministic policy, Represents the evaluation value of the critic network under the current state information and action strategy; The Actor target network is based on the sampled Get the corresponding action strategy , and the and the Output to the Critic target network to calculate the evaluation value of the next state ; According to the evaluation value of the Critic network And the evaluation value of the next state of the Critic target network ; By minimizing the loss function shown below To update the network parameters of the Critic network : in, ; express The state distribution obeys the deterministic policy, express Obey and The same probability distribution, express Obey and The same probability distribution, is the attenuation factor, which indicates the impact of the current moment reward on the future moment; Indicates that the Actor target network is based on the sampled samples The corresponding action strategy with random noise added; Update the network parameters of the Actor target network and the Critic target network according to the following formula: in, is the sliding average coefficient, which is less than 1.

7. A computer storage medium, characterized in that The computer storage medium stores a monocular drone obstacle avoidance program based on deep deterministic policy gradient. When the monocular drone obstacle avoidance program based on deep deterministic policy gradient is executed by at least one processor, the steps of the monocular drone obstacle avoidance method based on deep deterministic policy gradient according to any one of claims 1 to 3 are implemented.

Images