Unmanned aerial vehicle visual obstacle avoidance control method based on non-expert strategy reinforcement learning guidance

Abstract

The invention discloses an unmanned aerial vehicle visual obstacle avoidance control method based on non-expert strategy reinforcement learning guidance. The unmanned aerial vehicle visual obstacle avoidance control method realizes collision-free autonomous flight of an unmanned aerial vehicle from a starting point to a target point. In the online learning process of the intelligent agent, an artificial potential field method is converted into discrete actions through fuzzy logic, the intelligent agent is guided in a non-expert strategy mode in the initial learning stage to optimize and compete for the double Q network through behavior cloning, and then the intelligent agent is gradually transited to deep reinforcement learning. An environment is explored autonomously, a color image is subjected to visual coding to generate a state vector, and a more accurate and robust competitive double Q network is trained according to a reward signal. The trained neural network model generates the Q value of each action under the current time step according to the state obtained in real time, and the unmanned aerial vehicle executes the corresponding linear speed according to the maximum Q value, so that the obstacle avoidance task is completed. According to the method, the autonomous obstacle avoidance capability of the unmanned aerial vehicle in a partially observable and dense obstacle environment is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and unmanned aerial vehicle system technology, and in particular relates to a unmanned aerial vehicle visual obstacle avoidance control method based on non-expert strategy reinforcement learning guidance. Background Art

[0002] Quadrotors have the advantages of high maneuverability and versatility, and have broad application prospects in the civil field. In UAV systems, autonomous obstacle avoidance is one of the core issues of autonomous flight of UAVs. In recent years, with the rapid development of artificial intelligence and deep reinforcement learning technology, obstacle avoidance methods based on deep reinforcement learning have gradually received widespread attention. Deep reinforcement learning combines the feature extraction of deep learning and the decision optimization capabilities of reinforcement learning, and shows good potential in improving the environmental adaptability and obstacle avoidance performance of UAVs. However, with the increase in the complexity, duration and scale of UAV missions, the challenge of autonomous obstacle avoidance is becoming increasingly severe, especially in crowded and uncertain environments. The uncertainty of the environment and the high degree of freedom of UAVs will cause the dimensionality disaster of the system state-action space. Deep reinforcement learning optimizes the control strategy by collecting samples through continuous interaction with the environment, but when the dimension of the state-action space grows exponentially, the trial-and-error mechanism of the algorithm requires a large number of state-action pairs to be optimized, resulting in a very resource-intensive convergence process in complex environments. Low sampling efficiency has therefore become a key challenge. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a UAV visual obstacle avoidance control method based on non-expert strategy reinforcement learning guidance. It can effectively solve the problem of autonomous obstacle avoidance control of UAVs in partially observable environments and significantly improve the sampling efficiency and learning effect of UAVs.

[0004] Technical solution: The present invention provides a UAV visual obstacle avoidance control method based on non-expert strategy reinforcement learning guidance, comprising the following steps:

[0005] Step 1: Construct a non-expert strategy based on the artificial potential field method and discretize the continuous actions generated by the artificial potential field method using fuzzy logic;

[0006] Step 2: Update the main network and target network of the competitive dual Q network by autonomously exploring the environment, obtain images taken by the onboard monocular camera from the environment and perform visual encoding, and evaluate the reward or penalty of the agent at the current time step based on the obtained state vector;

[0007] Step 3: Construct a competitive dual Q network and load the trained network parameters. The agent obtains the state vector during the interaction with the simulation environment. The competitive dual Q network uses the state vector as input to solve the Q value of each action of the agent in the current state. The action decision network selects the action with the maximum expected return based on the maximum Q value. The agent executes the action to avoid obstacles and fly to the preset target point, completing the autonomous obstacle avoidance task.

[0008] Furthermore, step 1 specifically includes the following steps:

[0009] Step 1.1: Based on the artificial potential field method, the position information of obstacles and targets is used to artificially construct the gravitational field and repulsive field, so that the intelligent agent can navigate to the preset target point while avoiding collision;

[0010] Step 1.2: Fuzzification: The continuous action value is converted into a fuzzy value through the triangular membership function. The fuzzy rules evaluate the applicability of each discrete action. The results of these rules are then aggregated to form a fuzzy output set. The aggregated fuzzy output is converted back to an exact value through defuzzification to determine the final discrete action.

[0011] Step 1.3, Behavior cloning: At the beginning of training, the initial parameters of the competitive dual Q network and the non-expert strategy π based on the artificial potential field will be used. n (s) Generate two sets of velocities and calculate the imitation loss relative to the non-expert strategy

[0012] Furthermore, step 1.1 is as follows: Given that the action space of the agent is two-dimensional, the dynamic equation of the agent with mass m in the resultant force field F is as follows:

[0013]

[0014] Among them, θ represents the angle between the resultant force field F and the x-axis in the world coordinate system;

[0015] When the agent is at position p at time step t a When it is at position p g The gravitational field exerted by the target point will guide the movement of the agent. The calculation formula is as follows:

[0016]

[0017] Among them, τ is the gravitational field coefficient, d(p a ,p g ) represents the Euclidean distance between the agent and the target point, d * It is a distance parameter that needs to be adjusted manually;

[0018] The repulsive field exerted by the obstacle will guide the agent to perform avoidance actions. Only the repulsive field generated by the obstacle closest to the agent is considered, while the influence of other obstacles is ignored. It is defined as follows:

[0019]

[0020] Among them, ξ is the repulsive field coefficient, d(p a ,p o ) represents the Euclidean distance between the agent and the nearest obstacle, Represents the maximum collision radius that needs to be manually adjusted. The resultant force generated by the potential field is obtained by the following formula:

[0021]

[0022] Among them, U(p a )=U att (p a )+U rep (p a ), is the gradient operator about the agent's position, U att (p a ) and U rep (p a ) is as follows:

[0023]

[0024] Let d(p a ,p o )=D to obtain:

[0025]

[0026] Furthermore, step 1.2 is as follows: fuzzy logic allows truth values ​​to be between 0 and 1, and the fuzzy set A defined on the domain X is represented by the membership function μ A :X→[0,1] representation, where μ A (x) = 0 means it does not belong to the set at all, μ A (x) = 1 means that it completely belongs to the set; the triangular membership function is selected to discretize the continuous actions generated by the artificial potential field, and the fuzzy operation includes the union μ A∪B (x) = max(μ A (x),μ B (x)), the intersection μ A∩B (x) = min(μ A (x),μ B (x)) and complement The fuzzy inference system converts precise input into fuzzy values, evaluates fuzzy rules, summarizes the results and generates precise output values ​​through defuzzification. Its calculation formula is as follows:

[0027]

[0028] where μ B (y) is the membership function of the output fuzzy set B, and y represents the output variable.

[0029] Furthermore, step 1.3 is as follows: At the beginning of training, the initial parameters of the competitive dual Q network and the non-expert strategy π based on the artificial potential field are used. n (s) Generates two sets of speeds, the former is used for the agent to execute and interact with the environment, and the latter is stored in the experience replay pool together with the corresponding state vector; after filling the experience replay pool, the algorithm samples a small batch of data from it, generates speeds based on the current Q network, and calculates the imitation loss relative to the non-expert strategy To calculate First, we need to convert the Q value output by the network into a probability distribution:

[0030]

[0031] Then, by calculating P(s;θ) and π n (s) to obtain the cross entropy loss between

[0032]

[0033] Furthermore, step 2 specifically includes the following steps:

[0034] Step 2.1, agent state vector acquisition: The state vector has three elements, including a color image taken by an onboard monocular camera from the environment, the Euclidean distance between the preset target point and the agent's current position, and the angle difference between the preset target point and the agent's current orientation;

[0035] Step 2.2, construct the main network and target network of the competitive dual Q network, and further optimize the network by autonomously exploring the environment on the basis of updating the network parameters based on behavior cloning. Specifically, the agent continuously interacts with the environment, and the historical trajectory generated in this process will be stored in the experience replay pool. The historical trajectory is a four-tuple (s, a, r, s′), where s is the state vector of the agent at the current time step, a is the action it performs, that is, the linear velocity of the agent in the x and y directions, r is the reward or penalty obtained by performing action a, and s′ is the new state vector of the environment transferred to the next time step after performing action a; the interaction process is: the agent obtains visual input, state information of itself and the target point, and forms a state vector containing three elements. The competitive dual Q network uses the state vector as input to solve the Q value of each action of the agent in the current state. The action decision network selects the action with the maximum expected return according to the maximum Q value. The agent performs the action and obtains the state vector and reward or penalty for the next time step;

[0036] Step 2.3: After the experience replay pool is filled, small batches of random sampling are performed from it, and the network parameters of the competitive dual Q network are updated using stochastic gradient descent. Specifically, the Adam algorithm is used to batch update the main network parameters each time; at regular time steps, the target network parameters are replaced and updated, that is, the main network parameters are used to replace the target network parameters;

[0037] Step 2.4: When the number of training episodes reaches a preset condition or when the network parameter update amplitude is less than a preset threshold, the update process is terminated and the model parameters of the competitive dual Q network are saved; the update process is performed in units of each episode. During this process, each episode starts from the initial state. When the agent meets any of the following conditions, the current episode ends and the learning of the next episode is restarted. The conditions include: the agent flies out of the boundary of the definition domain, the agent collides with an obstacle in the environment, the total time step of the agent's flight exceeds the preset maximum time step, and the agent reaches the preset target point.

[0038] Furthermore, in step 2.3, the target network parameters are replaced and updated, and the total loss function calculation formula of the training process is as follows:

[0039]

[0040] Where α represents the weight decay factor and γ is a scaling parameter used to adjust the size of the imitation loss term to make it consistent with the temporal difference loss. M represents the size of the mini-batch, β is a hyperparameter, V(s) and A(s,a) represent the state value function and action advantage function respectively, and |A| represents the size of the set of candidate actions.

[0041] Furthermore, step 3 specifically includes the following steps:

[0042] Step 3.1, agent state vector acquisition: The state vector has three elements, including a color image taken by an onboard monocular camera from the environment, the Euclidean distance between the preset target point and the agent's current position, and the angle difference between the preset target point and the agent's current orientation;

[0043] Step 3.2, Q function evaluation based on competitive dual Q network: Construct a competitive dual Q network, load the trained network parameters, use the state vector of the agent at the current time step as the network input, and solve the Q value of each action of the agent in the current state;

[0044] Step 3.3, action selection: The action decision network selects the action with the maximum expected reward based on the maximum Q value and determines the linear velocity of the agent in the x and y directions at the current time step;

[0045] Step 3.4: The agent performs the action and obtains the state vector of the next time step;

[0046] Step 3.5: Repeat steps 3.1 to 3.4 until the intelligent agent flies to the preset target point without collision and completes the autonomous obstacle avoidance task.

[0047] The present invention also discloses a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0048] The present invention also discloses a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method of the present invention are implemented.

[0049] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0050] (1) The present invention proposes a visual autonomous obstacle avoidance control method for unmanned aerial vehicles guided by non-expert strategy reinforcement learning. In the early stage of learning, a non-expert strategy based on the artificial potential field method is introduced as a teacher, and the intelligent agent is used as a student. The action decision network is optimized through behavior cloning, thereby solving the problem of low sampling efficiency of deep reinforcement learning training. In order to further solve this problem, the method introduces fuzzy logic to convert the continuous actions of the artificial potential field method into discrete actions, and reduces the demand for the number of state-action pairs by reducing the dimension of the action space. In the later stage of learning, the method gradually turns to relying on deep reinforcement learning to independently explore the environment, generates state vectors by visually encoding color images, and trains a more accurate and robust action decision network based on reward (penalty) signals, thereby improving the sampling efficiency and learning effect of the intelligent agent at the same time, and realizing autonomous obstacle avoidance control of unmanned aerial vehicles in partially observable environments.

[0051] (2) The UAV visual autonomous obstacle avoidance control method guided by non-expert strategy reinforcement learning in this invention relies only on visual input and builds a pixel-action end-to-end autonomous obstacle avoidance framework. This framework can be used in situations where the UAV’s perception is limited, effectively improving the UAV’s ability to safely navigate in complex and uncertain environments, ensuring that the UAV can avoid obstacles and achieve autonomous flight.

[0052] (3) The UAV visual autonomous obstacle avoidance control method guided by non-expert strategy reinforcement learning of the present invention introduces a non-expert strategy based on the artificial potential field method in the early stage of training, provides prior knowledge, reduces the algorithm's demand for state-action pairs, and significantly improves the sampling efficiency in the early stage of learning; in the later stage of training, the algorithm gradually transitions to the stage of autonomous exploration of the environment, further improving the convergence speed and overall performance of the algorithm, enabling the UAV to adapt to and optimize its obstacle avoidance strategy more quickly in a complex environment.

[0053] (4) The visual autonomous obstacle avoidance control method for unmanned aerial vehicles guided by non-expert strategy reinforcement learning of the present invention proposes a solution to convert the continuous action of the artificial potential field method into discrete actions through fuzzy logic, avoiding the dimensionality curse problem of the state-action space. While ensuring the flight performance of the unmanned aerial vehicle, it further improves the sampling efficiency and ensures the stability and effectiveness of the algorithm when facing high-dimensional state space. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a block diagram of a UAV visual autonomous obstacle avoidance control system guided by non-expert strategy reinforcement learning;

[0055] Figure 2 is a schematic diagram of a competitive dual-Q network structure in an embodiment of the present invention;

[0056] Figure 3is the membership function used in the fuzzy logic in the embodiment of the present invention;

[0057] Figure 4 1 is a diagram of the flight trajectory of a UAV in a simple cylindrical obstacle scenario in an embodiment of the present invention; wherein (a) is a scenario with 10 cylindrical obstacles, and (b) is a scenario with 15 cylindrical obstacles;

[0058] Figure 5 1 is a flight trajectory diagram of a UAV in a complex furniture obstacle scene in an embodiment of the present invention; wherein (c) is a scene with 10 furniture obstacles, and (d) is a scene with 15 furniture obstacles. DETAILED DESCRIPTION

[0059] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings. Figure 1 and Figure 2 As shown, the present invention is a method for autonomous obstacle avoidance control of unmanned aerial vehicle based on non-expert strategy reinforcement learning guidance, and its specific contents include:

[0060] 1. Construct a non-expert strategy based on the artificial potential field method, and then use fuzzy logic to discretize the continuous actions generated by the artificial potential field method. In the early stage of learning, non-expert discrete actions are introduced as teachers. The intelligent agent imitates the teacher's actions through behavioral cloning, thereby improving sampling efficiency and accelerating the convergence process of learning.

[0061] 1.1 Artificial potential field method

[0062] The artificial potential field method uses the position information of obstacles and targets to artificially construct gravitational fields and repulsive fields. This manual design enables the agent to navigate to the preset target point without collision. Given that the action space of the agent is two-dimensional, the dynamic equation of the agent with mass m in the resultant force field F is as follows:

[0063]

[0064] Among them, θ represents the angle between the resultant force field F and the x-axis in the world coordinate system, and the mass of the drone is m = 1.48 kg.

[0065] Specifically, when the agent is at position p at time step t a When it is at position p g The gravitational field exerted by the target point will guide the movement of the agent. The calculation formula is as follows:

[0066]

[0067] Among them, τ is the gravitational field coefficient, d(p a ,p g ) represents the Euclidean distance between the agent and the target point, d* It is a distance parameter that needs to be adjusted manually.

[0068] Similarly, the repulsive field exerted by the obstacle will guide the agent to perform avoidance actions. This method only considers the repulsive field generated by the obstacle closest to the agent, while ignoring the influence of other obstacles. It is defined as follows:

[0069]

[0070] Among them, ξ is the repulsive field coefficient, d(p a ,p o ) represents the Euclidean distance between the agent and the nearest obstacle, represents the maximum collision radius that needs to be manually adjusted. Therefore, the resultant force generated by the potential field can be obtained by the following formula:

[0071]

[0072] Among them, U(p a )=U att (p a )+U rep (p a ), is the gradient operator about the agent’s position. att (p a ) and U rep (p a ) is as follows:

[0073]

[0074] Let d(p a ,p o )=D, we can get:

[0075]

[0076] Among them, τ = 1.0, ξ = 0.5, d * =1m,

[0077] 1.2 Fuzzy Logic

[0078] Fuzzy logic allows truth values ​​to be between 0 and 1. A fuzzy set A defined on the domain X is represented by the membership function μ A :X→[0,1] representation, where μ A (x) = 0 means it does not belong to the set at all, μ A (x) = 1 means it completely belongs to the set. The invention selects a triangular membership function to discretize the continuous actions generated by the artificial potential field. Fuzzy operations include the union (μ A∪B (x) = max(μA (x),μ B (x))), intersection (μ A∩B (x) = min(μ A (x),μ B (x))) and complement A fuzzy inference system converts precise inputs into fuzzy values, evaluates fuzzy rules (e.g., "if x_1 is A_1 and x_2 is A_2, then y is B"), aggregates the results, and produces precise output values ​​by defuzzification (usually using the centroid method), using the following calculation formula:

[0079]

[0080] where μ B (y) is the membership function of the output fuzzy set B, and y represents the output variable. Using fuzzy logic to map continuous actions to discrete actions involves three steps. First, fuzzification converts the continuous action values ​​into fuzzy values ​​using triangular membership functions. Next, fuzzy rules evaluate the applicability of each discrete action, and the results of these rules are then aggregated to form a fuzzy output set. Finally, defuzzification converts the aggregated fuzzy output back to an exact value to determine the final discrete action.

[0081] like Figure 3 As shown, the discrete action set includes dividing the speed range of -1.5m / s to 1.5m / s into intervals of 0.5m / s, and calculating the membership values ​​of these specific areas based on the continuous speed, and finally the discrete action with the highest membership value is output. For the linear speed of the drone in the x-axis direction (controlling the forward movement of the drone), the fuzzy logic uses a triangular membership function with a width of 1.0m / s centered on each discrete action. Specifically, for the endpoint membership function, the membership of the speed exceeding ±1.5m / s is set to 1, and the membership in the range of 1.0m / s to 1.5m / s is set to 0, thereby mapping it to a smaller discrete action. This method tends to select a lower speed, effectively limiting the linear speed of the drone in the x-axis direction, thereby enhancing the forward obstacle avoidance capability. For the membership of other discrete speeds, it is calculated by a triangular function, and the smaller discrete speed is preferred when the membership is equal. For the x-axis, the membership function at its endpoint is defined as:

[0082]

[0083] The membership functions of other points are defined as:

[0084]

[0085] Different from the linear velocity in the x-axis direction, the membership function of the linear velocity in the y-axis direction is always 1 at the endpoint, and the other points use a triangular membership function similar to the x-axis. When the y-axis velocity membership is equal, the velocity with a larger absolute value is preferred to improve the lateral maneuverability of the drone, thereby providing more flexible lateral obstacle avoidance actions.

[0086] 1.3 Behavioral cloning

[0087] At the beginning of training, the initial parameters of the competitive dual Q network and the non-expert strategy π based on the artificial potential field are used. n (s) Generate two sets of speeds, the former is used for the execution of the agent and interaction with the environment, and the latter is stored in the experience replay pool together with the corresponding state vector, which is initialized to empty. In this embodiment, N = 100000. First, initialize the main network parameters ω of the competitive dual Q network and initialize the target network parameters ω - ←ω. After filling the experience replay pool, the algorithm samples a small batch of data from it, generates it according to the current Q network speed, and calculates the imitation loss relative to the non-expert strategy To calculate First, we need to convert the Q value output by the network into a probability distribution:

[0088]

[0089] Then we can calculate P(s;θ) and π n (s) to obtain the cross entropy loss between

[0090]

[0091] 2. Update the main network and target network of the competitive dual Q network by autonomously exploring the environment, thereby solving the shortcoming that non-expert strategies are easily trapped in local optimal values, thereby further improving the autonomous obstacle avoidance capability of the drone. The main process includes obtaining color images taken by the onboard monocular camera from the environment and performing visual encoding, evaluating the reward (penalty) of the agent at the current time step based on the obtained state vector, and further updating the main network and target network of the competitive dual Q network to realize the action selection of the agent in the current state space.

[0092] 2.1 Agent state vector acquisition: The state vector has three elements, including three consecutive frames of 224×224 color images taken by the onboard monocular camera from the environment, the Euclidean distance d between the preset target point and the current position of the agent, and the angle difference θ between the preset target point and the current orientation of the agent;

[0093] 2.2 Construct the main network and target network of the competitive dual Q network. On the basis of updating the network parameters based on behavioral cloning, the network is further optimized by autonomously exploring the environment. The specific steps include: the agent continues to interact with the environment, and the historical trajectory generated in this process will be stored in the experience replay pool with a capacity of N. If the amount of data in the experience pool is greater than the maximum capacity of the experience pool, the latest experience data will replace the oldest experience data. The historical trajectory is a four-tuple (s, a, r, s′), where s is the state vector of the agent at the current time step, a is the action it performs, that is, the linear velocity of the agent in the x and y directions, r is the reward (penalty) obtained by performing action a, and s′ is the new state vector of the environment transferred to the next time step after performing action a. The interaction process is as follows: the agent obtains visual input, state information of itself and the target point, and forms a state vector containing three elements. The competitive dual Q network uses the state vector as input to solve the Q value of each action of the agent in the current state. The action decision network selects the action with the maximum expected return according to the maximum Q value. The agent executes the action and obtains the state vector and reward (penalty) of the next time step. The reward function value r can be obtained by the following formula:

[0094]

[0095] 2.3 After the experience replay pool is filled, small batches of random sampling are performed from it, and the network parameters of the competitive dual Q network are updated using stochastic gradient descent. The specific steps include: each update uses the Adam algorithm to update the main network parameters in batches with a learning rate of η = 0.0001; every 4 episodes, the target network parameters are replaced and updated, that is, the main network parameters are used to replace the target network parameters. The total loss function calculation formula of the training process is as follows:

[0096]

[0097] where α varies between 0.05 and 0.95, decays by 0.05 every 500 time steps, γ = 0.99, M = 64, β = 2.5, and |A| = 7.

[0098] In this embodiment, the network structure includes two parts: a visual encoder and a competitive dual Q network. The visual encoder consists of two layers of two-dimensional convolutional layers, and outputs a feature vector with a dimension of 18×18. Specifically, the first convolutional layer uses 32 convolution kernels of size 4×3 and a step size of 4; the second convolutional layer uses 64 convolution kernels of the same size with a step size of 3. Both convolutional layers use the ReLU activation function. After flattening, the feature vector is concatenated with the Euclidean distance d between the preset target point and the current position of the agent, and the angle difference θ between the preset target point and the current orientation of the agent, and input into the competitive dual Q network. The competitive dual Q network contains two branches: a state value function branch and an advantage function branch. Both the state value function branch and the advantage function branch contain three fully connected layers; the number of hidden nodes in the first fully connected layer and the second fully connected layer of the two branches is 256, and both use the ReLU activation function. The number of hidden nodes in the third fully connected layer of the state value function branch and the advantage function branch are 1 and 7, respectively, representing the value function of the current state and the advantage function of the seven candidate actions in the current state.

[0099] 2.4 When the number of training episodes reaches the preset condition or when the network parameter update amplitude is less than the preset threshold ∈ = 1e -6 When , the updating process ends and the model parameters of the competitive dual Q network are saved.

[0100] The above update process is performed in units of each episode. In this process, each episode starts from the initial state. When the agent meets any of the following conditions, the current episode ends and the learning of the next episode starts again: 1) the agent flies out of the boundary of the definition domain; 2) the agent collides with obstacles in the environment; 3) the total time step of the agent's flight exceeds the preset maximum time step; 4) the agent reaches the preset target point. In this embodiment, the maximum time step is set to 80.

[0101] 3. Construct a competitive dual Q network and load the trained network parameters. The agent obtains the state vector during the interaction with the simulation environment. The competitive dual Q network uses the state vector as input to solve the Q value of each action of the agent in the current state. The action decision network selects the action with the maximum expected return based on the maximum Q value. The agent executes the action to avoid obstacles and fly to the preset target point, completing the autonomous obstacle avoidance task.

[0102] 3.1 Agent state vector acquisition: The state vector has three elements, including three consecutive frames of 224×224 color images taken by the onboard monocular camera from the environment, the Euclidean distance d between the preset target point and the current position of the agent, and the angle difference θ between the preset target point and the current orientation of the agent;

[0103] 3.2 Q function evaluation based on competitive dual Q network: Construct a competitive dual Q network and load the trained network parameters. Use the state vector of the agent at the current time step as the network input to calculate the Q value of each action of the agent in the current state.

[0104] 3.3 Action selection: The action decision network selects the action with the maximum expected reward based on the maximum Q value and determines the linear velocity of the agent in the x and y directions at the current time step;

[0105] 3.4 The agent performs the action and obtains the state vector of the next time step;

[0106] 3.5 Repeat the above process until the agent flies to the preset target point without collision and completes the autonomous obstacle avoidance task. Figure 4 As shown in Figure 2, the flight trajectory of the drone in the actual environment is as follows: Figure 5 shown.

Claims

1. A visual obstacle avoidance control method for unmanned aerial vehicles based on non-expert strategy reinforcement learning guidance, characterized in that: The steps include: Step 1: Construct a non-expert strategy based on the artificial potential field method and discretize the continuous actions generated by the artificial potential field method using fuzzy logic; Step 2: Update the main network and target network of the competitive dual Q network by autonomously exploring the environment, obtain images taken by the onboard monocular camera from the environment and perform visual encoding, and evaluate the reward or penalty of the agent at the current time step based on the obtained state vector; Step 3: Construct a competitive dual Q network and load the trained network parameters. The agent obtains the state vector during the interaction with the simulation environment. The competitive dual Q network uses the state vector as input to solve the Q value of each action of the agent in the current state. The action decision network selects the action with the maximum expected return based on the maximum Q value. The agent executes the action to avoid obstacles and fly to the preset target point, completing the autonomous obstacle avoidance task.

2. According to claim 1, a method for controlling unmanned aerial vehicle visual obstacle avoidance based on non-expert strategy reinforcement learning guidance is characterized in that: Step 1 specifically includes the following steps: Step 1.1: Based on the artificial potential field method, the position information of obstacles and targets is used to artificially construct the gravitational field and repulsive field, so that the intelligent agent can navigate to the preset target point while avoiding collision; Step 1.2: Fuzzification: The continuous action value is converted into a fuzzy value through the triangular membership function. The fuzzy rules evaluate the applicability of each discrete action. The results of these rules are then aggregated to form a fuzzy output set. The aggregated fuzzy output is converted back to an exact value through defuzzification to determine the final discrete action. Step 1.3, Behavior cloning: At the beginning of training, the initial parameters of the competitive dual Q network and the non-expert strategy π based on the artificial potential field will be used. n (s) Generate two sets of velocities and calculate the imitation loss relative to the non-expert policy 3. The unmanned aerial vehicle visual obstacle avoidance control method based on non-expert strategy reinforcement learning guidance according to claim 2 is characterized in that: Step 1.1 is as follows: Given that the action space of the agent is two-dimensional, the dynamic equation of the agent with mass m in the resultant force field F is as follows: Among them, θ represents the angle between the resultant force field F and the x-axis in the world coordinate system; When the agent is at position p at time step t a When it is at position p g The gravitational field exerted by the target point will guide the movement of the agent. The calculation formula is as follows: Among them, τ is the gravitational field coefficient, d(p a ,p g ) represents the Euclidean distance between the agent and the target point, d * is a distance parameter that needs to be adjusted manually; The repulsive field exerted by the obstacle will guide the agent to perform avoidance actions. Only the repulsive field generated by the obstacle closest to the agent is considered, while the influence of other obstacles is ignored. It is defined as follows: Among them, ξ is the repulsive field coefficient, d(p a ,p o ) represents the Euclidean distance between the agent and the nearest obstacle, Represents the maximum collision radius that needs to be manually adjusted. The resultant force generated by the potential field is obtained by the following formula: Among them, U(p a )=U att (p a )+U rep (p a ), is the gradient operator about the agent's position, U att (p a ) and U rep (p a ) is as follows: Let d(p a ,p o )=D to obtain:

4. The method for controlling unmanned aerial vehicle visual obstacle avoidance based on non-expert strategy reinforcement learning guidance according to claim 2 is characterized in that: Step 1.2 is as follows: Fuzzy logic allows truth values ​​to be between 0 and 1. The fuzzy set A defined on the domain X is represented by the membership function μ A :X→[0,1] representation, where μ A (x) = 0 means it does not belong to the set at all, μ A (x) = 1 means that it completely belongs to the set; the triangular membership function is selected to discretize the continuous actions generated by the artificial potential field, and the fuzzy operation includes the union μ A∪B (x) = max(μ A (x),μ B (x)), the intersection μ A∩B (x) = min(μ A (x),μ B (x)) and the complement μ -A (x) = 1-μ A (x); The fuzzy inference system converts precise input into fuzzy values, evaluates fuzzy rules, summarizes the results and generates precise output values ​​through defuzzification. Its calculation formula is as follows: where μ B (y) is the membership function of the output fuzzy set B, and y represents the output variable.

5. The method for controlling unmanned aerial vehicle visual obstacle avoidance based on non-expert strategy reinforcement learning guidance according to claim 2 is characterized in that: Step 1.3 is as follows: At the beginning of training, the initial parameters of the competitive dual Q network and the non-expert strategy π based on the artificial potential field are used. n (s) Generates two sets of speeds, the former is used for the agent to execute and interact with the environment, and the latter is stored in the experience replay pool together with the corresponding state vector; after filling the experience replay pool, the algorithm samples a small batch of data from it, generates speeds based on the current Q network, and calculates the imitation loss relative to the non-expert strategy To calculate First, we need to convert the Q value output by the network into a probability distribution: Then, by calculating P(s;θ) and π n (s) to obtain the cross entropy loss between 6. The method for controlling unmanned aerial vehicle visual obstacle avoidance based on non-expert strategy reinforcement learning guidance according to claim 1 is characterized in that: Step 2 specifically includes the following steps: Step 2.1, agent state vector acquisition: The state vector has three elements, including a color image taken by an onboard monocular camera from the environment, the Euclidean distance between the preset target point and the agent's current position, and the angle difference between the preset target point and the agent's current orientation; Step 2.2, construct the main network and target network of the competitive dual Q network, and further optimize the network by autonomously exploring the environment on the basis of updating the network parameters based on behavior cloning. Specifically, the agent continuously interacts with the environment, and the historical trajectory generated in this process will be stored in the experience replay pool. The historical trajectory is a four-tuple (s, a, r, s′), where s is the state vector of the agent at the current time step, a is the action it performs, that is, the linear velocity of the agent in the x and y directions, r is the reward or penalty obtained by performing action a, and s′ is the new state vector of the environment transferred to the next time step after performing action a; the interaction process is: the agent obtains visual input, state information of itself and the target point, and forms a state vector containing three elements. The competitive dual Q network uses the state vector as input to solve the Q value of each action of the agent in the current state. The action decision network selects the action with the maximum expected return according to the maximum Q value. The agent performs the action and obtains the state vector and reward or penalty for the next time step; Step 2.3: After the experience replay pool is filled, small batches of random sampling are performed from it, and the network parameters of the competitive dual Q network are updated using stochastic gradient descent. Specifically, the Adam algorithm is used to batch update the main network parameters each time; at regular time steps, the target network parameters are replaced and updated, that is, the main network parameters are used to replace the target network parameters; Step 2.4: When the number of training episodes reaches a preset condition or when the network parameter update amplitude is less than a preset threshold, the update process is terminated and the model parameters of the competitive dual Q network are saved; the update process is performed in units of each episode. During this process, each episode starts from the initial state. When the agent meets any of the following conditions, the current episode ends and the learning of the next episode is restarted. The conditions include: the agent flies out of the boundary of the definition domain, the agent collides with an obstacle in the environment, the total time step of the agent's flight exceeds the preset maximum time step, and the agent reaches the preset target point.

7. The method for controlling unmanned aerial vehicle visual obstacle avoidance based on non-expert strategy reinforcement learning guidance according to claim 6 is characterized in that: In step 2.3, the target network parameters are replaced and updated, and the total loss function calculation formula of the training process is as follows: Where α represents the weight decay factor and γ is a scaling parameter used to adjust the size of the imitation loss term to make it consistent with the temporal difference loss. M represents the size of the mini-batch, β is a hyperparameter, V(s) and A(s,a) represent the state value function and action advantage function respectively, and |A| represents the size of the set of candidate actions.

8. The method for controlling unmanned aerial vehicle visual obstacle avoidance based on non-expert strategy reinforcement learning guidance according to claim 1 is characterized in that: Step 3 specifically includes the following steps: Step 3.1, agent state vector acquisition: The state vector has three elements, including a color image taken by an onboard monocular camera from the environment, the Euclidean distance between the preset target point and the agent's current position, and the angle difference between the preset target point and the agent's current orientation; Step 3.2, Q function evaluation based on competitive dual Q network: Construct a competitive dual Q network, load the trained network parameters, use the state vector of the agent at the current time step as the network input, and solve the Q value of each action of the agent in the current state; Step 3.3, action selection: The action decision network selects the action with the maximum expected reward based on the maximum Q value and determines the linear velocity of the agent in the x and y directions at the current time step; Step 3.4: The agent performs the action and obtains the state vector of the next time step; Step 3.5: Repeat steps 3.1 to 3.4 until the intelligent agent flies to the preset target point without collision and completes the autonomous obstacle avoidance task.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 1.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

Images