A method for generating obstacle avoidance strategies for distributed UAV swarms

Abstract

The application provides a distributed unmanned aerial vehicle cluster obstacle avoidance strategy generation method, comprising the following steps: one, establishing an unmanned aerial vehicle cluster trajectory model; two, state space design; three, action space design; four, obtaining a state transition equation; five, reward function design; six, updating Q_Table by using a Q-Learning algorithm. The distributed unmanned aerial vehicle cluster obstacle avoidance strategy can determine the passing order of each unmanned aerial vehicle under a preset trajectory, and avoid movement deadlock of the unmanned aerial vehicle cluster. The application does not need to set a long machine in the unmanned aerial vehicle cluster, and when a single unmanned aerial vehicle fails, the whole system controlled by the application can still operate normally to complete the unmanned aerial vehicle cluster obstacle avoidance task.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a method for generating a distributed UAV swarm obstacle avoidance strategy. Background Technology

[0002] The motion strategy of a drone swarm on a specific trajectory refers to the movement strategy of a drone swarm (multiple drones) flying on the same trajectory to ensure that they do not collide with each other. Since the trajectory may have one or more intersections, when there is only one drone at the intersection node, it is only necessary to ensure a safe distance between each drone and its neighboring drones. However, when multiple drones are near the intersection point, a motion strategy is needed to determine the passing order of each drone in order to avoid motion deadlock.

[0003] Chinese patent publication number CN114428516A discloses a method for obstacle avoidance in drone swarms. The method involves the following steps: First, a leader drone is selected within the swarm, with the other drones acting as wingmen. Second, the leader drone identifies the initial formation of the swarm by acquiring the position information of the wingmen. Third, both the leader and wingmen sense obstacles ahead, and the wingmen relay this information back to the leader. Fourth, the leader determines the new formation and position of the swarm based on the obstacle information, updates the target position of each drone in the swarm for the next moment, and sends this target position to the wingmen. Fifth, the swarm changes formation based on the target position for the next moment. Sixth, it determines whether the swarm has passed the obstacle. Seventh, it determines whether the current moment is the last moment. While this method provides an effective obstacle avoidance mechanism for drone swarms, the entire system will be unable to effectively avoid obstacles if the leader drone fails. Summary of the Invention

[0004] The purpose of this invention is to provide a method for generating a distributed drone swarm obstacle avoidance strategy, which can effectively solve the problem of collisions in drone swarms under a preset trajectory.

[0005] The technical solution of this invention is as follows:

[0006] Step 1: Establish a drone swarm trajectory model and control the drones to move between adjacent trajectory points using different motion control commands.

[0007] Step 2: The drone swarm moves along a preset flight path. Design a state space S and an action space A.

[0008] Step 3: Obtain the state transition equation.

[0009] Step 4: Design the reward function:

[0010] The drone swarm moves along a pre-set flight path. The reward function is determined by the state s and the action a. The reward function at time t is designed as follows:

[0011]

[0012] Among them, (s t ) represents the Euclidean distance between any two drones in the cluster, and mini_distance represents the minimum Euclidean distance between two drones.

[0013] Step 5: Update the Q_Table using the Q-Learning algorithm:

[0014] The Q-Learning value is represented as Q(s,a), s∈S, a∈A. Its actual physical meaning is: the expected cumulative reward obtained by choosing action a in state s, and the expected cumulative reward obtained at time t is:

[0015]

[0016] Where γ is the reward discount factor.

[0017] A table Q_Table is constructed based on the state space and action space of the drone swarm. Q_Table stores the expected cumulative reward Q(s,a) that can be obtained by taking different actions s in each state.

[0018] The Q-Learning algorithm is used to adjust and update the values ​​in the Q_Table. As the number of episodes of exploration and learning increases sufficiently, the values ​​in the Q_Table will gradually converge.

[0019] After completing the exploration and learning process, the drone swarm uses the Q_Table to make corresponding decisions. That is, it selects the action that brings the maximum Q(s,a) by querying the Q_Table, which is the optimal action, and thus completes obstacle avoidance.

[0020] The beneficial effects of this invention are as follows: The distributed drone swarm obstacle avoidance strategy of this invention can determine the passage order of each drone under a preset trajectory, avoiding motion deadlock in the drone swarm. This invention does not require setting a leader drone in the drone swarm; when a single drone malfunctions, the entire system controlled by this invention can still operate normally and complete the drone swarm obstacle avoidance task. Attached Figure Description

[0021] Figure 1 This is a flowchart of the invention;

[0022] Figure 2 This is a flowchart of the Q-Learning algorithm for updating the Q_Table. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0024] Please refer to Figure 1 As shown, this embodiment includes the following steps:

[0025] Step 1: Establish a drone swarm trajectory model:

[0026] The trajectory in space is discretized to obtain an ordered sequence P of trajectory points. L ={1,2,3,…,L}, where L is the number of trajectory points, and the larger L is, the smoother the trajectory.

[0027] This embodiment specifies that each UAV has only three motion control commands: forward, backward, and wait, denoted as Cmd = {forward, back, wait}. Here, forward means moving forward one trajectory point, wait means staying at the current trajectory point, and back means moving backward one trajectory point. Different motion control commands are used to control the UAV's movement between adjacent trajectory points.

[0028] Step 2, State Space Design:

[0029] In an environment where a drone swarm moves along a pre-defined flight path, the state space S is related to the number of drones N and the sequence of trajectory points P in the swarm. L This is relevant. Design a state space accordingly. Where s = [p1, p2, p3, ... p n …,p N ] is an N-dimensional vector, p n p represents the position of the nth drone on a specific trajectory. n ∈P L .

[0030] Step 3: Motion Space Design

[0031] In an environment where a swarm of drones moves along a pre-defined flight path, the number of drones is N, and the motion control command Cmd has 3 types. Design an action space A = {a1, a2, a3, ..., a...} N}, where a = [c1, c2, c3, ... c n …,c N ] is an N-dimensional vector, c n This represents the instruction received by the nth drone. n A value of 0 indicates that the drone is forwarding. nA value of 1 indicates that the drone is waiting, c n A value of 2 indicates that the drone is back.

[0032] Step 4: Obtain the state transition equation:

[0033] Based on the definitions of state space and action space, we can deduce that action a is executed at time t. t Subsequently, the state change △s of the drone swarm t The expression is: △s t =1-a t = [1-c11-c2…1-c N ], where a t ∈A. Therefore, at time t, in action a... t The state transition relationship under the action is: s t+1 =△s t +s t , where s t+1 ∈S, S t ∈S, a t ∈A.

[0034] Step 5: Reward Function Design

[0035] In an environment where a swarm of drones moves along a pre-defined flight path, the reward function is determined by the state s and the action a. To ensure that the swarm can successfully complete its flight mission along a specific trajectory without collisions, a reward function at time t is designed as follows:

[0036]

[0037] Where N is the number of drones, c n d(s) represents the instruction received by the nth drone. t ) represents the Euclidean distance between any two drones in the cluster, and mini_distance is determined based on the size of the drone.

[0038] Step 6: Converge to the optimal policy using the Q-Learning algorithm and update the Q_Table:

[0039] Q-learning is a reinforcement learning algorithm that converges to the optimal policy under conditions of parameter selection conforming to the norm. Therefore, Q-learning can be used to score the corresponding actions in all states of a drone swarm and select the action based on the highest score. The Q-Learning value is represented as Q(s,a), s∈S, a∈A, and its actual physical meaning is the expected cumulative reward obtained by choosing action a in state s. Then, the expected cumulative reward at time t is:

[0040]

[0041] Here, γ is the reward discount factor. A table Q_Table is constructed based on the state space and action space of the drone swarm. Q_Table stores the expected cumulative reward Q(s,a) obtained by taking different actions s in each state. The Q-Learning algorithm is used to adjust and update the values ​​in Q_Table. As the number of exploration and learning episodes is sufficient, the values ​​in Q_Table will gradually converge. After completing the exploration and learning, the drone swarm uses Q_Table to make corresponding decisions, that is, by querying Q_Table to select the action that brings the maximum Q(s,a), which is the optimal policy action.

[0042] In some preferred embodiments, the Q-Learning algorithm updates the Q_Table as follows: Figure 2 As shown, it includes the following steps:

[0043] A1. Set the learning rate α∈(0,1), set the reward discount rate γ∈(0,1), and set the parameter ∈-greedy ∈∈(0,1);

[0044] A2. Initialize Q_Table, set all values ​​to 0. The rows of Q_Table correspond to the state space S, and the columns correspond to the action space A.

[0045] A3. At time t, a state s is randomly selected from the state space. t ;

[0046] A4. According to state s t Use the ∈-greedy strategy to select an action a from the action space A. t ;

[0047] A5. Drone swarm execution actions a t The state s at the next time step is obtained according to the state transition equation. t+1 The immediate reward r is obtained based on the reward function. t+1 ;

[0048] A6. Update the state s in Q_Table t and action a t The corresponding Q value is updated using the following formula:

[0049] Q(s t ,a t )←Q(s t ,a t )+α[r t+1 +γmaxQ(s t+1 ,a)-Q(s t ,a t)];

[0050] A7. Update the initial state, s t ←s t+1 ;

[0051] A8. Repeat steps A3-A7 above until the set number of explorations is reached, and obtain the converged Q_Table.

Claims

1. A method for generating a distributed UAV cluster obstacle avoidance strategy, characterized in that The method comprises the following steps: Step one, establishing a UAV cluster trajectory model, controlling the UAV to move between adjacent trajectory points through different motion control instructions; Step two, the UAV cluster moves under the preset flight trajectory, a state space S and an action space A are designed; Step three, a state transition equation is obtained; Step four, a reward function is designed: The UAV cluster moves under the preset flight trajectory, and the reward function is determined by the state s and the action a. The reward function at time t is designed as: wherein d(s t ) represents the Euclidean distance between any two UAVs in the cluster, and mini_distance represents the minimum Euclidean distance between the two UAVs; s t represents the state at time t, a t represents the action at time t, N represents the number of UAVs in the UAV cluster, and c n represents the instruction received by the nth UAV, and c n is set to 0 to indicate that the UAV moves forward, c n is set to 1 to indicate that the UAV waits, and c n is set to 2 to indicate that the UAV moves backward. Step five, Q_Table is updated by using the Q-Learning algorithm: The Q-Learning value is represented as Q(s, a), s∈S, a∈A, and the actual physical meaning is that the expected cumulative reward can be obtained by selecting the action a under the state s. Therefore, the expected cumulative reward at time t is: Wherein, γ is a reward discount factor, and k represents time k; Based on the state space and the action space of the UAV cluster, a table Q_Table is formed, and Q_Table stores the expected cumulative reward Q(s, a) that can be obtained by selecting different actions s under each state; The values in Q_Table are adjusted and updated by using the Q-Learning algorithm. When the number of exploration and learning times episode is sufficient, the values in Q_Table will gradually converge; After completing the exploration and learning, the UAV cluster makes corresponding decisions by using Q_Table, that is, by querying Q_Table to select the action that can bring the maximum Q(s, a), that is, the optimal action, so as to complete obstacle avoidance.

2. The method of claim 1, wherein, Step one is specifically: Discretize the trajectory in space to obtain a set of ordered trajectory point sequences P L ={1,2,3,…,L}, where L is the number of trajectory points; It is specified that the motion control instruction of each UAV is only forward, backward and wait, denoted as Cmd={forward, back, wait}, wherein forward means moving forward by one trajectory point, wait means keeping at the current trajectory point, and back means moving backward by one trajectory point.

3. The method of claim 2, wherein, State space in step two where s = [p1, p2, p3,... pn], p n n represents the position of the nth UAV in the trajectory. N ], p n n represents the position of the nth UAV in the trajectory.

4. The method of claim 2, wherein, The action space A in step two is {a1, a2, a3, ..., a...} N }, where a = [c1, c2, c3, ... c n …,c N ].

5. The distributed UAV cluster obstacle avoidance strategy generation method according to claim 3 or 4, characterized in that, According to the definition of state space and action space, the action a at time t is obtained t After that, the change state of the UAV cluster state is obtained t , △s t = 1-a t = [1-c1 1-c2 … 1-c N ], wherein a t ∈A; At time t, under the action of action a t The transition relationship between states is: s t+1 = Δs t + s t where s t+1 ∈ S, S t ∈ S, a t ∈ A.

6. The method of claim 5, wherein, In step five, Q_Table is updated by using the Q-Learning algorithm, and the step specifically comprises: A1, setting the learning rate α∈(0,1), setting the reward discount rate γ∈(0,1), and setting the parameter ∈ of the -greedy ∈∈(0,1); A2, initializing Q_Table, setting all values to 0, and the rows of Q_Table correspond to the state space S and the columns correspond to the action space A; A3. At time t, a state s is randomly selected from the state space t ; A4、 according to state s t , select an action a from action space A using an e-greedy policy t ; A5, the UAV group executes action a t , the next time state s is obtained according to the state transition equation t+1 , the instant reward r is obtained according to the reward function t+1 ; A6. Update the state s in Q_Table t and action a t corresponding Q value, the update formula is: Q(s t ,a t )←Q(s t ,a t )+α[r t+1 +γmaxQ(s t+1 ,a)-Q(s t ,a t )]; A7, update initial state, s t ← s t+1 ; A8, repeat the above steps A3-A7 until the set exploration times are reached, and obtain the converged Q_Table.

Images