An unmanned aerial vehicle swarm intelligent cooperative obstacle avoidance method

Abstract

This application relates to the field of intelligent cooperative UAV technology and discloses an intelligent cooperative obstacle avoidance method for UAV swarms. The method includes: setting initial parameters for the ISSLO and MADDPG algorithms; treating the UAVs as agents; using the environment in which the UAVs perform their tasks as a simulated obstacle avoidance environment; and having the ISSLO and MADDPG algorithms interact independently with the simulated obstacle avoidance environment using their respective probabilities. The UAV swarm continuously interacts with the simulated obstacle avoidance environment, receiving corresponding rewards at each time step and updating the weights of the policy network and value network until the UAV swarm reaches its destination or encounters an obstacle, at which point the current round ends. After several rounds of interaction, until the maximum number of training rounds is reached, the obstacle avoidance training of the UAV swarm is completed, and a pre-trained model is output. The MADDPG algorithm loads the pre-trained model and interacts with the simulated obstacle avoidance environment to achieve online path planning during UAV swarm obstacle avoidance. This application effectively improves the path planning capability of UAVs during obstacle avoidance.

Description

Technical Field

[0001] This application relates to the field of intelligent collaborative technology for unmanned aerial vehicles (UAVs), and in particular to an intelligent collaborative obstacle avoidance method for UAV swarms. Background Technology

[0002] Small multi-rotor drones have been widely used in various fields due to their small size, agile maneuvers, and suitability for maneuvering in complex environments. Drone-related technologies include flight control, formation control, cooperative combat, and path planning, with path planning being one of the key technologies for improving drone autonomy and intelligence. Drone obstacle avoidance is essentially a path planning problem, which involves finding a collision-free, optimal or near-optimal flight path within a given environment that allows the drone to complete its mission while satisfying both its own and environmental constraints. However, achieving real-time autonomous navigation and obstacle avoidance in unfamiliar environments remains a pressing issue.

[0003] Reinforcement learning, with its powerful autonomous learning and decision-making capabilities, has become a cutting-edge and core direction in UAV obstacle avoidance technology, demonstrating enormous potential, especially in handling complex, dynamic, and unknown environments. Compared to traditional UAV obstacle avoidance methods, reinforcement learning can interact with the environment to autonomously learn and improve strategies without requiring precise mathematical models of the environment, thus better adapting to unknown and dynamically changing environments. Reinforcement learning methods can bypass the cumulative delays in perception, planning, and control inherent in traditional methods, enabling more agile obstacle avoidance responses in highly dynamic scenarios. Furthermore, in multi-UAV systems, multi-agent reinforcement learning can solve complex cooperative obstacle avoidance problems that are difficult for centralized control to handle by learning collaborative strategies, showing great potential in swarm applications.

[0004] MADDPG is a widely used algorithm in multi-agent reinforcement learning, especially in the intelligent collaboration of UAV swarms. However, MADDPG still suffers from problems such as overestimation bias in the value function, training instability, and slow convergence speed. Summary of the Invention

[0005] In view of this, this application provides an intelligent cooperative obstacle avoidance method for drone swarms to help drone swarms find better obstacle avoidance planning strategies.

[0006] This application discloses an intelligent cooperative obstacle avoidance method for unmanned aerial vehicle (UAV) swarms, which includes: Step 1: Initialize the location information of the drone, obstacles, and destination; establish the drone dynamics model, the drone's state space, action space, and the reward function required for training. Step 2: Set the initial parameters for the ISSLO and MADDPG algorithms. Treat the UAV as an intelligent agent and the environment in which the UAV performs its mission as the simulated obstacle avoidance environment. The ISSLO and MADDPG algorithms interact with the simulated obstacle avoidance environment independently with their respective probabilities. After interacting with the simulated obstacle avoidance environment, the ISSLO algorithm uses population optimization to obtain the optimal solution. The optimal solution is then converted into an experience sample and imported into the experience pool of the MADDPG algorithm. The MADDPG algorithm samples from the experience pool to complete the reuse and learning of high-quality experience. Step 3: The drone swarm continuously interacts with the simulated obstacle avoidance environment, receiving corresponding rewards at each time step and updating the weights of the policy network and value network until the drone swarm reaches the destination or encounters an obstacle, at which point the round ends; after several rounds of interaction, until the maximum number of rounds is reached, the obstacle avoidance training of the drone swarm is completed, and the pre-trained model is output. Step 4: The MADDPG algorithm loads the pre-trained model and interacts with the simulated obstacle avoidance environment to realize online path planning when the UAV swarm avoids obstacles.

[0007] Furthermore, the expression for the UAV dynamics model is:

[0008] In the formula, , and These represent the speed, track angle, and heading angle of the UAV, respectively. , , These respectively represent the drones in The x-axis coordinate, y-axis coordinate, and z-axis coordinate at time t. Indicates the decision time interval. , , These respectively represent the drones in The x-axis, y-axis, and z-axis coordinates at any given time; the x-axis represents the direction of the UAV's movement, the y-axis is perpendicular to the x-axis, and the z-axis is perpendicular to the xoy plane.

[0009] Furthermore, constructing the state space of the UAV includes: The location and distance information of the target point and obstacles together constitute the UAV's status information. :

[0010] In the formula, and These represent the distances between the drone's current position and the target point and the obstacle, respectively. and These represent the differences between the drone's trajectory angle and the trajectory angles of the target point and the obstacle, respectively. and These represent the differences between the heading angle of the drone and the heading angles of the target point and the obstacle, respectively.

[0011] Furthermore, the operational space of the drone includes: The discrete velocity range of the UAV is:

[0012] in, For the speed of the drone, , , All of these are the speeds of the drone, and all are related to the magnitude of the change in the drone's turning angle; If it is known that the drone is at the current moment Location track angle and heading angle When the drone selects a certain turning action and The speed at this time is V After the drone completes the turning maneuver, its new position will be:

[0013] In summary, the action variables of the drone are: A is the action, and there are 3 variables. It is the output value of the policy network; , These are the new track angle and heading angle, respectively, and their angular changes are: and .

[0014] Furthermore, the expression for the reward function is:

[0015] In the formula, For the reward function, To maximize the reward value when the drone reaches the target point after performing the action, for The distance between the drone and the target at all times. for The distance between the drone and the target at all times. for The distance between the drone and the obstacle in real time. for The distance between the drone and the obstacle in real time; For the drone formation radius, The threat radius of the obstacle; The distance between the drone and the nearest known obstacle; The drone receives the maximum reward value when it reaches the target point after performing the action. A larger reward value is obtained when the drone approaches the target point and moves away from obstacles after performing the action. When the drone approaches the target point after performing the action and is within the threat range of the obstacle, it may not necessarily collide with the obstacle, thus receiving a smaller reward value. If the drone moves away from the target point after performing the action, it will incur a significant penalty. When the distance between the drone and the obstacle is less than 10 after the action is performed. When this happens, it is considered that a collision has occurred with an obstacle, and the maximum penalty value will be obtained. ; .

[0016] Further, step 2 includes: During training, the hyperparameters of the ISSLO and MADDPG algorithms are initialized. Each agent in the MADDPG algorithm represents a drone, and multiple agents form a drone swarm. The ISSLO algorithm and the MADDPG algorithm each have their own probability. , It can independently interact with the simulated obstacle avoidance environment, in which Before interaction, first generate a random number between 0 and 1, rand. The ISSLO algorithm interacts with the environment to obtain the optimal solution, which is stored as high-quality experience in the experience pool of the MADDPG algorithm for sampling. If Then the agent of the MADDPG algorithm interacts with the environment and completes sampling training until the iteration ends.

[0017] Furthermore, the ISSLO algorithm interacts with the environment to obtain the optimal solution, including: The search begins by randomly initializing the population within the decision space using the following formula:

[0018] in, Indicates the first The candidate solution positions represent the individual's initial learning state; Refers to the first The nth candidate solution, the action variable of the nth... The position value of the dimension; , They refer to the first Lower and upper limits of dimensional position values; rand It is a random variable in the range [0,1]. The optimal objective function value can be obtained using the following formula:

[0019] In the formula, (·) refers to the objective function; It is the optimal objective function value; Represents the optimal individual; The independent variable is the one that takes the minimum value. This represents the total number of candidate solutions. The learning state, i.e., the candidate solution, for each individual is obtained using the following formula:

[0020] In the formula, Refers to the new first Individual learning status; This represents the learning state of another randomly selected individual; This refers to the corresponding fitness value; D-dimensional random vectors within, The contraction-expansion coefficient is a fixed constant. For Hadarmar; The position of candidate solution S is updated as follows:

[0021]

[0022] in, The updated candidate solution, This indicates taking the absolute value. express random numbers, This represents exponentiation with base e. b It is a constant. Indicates the maximum number of iterations. Indicates the current number The next iteration; Update the position again after the initial update:

[0023] In the formula, S Represents the current candidate solution. Indicates the updated candidate solution; express Random numbers; , These represent the lower and upper limits of the candidate solution, respectively. Update the following formula: Position of candidate solutions :

[0024]

[0025] In the formula, (·) is an indicator function. Used to generate a vector of -1 or 1, which determines the direction of position update; The Hadamard product represents the element-wise multiplication of two vectors. The exponential decay factor is used to determine... The decay rate; Indicates the first The number of update failures for each individual; This refers to the maximum number of iterations. Update the following formula: Position of candidate solutions :

[0026] If the weakest learner fails to meet the requirement within the specified number of attempts, the following formula will be used to update the [number]th learner. Position of candidate solutions The learner refers to a specific solution, i.e., an action value; the weakest learner is the worst solution.

[0027] In the formula, Refers to the individual with the best current fitness; It is obtained through the following formula:

[0028]

[0029]

[0030] in, To Updated candidate solution positions Refers to the first Individual, the first Knowledge exchange operations across dimensions Refers to the first Individual, the first Logical negation of the knowledge exchange mask in dimensions Indicates the first in the crowd Knowledge exchange rate of individuals; For the first Individual, the first A random number with dimensions in the range (0,1); The dimension that is randomly selected; It is a positive number, and its value range is [0.2, 0.3]. , The result is a random number, and its value range is (0,1].

[0031] In the formula, Indicates the first Individual, the first Updated in dimension ; , They represent the first The lower and upper limits of the position values ​​in each dimension.

[0032] Furthermore, the following formulas are used to obtain the knowledge exchange rate for each individual and the number of update failures for each learner:

[0033] In the formula, Indicates the first In the nth iteration Individual, , The first The individual The exchange rate of old knowledge saved in the previous iteration and the... The learner Number of update failures in the next iteration For the first The individual The knowledge exchange rate corresponding to +1 iteration I This indicates whether the current iteration update was successful. At that time, the iteration update was successful. I When it is 1, At that time, this iteration update failed. I =0, For the first The individual The knowledge exchange rate corresponding to each iteration.

[0034] Furthermore, the contraction-expansion coefficient The expression is:

[0035] in, ; This represents the maximum number of iterations. Will Replace with a chaotic vector:

[0036] in, In the first The chaotic vector at the next iteration In the first The chaotic vector at the next iteration; The expression for the candidate solution becomes:

[0037] In the formula, Indicates the first The individual in the first The chaotic vector at the next iteration; By employing a novel strategy combining oppositional learning and population-differential directional perturbation, the expression for the candidate solution is transformed into:

[0038] In the formula, This is the scaling factor; , These are two randomly selected candidate solutions.

[0039] Furthermore, the MADDPG algorithm samples the experience sample data in the experience pool using the following sampling strategy: If n < :

[0040] If n ≥ :

[0041] in, It is a positive number, and its value range is [0.6, 0.7]. n A random number in the range (0,1); Indicates the first The interval priority value selected in the next iteration; Indicates the first In the next iteration, the maximum priority value of the experience samples in the updated experience pool is determined. Indicates the first In the next iteration, the minimum priority value in the updated experience pool of experience samples; For the first During the sampling, a priority interval is selected, and the minimum batch size of the empirical sample is randomly drawn from that interval. The storage format of the empirical pool is as follows: , The first in the experience pool An empirical sample, , , , These refer to the experience pool number 1 and 2 respectively. The state observed in each empirical sample, the action taken, the reward received after taking the action, and the priority value. The new environmental state after taking action; During each training sampling, the temporal difference error and corresponding priority value of the empirical sample are recalculated; each time a new empirical sample is generated, it is updated in the empirical pool; among which, the priority of the new empirical sample... Y The calculation formula is as follows:

[0042] in, Refers to reward value; Discount factor; Let Q be the target policy network value. Q-value of the main policy network; It is a constant to prevent 0; σ These are hyperparameters used to balance priority distribution. σ The larger the value, the higher the probability that the empirical sample with the higher the time-series difference error will be sampled.

[0043] Due to the adoption of the above technical solution, this application has the following advantages: 1. This application adopts the MADDPG algorithm architecture guided by the ISSLO algorithm. This architecture makes full use of the global search capability of the ISSLO algorithm to improve the policy exploration capability of the reinforcement learning-type MADDPG algorithm and avoid it from getting trapped in local optima.

[0044] 2. This application adopts a new empirical sampling strategy to improve the problem of large deviation in Q-evaluation and slow convergence speed of the Critic network in the MADDPG algorithm.

[0045] 3. This application can effectively improve the path planning capability of UAVs when avoiding obstacles. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0047] Figure 1This is a schematic diagram of the improved SSLO algorithm according to an embodiment of this application; Figure 2 This is a schematic diagram of the drone simulation obstacle avoidance algorithm architecture based on ISSLO-MADDPG according to an embodiment of this application. Detailed Implementation

[0048] The present application will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of the present application.

[0049] See Figure 1 This application provides an embodiment of an intelligent cooperative obstacle avoidance method for unmanned aerial vehicle (UAV) swarms, comprising: Step 1: Initialize the location information of the drone, obstacles, and destination; establish the drone dynamics model, the drone's state space, action space, and the reward function required for training. Step 2: Set the initial parameters for the ISSLO and MADDPG algorithms. Treat the UAV as an intelligent agent and the environment in which the UAV performs its mission as the simulated obstacle avoidance environment. The ISSLO and MADDPG algorithms interact with the simulated obstacle avoidance environment independently with their respective probabilities. After interacting with the simulated obstacle avoidance environment, the ISSLO algorithm uses population optimization to obtain the optimal solution. The optimal solution is then converted into an experience sample and imported into the experience pool of the MADDPG algorithm. The MADDPG algorithm samples from the experience pool to complete the reuse and learning of high-quality experience. Step 3: The drone swarm continuously interacts with the simulated obstacle avoidance environment, receiving corresponding rewards at each time step and updating the weights of the policy network and value network until the drone swarm reaches the destination or encounters an obstacle, at which point the round ends; after several rounds of interaction, until the maximum number of rounds is reached, the obstacle avoidance training of the drone swarm is completed, and the pre-trained model is output. Step 4: The MADDPG algorithm loads the pre-trained model and interacts with the simulated obstacle avoidance environment to realize online path planning when the UAV swarm avoids obstacles.

[0050] Optionally, drones come in a wide variety of types and have a broad range of applications, resulting in multiple and inconsistent mathematical models. Currently, one of the more complex drone mathematical models is a six-degree-of-freedom model, which mainly includes the drone's kinematic and dynamic models. If we assume that the drone's sideslip and angle of attack are very small and ignore the influence of wind, the six-degree-of-freedom model can be written in another commonly used form: a three-degree-of-freedom model. The expression for the drone's dynamic model is as follows: (1.1) In the formula, , and These represent the speed, track angle, and heading angle of the UAV, respectively. , , These respectively represent the drones in The x-axis coordinate, y-axis coordinate, and z-axis coordinate at time t. Indicates the decision time interval. , , These respectively represent the drones in The x-axis, y-axis, and z-axis coordinates at any given time; the x-axis represents the direction of the UAV's movement, the y-axis is perpendicular to the x-axis, and the z-axis is perpendicular to the xoy plane.

[0051] Optionally, constructing the state space of the UAV includes: In path planning, the distance information between the drone's current position and surrounding obstacles is crucial, while the distance and orientation information between the drone's current position and the target point are also essential factors. The drone's current position determines both its distance from obstacles and the target point, as well as its orientation relative to them. Typically, if the drone is equipped with an electro-optical pod containing a series of sensors, it can detect relevant information about obstacles and the target point. Therefore, the orientation and distance information of the target point and obstacles together constitute the drone's state information. : (1.2) In the formula, and These represent the distances between the drone's current position and the target point and the obstacle, respectively. and These represent the differences between the drone's trajectory angle and the trajectory angles of the target point and the obstacle, respectively. and These represent the differences between the heading angle of the drone and the heading angles of the target point and the obstacle, respectively.

[0052] Optionally, the drone's actions refer to the changes in its heading angle, track angle, and desired flight speed within a time interval. Assuming the change in the drone's heading angle ranges from -30° to 30°, and to reduce computational complexity and improve the speed of online autonomous learning, the changes in these two angles are divided into seven actions at 10° intervals, thus discretizing the action space. On the other hand, the drone's speed changes when it turns; therefore, its flight speed is set based on the magnitude of the turning angle. When the drone does not turn or the turning angle is small (i.e., the turning angle changes between 0° and ±10°), a larger speed is set, denoted as... When the drone's turning angle is large, i.e., the turning angle changes between 20° and 30°, the drone's speed is set to be relatively low, denoted as... When the drone's turning angle is between 10° and 20°, the drone's speed is recorded as... The yaw angle action space of the drone It is expressed as follows: = = [-30°, -20°, -10°, 0°, 10°, 20°, 30°](1.3) Optionally, the action space of the drone includes: The discrete velocity range of the UAV is: (1.4) in, For the speed of the drone, , , All of these are the speeds of the drone, and all are related to the magnitude of the change in the drone's turning angle; If it is known that the drone is at the current moment Location track angle and heading angle When the drone selects a certain turning action and The speed at this time is V After the drone completes the turning maneuver, its new position will be: (1.5) In summary, the action variables of the drone are: A is the action, and there are 3 variables. It is the output value of the policy network; , These are the new track angle and heading angle, respectively, and their angular changes are: and .

[0053] Optionally, the main function of the reward / penalty function R is to evaluate the effect of the action taken in the current state, guiding the UAV to learn the state-action mapping relationship. Therefore, a reasonable reward / penalty function is beneficial to improving algorithm performance. Reinforcement learning aims to find the optimal strategy to achieve collision-free flight of the UAV, thereby maximizing the reward value. To improve the system's learning efficiency, let the distance between the UAV and the target point be... Smaller than the drone formation radius When the distance between the drone and the obstacle is reached, it is considered that the drone has successfully reached the target location; similarly, when the distance between the drone and the obstacle is reached... Smaller than the drone formation radius At that time, it is assumed that a collision between the drone and the obstacle is inevitable; when The threat radius is greater than that of the obstacle. If the condition is met, it is assumed that the drone will definitely not collide with the obstacle. Based on the above analysis, to promptly reflect the effectiveness of the strategy, the expression for the reward function is: (1.6) In the formula, For the reward function, To maximize the reward value when the drone reaches the target point after performing the action, for The distance between the drone and the target at all times. for The distance between the drone and the target at all times. for The distance between the drone and the obstacle in real time. for The distance between the drone and the obstacle in real time; For the drone formation radius, The threat radius of the obstacle; The distance between the drone and the nearest known obstacle; The drone receives the maximum reward value when it reaches the target point after performing the action. A larger reward value is obtained when the drone approaches the target point and moves away from obstacles after performing the action. When the drone approaches the target point after performing the action and is within the threat range of the obstacle, it may not necessarily collide with the obstacle, thus receiving a smaller reward value. If the drone moves away from the target point after performing the action, it will incur a significant penalty. When the distance between the drone and the obstacle is less than 10 after the action is performed. When this happens, it is considered that a collision has occurred with an obstacle, and the maximum penalty value will be obtained. ; .

[0054] Optionally, step 2 includes: Constructing the ISSLO-MADDPG algorithm training architecture, such as Figure 1 As shown. During training, the hyperparameters of the ISSLO and MADDPG algorithms are initialized. In the MADDPG algorithm, each agent represents a drone, and multiple agents form a drone swarm. The ISSLO algorithm and the MADDPG algorithm each have their own probability. , It can independently interact with the simulated obstacle avoidance environment, in which , The value can be 0.2; before interaction, generate a random number between 0 and 1, rand. The ISSLO algorithm interacts with the environment to obtain the optimal solution, which is stored as high-quality experience in the experience pool of the MADDPG algorithm for sampling. If The MADDPG algorithm's agent then interacts with the environment and completes sampling training until the iteration ends. During sampling, a novel sampling strategy proposed in the embodiments of this application is employed.

[0055] Figure 1 This is a diagram of the UAV obstacle avoidance algorithm architecture based on ISSLO-MADDPG. This architecture fully utilizes the global search capability of metaheuristic algorithms, adding high-quality experiences found through the search to the experience pool generated by the MADDPG policy network, thus addressing the problem of traditional MADDPG algorithms easily getting trapped in local optima. The ISSLO-MADDPG algorithm improves policy diversity and ensures high-quality policy experience learning. Furthermore, the new experience sampling strategy can also solve the problems of large variance in Q-values ​​and slow convergence speed.

[0056] The Improved SSLO Algorithm (ISSLO): The Stochastic Social Learning Optimization Algorithm (SSLO) is a novel metaheuristic algorithm inspired by social learning theory and bucket theory. It simulates human knowledge acquisition mechanisms, including observation, imitation, and social interaction.

[0057] Optionally, the ISSLO algorithm interacts with the environment to obtain the optimal solution, including: The search begins by randomly initializing the population within the decision space using the following formula: (2.1) in, Indicates the first The candidate solution positions represent the individual's initial learning state; Refers to the first The nth candidate solution, the action variable of the nth... The position value of the dimension; , They refer to the first Lower and upper limits of dimensional position values; rand It is a random variable in the range [0,1]. The optimal objective function value can be obtained using the following formula: (2.2) In the formula, (·) refers to the objective function; It is the optimal objective function value; Represents the optimal individual; The independent variable is the one that takes the minimum value. This represents the total number of candidate solutions. Inspired by the exponential tail exploration kernel, the SSLO operator utilizes quantum performance to establish a unified computational architecture for knowledge accumulation and inspiration triggering. In the daily learning phase, individuals gradually accumulate knowledge by imitating high-quality solutions. The learning state of each individual, i.e., the candidate solution, is obtained through the following formula: (2.3) In the formula, Refers to the new first Individual learning status; This represents the learning state of another randomly selected individual; This refers to the corresponding fitness value; D-dimensional random vectors within, The contraction-expansion coefficient is a fixed constant. For Hadarmar; The "bucket effect" refers to the phenomenon where the amount of water a bucket can hold depends on its shortest stave. Therefore, special interventions are implemented for the worst-performing individuals to ensure their rapid improvement. A two-stage strategy is employed here.

[0058] Phase 1: In this phase, the algorithm primarily focuses on extensive exploration within the solution space to discover potential regions. The positions of candidate solutions S are updated as follows: (2.4) (2.5) in, The updated candidate solution, This indicates taking the absolute value. express random numbers, This represents exponentiation with base e. b It is a constant. Indicates the maximum number of iterations. Indicates the current number The next iteration; Phase 2: In this phase, the algorithm focuses on in-depth exploration of the discovered potential regions. The updated positions are then updated again. (2.6) In the formula, S Represents the current candidate solution. Indicates the updated candidate solution; express Random numbers; , These represent the lower and upper limits of the candidate solution, respectively. In the early exploration phase, the weakest individual actively seeks the most suitable learning model through self-exploration or reflexivity, thereby gradually improving its knowledge level. The algorithm assigns a larger step size and higher randomness to the weakest individual, which helps to quickly discover new learning methods, thus accelerating the learning process. The following formula updates the... Position of candidate solutions : (2.7) (2.8) In the formula, (·) is an indicator function. Used to generate a vector of -1 or 1, which determines the direction of position update; The Hadamard product represents the element-wise multiplication of two vectors. The exponential decay factor is used to determine... The decay rate; Indicates the first The number of update failures for each individual; This refers to the maximum number of iterations. In the later stages, the weakest individual continuously learns from the best-performing individual, while simultaneously applying newly acquired knowledge to explore the search space more deeply, thus steadily accumulating knowledge reserves. Therefore, the algorithm employs smaller step sizes and a more refined search strategy. The following formula updates the... Position of candidate solutions : (2.9) If the weakest learner fails to meet the requirement within the specified number of attempts, the following formula will be used to update the [number]th learner. Position of candidate solutions It helps learners break through cognitive barriers and break free from the constraints of localized thinking; a learner refers to a specific solution, i.e., an action value, and the weakest learner is the worst solution; (2.10) In the formula, Refers to the individual with the best current fitness; At this stage, each individual provides unique information based on their own knowledge base. Through information exchange mechanisms, individuals interact and share knowledge, achieving broad knowledge exchange. This method can effectively reduce cognitive biases, stimulate new insights and creativity, and gradually improve the knowledge system. This is achieved through the following formula: (2.11) (2.12) (2.13) in, To Updated candidate solution positions Refers to the first Individual, the first Knowledge exchange operations across dimensions Refers to the first Individual, the first Logical negation of the knowledge exchange mask in dimensions Indicates the first in the crowd Knowledge exchange rate of individuals; For the first Individual, the first A random number with dimensions in the range (0,1); The dimension that is randomly selected; It is a positive number, and its value range is [0.2, 0.3]. , The result is a random number, and its value range is (0,1]. (2.14) In the formula, Indicates the first Individual, the first Updated in dimension ; , They represent the first The lower and upper limits of the position values ​​in each dimension.

[0059] Optionally, finally, the algorithm selects a suitable learning strategy for each individual by choosing a strategy. Simultaneously, it carefully tracks the number of update failures for each learner. Once an update failure occurs, it automatically reverts to the latest information exchange rate. Through this mechanism, the algorithm achieves dynamic adaptation. The following formulas are used to obtain each individual, its corresponding knowledge exchange rate, and the number of update failures for each learner: (2.15) In the formula, Indicates the first In the nth iteration Individual, , The first The individual The exchange rate of old knowledge saved in the previous iteration and the... The learner The number of update failures in each iteration For the first The individual The knowledge exchange rate corresponding to +1 iteration IThis indicates whether the current iteration update was successful. At that time, the iteration update was successful. I When it is 1, At that time, this iteration update failed. I =0, For the first The individual The knowledge exchange rate corresponding to each iteration.

[0060] Optionally, Figure 2 The flowchart for the improved SSLO algorithm is shown. In the global exploration strategy of the SSLO algorithm, equation (2.2) will be affected by the fixed contraction-expansion coefficient. =1.4588 leads to large skips over the optimal region and subsequent convergence oscillations. Therefore, this application proposes a strategy combining adaptive control's contraction-expansion coefficient and chaotic mapping to improve the original formula (2.3) contraction-expansion coefficient. The expression is: (2.16) in, ; Maximum number of iterations; early stage of the strategy It has a large value and strong global exploration capability. It tends to 0 in the later stage, avoiding skipping the optimal solution and improving the local convergence accuracy. In addition, in formula (2.3) This is a purely random behavior, which may lead to local clustering and uneven traversal. Here, we can... Replace with chaotic mapping, that is Replace with a chaotic vector: (2.17) in, In the first The chaotic vector at the next iteration In the first The chaotic vector at the next iteration; Equation (2.3) becomes Equation (2.18), and Equation (2.18) is the expression for the candidate solution: (2.18) In the formula, Indicates the first The individual in the first The chaotic vector at the next iteration; Equation (2.10) is a strategy proposed to address the stagnation problem of the "barrel effect." However, this strategy lacks directionality in escaping local optima, leading to numerous ineffective global jumps. Therefore, this application proposes a novel strategy combining opposition learning and population difference-oriented perturbation. Specifically, by combining opposition learning and population difference-oriented perturbation, the expression for the candidate solution is transformed into: (2.19) In the formula, This is the scaling factor, and its value can range from 0.8 to 1.2. , Two randomly selected candidate solutions are given. This strategy utilizes population information for directional guidance during exploration, avoiding blind random jumps, excessive leaps, and making it easier to enter unexplored areas.

[0061] Optionally, since the traditional MADDPG algorithm suffers from large Q-estimate variance and slow convergence speed, this application proposes a new empirical sampling strategy: The MADDPG algorithm samples empirical data from the empirical pool using the following sampling strategy: If n < : (3.1) If n ≥ : (3.2) in, It is a positive number, and its value range is [0.6, 0.7]. n A random number in the range (0,1); Indicates the first The interval priority value selected in the next iteration; Indicates the first In the next iteration, the maximum priority value of the experience samples in the updated experience pool is determined. Indicates the first In the next iteration, the minimum priority value in the updated experience pool of experience samples; For the first During the sampling, a priority interval is selected, and the minimum batch size of the empirical sample is randomly drawn from that interval. The storage format of the empirical pool is as follows: , The first in the experience pool An empirical sample, , , , These refer to the experience pool number 1 and 2 respectively. The state observed in each empirical sample, the action taken, the reward received after taking the action, and the priority value. The new environmental state after taking action; During each training sampling, the temporal difference error and corresponding priority value of the empirical sample are recalculated; each time a new empirical sample is generated, it is updated in the empirical pool; among which, the priority of the new empirical sample... Y The calculation formula is as follows: (3.3) in, Refers to reward value; Discount factor; Let Q be the target policy network value. Q-value of the main policy network; A constant (e.g., 1) ),prevent 0, with values ​​ranging from 0.6 to 0.7, is used to balance priority distribution; σ These are hyperparameters used to balance priority distribution. σ The larger the value, the higher the probability that empirical samples with higher time-series difference (TD) errors will be sampled.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of the claims of this application.

Claims

1. A method for intelligent collaborative obstacle avoidance in a swarm of unmanned aerial vehicles (UAVs), characterized in that, include: Step 1: Initialize the location information of the drone, obstacles, and destination; establish the drone dynamics model, the drone's state space, action space, and the reward function required for training. Step 2: Set the initial parameters for the ISSLO and MADDPG algorithms. Treat the UAV as an intelligent agent and the environment in which the UAV performs its mission as the simulated obstacle avoidance environment. The ISSLO and MADDPG algorithms interact with the simulated obstacle avoidance environment independently with their respective probabilities. After interacting with the simulated obstacle avoidance environment, the ISSLO algorithm uses population optimization to obtain the optimal solution. The optimal solution is then converted into an experience sample and imported into the experience pool of the MADDPG algorithm. The MADDPG algorithm samples from the experience pool to complete the reuse and learning of high-quality experience. Step 3: The drone swarm continuously interacts with the simulated obstacle avoidance environment, receiving corresponding rewards at each time step and updating the weights of the policy network and value network until the drone swarm reaches the destination or encounters an obstacle, at which point the round ends; after several rounds of interaction, until the maximum number of rounds is reached, the obstacle avoidance training of the drone swarm is completed, and the pre-trained model is output. Step 4: The MADDPG algorithm loads the pre-trained model and interacts with the simulated obstacle avoidance environment to realize online path planning when the UAV swarm avoids obstacles.

2. The method according to claim 1, characterized in that, The expression for the UAV dynamics model is: In the formula, , and These represent the speed, track angle, and heading angle of the UAV, respectively. , , These respectively represent the drones in The x-axis coordinate, y-axis coordinate, and z-axis coordinate at time t. Indicates the decision time interval. , , These respectively represent the drones in The x-axis, y-axis, and z-axis coordinates at any given time; the x-axis represents the direction of the UAV's movement, the y-axis is perpendicular to the x-axis, and the z-axis is perpendicular to the xoy plane.

3. The method according to claim 1, characterized in that, Constructing the state space of the UAV includes: The location and distance information of the target point and obstacles together constitute the UAV's status information. S' : In the formula, and These represent the distances between the drone's current position and the target point and the obstacle, respectively. and These represent the differences between the drone's trajectory angle and the trajectory angles of the target point and the obstacle, respectively. and These represent the differences between the heading angle of the drone and the heading angles of the target point and the obstacle, respectively.

4. The method according to claim 1, characterized in that, The operational space of the drone includes: The discrete velocity range of the UAV is: in, For the speed of the drone, , , All of these are the speeds of the drone, and all are related to the magnitude of the change in the drone's turning angle; If it is known that the drone is at the current moment Location track angle and heading angle When the drone selects a certain turning action and The speed at this time is V After the drone completes the turning maneuver, its new position will be: In summary, the action variables of the drone are: A is the action, and there are 3 variables. It is the output value of the policy network; , These are the new track angle and heading angle, respectively, and their angular changes are: and .

5. The method according to claim 1, characterized in that, The expression for the reward function is: In the formula, For the reward function, To maximize the reward value when the drone reaches the target point after performing the action, for The distance between the drone and the target at all times. for The distance between the drone and the target at all times. for The distance between the drone and the obstacle in real time. for The distance between the drone and the obstacle in real time; For the drone formation radius, The threat radius of the obstacle; The distance between the drone and the nearest known obstacle; The drone receives the maximum reward value when it reaches the target point after performing the action. A larger reward value is obtained when the drone approaches the target point and moves away from obstacles after performing the action. When the drone approaches the target point after performing the action and is within the threat range of the obstacle, it may not necessarily collide with the obstacle, thus receiving a smaller reward value. If the drone moves away from the target point after performing the action, it will incur a significant penalty. When the distance between the drone and the obstacle is less than 10 after the action is performed. When this happens, it is considered that a collision has occurred with an obstacle, and the maximum penalty value will be obtained. ; .

6. The method according to claim 1, characterized in that, Step 2 includes: During training, the hyperparameters of the ISSLO and MADDPG algorithms are initialized. Each agent in the MADDPG algorithm represents a drone, and multiple agents form a drone swarm. The ISSLO algorithm and the MADDPG algorithm each have their own probability. , It can independently interact with the simulated obstacle avoidance environment, in which Before interaction, first generate a random number between 0 and 1, rand. The ISSLO algorithm interacts with the environment to obtain the optimal solution, which is stored as high-quality experience in the experience pool of the MADDPG algorithm for sampling. If Then the agent of the MADDPG algorithm interacts with the environment and completes sampling training until the iteration ends.

7. The method according to claim 6, characterized in that, The ISSLO algorithm interacts with the environment to obtain the optimal solution, including: The search begins by randomly initializing the population within the decision space using the following formula: in, Indicates the first The candidate solution positions represent the individual's initial learning state; Refers to the first The nth candidate solution, the action variable of the nth... The position value of the dimension; , They refer to the first Lower and upper limits of dimensional position values; rand It is a random variable in the range [0,1]. The optimal objective function value can be obtained using the following formula: In the formula, (·) refers to the objective function; It is the optimal objective function value; Represents the optimal individual; The independent variable is the one that takes the minimum value. This represents the total number of candidate solutions. The learning state, i.e., the candidate solution, for each individual is obtained using the following formula: In the formula, Refers to the new first Individual learning status; This represents the learning state of another randomly selected individual; This refers to the corresponding fitness value; D-dimensional random vectors within, The contraction-expansion coefficient is a constant. For Hadarmar; The position of candidate solution S is updated as follows: in, The updated candidate solution, This indicates taking the absolute value. express random numbers, This represents exponentiation with base e. b It is a constant. Indicates the maximum number of iterations. Indicates the current number The next iteration; Update the position again after the initial update: In the formula, S Represents the current candidate solution. Indicates the updated candidate solution; express Random numbers; , These represent the lower and upper limits of the candidate solution, respectively. Update the following formula: Position of candidate solutions : In the formula, (·) is an indicator function. Used to generate a vector of -1 or 1, which determines the direction of position update; The Hadamard product represents the element-wise multiplication of two vectors. The exponential decay factor is used to determine... The decay rate; Indicates the first The number of update failures for each individual; This refers to the maximum number of iterations. Update the following formula: Position of candidate solutions : If the weakest learner fails to meet the requirement within the specified number of attempts, the following formula will be used to update the [number]th learner. Position of candidate solutions The learner refers to a specific solution, i.e., an action value; the weakest learner is the worst solution. In the formula, Refers to the individual with the best current fitness; It is obtained through the following formula: in, To Updated candidate solution positions Refers to the first Individual, the first Knowledge exchange operations across dimensions Refers to the first Individual, the first Logical negation of the knowledge exchange mask in dimensions Indicates the first in the crowd The knowledge exchange rate of an individual; For the first Individual, the first A random number with dimensions in the range (0,1); The dimension that is randomly selected; It is a positive number; , It is a random number; In the formula, Indicates the first Individual, the first Updated in dimension ; , They represent the first The lower and upper limits of the position values ​​in each dimension.

8. The method according to claim 7, characterized in that, The following formulas can be used to obtain the knowledge exchange rate for each individual and the number of update failures for each learner: In the formula, Indicates the first In the nth iteration Individual, , The first The individual The exchange rate of old knowledge saved in the previous iteration and the... The learner The number of update failures in each iteration For the first The individual The knowledge exchange rate corresponding to +1 iteration I This indicates whether the current iteration update was successful. At that time, the iteration update was successful. I When it is 1, At that time, this iteration update failed. I =0, For the first The individual The knowledge exchange rate corresponding to each iteration.

9. The method according to claim 7, characterized in that, Contraction-expansion coefficient The expression is: in, ; This represents the maximum number of iterations. Will Replace with a chaotic vector: in, In the first The chaotic vector at the next iteration In the first The chaotic vector at the next iteration; The expression for the candidate solution becomes: In the formula, Indicates the first The individual in the first The chaotic vector at the next iteration; By employing a novel strategy combining oppositional learning and population-differential directional perturbation, the expression for the candidate solution is transformed into: In the formula, This is the scaling factor; , These are two randomly selected candidate solutions.

10. The method according to claim 6, characterized in that, The MADDPG algorithm samples the experience sample data in the experience pool using the following sampling strategy: If n < : If n ≥ : in, It is a positive number. n A random number in the range (0,1); Indicates the first The interval priority value selected in the next iteration; Indicates the first In the next iteration, the maximum priority value of the experience samples in the updated experience pool is determined. Indicates the first In the next iteration, the minimum priority value in the updated experience pool of experience samples; For the first During the sampling, a priority interval is selected, and the minimum batch size of the empirical sample is randomly drawn from that interval. The storage format of the empirical pool is as follows: , The first in the experience pool An empirical sample, , , , These refer to the experience pool number 1 and 2 respectively. The state observed in each empirical sample, the action taken, the reward received after taking the action, and the priority value. The new environmental state after taking action; During each training sampling, the temporal difference error and corresponding priority value of the empirical sample are recalculated; each time a new empirical sample is generated, it is updated in the empirical pool; among which, the priority of the new empirical sample... Y The calculation formula is as follows: in, Refers to reward value; Discount factor; Let Q be the target policy network value. Q-value of the main policy network; It is a constant to prevent 0; σ These are hyperparameters used to balance priority distribution. σ The larger the value, the higher the probability that the empirical sample with the higher the time-series difference error will be sampled.

Images