Unmanned aerial vehicle autonomous obstacle avoidance method and device based on cluster confrontation, and storage medium

Abstract

The invention provides an unmanned aerial vehicle autonomous obstacle avoidance method based on cluster confrontation. The unmanned aerial vehicle autonomous obstacle avoidance method comprises the following steps: constructing an obstacle avoidance task comprising a plurality of virtual unmanned aerial vehicles in an obstacle scene in a simulation environment; in the process of executing the competition task, own state codes and visual information codes of the virtual unmanned aerial vehicles are utilized to construct respective body state information, relative information between the virtual unmanned aerial vehicles and other unmanned aerial vehicles is utilized to construct competition information of the virtual unmanned aerial vehicles, and the body state information and the competition information are utilized to construct states of the virtual unmanned aerial vehicles; performing joint reinforcement learning training on the autonomous obstacle avoidance strategies of all the virtual unmanned aerial vehicles by using the shared strategy network and value network, so that each virtual unmanned aerial vehicle can obtain the current action according to the current state; and deploying the trained strategy network on the target unmanned aerial vehicle, wherein the strategy network obtains an obstacle avoidance action slightly based on the body state information of the target unmanned aerial vehicle. The problems of low obstacle avoidance success rate and poor adaptability of existing unmanned aerial vehicle reinforcement learning in a complex unknown environment are solved.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation technology for unmanned aerial vehicles (UAVs), and more particularly to a method, apparatus, and storage medium for autonomous obstacle avoidance of UAVs based on swarm warfare. Background Technology

[0002] In existing autonomous obstacle avoidance technologies for unmanned aerial vehicles (UAVs), single-UAV reinforcement learning typically simplifies the task to an interaction between a single UAV and a static environment, using "reaching the target" as the reward signal. Limited by the singularity of the training scenario and the lack of dynamic adversarial stimulation, the resulting strategies tend to converge to local suboptimal solutions, and the success rate drops sharply as obstacle density or flight speed increases. On the other hand, while multi-UAV cooperative methods improve robustness through information sharing, they rely on real swarm deployment, which is costly, and the cooperation mechanism cannot be directly translated into improved single-UAV performance. Furthermore, existing technologies have not yet introduced "virtual swarm competition" into single-UAV training: they lack both a simulation framework centered on competitive adversarial elements and a unified representation integrating relative pose and collision distance in adversarial observation, resulting in insufficient adaptability of single UAVs in unknown and complex environments. Therefore, there is an urgent need for a method that can introduce swarm competition stimulation during the training phase without deploying real multi-UAVs, to significantly improve the obstacle avoidance success rate and generalization ability of single UAVs in high-speed, unknown scenarios. Summary of the Invention

[0003] The present invention aims to at least partially solve one of the technical problems existing in the prior art.

[0004] To address this, the present invention provides a method, apparatus, and storage medium for autonomous obstacle avoidance of unmanned aerial vehicles (UAVs) based on swarm adversarial learning. By introducing swarm adversarial learning into a simulation environment and combining it with reinforcement learning, the obstacle avoidance performance of a single UAV is significantly improved, solving the problems of low success rate and poor adaptability of existing single UAV reinforcement learning in complex and unknown environments.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The first aspect of this invention provides an autonomous obstacle avoidance method for unmanned aerial vehicles (UAVs) based on swarm warfare, comprising:

[0007] Construct a competitive obstacle avoidance task involving multiple virtual drones in an obstacle scenario within a simulated environment;

[0008] During the execution of competitive tasks, the ontological state information of each virtual drone is constructed by using its own state encoding and visual information encoding, the competitive information of each virtual drone is constructed by using the relative information between each virtual drone and all other virtual drones, and the state of each virtual drone is constructed by using the ontological state information and competitive information of the virtual drone.

[0009] The autonomous obstacle avoidance strategies of all virtual drones are jointly reinforced and trained using a shared policy network and value network, so that each virtual drone can determine its current action based on its current state.

[0010] The trained policy network is deployed on the target drone, and the policy network obtains obstacle avoidance actions based on the target drone's own state information.

[0011] In some embodiments, the obstacle scenario is a randomly generated obstacle scenario in the simulation environment.

[0012] In some embodiments, the information data of the self-state encoding comes from the inertial navigation module of the UAV flight controller, including filtered IMU data and GPS information; the visual state encoding is the depth feature obtained by inputting the depth map acquired by the UAV into a pre-trained visual encoder for compression encoding.

[0013] In some embodiments, the training process of the visual encoder includes:

[0014] Data was collected in an obstacle scenario constructed within the simulation environment. Zhang's simulated depth map was collected in a real environment containing obstacles. Each of the collected depth maps is dilated, and the training dataset is constructed using all the dilated depth maps.

[0015] The visual encoder is obtained by using a variational autoencoder as the training architecture and training the variational autoencoder with the training dataset and a preset loss function.

[0016] In some embodiments, the competition status information includes: normalized relative information of the self-machine and all other virtual drones and / or normalized collision distance of the self-machine.

[0017] In some embodiments, the relative information is relative position and / or relative velocity; the collision distance is characterized by the Euclidean symbol field distance of the self at the current time and current position.

[0018] In some embodiments, the joint reinforcement learning training of the autonomous obstacle avoidance strategies of all virtual drones using a shared policy network and value network includes:

[0019] Each virtual drone's flight competition task is modeled as a Markov Decision Process (MDP): ,in, This represents the state space of a single drone. This represents the action space of a single drone. Used to describe the state Execute action Thus transitioning to state The probability, Used to provide status Execute action The immediate reward received afterward; all N virtual drones in the virtual cluster share the same strategy. Let the joint state space and joint action space of N virtual drones be respectively... and , and Define the state space and action space of the i-th virtual drone in the virtual cluster, respectively.

[0020] The optimization objective of virtual cluster warfare is to :

[0021]

[0022] in, Indicates the discount factor; Represents the expectation operator; This represents the initial state of the i-th virtual drone. This represents the state of the i-th virtual drone at time t. This represents the action of the i-th virtual drone at time t; This represents the reward function in the optimization objective; N is the number of virtual drones in the cluster.

[0023] The optimal strategy To maximize the expected cumulative discount reward, i.e.:

[0024]

[0025] A policy network and a value network are constructed, each network including a visual feature processing part and an action generation part, wherein the visual feature processing part adopts a visual encoder for acquiring the visual information encoding;

[0026] The policy network and the value network are trained using a proximal policy optimization reinforcement learning method, wherein the parameters of the visual encoder are frozen, and all virtual drones share the same experience replay pool.

[0027] In some embodiments, the reward function R in the optimization objective is expressed as:

[0028]

[0029] in:

[0030] For smooth reward, , Let t be the propeller speed of the virtual drone. for The actions of a virtual drone generated in real time;

[0031] As a thrust reward, , The preset hover throttle, Let t be the drone throttle position at time t;

[0032] Rewards for guiding flight direction. , Location of the target object and They are Time and The drone's location at any given time;

[0033] As an obstacle avoidance reward, , For competition status information The Euclidean symbol field distance described in the text;

[0034] The survival reward is a positive constant.

[0035] For maximum speed bonus, , Let be the speed of the drone at time t. This is the maximum speed of the drone;

[0036] Towards rewards, ;

[0037] For horizontal attitude reward, the expression is as follows:

[0038]

[0039] In the formula, Let be the quaternion of the UAV at time t, representing the rotation of the UAV's attitude relative to the world coordinate system at time t; It is a vertically upward unit vector in the world coordinate system; It is an operation on the z-component of the orientation quantity;

[0040] The reward for completing the mission is a positive constant, awarded when the drone completes the mission and reaches the destination.

[0041] The weights for each of the above reward items are given, and the sum of the weights is 1.

[0042] A second aspect of the present invention provides an autonomous obstacle avoidance device for unmanned aerial vehicles (UAVs) based on swarm warfare, comprising:

[0043] The first module is configured to construct a competitive obstacle avoidance task involving multiple virtual drones in an obstacle scenario within a simulation environment;

[0044] The second module is configured to construct the ontological state information of each virtual drone by using its own state encoding and visual information encoding during the execution of the competitive task, construct the competitive information of each virtual drone by using the relative information between each virtual drone and all other virtual drones, and construct the state of each virtual drone by using the ontological state information and competitive information of the virtual drone.

[0045] The third module is configured to use a shared policy network and value network to jointly reinforce the autonomous obstacle avoidance strategy of all virtual drones, so that each virtual drone can obtain the action at the current moment according to the current state.

[0046] The fourth module is configured to deploy the trained policy network onto the target drone, which obtains obstacle avoidance actions based on the target drone's own state information.

[0047] A third aspect of the present invention provides a computer-readable storage medium storing computer instructions for causing the computer to execute the autonomous obstacle avoidance method for unmanned aerial vehicles according to any embodiment of the first aspect of the present invention.

[0048] This invention has the following characteristics and beneficial effects:

[0049] This invention proposes a swarm-based adversarial method, device, and storage medium for autonomous obstacle avoidance by unmanned aerial vehicles (UAVs). By introducing a competition mechanism in a multi-agent environment, the performance of single-UAV policies is significantly improved. The core of this method lies in the fact that during the training phase, multiple UAVs share a policy and compete in the same environment. Their observed states not only include their own deep visual encoding and ontology state but also incorporate key information such as the relative position and speed of their competitors. This design provides richer comparison signals for policy learning, motivating UAVs to pursue high-speed flight while ensuring obstacle avoidance safety. Experiments demonstrate that this competitive framework effectively improves the robustness and sample efficiency of the policy, enabling it to achieve higher success rates and stronger generalization capabilities in complex and unknown environments, showing significant advantages over traditional single-agent reinforcement learning methods. Attached Figure Description

[0050] Figure 1This is an overall flowchart of an autonomous obstacle avoidance method for unmanned aerial vehicles based on swarm confrontation, provided by the first aspect of the present invention.

[0051] Figure 2 This is a visual schematic diagram of the VAE network structure and collision inflation depth map involved in the UAV autonomous obstacle avoidance method provided in the first aspect embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the adversarial observation information constructed in the UAV autonomous obstacle avoidance method provided in the first aspect embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the structure of an electronic device provided in a third aspect embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for explaining this application and are not intended to limit this application.

[0055] Conversely, this application covers any alternatives, modifications, equivalent methods, and schemes made within the spirit and scope of this application as defined by the claims. Furthermore, to provide the public with a better understanding of this application, certain specific details are described in detail below. However, this application can be fully understood by those skilled in the art even without these detailed descriptions.

[0056] See Figure 1 The first aspect of this invention provides a method for autonomous obstacle avoidance of unmanned aerial vehicles (UAVs) based on swarm warfare. This method includes the following steps:

[0057] Step S100: Construct a competitive task involving multiple virtual drones in an obstacle scenario within a simulation environment;

[0058] Step S200: During the execution of the competition task, the ontological state information of each virtual drone is constructed by using the self-state encoding and visual information encoding of each virtual drone, the competition information of each virtual drone is constructed by using the relative information between each virtual drone and all other virtual drones, and the state of each virtual drone is constructed by using the ontological state information and competition information of the virtual drone.

[0059] Step S300: Use a shared policy network and value network to jointly reinforce the autonomous obstacle avoidance strategy of all virtual drones, so that each virtual drone can obtain the action at the current moment according to the current state.

[0060] Step S400: Deploy the trained policy network onto the target UAV. The policy network obtains obstacle avoidance actions based on the target UAV's own state information.

[0061] In some embodiments, step S100 includes:

[0062] First, create a rectangular region in an arbitrary physical simulation platform. The dimensions of this region are 8 m × 16 m × 3 m. At the start of each round of competition, an obstacle scene is randomly generated on the bottom surface of the region. Specifically, 40 cylinders with radii of 0.1 m to 0.3 m and heights of 2 m to 3 m are generated in a uniform and random distribution as tree trunk obstacles.

[0063] Subsequently, N isomorphic virtual drones were deployed in the simulation environment to form a virtual cluster. All virtual drones share the same obstacle environment, and each virtual drone has the task of "competitive obstacle avoidance to reach the target". Specifically, all virtual drones start from the same starting position and reach the same target position. The virtual drone that reaches the target position first without colliding with the obstacle wins in this round of the task.

[0064] In some embodiments, step S200 includes:

[0065] During the execution of competitive tasks in the virtual cluster, it is necessary to obtain the state of each virtual drone at time t. The superscript 'i' represents the virtual drone's ID, indicating its status. Including the ontology state information of the i-th virtual drone at time t and competition status information .in,

[0066] Ontology status information This includes self-state coding and visual state coding; the self-state coding information comes from the inertial navigation module mounted on the virtual drone flight controller, including filtered IMU data (such as angle information, angular velocity information, and linear velocity information) and GPS information. The self-state coding is denoted as... , , , and Let be the rotation matrix, position, velocity, and angular velocity of the i-th virtual drone at time t, respectively; the visual state encoding is the depth feature obtained by inputting the depth map acquired by the virtual drone at time t into a pre-trained visual encoder for compression encoding, denoted as . This deep feature specifically encodes the latent space state of visual information. Therefore, the ontology state information... It can be represented as .

[0067] Furthermore, in one specific embodiment of this application, the visual encoder is obtained through offline training. After offline pre-training, the parameters and structure of the visual encoder are frozen to avoid visual feature drift. The training of the visual encoder includes the following steps:

[0068] Step S211: Training Data Acquisition

[0069] Using the random obstacle scenario constructed in step S100, data is collected from the simulation environment using the IsaacGym simulator. Zhang's simulated depth map (in this embodiment) Using any depth sensor device (such as the RealSense D430 camera), data is collected in a real indoor / outdoor environment containing obstacles (this real indoor / outdoor environment may or may not be related to the simulation environment). Zhang's true depth map (in this embodiment) The real depth map and the simulated depth map have the same resolution, L×W (in this embodiment, the resolution is 212×120). The acquired real depth map can improve the generalization ability of the strategy to real scenes, bridging part of the gap between simulation and real scenes at the data source level. Zhang's simulated depth map and Each of the three real depth maps is dilated to make obstacles in the depth maps have wider obstacle regions. All the dilated depth maps are then randomly shuffled to form a training dataset for subsequent training of the visual encoder. The specific steps of the dilation process include: constructing a 3D obstacle map based on each acquired depth map; dilating all obstacles in each 3D obstacle map (the dilation parameter can be set according to the application scenario, for example, dilating to 1.5 times the original size); and then generating a depth map from the same depth map acquisition perspective using the dilated 3D obstacle map.

[0070] Step S212: Construct the visual encoder

[0071] The visual encoder is pre-trained using variational autoencoders (VAEs) as the main training architecture. The visual encoder contains encoders. With decoder ,like Figure 2 As shown. Among them,

[0072] encoder It can be a neural network of any structure, such as a convolutional neural network or a transformer neural network. Encoder The input is a depth map with a resolution of L×W. via encoder Depth map Compressed to a latent space state of length Z encoder The mapping distribution is typically a Gaussian distribution with diagonal covariance; the encoder's mapping... It can be represented as:

[0073]

[0074] in, It is a Gaussian distribution. It is a diagonal matrix; and It is an encoder that needs to be trained. The parameters are the mean and standard deviation of the Gaussian distribution, respectively.

[0075] decoder Structure and encoder It forms a symmetrical structure. (Through the decoder) encoder The output is a latent space state of length Z. Reconstructed to match the original depth map Reconstructed depth maps with the same resolution Decoder mapping It can be represented as:

[0076]

[0077] in, and It is a decoder that needs to be trained. The parameters are the mean and standard deviation of the Gaussian distribution, respectively.

[0078] By minimizing the comprehensive loss function Training the visual encoder and the overall loss function It can be represented as:

[0079]

[0080]

[0081]

[0082]

[0083] in, and These are the comprehensive loss functions. The mean squared error term and the mean absolute error term together encourage accurate reconstruction; Let Z be the KL divergence, which regularizes the latent space, and let Z be the dimension of the latent space states. and They are the first Mean and variance of the dimension; and It is used for adjustment , The coefficients involved in the loss calculation, along with the KL divergence, are used to control the trade-off between fidelity and potential compactness.

[0084] Competition status information Including the i-th virtual drone and others The normalized relative information of the i-th virtual drone and / or the normalized collision distance between the i-th virtual drone and the obstacle are used as adversarial competition observation information. Let the i-th virtual drone and the m-th other virtual drones at time t have respect to parameters. The relative value is Parameters Available locations and / or speed Collision distance is determined by any combination of one or more of the following relative information: The distance of the virtual drone (self) at its current time and position, represented by the Euclidean symbolic field (ESDF), can be replaced by the distance of the virtual drone to the nearest static obstacle as observed by itself; the distance of the i-th virtual drone to other... Regarding the parameters of virtual drones The initial competition state information is obtained by stitching together the relative values ​​and / or the collision distance of the i-th virtual drone. ,right After normalization, the competition state information is obtained. Initial competition state information The expression is as follows:

[0085]

[0086] In the formula, This indicates a splicing operation.

[0087] Furthermore, in one specific embodiment of this application, see [link to relevant documentation]. Figure 3 The example shows three virtual drones, representing the intelligent agent, opponent 0, and opponent 1, in this embodiment, with their relative positions... and relative velocity As relative information, the relative position and relative velocity between the agent and its opponent 0 are respectively... The relative positions and relative velocities of the agent and opponent 1 are respectively The initial competitive state information of the agent is then represented as: .

[0088] It should be noted that the competitive state information designed in this embodiment is intended to carry the correlation information within the virtual cluster, and the construction of adversarial information is closely coupled with the task objective. The task of this embodiment is to guide the drone to traverse an unknown environment based solely on visual depth input, with the goal of achieving high-speed and collision-free passage in a completely unknown forest scene. Therefore, the objective of this embodiment includes not only speed and positional relationships (flying faster) but also collision data between the drone and obstacles (flying more safely). In this embodiment, the Euclidean symbolic field distance of the i-th virtual drone is used to represent the collision distance of the i-th virtual drone. This is an indirect method for evaluating the flight safety of the drone and determining whether a collision has occurred. If the obstacle distance is less than a set distance threshold, the drone will be judged to have collided.

[0089] In some embodiments, step S300 includes:

[0090] Step S310: Design a Markov decision process representation for virtual cluster adversarial scenarios.

[0091] Each virtual drone's flight competition task is modeled as a Markov Decision Process (MDP): ,in, This represents the state space of a single drone. This represents the action space of a single drone. Used to describe the state Execute action Thus transitioning to state The probability, Used to provide status Execute action The instant reward is then obtained. All N virtual drones in the virtual cluster share the same policy. The joint state space and joint action space of N virtual drones are respectively and , and The state space and action space of the i-th virtual drone in the virtual cluster are respectively defined.

[0092] Step S320: Setting the optimization objective and reward function considering the competitive state.

[0093] Let the optimization objective of virtual cluster adversarial after adding adversarial observation be: :

[0094]

[0095] in, Indicates the discount factor. The superscript t in the text represents t raised to the power of t; Represents the expectation operator; This represents the initial state of the i-th virtual drone. This represents the action of the i-th virtual drone at time t; The reward function in the optimization objective is designed in the following form:

[0096]

[0097] in:

[0098] For smooth reward, This reward encourages drones to produce smooth movements. Let t be the propeller speed of the virtual drone. The actions of the virtual drone generated in the previous moment;

[0099] As a thrust reward, The reward encourages drones to learn hover throttle control. , Let t be the drone throttle position at time t;

[0100] Rewards for guiding flight direction. This incentive is used to drive the drone toward the target location. flight, and They are Time and The drone's location at any given time;

[0101] As an obstacle avoidance reward, The reward is used to pull the drone away from the obstacle, where For competition status information The Euclidean Symbolic Field (ESDF) distance described in the text;

[0102] The survival reward is a positive constant used to encourage drones to fly for longer periods of time.

[0103] For maximum speed bonus, Used to encourage drones to operate at the maximum speed limit. flight, Let be the speed of the drone at time t;

[0104] Towards rewards, This is used to encourage the drone's speed direction to be in the positive direction of the drone;

[0105] For horizontal attitude reward, used to encourage the drone to maintain a stable, vertically upward attitude, the expression is as follows:

[0106]

[0107] In the formula, Let be the quaternion of the UAV at time t, representing the rotation of the UAV's attitude relative to the world coordinate system at time t; It is a vertically upward unit vector (in world coordinates), usually represented as ; It is a quaternion The inverse (conjugate quaternion) represents the relationship between the quaternion and the quaternion. Opposite rotation; It is an operation on the z-component of the orientation quantity (i.e., the vertical component).

[0108] The reward for completing the mission is a large positive constant, which is given when the drone completes the mission and reaches the destination.

[0109] The weights corresponding to the above reward items can be determined through ablation experiments, and the sum of the weights is 1.

[0110] Optimal Strategy To maximize the expected cumulative discount reward, i.e.:

[0111]

[0112] Step S330: Reinforcement Learning Network Design

[0113] Reinforcement learning training employs an Actor-Critic network, which consists of two sub-networks: a policy network that outputs the action probability distribution and is responsible for policy optimization; and a value network that estimates the state-value function, thereby calculating the advantage function. The policy network and the value network maintain a consistent network structure.

[0114] Both the policy network and the value network are divided into a visual feature processing part and an action generation part. The visual feature processing part uses a visual encoder pre-trained in steps S211-S212. This visual encoder remains frozen after pre-training and does not participate in the optimization of virtual swarm adversarial learning. The action generation part uses any type of neural network structure, such as a convolutional neural network, transformer, or multilayer perceptron (MLP). In this embodiment, the policy network uses a 3-layer MLP (512, 256, 128) with tanh activation; the value network also uses a 3-layer MLP (512, 256, 1) with tanh activation. These networks are shared by all virtual drones.

[0115] Step S340, Virtual Cluster Adversarial Learning

[0116] Virtual cluster adversarial learning employs reinforcement learning training methods, preferentially using proximal policies to optimize reinforcement learning PPO (Proximity-Based Probability) approaches. Specifically, this includes:

[0117] Decisions are generated based on the current strategy; all collected information on virtual drone actions, states, and rewards is stored in an experience replay pool; data is randomly selected from the experience replay pool and the reward value for reinforcement learning is calculated; the policy network and value network are updated separately, with the parameters of the visual encoder frozen. Training terminates when the reward value for reinforcement learning converges and no longer shows significant improvement.

[0118] In some embodiments, in step S400, the policy network finally trained in step S300 is deployed on the target drone (which can be a real drone or a virtual drone). The state of the target drone is collected in real time during the flight mission, wherein the competitive state information of the target drone is set to zero, that is, the effective part of the state of the target drone is only the body state information of the target drone. The state of the target drone is input into the trained policy network to generate obstacle avoidance actions.

[0119] In this embodiment, the following training hyperparameters are selected: the number of reinforcement learning parallel environments is 256, mini_batch_size (minimum batch size) = 2048, learning_rate (learning rate) = 3e-4, e_clip (PPO cutoff parameter) = 0.2, and grad_norm (gradient norm) = 1.5.

[0120] Understandably, this invention proposes a competition-based autonomous flight training framework for unmanned aerial vehicles (UAVs). Its core feature lies in constructing a multi-agent competitive environment, incorporating competitive information into the state representation, and utilizing adversarial interactions between virtual agents to incentivize individuals to learn more efficient and robust end-to-end flight strategies. This method employs a shared strategy and experience replay mechanism, combined with a pre-trained visual encoder to compress depth images, effectively improving the convergence speed and generalization ability of the strategy. This invention significantly improves the navigation success rate and flight efficiency of UAVs in complex and unknown environments; it spontaneously guides strategy optimization through a competition mechanism, reducing reliance on manual reward design; and it possesses good transferability from simulation to real-world applications, verifying its practicality and robustness in real-world scenarios.

[0121] A second aspect of the present invention provides an autonomous obstacle avoidance device for unmanned aerial vehicles (UAVs) based on swarm warfare, comprising:

[0122] The first module is configured to construct a competitive obstacle avoidance task involving multiple virtual drones in an obstacle scenario within a simulation environment;

[0123] The second module is configured to construct the ontological state information of each virtual drone by using its own state encoding and visual information encoding during the execution of the competitive task, construct the competitive information of each virtual drone by using the relative information between each virtual drone and all other virtual drones, and construct the state of each virtual drone by using the ontological state information and competitive information of the virtual drone.

[0124] The third module is configured to use a shared policy network and value network to jointly reinforce the autonomous obstacle avoidance strategy of all virtual drones, so that each virtual drone can obtain the action at the current moment according to the current state.

[0125] The fourth module is configured to deploy the trained policy network onto the target drone, which obtains obstacle avoidance actions based on the target drone's own state information.

[0126] It should be noted that the foregoing explanation of the embodiments of the UAV autonomous obstacle avoidance method also applies to the UAV autonomous obstacle avoidance device of this embodiment, and will not be repeated here.

[0127] To implement the above embodiments, this invention also proposes a computer-readable storage medium storing a computer program thereon, which is executed by a processor to perform the unmanned aerial vehicle (UAV) autonomous obstacle avoidance method of the above embodiments.

[0128] The following is for reference. Figure 4The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present invention. It should be noted that the electronic device in the embodiments of the present invention may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs, desktop computers, and servers. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0129] like Figure 4 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 108 into a random access memory (RAM) 103. The RAM 103 also stores various programs and data required for the operation of the electronic device. The processing unit 101, ROM 102, and RAM 103 are interconnected via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0130] Typically, the following devices can be connected to I / O interface 105: input devices 106 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, etc.; output devices 107 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 108 including, for example, magnetic tapes, hard disks, etc.; and communication devices 109. Communication device 109 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0131] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, this embodiment includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via communication device 109, or installed from storage device 108, or installed from ROM 102. When the computer program is executed by processing device 101, it performs the functions defined above in the methods of embodiments of this disclosure.

[0132] It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0133] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0134] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the aforementioned autonomous obstacle avoidance method for unmanned aerial vehicles.

[0135] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and Python, as well as conventional procedural programming languages ​​such as the "C-" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0137] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0138] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0139] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0140] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0141] Those skilled in the art will understand that implementing all or part of the steps of the methods in the above embodiments can be accomplished by instructing related hardware through a program. The developed program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0142] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0143] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for autonomous obstacle avoidance of unmanned aerial vehicles based on swarm confrontation, characterized in that, The method comprises the following steps: constructing a competitive obstacle avoidance task of multiple virtual unmanned aerial vehicles (UAVs) in an obstacle scene in a simulation environment; during the execution of the competitive task, constructing, by each virtual UAV, its own ontology state information by using its own state coding and visual information coding, constructing, by each virtual UAV, its own competitive information by using the relative information between each virtual UAV and all other virtual UAVs, and constructing, by each virtual UAV, its own state by using the ontology state information and the competitive information of the virtual UAV; training, by using a shared policy network and a value network, the autonomous obstacle avoidance strategies of all virtual UAVs through joint reinforcement learning, so that each virtual UAV can obtain an action at the current time according to the state at the current time; deploying the trained policy network on a target UAV, and obtaining, by the policy network, an obstacle avoidance action based on the ontology state information of the target UAV. 2.The method of claim 1, wherein, The obstacle scene is a randomly generated obstacle scene in the simulation environment. 3.The method of claim 1, wherein, The information data of the own state coding is obtained from an inertial navigation module of a flight controller of the UAV, and comprises filtered IMU data and GPS information; the visual state coding is obtained by inputting a depth map obtained by the UAV into a pre-trained visual encoder for compression coding to obtain depth features.

4. The method of claim 3, wherein, The training process of the visual encoder comprises the following steps: collecting obstacle scene under the construction of the simulation environment simulated depth map, collecting real environment containing obstacles real depth map, respectively, the collected depth map is expanded, and all the expanded depth maps are used to construct a training data set; training, by using a variational autoencoder as a training architecture, the variational autoencoder by using a training data set and a preset loss function, to obtain the visual encoder.

5. The method of claim 1, wherein, The competitive state information comprises normalized relative information of the virtual UAV and all other virtual UAVs and / or normalized collision distance of the virtual UAV.

6. The method of claim 5, wherein, The relative information adopts relative position and / or relative speed; and the collision distance is represented by a Euclidean signed distance field distance of the virtual UAV at the current time and the current position.

7. The method of claim 1, wherein, The training, by using the shared policy network and the value network, of the autonomous obstacle avoidance strategies of all virtual UAVs through joint reinforcement learning comprises the following steps: The flight competition task of each virtual UAV is modeled as a Markov Decision Process (MDP): wherein, represents the state space of a single UAV, represents the action space of a single UAV, is used to describe the probability of transitioning from state to state by performing action , is used to provide the immediate reward obtained after performing action in state ; all N virtual UAVs in the virtual swarm share the same policy , let the joint state space and joint action space of the N virtual UAVs be and , and represent the state space and action space of the i-th virtual UAV in the virtual swarm, respectively; Let the optimization goal of the virtual cluster confrontation be : wherein, denotes a discount factor; denotes an expectation operator; denotes an initial state of the i-th virtual UAV, denotes a state of the i-th virtual UAV at time t, denotes an action of the i-th virtual UAV at time t; denotes a reward function in the optimization objective; N is the number of virtual UAVs contained in the swarm; Then the optimal policy maximizes the expected cumulative discounted reward, i.e.: constructing the policy network and the value network, each of which comprises a visual feature processing part and an action generation part, wherein the visual feature processing part adopts a visual encoder used for obtaining the visual information coding; training, based on a proximal policy optimization reinforcement learning method, the policy network and the value network, wherein the parameters of the visual encoder are frozen, and all virtual UAVs share the same experience replay pool.

8. The method of claim 7, wherein, The reward function R in the optimization target is expressed as: wherein: a smoothness reward, , a propeller rotation speed of the virtual UAV at time t, is an action of the virtual UAV generated at time t; for thrust reward, , for preset hover throttle, for drone throttle at time t; a flight direction guidance reward, a target object position, are respectively a UAV position at a time instant and a UAV position at a time instant​​ obstacle avoidance reward, competitive state information Euclidean signed field distance described in the section​ For survival reward, for a positive constant; rewarding for maximum speed, a speed of the UAV at time t, a maximum speed of the UAV;​ To reward, ; The reward for the horizontal pose is expressed as follows: In the formula, is the quaternion of the UAV at time t, indicating the rotation of the attitude of the UAV at time t relative to the world coordinate system; is a unit vector vertically upward in the world coordinate system; is the z component of the orientation vector. Reward for task completion, a positive constant given when the drone completes the task and reaches the end point; are weights corresponding to the above-mentioned each award item, and the sum of each weight is taken as 1.

9. An unmanned aerial vehicle autonomous obstacle avoidance device based on swarm confrontation, characterized in that, The method comprises the following steps: a first module configured to construct a competitive obstacle avoidance task of multiple virtual unmanned aerial vehicles (UAVs) in an obstacle scene in a simulation environment; a second module configured to, during the execution of the competitive task, construct, by each virtual UAV, its own ontology state information by using its own state coding and visual information coding, construct, by each virtual UAV, its own competitive information by using the relative information between each virtual UAV and all other virtual UAVs, and construct, by each virtual UAV, its own state by using the ontology state information and the competitive information of the virtual UAV; The third module is configured to perform joint reinforcement learning training on the autonomous obstacle avoidance strategies of all virtual drones using the shared policy network and the value network, so that each virtual drone can obtain an action at the current time according to a state at the current time. The fourth module is configured to deploy the trained policy network on the target drone, so that the policy network can obtain an obstacle avoidance action based on the ontology state information of the target drone.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the computer to perform the method for autonomous obstacle avoidance of the drone according to any one of claims 1-8.

Images