Unmanned aerial vehicle racing method and simulation system in complex environment

Abstract

The invention discloses an unmanned aerial vehicle racing method in a complex environment and a simulation system, and relates to the technical field of unmanned aerial vehicles, and the method comprises the steps: a soft constraint training stage: canceling an unmanned aerial vehicle collision body, processing multi-source observation data under a symmetric actor-commentator framework, introducing local Lipschitz continuous constraint, and carrying out the soft constraint training stage; optimizing by adopting a near-end strategy optimization algorithm to obtain a basic racing strategy; in the hard constraint training stage, a cuboid collision body is configured, an asymmetric actor-commentator framework is adopted, noise is added to the input of an actor network, a basic strategy is finely adjusted by adopting a near-end strategy optimization algorithm, and a target strategy is obtained; and a real unmanned aerial vehicle deployment stage: deploying the target strategy to a real unmanned aerial vehicle, constructing a control compensation logic, and completing autonomous racing in a complex environment. According to the method, the training efficiency and the strategy performance are remarkably improved through a double-stage training framework, meanwhile, the research and development efficiency is greatly improved through the high fidelity and modular design of a simulation platform, and seamless migration from simulation to reality is ensured.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a UAV racing method and simulation system in complex environments. Background Technology

[0002] Reinforcement learning (RL) has shown great application potential in the field of robot control in recent years, especially for controlling complex, high-dimensional, and nonlinear dynamic systems. However, training reinforcement learning directly on physical robots is not only costly and risky, but also inefficient. Therefore, researchers generally rely on high-fidelity physical simulators as training environments to quickly collect data and iteratively optimize control strategies under safe and controllable conditions.

[0003] Unmanned aerial vehicle (UAV) simulation platforms are computer-based virtual testing environments that provide core support for the research and verification of UAV algorithms through physical environment modeling, task simulation, and sensor simulation. Their user-friendly interfaces and multi-language support capabilities effectively integrate control algorithms and robot middleware, thereby significantly reducing the risks and development costs of real-world debugging.

[0004] Currently, NVIDIA's Isaac Lab simulation platform is one of the leading examples in this field. Built on the NVIDIA PhysX physics engine and Isaac Sim, Isaac Lab enables large-scale parallel simulations through GPU acceleration, allowing tens of thousands of simulation instances to run simultaneously, significantly improving training speed and data acquisition efficiency. However, despite its powerful performance, Isaac Lab still has significant limitations in several key technical aspects, restricting its further application in cutting-edge research: 1. Insufficient support for drone scenarios Isaac Lab's development and examples primarily focus on motion control and robotic arm manipulation tasks for legged robots (bipedal, humanoid, quadrupedal), offering limited support in terms of models, sensors, and environments for unmanned aerial vehicles (UAVs). The underactuated, high-speed, and three-dimensional maneuvering dynamics of UAVs, coupled with the lack of highly customized simulation environments, severely limit the application of reinforcement learning in agile UAV control research.

[0005] 2. Difficulties exist in transferring simulation to reality.

[0006] Isaac Lab offers high-fidelity dynamics simulations for quadrupedal and humanoid robots, but its dynamics modeling accuracy for drones is insufficient. Currently, it only provides simulation parameters for a small drone (Crazyflie), which is insufficient to cover the characteristics of different models. Furthermore, the lack of low-level controller support for underactuated systems makes rapid deployment of classic controllers inconvenient, impacting the development efficiency and portability of drone algorithms.

[0007] These shortcomings make it difficult for researchers to efficiently train drone control strategies that can be directly transferred to real platforms. Although existing research has shown that simulation-based reinforcement learning can train drone controllers that outperform human pilots, such results heavily rely on well-designed high-fidelity simulation environments and effective simulation-to-real transfer techniques, highlighting the importance of building dedicated high-fidelity simulation systems.

[0008] On the other hand, drone racing, as a challenging scenario integrating perception, decision-making, and control, has gradually become a cutting-edge direction in autonomous flight research in recent years. Racing tasks are characterized by high speed, nonlinearity, and strong time-varying features: drones must navigate narrow gates at high speeds on complex tracks while maintaining attitude stability and safety. This places extremely high demands on perception, state estimation, path planning, and control algorithms. Traditional rule-based or hierarchical control-based navigation methods heavily rely on detailed mapping and high-precision positioning, making it difficult to balance real-time performance and robustness. In recent years, reinforcement learning, imitation learning, and model-based predictive control (MPC) have been introduced into drone racing research. These methods can directly learn end-to-end policies from raw inputs such as cameras, IMUs, and depth sensors, ensuring both real-time performance and strong generalization capabilities.

[0009] Research on drone racing algorithms has not only promoted the development of high-performance flight control and intelligent decision-making, but also has important application value: on the one hand, it can serve high-speed autonomous tasks such as disaster relief, search and rescue, and inspection of complex environments; on the other hand, racing scenarios provide a standardized platform for verifying the performance of the perception-decision-control closed loop, which helps to promote aerial robots to a higher level of autonomous intelligence.

[0010] Therefore, those skilled in the art are dedicated to developing a method and simulation system for UAV racing in complex environments, providing more powerful algorithm support and verification platform for autonomous UAV racing. This system can achieve efficient strategy training and seamless transfer to real UAVs, enabling high-speed maneuvering, obstacle avoidance, and precise lap flight. Summary of the Invention

[0011] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is how to enable UAVs to learn a flight strategy that balances high-speed racing and safe obstacle avoidance in unknown and complex environments, and how to achieve efficient training of UAV control strategies and zero-sample transfer from simulation to reality.

[0012] To achieve the above objectives, the present invention provides a method for drone racing in complex environments, the method comprising the following steps: Step 1, Soft Constraint Training Phase: Initialize the basic drone racing strategy, including: Step 1.1: Remove the cuboid collider from the UAV's 3D model; Step 1.2: Acquire multi-source observation data, which includes environmental depth maps, UAV status information, and mission command information; Step 1.3: Under the symmetrical actor-critic framework, a convolutional neural network is used to process the environmental depth image, and a multilayer perceptron is used to process the UAV's own state information and the mission instruction information; the processed multi-source observation data is input into the actor network; the actor network outputs control actions based on the processed information and interacts with the simulation environment; Step 1.4: Calculate the reward function based on the interaction results, introduce local Lipshitz continuity constraints, adopt the proximal policy optimization algorithm, iteratively update the parameters of the actor network and the critic network, and obtain the basic racing strategy; Step 2, Hard Constraint Training Phase: Enhance policy safety and robustness, including: Step 2.1: Configure a cuboid collider for the UAV's 3D model; Step 2.2: Using an asymmetric actor-critic framework, noise is added to the input of the actor network; Step 2.3: Calculate the reward function based on the interaction results; Step 2.4: Fine-tune the basic policy using the near-end policy optimization algorithm to obtain the target policy; Step 3, the actual deployment phase, involves deploying the target strategy to real drones, building control compensation logic, and completing autonomous racing in complex environments.

[0013] Furthermore, the WARP parallel collision point calculation logic is used to obtain the immersion depth of the drone and scene obstacles, and the collision constraint is relaxed into a soft constraint through the collision penalty formula.

[0014] Furthermore, the environmental depth map is used to perceive the geometric information of obstacles and door frames; the UAV's own state variables include its position, velocity, and attitude angle; the mission instruction information includes the position coordinates and direction vector of the next target door frame.

[0015] Furthermore, the reward function includes orientation reward, body angular velocity penalty, motion rate penalty, collision penalty, perception reward, successful gate crossing reward, and poor posture penalty.

[0016] Further, in step 2.2, adding noise to the input of the actor network includes: The environmental depth map is multiplied by a random scaling factor to simulate environmental depth map noise; The velocity of the aircraft is multiplied by a random scaling factor to simulate velocity estimation noise; random angular noise is added to the attitude angle of the aircraft to simulate attitude estimation error. The course learning involves noise enhancement of the task instruction information.

[0017] Furthermore, the noise-enhanced learning of the task instruction information includes: based on the UAV's racing performance, adding evolving noise to the position coordinates and direction vector of the next target gate, as described in the following formula:

[0018] in, μ It is the mean of the added Gaussian noise. , These are the noise enhancement factor and the attenuation factor, respectively. n This represents the number of gates the drone traverses on the current track; that is, the better the drone performs, the louder the command noise becomes, and vice versa.

[0019] Furthermore, the hard-constraint training phase also includes: Collision Termination: When the distance between the drone's 3D model and an obstacle is detected to be less than a critical value, the current training round is immediately terminated, and a high collision penalty is imposed; Action boundary constraint steps: Limit the boundaries of the control actions output by the actor's network.

[0020] A drone racing simulation system for complex environments includes: Installation script module: Used to automatically deploy the runtime environment of the simulation system; Training and testing script module: used to start the training or testing of a strategy with one click by configuring parameters; Configuration interface module: used to configure the simulation environment and set physical parameters for different tasks. Its design conforms to the Isaac Lab Manager-based environment interface and the RSL-RL interface training interface. Control script module: Used to call the parallel control interface unit to control the state of the UAV model unit; The UAV simulation platform module is a secondary encapsulation based on NVIDIA's Isaac Lab, including: a UAV model unit, providing a 3D simulation model of the UAV with real-world calibrated physical parameters; a dynamics model unit, supporting forward dynamics calculations and including configurable drag coefficients and response delay parameters; a sensor simulation unit, used for high-fidelity simulation of the data and noise characteristics of depth cameras and IMUs; and a parallel control interface unit, providing four levels of control interfaces for position, velocity, thrust and angular velocity, and propeller thrust, used to control the state of the UAV model unit.

[0021] Furthermore, the drone simulation platform module also includes a sample task library, which includes drone racing, obstacle avoidance, and hovering tasks.

[0022] Furthermore, the drone simulation platform module also includes a trajectory primitive generator, used to predefine basic trajectory patterns such as circles, polygons, and ellipses to generate diverse racing track environments.

[0023] Compared with the prior art, the present invention has at least the following beneficial technical effects: 1. This invention adopts a two-stage reinforcement learning training framework and performs policy learning on a self-developed highly parallel high-fidelity simulation platform. This framework significantly improves the sample efficiency of reinforcement learning training and learns a policy that combines obstacle avoidance and racing capabilities. 2. This invention introduces asymmetric frames and noise-enhanced learning to force the policy network to develop deep visual understanding capabilities, enabling it to perceive door frames and learn control and recognition capabilities end-to-end without the need for additional door frame recognition algorithms, thus reducing computational power requirements. 3. This invention is based on the large-scale simulation framework Isaac Lab. Through meticulous dynamic modeling and system identification, it narrows the gap between simulation and reality, ensuring the effectiveness of policy transfer. The modular architecture and open interface design enable users to quickly expand models, sensors and tasks, and seamlessly integrate with mainstream learning frameworks. The hierarchical control interface and example task library provided greatly reduce the difficulty and cost of algorithm development, verification and iteration.

[0024] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0025] Figure 1 This is a flowchart of a preferred embodiment of the present invention; Figure 2 This is a block diagram of a simulation system structure according to a preferred embodiment of the present invention; Figure 3This is a flowchart of the simulation and training loop of a preferred embodiment of the present invention; Figure 4 This is a structural diagram of a drone simulation platform module according to a preferred embodiment of the present invention. Detailed Implementation

[0026] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0027] This embodiment provides a method for drone racing in complex environments, which can be applied to a drone racing simulation system in a complex environment to achieve safe and efficient autonomous track navigation, such as... Figure 1 As shown, it is divided into a soft training phase, a hard training phase, and a deployment phase, including the following steps: Step 1, Soft Constraint Training Phase: Initialize the basic drone racing strategy and trajectory perception learning, including: Step 1.1: Remove the cuboid collider from the UAV's 3D model. Since the collider is removed, the simulation environment will not perform collision detection on the UAV to terminate its flight. Simultaneously, a parallel collision point calculation method written in WARP is used to calculate the immersion depth of the UAV and the scene, relaxing the collision constraints to soft constraints. Then, a penalty is applied based on the number of collision points, i.e., the penalty action is based on the following formula:

[0028] in, p These are predefined collision points on the drone mesh. It is an indicator function; its value is 1 when the condition is met, and 0 otherwise.

[0029] Step 1.2: Acquire multi-source observation data, including environmental depth maps, UAV status information, and mission command information.

[0030] The environmental depth map is a depth image of the track environment captured by the depth camera on the drone, used to perceive the geometric information of obstacles and gates; the drone's own state variables include the drone's position, speed, and attitude angle; the mission command information includes the position coordinates and direction vector of the next target gate.

[0031] Step 1.3: Under the Symmetric Actor-Critic framework, an observation encoder is used to process multi-source observation data and extract feature vectors of the environment and task, providing a unified input representation for the subsequent Actor-Critic network. The observation encoder consists of a Convolutional Neural Network (CNN) and a Multilayer Perceptron (MLP). The CNN processes the environmental depth image, and the MLP processes the UAV's own state information and task command information. The processed multi-source observation data is then input into the actor network. Based on the processed information, the actor network outputs control actions to interact with the simulation environment.

[0032] The actor network receives the encoded feature vectors and outputs continuous control actions, including the combined thrust and angular velocity. The critic network, also based on the feature vectors, evaluates the value of the actor's actions through temporal difference (TD) error, providing a supervision signal for the actor's policy gradient update.

[0033] Step 1.4: Calculate the reward function based on the interaction results to encourage drones to complete the track efficiently. The reward function settings are shown in Table 1.

[0034] Table 1

[0035] This indicates the settings for the hard constraint training phase.

[0036] Locally Lipschitz Continuous Constraints are introduced to enhance the robustness of the policy and reduce policy action oscillations. Specifically, a Locally Lipschitz Continuous Constraint module is designed to constrain multi-source observation data, including the environmental depth map (Depth), the UAV's own state information (State), and the mission instruction information (Next Gate), ensuring that the output of the subsequent policy network meets the continuity requirements and avoiding gradient explosion or policy oscillations during training.

[0037] The expression is as follows:

[0038]

[0039] Then, the Proximal Policy Optimization (PPO) algorithm is used to iteratively update the parameters of the actor network and the critic network to obtain the basic racing strategy.

[0040] Step 2, Hard Constraint Training Phase: Enhancing Policy Safety and Robustness. Building upon the Soft Phase, this phase trains the drone's ability to handle complex scenarios through adaptive difficulty and increased safety priority, including: Step 2.1: Configure a cuboid collider for the UAV's 3D model; the UAV will then terminate its trajectory due to the collision, further enhancing the strategy's collision perception of obstacles and improving the strategy's robustness.

[0041] Step 2.2: Initialize the asymmetric actor-critic framework based on the policy parameters obtained from soft-constraint training. During the soft-constraint training phase, the actor network and the critic network use consistent inputs. However, during the hard-constraint training phase, considering that depth measurement data collected by sensors in the real world is always affected by errors and noise, and that the UAV's own attitude estimation also has errors, a noise simulation technique is added. This adds noise to the input of the actor network, improving the robustness and generalization of the actor while ensuring the Critic accurately estimates the state-value function. Specifically: The environmental depth map is multiplied by a random scaling factor to simulate noise in the environmental depth map. For the aircraft's state, such as multiplying the aircraft's velocity by a random scaling factor, simulate velocity estimation noise; for the selection of the aircraft's attitude angle, add random angle noise to simulate attitude estimation error. The course learning involves noise enhancement of mission command information. This means that, based on the drone's racing performance, evolving noise is added to the position and orientation command of the next target gate, as described in the following formula:

[0042] in, μ It is the mean of the added Gaussian noise. , These are the noise enhancement factor and the attenuation factor, respectively. n This represents the number of gates the drone traverses on the current track. The better the drone performs, the louder the command noise becomes, and vice versa. This allows the strategy to reduce its reliance on precise commands without sacrificing racing performance, thus enabling the development of gate location perception capabilities within the depth map, ultimately integrating control and perception.

[0043] Furthermore, this phase employs an adaptive curriculum to dynamically adjust the difficulty of the training scenarios. Initially, a simple track (large gate spacing, few obstacles) is used, and as training progresses, the track complexity is gradually increased (reducing gate spacing, adding more obstacles), forcing the policy to learn the generalization ability of the scenarios; and the aforementioned noisy curriculum is also added.

[0044] Step 2.3: Calculate the reward function based on the interaction results; set collision termination and safety constraints: Collision Termination: When the distance between the drone and an obstacle is detected to be less than the threshold (or the physical collision triggers the sensor), the current training episode is immediately terminated, and a high collision penalty is applied to force the policy to learn the "obstacle avoidance priority" logic instead of the previous soft collision penalty. Bounded Action: Limits the actions output by the Actor (e.g., angular velocity does not exceed the maximum safe value, attitude angle range ±60°) to prevent extreme actions from causing drone instability and ensure training safety.

[0045] Step 2.4: Fine-tune the basic policy during the soft-constraint training phase using the proximal policy optimization algorithm to obtain the target policy. The reward function used is shown in Table 1.

[0046] Step 3, the actual deployment phase, involves deploying the target strategy to real drones, constructing control compensation logic, and completing autonomous racing in complex environments. This includes: 1) Hardware carrier selection and integration Main control unit: The Radxa Zero3w embedded development board is used, which has the characteristics of low power consumption and high performance. The reinforcement learning policy network and control algorithm are run on the NPU with the RKNN framework to achieve millisecond-level real-time decision-making. Sensing Unit: Equipped with a Realsense D435i depth camera (to acquire depth images in real time and perceive environmental geometry) and Vicon Marker motion capture markers (used for high-precision positioning calibration in a laboratory environment to ensure the mapping accuracy between the sensor and the real location). Execution unit: The drone platform is equipped with a brushless motor, which responds to control commands to adjust attitude and speed.

[0047] 2) Control compensation and delay processing

[0048] Air drag compensation: Establish a mathematical model of air drag, estimate relevant parameters based on UAV test flight data, integrate it into the simulation platform for domain randomization processing, and improve the strategy's sim-to-real capability; PID closed-loop control (PID Gain): A PD controller is designed for the angular velocity loop, and a fast response to control commands is achieved through the proportional-derivative element; Control Delay Compensation: This involves analyzing system delays (sensor acquisition → algorithm calculation → actuator response) and compensating for these delays in the simulation system to offset the impact of delays on control accuracy and ensure timely command execution.

[0049] 3) Real track flight

[0050] In a real racetrack environment, the drone autonomously completes the closed-loop process of "environmental perception (depth camera) → trajectory planning (Actor output) → control execution (PID + compensation)" based on the strategies learned during the training phase, achieving safe and efficient racetrack crossing.

[0051] This embodiment also provides a drone racing simulation system for complex environments, including: 1) Install the script module to automatically deploy the runtime environment of the simulation system.

[0052] A script is a set of instructions written in text that can be executed automatically by a computer. Common scripting languages ​​include Python, JavaScript, and Bash.

[0053] In this embodiment, the installation script is used to automatically deploy the runtime environment of the UAV simulation system and install Python dependency packages. After successfully running the installation script, the simulation system can be used immediately. Using the installation script can avoid compatibility issues that may occur during manual installation, ensuring that the UAV simulation system is "install-and-use".

[0054] 2) Training and testing script module, used to start the training or testing of the strategy with one click by configuring parameters.

[0055] When using this drone simulation system, the training and testing scripts in this embodiment can be executed for initial testing. Thanks to the user-friendly language interface of this embodiment, users can write their own scripts to perform different training and performance tests for various drone missions. After the training and testing scripts are executed, the Isaac Sim software will be launched to execute the specific content of the scripts.

[0056] 3) The configuration interface module is used to configure the simulation environment and set physical parameters for different tasks. Its design conforms to the Isaac Lab Manager-based environment interface and the RSL-RL interface for training. Users can flexibly adjust training elements such as the UAV model, sensor suite, and scene environment for different UAV flight control tasks via the Python interface. Interaction through control scripts ensures that changes to the underlying engine are unnecessary when modifying the task; different experimental scenarios can be quickly switched by adjusting the configuration.

[0057] 4) Control script module, used to call the parallel control interface unit to control the state of the UAV model unit.

[0058] 5) Unmanned Aerial Vehicle (UAV) Simulation Platform Module

[0059] In this embodiment, the drone simulation platform module is a secondary encapsulation based on NVIDIA's Isaac Lab, including: The UAV model unit provides a 3D simulation model of the UAV with real-world calibrated physical parameters; the dynamics model unit supports forward dynamics calculations and includes configurable drag coefficients and response delay parameters; the sensor simulation unit is used to simulate the data and noise characteristics of depth cameras and IMUs with high fidelity; and the parallel control interface unit provides control interfaces at four levels: position, velocity, thrust and angular velocity, and propeller thrust, for controlling the state of the UAV model unit.

[0060] The drone simulation platform module also includes a sample task library and a trajectory primitive generator. The sample task library includes drone racing, obstacle avoidance, and hovering tasks. The trajectory primitive generator is used to predefine basic trajectory patterns such as circles, polygons, and ellipses to generate diverse racing track environments.

[0061] like Figure 3 As shown, the drone simulation process can be implemented using Isaac Sim 4.2. Isaac Sim 4.2 provides users with a high-fidelity and configurable drone simulation environment, capable of supporting drone motion execution and feedback in various complex scenarios. The motion control commands output by the controller are executed within this platform. Different control inputs drive the drone's dynamic model, thereby changing its attitude, velocity, position, and other state variables. Subsequently, Isaac Sim 4.2 outputs the updated state variables in real time, forming a closed loop of perception and feedback.

[0062] In this process, the controller plays a crucial role. Higher-level policy modules (such as reinforcement learning-based policy networks) output the desired high-level actions, while the controller maps these high-level commands to lower-level control variables that conform to physical constraints, such as motor speed or servo angle, ensuring accurate execution of actions in the simulation environment. After an action is applied, the UAV's state changes due to different inputs. Isaac Sim 4.2 feeds back the changed state to the control policy module and the Proximal Policy Optimization (PPO) trainer. Upon receiving the state feedback, the PPO module calculates and updates the current policy parameters; the updated policy, combined with the state information output from the environment, recalculates the desired action and applies it again to the UAV model through the controller. Thus, a complete training loop of "state-decision-execution-feedback-update" is formed.

[0063] The entire simulation and training process is scheduled and managed by the Manager-based environment interface integrated into IsaacLab. Manager is responsible for synchronizing the environment state, ending and resetting training episodes, recording and statistically analyzing performance metrics, and decoupling configuration files and control scripts during training. This design ensures researchers have high flexibility during experiments, enabling them to quickly modify and expand different UAV missions; it also ensures the training process meets the requirements of high-fidelity simulation and verifiability for industrial applications. This allows the UAV's learning process in the virtual environment to closely resemble real physical laws while simultaneously meeting the dual needs of research exploration and engineering implementation.

[0064] like Figure 4 The diagram shows the structural composition of the core part of the simulation system, which is built based on the Isaac Sim 4.2 environment.

[0065] Here, users can refer to the formal framework of Markov decision process (MDP) to define reinforcement learning-related elements for different UAV simulation tasks. This process includes multiple aspects such as reward definition, environment definition, curriculum setting, and action definition.

[0066] The reward function is the core driving force of reinforcement learning training. It provides quantitative feedback on the consequences of the drone's actions in the environment, thereby guiding the policy network to continuously improve. In this embodiment, positive rewards are used to encourage the drone to move towards the target direction, while negative rewards are used to reinforce the drone's obstacle avoidance behavior during flight.

[0067] The environment definition forms the foundation for UAV training. In this embodiment, the environment definition includes a depth camera, a 3D model of the UAV, and an inertial measurement unit (IMU). The depth camera provides the UAV with visual perception capabilities, the 3D model ensures that the UAV's motion in the simulation conforms to real physical characteristics, achieving high-fidelity physical reproduction, while the IMU provides motion information such as attitude and angular velocity, ensuring the integrity and authenticity of state observation.

[0068] Curriculum design helps drones achieve progressive training in reinforcement learning. By flexibly setting training difficulty, the initial difficulty of complex tasks can be effectively reduced during the learning process, allowing the strategy to gradually accumulate experience and acquire stronger generalization capabilities. The introduction of curriculum learning not only improves training stability but also shortens model convergence time.

[0069] Action definition specifies the control methods that a UAV can execute in a simulation. Within this framework, actions not only include action commands translated from controller commands to low-level execution, but also allow users to customize dynamic parameters based on the characteristics of different UAV models. This flexible action modeling approach enables the simulation system to be compatible with various UAV models and control requirements.

[0070] The various definitions and configurations here all adopt a modular design approach, which not only enhances the flexibility and scalability of the simulation system during use, but also enables researchers to quickly replace, modify, and combine them according to task requirements. Based on this framework, users can efficiently conduct UAV reinforcement learning training and testing on the Isaac Sim 4.2 platform, meeting both the experimental needs of scientific research and providing an scalable verification tool for engineering implementation.

[0071] This embodiment achieves improved obstacle avoidance capabilities, optimized racing performance, and enhanced robustness of UAVs in complex track scenarios by combining "policy initialization + trajectory generalization" in the soft training phase, "safety enhancement + perception learning" in the hard training phase, and "hardware adaptation + control compensation" in the deployment phase. Compared with traditional rule-based or single-stage training methods, the speed is improved by nearly 50%, and the success rate (measured by collision) is improved by nearly 60% (from 20% to 80%).

[0072] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for drone racing in complex environments, characterized in that, The method includes the following steps: Step 1, Soft Constraint Training Phase: Initialize the basic drone racing strategy, including: Step 1.1: Remove the cuboid collider from the UAV's 3D model; Step 1.2: Acquire multi-source observation data, which includes environmental depth maps, UAV status information, and mission command information; Step 1.3: Under the symmetrical actor-critic framework, a convolutional neural network is used to process the environmental depth image, and a multilayer perceptron is used to process the UAV's own state information and the mission instruction information; the processed multi-source observation data is input into the actor network; the actor network outputs control actions based on the processed information and interacts with the simulation environment; Step 1.4: Calculate the reward function based on the interaction results, introduce local Lipshitz continuity constraints, adopt the proximal policy optimization algorithm, iteratively update the parameters of the actor network and the critic network, and obtain the basic racing strategy; Step 2, Hard Constraint Training Phase: Enhance policy safety and robustness, including: Step 2.1: Configure a cuboid collider for the UAV's 3D model; Step 2.2: Using an asymmetric actor-critic framework, noise is added to the input of the actor network; Step 2.3: Calculate the reward function based on the interaction results; Step 2.4: Fine-tune the basic policy using the near-end policy optimization algorithm to obtain the target policy; Step 3, the actual deployment phase, involves deploying the target strategy to real drones, building control compensation logic, and completing autonomous racing in complex environments.

2. The method for drone racing in complex environments as described in claim 1, characterized in that, Step 1 further includes: using WARP parallel collision point calculation logic to obtain the immersion depth of the drone and scene obstacles, and relaxing the collision constraint into a soft constraint through the collision penalty formula.

3. The method for drone racing in complex environments as described in claim 1, characterized in that, The environmental depth map is used to perceive the geometric information of obstacles and door frames; the UAV's own state variables include the aircraft's position, velocity, and attitude angle; the mission instruction information includes the position coordinates and direction vector of the next target door frame.

4. The method for drone racing in complex environments as described in claim 1, characterized in that, The reward function includes orientation reward, body angular velocity penalty, motion rate penalty, collision penalty, perception reward, successful gate crossing reward, and poor posture penalty.

5. The method for drone racing in complex environments as described in claim 4, characterized in that, In step 2.2, adding noise to the input of the actor network includes: The environmental depth map is multiplied by a random scaling factor to simulate environmental depth map noise; The velocity of the aircraft is multiplied by a random scaling factor to simulate velocity estimation noise; random angular noise is added to the attitude angle of the aircraft to simulate attitude estimation error. The course learning involves noise enhancement of the task instruction information.

6. The method for drone racing in complex environments as described in claim 5, characterized in that, The noise enhancement process for the task instruction information includes: based on the UAV's racing performance, adding evolving noise to the position coordinates and direction vector of the next target gate, as described in the following formula: in, μ It is the mean of the added Gaussian noise. , These are the noise enhancement factor and the attenuation factor, respectively. n This represents the number of gates the drone traverses on the current track; that is, the better the drone performs, the louder the command noise becomes, and vice versa.

7. The method for drone racing in complex environments as described in claim 1, characterized in that, The hard-constraint training phase also includes: Collision Termination: When the distance between the drone's 3D model and an obstacle is detected to be less than a critical value, the current training round is immediately terminated, and a high collision penalty is imposed; Action boundary constraint steps: Limit the boundaries of the control actions output by the actor's network.

8. A simulation system for executing the UAV racing method under complex environments as described in any one of claims 1 to 7, characterized in that, include: Installation script module: Used to automatically deploy the runtime environment of the simulation system; Training and testing script module: Used to start the training or testing of a strategy with one click by configuring parameters; Configuration interface module: used to configure the simulation environment and set physical parameters for different tasks. Its design conforms to the Isaac Lab Manager-based environment interface and the RSL-RL interface training interface. Control script module: Used to call the parallel control interface unit to control the state of the UAV model unit; Drone simulation platform module: Based on NVIDIA's Isaac Lab, it is repackaged and includes: Drone model unit, which provides a 3D simulation model of a drone with real-world calibrated physical parameters; The dynamics model unit supports forward dynamics calculations and includes configurable drag coefficients and response delay parameters; the sensor simulation unit is used to simulate the data and noise characteristics of depth cameras and IMUs with high fidelity; the parallel control interface unit provides control interfaces at four levels: position, velocity, thrust and angular velocity, and propeller thrust, for controlling the state of the UAV model unit.

9. The UAV racing simulation system under complex environments as described in claim 8, characterized in that, The drone simulation platform module also includes a sample task library, which includes drone racing, obstacle avoidance, and hovering tasks.

10. The UAV racing simulation system under complex environments as described in claim 8, characterized in that, The drone simulation platform module also includes a trajectory primitive generator, which is used to predefine basic trajectory patterns such as circles, polygons, and ellipses to generate diverse racing track environments.

Images