Quadrotor random obstacle racing method based on course reinforcement learning

Abstract

The invention discloses a four-rotor random obstacle racing method based on course reinforcement learning, and aims to solve the problems that unmanned aerial vehicle racing conflicts with an obstacle avoidance target, the generalization of an obstacle scene is poor, and migration from simulation to reality is difficult in the prior art. The method comprises the following steps: constructing an unmanned aerial vehicle system comprising a flight platform, a sensing unit, a calculation unit and a bottom layer controller; defining an observation space consisting of state observation and visual observation, and an action space containing collective thrust and body angular velocity; through a multi-stage curriculum reinforcement learning framework with gradually increased difficulty and a multi-scene parallel updating mechanism, obstacle avoidance and door-crossing racing requirements are balanced in combination with a composite reward function; training an end-to-end visual control strategy by adopting a lightweight network architecture combining a CNN, an MLP and a GRU; after training is completed, the strategy network weight is exported to be in an ONNX format, and the strategy network weight is deployed in the Raspberry Pi to achieve zero sample migration. According to the invention, 8m / s high-speed collision-free flight in a complex obstacle environment is realized, the success rate reaches 100%, and the generalization is strong.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of autonomous robots and artificial intelligence, and particularly relates to a quadrotor random obstacle racing method based on curriculum reinforcement learning. BACKGROUND

[0002] Autonomous unmanned aerial vehicle navigation technology has become a research hotspot in the field of artificial intelligence and robots due to its wide application in many industries such as express delivery, facility inspection, emergency response, etc. Traditional unmanned aerial vehicle racing only emphasizes "the shortest navigation time" and needs to achieve high-speed flight in an obstacle-free track. However, in actual applications, unmanned aerial vehicles often need to complete tasks in an environment containing random obstacles. Therefore, the two goals of "racing" and "obstacle avoidance" are integrated, and the vehicle is required to avoid obstacles in a partially or completely unknown environment while driving at high speed, forming a "random obstacle racing" task that is closer to actual demand. This task has become an important benchmark for testing the performance of advanced autonomous systems.

[0003] Unmanned aerial vehicle racing originated from competitive sports, when human pilots controlled agile unmanned aerial vehicles on a track to fly as fast as possible while avoiding potential obstacles, which relied on the precise operation, quick reaction ability and professional navigation skills of the pilot.

[0004] Autonomous unmanned aerial vehicle racing in a cluttered environment is a major challenge, which requires the vehicle to avoid obstacles while flying as fast as possible. Although autonomous quadrotors have shown excellent agility in controlled laboratory environments, achieving collision-free racing in a densely cluttered environment is still an unsolved problem. Obstacle perception racing requires unmanned aerial vehicles to maximize their flight speed while avoiding obstacles, thus pushing the platform dynamics to its operating limit. However, as shown in Figure 1 Since the door is also considered as an obstacle, the two goals of passing through the door and avoiding obstacles are essentially in conflict, which poses a serious challenge to both tasks. These competing demands, combined with environmental variability and model mismatch, can lead to catastrophic failures, highlighting the urgent need for robust and adaptive solutions.

[0005] To solve the above problems, researchers have proposed a variety of technical solutions. Traditional path planning and optimization methods can achieve a certain balance between racing efficiency and computational efficiency, but they rely too much on carefully designed algorithms and their performance easily degrades when the model and the actual environment mismatch. Current methods based on reinforcement learning and imitation learning can achieve low-latency collision-free control, but the trained strategies often overfit to a single track and obstacle configuration, have poor generalization to obstacle scenarios, and have very low success rate of transferring from simulation environment to real world, making it difficult to meet the actual application requirements.

[0006] Therefore, in view of the core pain points of the target conflict, poor generalization, and difficulty in simulation migration of autonomous unmanned aerial vehicle racing and obstacle avoidance, a robust and adaptive technical solution is urgently needed to promote the industrial application of autonomous unmanned aerial vehicles flying at high speed and safely in complex and chaotic environments. SUMMARY

[0007] In view of the above-mentioned defects of the prior art, the present application at least solves the following technical problems: 1. The existing learning-based method is difficult to balance the inherent conflict between the "high-speed racing through the door" and "obstacle avoidance" of the unmanned aerial vehicle, and the two goals are contradictory, resulting in a single strategy that cannot achieve efficient racing and safe obstacle avoidance at the same time, which is prone to flight failure; 2. The traditional learning-based racing and obstacle avoidance strategy network is over-fitted to a single race track or obstacle configuration, and when faced with complex scenes such as unobserved obstacle density changes and door frame position disturbances, it has poor adaptability and is difficult to work stably; 3. The success rate of the strategy trained by the existing technology is low when migrating from the simulation environment to the real unmanned aerial vehicle, and a large amount of real scene data is needed to debug the strategy network parameters, resulting in a long development cycle and high deployment cost; 4. The traditional path planning and optimization method relies on accurate environment modeling, and the performance degrades when the model and the actual environment do not match, which cannot meet the high-speed agile flight requirements in complex dynamic environments.

[0008] The application discloses a quadrotor random obstacle racing method based on curriculum reinforcement learning, which comprises the following steps: S1: constructing a quadrotor unmanned aerial vehicle system comprising a flight platform, a perception unit, a computing unit, and a bottom controller; S2: defining the observation space and action space of the unmanned aerial vehicle, wherein the observation space comprises unmanned aerial vehicle state observation and visual observation, and the action space is the control instruction of the unmanned aerial vehicle; S3: training an end-to-end visual control strategy using a multi-stage curriculum reinforcement learning framework, wherein the framework comprises a difficulty-increasing training stage and a multi-scene parallel updating mechanism; S4: balancing the obstacle avoidance and door racing goals through a composite reward function, and the reward function expression is:

[0009] wherein, is the forward reward, is the alignment reward, is the control smoothing reward, is the speed constraint reward, is the obstacle avoidance reward, is the door racing reward, is the collision penalty reward; S5: extracting visual features and state features through a lightweight network architecture and fusing decisions to output the control instruction; S6: exporting the trained policy network weight in ONNX format and deploying it on the computing unit to realize zero-shot transfer from simulation to real environment and complete high-speed obstacle avoidance racing; Further, the hardware parameters of the quadcopter unmanned aerial vehicle system are: The flight platform is a quadcopter unmanned aerial vehicle with a mass of 0.58 kg, an inertia moment of [1.01, 1.53, 2.03] g·m², a maximum thrust of 14 N, and an arm length of 0.075 m. The perception unit is an Intel D435i depth camera used for real-time collection of environmental depth images. The computing unit is a Raspberry Pi single-board computer used for real-time inference of the policy. The bottom layer controller is an STM32F7 flight control firmware supported by BetaFlight, which is used to convert high-level control instructions into bottom layer instructions and stabilize the flight of the unmanned aerial vehicle. Further, the observation space is specifically defined as: The state observation and the visual observation :

[0010] wherein, and are the relative positions of the unmanned aerial vehicle to the centers of the nearest two door frames, is the linear velocity of the unmanned aerial vehicle itself, is the desired velocity, is the attitude represented by a quaternion, is the angular velocity. is a 64x64 pixel down-sampled depth image of the depth camera, and is processed by inverse depth conversion and Gaussian noise. Further, the action space is specifically defined as: wherein, is the collective thrust, are the angular velocities of the x, y, and z axes of the body, respectively, corresponding to the control instruction. Further, the training stages of the multi-stage curriculum reinforcement learning are set in increasing difficulty, including: The first stage: obstacle-free track environment, training unmanned aerial vehicle basic flight and door frame crossing skills; The second stage: randomly generating safety margin obstacles in the track through an obstacle generator; The third stage: setting the initial position of the UAV randomization and door frame position disturbance, the disturbance range is x, y axis ± 1m, z axis ± 0.3m, and the UAV is required to fly at high speed; Further, each reward item in the reward function is respectively expressed by a formula:

[0011]

[0012]

[0013]

[0014]

[0015]

[0016]

[0017] wherein, is the distance from the center of the UAV to the target door frame at time t, is the current yaw angle of the UAV, is the door frame alignment yaw angle, is the control instruction at time t, is the distance from the UAV to the nearest collision point, is a constant offset, - is a weight coefficient; Further, the lightweight network architecture comprises: a visual encoder: a three-layer CNN is used to extract features from the 64x64 pixel depth image collected by the perception unit, and output a visual feature vector; a state processor: a two-layer MLP is used to process the state observation vector of the UAV and output a state feature vector; a feature fusion and decision module: the visual feature vector and the state feature vector are spliced and input into a GRU module with a hidden layer dimension of 256, and then mapped to the action space output through a policy network, wherein the policy network adopts a two-layer MLP; a value network: the value network has the same structure as the policy network, also adopts a two-layer MLP, the input is the splicing result of the visual feature vector and the state feature vector, and the output is a value estimate, which is used to assist the training and optimization of the end-to-end visual control policy; Further, the multi-scene parallel updating mechanism is that: multiple differentiated race track simulation environments are run in parallel and grouped, each group synchronously carries out training, real-time training data are collected, and the parameters of the strategy network and the value network are jointly updated to prevent the strategy from overfitting to a single scene to improve the generalization ability. Further, the training of the end-to-end visual control strategy adopts a proximal policy optimization algorithm, and the training object is the strategy network and the value network in the lightweight network architecture. Further, the parameters of the proximal policy optimization algorithm are: the learning rate is 1e-4 decaying to 1e-5, the discount factor is 0.99, the clipping range is 0.2, the GAE-lambda coefficient is 0.95, and the batch size is 51200.

[0018] The application provides a complete, industrial application-oriented unmanned aerial vehicle high-speed autonomous flight solution, solves the key bottleneck of the contradiction between unmanned aerial vehicle racing and obstacle avoidance tasks through an innovative algorithm framework and system design, and endows the agile flight technology with reliability, robustness and low cost, has a clear industrialization path and broad application prospect: 1. Core technical effect: Compared with the traditional method, the application has significant advantages in training efficiency, generalization ability and system architecture, which constitutes the core barrier of its industrialization: 1.1 Efficient training framework and excellent generalization ability: multi-stage course learning and multi-scene updating mechanism enable the strategy to learn complex skills from easy to difficult and efficiently, greatly improving the training efficiency and convergence stability compared with the traditional reinforcement learning. Through the system's domain randomization, the strategy shows strong robustness to obstacle configurations and door frame position changes that have not been seen in training, realizes "one training, widely applicable", and overcomes the pain points of traditional methods relying on accurate environment modeling and poor scene generalization ability; 1.2 End-to-end intelligent controller solving the core conflict: for the pair of internal conflict goals of "passing through the door frame" and "avoiding obstacles", the application designs a composite reward function, so that a single neural network can spontaneously learn a high-level strategy balancing the two, including performing large-angle yaw maneuvers at high speed to simultaneously achieve observation and control. Pure end-to-end visual-control mapping is adopted to avoid the delay and error accumulation of the traditional layered pipeline (perception-planning-control), laying a foundation for agile flight with millisecond-level response in dynamic environments; 1.3 Lightweight design and plug-and-play deployment characteristics: the lightweight network architecture has small parameter quantity, and the strategy can be easily deployed on low-cost embedded platforms such as Raspberry Pi, meeting the stringent weight, power consumption and computing resource requirements of unmanned aerial vehicles. Through full-link verification from hardware-in-the-loop simulation to real flight, combined with ONNX format export, the strategy ensures zero-sample simulation-to-reality migration, greatly reducing the cost and risk of on-site debugging.

[0019] 2. Excellent performance index effect: The performance of the application has been fully verified in simulation and physical experiments, and the key indicators far exceed the existing baseline method, providing solid data support for its industrialization: 2.1 Absolute reliability: In three different forms (S-shaped, J-shaped, 3D circular) of dense obstacle raceways, a success rate of 100% (10 consecutive simulation tests) is achieved, while the success rate of existing advanced methods is only 30%-40% 2.2 Extreme agility: The maximum speed in hardware-in-the-loop simulation is more than 10m / s, and in real-world flight it reaches 8m / s, and the lap speed is faster than the comparison baseline; 2.3 Strong generalization: In the extreme case of obstacle density increasing from 2 per two doors to 5, the success rate remains at 70%-80%; when the door frame position is randomly disturbed by ±1.0 meters, it can still maintain a success rate of 60%-80%; 2.4 System lightness: After deploying the strategy on Raspberry Pi, it can achieve real-time inference at 30Hz per second, fully meeting the real-time requirements of high-speed flight.

[0020] 3. Industrialization adaptation effect: The technical maturity and system design of the application have been highly close to the needs of industrialization: 3.1 Clear application scenarios and market space: In unmanned vehicle racing and performance, it can be used as a core technology to support professional racing unmanned vehicles or formation performance unmanned vehicles, achieving unprecedented high-speed and high-reliability performance in complex environments; In logistics and express delivery, in complex urban environments or warehouse interiors, unmanned vehicles can navigate at high speed and precision in dense obstacles, improving logistics efficiency; In search and inspection, in post-disaster ruins, forests or complex industrial facilities, unmanned vehicles can quickly and safely search and inspect without being limited by pre-set maps; In agricultural plant protection, in complex scenes such as orchards, unmanned vehicles can achieve more efficient autonomous flight spraying closer to crops; 3.2 Extensible technology platform and ecological potential: The VisFly simulator and open-source control package (betaflight-ctrl) relied on by the application form a complete research and development platform, which can quickly adapt to different unmanned vehicle hardware and task requirements; The end-to-end architecture makes it easy to combine with other AI models (such as target detection, speech recognition), expanding more composite functions (such as "tracking a target and avoiding obstacles"); 3.2 Low-cost, low-risk mass production and deployment: The algorithm has very low requirements for sensors (only one front-view depth camera) and the computing platform is low-cost (Raspberry Pi), greatly reducing the cost of single-machine hardware; the mode of full simulation training and zero-sample migration means that there is no need to collect a large amount of data in the real world or to conduct high-risk training, greatly shortening the development cycle, reducing the deployment threshold and safety risks, and being very suitable for large-scale replication and application.

[0021] The present application successfully converts laboratory-level agile flight AI into a stable, reliable, and easy-to-deploy industrialized solution through its efficient and reliable training framework, outstanding lightweight network architecture, and fully verified system engineering link, and has great potential for immediate landing and large-scale application in multiple high-value fields. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 Conflict diagram for the existing technology of four-rotor unmanned aerial vehicle passing through the gate and obstacle avoidance task; Figure 2 Architecture diagram of the four-rotor random obstacle racing method based on curriculum reinforcement learning of the present application; Figure 3 Composition and real-world experiment scene diagram of the four-rotor unmanned aerial vehicle obstacle avoidance racing system of the present application; Figure 4 Lightweight network architecture diagram of the present application. DETAILED DESCRIPTION

[0023] The following reference to the drawings of the specification introduces a plurality of preferred embodiments of the present application, so that the technical content is more clear and easy to understand. The present application can be embodied in many different forms of embodiments, and the protection scope of the present application is not limited to the embodiments mentioned in the text.

[0024] In the drawings, components of the same structure are denoted by the same reference numerals, and components having similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present application does not limit the size and thickness of each component. In order to make the drawing clearer, the thickness of some components is appropriately exaggerated in some places in the drawing.

[0025] Autonomous drone racing is a frontier field to test the agile flight and control algorithm of drones. In the prior art, the method based on reinforcement learning has shown better performance than traditional control methods in the obstacle-free track. However, when there are random obstacles in the track, the drone must deal with the two conflicting goals of 'passing through the door frame' and 'avoiding obstacles': the former requires the drone to actively and accurately approach the center of the door frame, while the latter encourages the drone to stay away from all obstacles (including the door frame itself). The existing methods either fail to converge due to the lack of effective training strategies, or the trained strategies have poor generalization and cannot adapt to unknown obstacle configurations, and the success rate of migration from simulation to real world is low. In view of this, the present application provides a quadrotor random obstacle racing method based on curriculum reinforcement learning, a curriculum reinforcement learning training framework, through multi-stage curriculum learning, multi-scene updating and a specially designed reward function, an end-to-end visual control strategy is trained, which is lightweight, efficient and can be migrated from simulation to real world with zero error, as shown in Figure 2 The above method comprises the following steps: S1: constructing a quadrotor drone system comprising a flight platform, a perception unit, a computing unit and a bottom controller; S2: defining the observation space and action space of the drone, wherein the observation space comprises drone state observation and visual observation, and the action space is the control instruction of the drone; S3: training an end-to-end visual control strategy using a multi-stage curriculum reinforcement learning framework, wherein the framework comprises a difficulty-increasing training stage and a multi-scene parallel updating mechanism, and the training object is a policy network and a value network in a lightweight network architecture; S4: balancing the obstacle avoidance and door passing racing goals through a composite reward function, wherein the reward function expression is:

[0026] wherein, is the forward reward, is the alignment reward, is the control smoothing reward, is the speed constraint reward, is the obstacle avoidance reward, is the door passing reward, is the collision penalty reward; S5: extracting visual features and state features through a lightweight network architecture and fusing decision-making, and outputting the control instruction by a policy network (two-layer MLP); S6: exporting the trained policy network weight in ONNX format, deploying it on the computing unit, realizing zero-sample migration from simulation to real environment, and completing high-speed obstacle avoidance racing.

[0027] In one specific embodiment, the quadcopter racing system of the present application is as shown in FIG. 1 (wherein a: hardware system composition, b: connection of Raspberry Pi and Intel D435i depth camera, c: real track obstacle and door frame distribution experimental scene), the selection, parameters and connection relationship of each hardware component are as follows: Figure 3 Flight platform: a quadcopter, the key physical parameters of which are: mass 0.58 kg, inertia moment [1.01, 1.53, 2.03] g·m², maximum thrust 14 N, arm length 0.075 m.

[0028] Sensing unit: an Intel D435i depth camera mounted on the quadcopter, used to obtain depth images of the front environment of the door and obstacles in real time.

[0029] Computing unit: a Raspberry Pi single-board computer, used to run the trained control strategy and perform real-time inference.

[0030] Bottom layer controller: an open source BetaFlight supported STM32F7 flight control firmware, used to convert high-level control instructions into bottom layer instructions and stabilize the quadcopter.

[0031] The lightweight network architecture of the present application is trained in the self-developed VisFly simulator, and the trained strategy can be directly deployed to the above hardware system. The core implementation details of the software and algorithm are as follows: The input observation space of the strategy network is composed of state observation and visual observation , all vectors are converted to the body coordinate system: State observation : includes the relative position of the quadcopter to the centers of the nearest two door frames and , the linear velocity of the quadcopter itself , the desired velocity , the attitude represented by the quaternion and the angular velocity .

[0032] Visual observation : the input comes from the 64x64 pixel down-sampled depth map of the depth camera. During training, the depth map is converted in reverse and random Gaussian noise is added to reduce the gap between simulation and reality.

[0033] The strategy network directly outputs control instructions, and the action space is wherein is the collective thrust, ​Respective body x, y, z axis angular velocity, corresponding to the control instruction; Lightweight network architecture such as Figure 4 As shown, specifically comprising: Visual encoder: adopt three-layer CNN, feature extraction of 64x64 pixel depth image collected by the perception unit, output visual feature vector.

[0034] State processor: adopt two-layer MLP, feature processing of UAV state observation vector , output state feature vector.

[0035] Feature fusion and decision: after splicing the visual feature vector and the state feature vector, input a GRU module with hidden layer dimension of 256 to capture the time sequence dependence relationship, and the GRU output is mapped to the action space output through the policy network (two-layer MLP [192, 96]) to obtain the control instruction. Value network: the value network is completely consistent with the above policy network structure, adopts two-layer MLP, the input is the splicing result of the visual feature vector and the state feature vector, and the output is the value estimation value, to assist the training optimization of the end-to-end visual control policy.

[0036] To balance the conflict between "door racing" and "obstacle avoidance safety", the embodiment designs a composite reward function , the specific expression and the meaning of each item are as follows:

[0037] The specific expression of each reward function is as follows:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044] For encouraging the UAV to move forward to the next door frame; For encouraging the UAV to adjust the yaw angle to align the door frame and observe the obstacles, which is the key to realize large-angle maneuvering flight; For punishing violent control actions, making the trajectory smooth; a speed constraint reward for punishing the behavior of exceeding the expected speed; is a core reward term for providing continuous obstacle avoidance reward signals, solving the conflict with passing through the door frame; and are the reward for successfully passing through the door frame and the punishment reward when a collision occurs, respectively; wherein, is the distance from the center of the UAV to the target door frame at time t, is the current yaw angle of the UAV, is the yaw angle of the door frame alignment, is the control instruction at time t, is the distance from the UAV to the nearest collision point, is a constant offset, - is a weight coefficient.

[0045] A multi-stage curriculum reinforcement learning framework combined with a multi-scene parallel updating mechanism is adopted to improve the training efficiency and strategy generalization: wherein the multi-stage curriculum learning is arranged in increasing difficulty, including: The first stage: training in a simple track environment without obstacles, training the UAV to master the basic flight and door frame passing skills, and making the strategy initially establish the behavior logic of "moving forward to the door frame and aligning"; The second stage: introducing an obstacle generator to randomly generate obstacles with a safety margin in the track, upgrading the task difficulty, and making the strategy learn to "avoid obstacles while keeping flying to the door frame"; The third stage: on the basis of the second stage containing random obstacles in the track, the initial position of the UAV is randomized and the door frame position is disturbed (randomly within ±1m in x and y axes and ±0.3m in z axis), while the UAV is required to complete the flight task at high speed, strengthening the adaptability of the strategy under dynamic conditions and further improving the robustness of the strategy.

[0046] wherein the multi-scene updating mechanism: in the training, based on the VisFly simulator, multiple groups of differentiated track simulation environments (for example, 100 groups) are run in parallel. Unlike the traditional method of updating all environments to the same scene simultaneously, the embodiment divides the environments into groups, each of which simultaneously trains and updates the strategy in different track scenes (for example, 10). This mechanism can effectively balance exploration and utilization, prevent the strategy from overfitting to a single scene, and significantly improve the training efficiency and generalization ability of the final strategy.

[0047] The training of the lightweight network architecture adopts a proximal policy optimization (PPO) algorithm, and key parameter configurations are as follows: a learning rate of 1e-4 decaying to 1e-5, a discount factor of 0.99, a clipping range of 0.2, a GAE-lambda of 0.95, and a batch size of 51200.

[0048] The deployment process includes: After the training is completed, the policy network weight is exported in an ONNX format, which is compatible with an embedded platform and can guarantee consistency of cross-framework deployment. The model file in the ONNX format is deployed to a Raspberry Pi single-board computer, an Intel D435i depth camera is configured, and an STM32F7 flight control firmware supported by BetaFlight is configured to complete hardware adaptation and communication debugging. After the deployment is completed, without additional fine-tuning in a real environment, the unmanned aerial vehicle can be directly started to realize zero-shot transfer from simulation to a real environment and complete a high-speed obstacle avoidance racing task.

[0049] The policy trained by the method described in the embodiment has achieved a significantly better effect than the prior art in simulation and real-world experiments: In three different shapes (S-shaped, J-shaped, and 3D circular) and obstacle-filled tracks, the success rate reaches 100%, the maximum speed in real-world flight reaches 8 m / s, the maximum speed in hardware-in-the-loop simulation exceeds 10 m / s, and the lap speed is faster than the comparative baseline, achieving high success rate and high speed; It can effectively cope with unobserved obstacle configurations and certain range of door frame position changes, can effectively adapt to unobserved complex scenes, and embodies excellent generalization; Thanks to comprehensive field randomization of vision, dynamics, and initial state and lightweight network design, without collecting data or fine-tuning in the real world, the policy can be stably run after deployment, greatly reducing debugging costs and safety risks, and realizing efficient zero-shot transfer; After the policy is deployed on the Raspberry Pi, it can achieve an instruction output of 30 Hz per second, fully meeting the millisecond-level reaction requirements of high-speed flight, adapting to the stringent restrictions of the unmanned aerial vehicle on weight, power consumption, and computing resources, and embodying real-time performance and lightweight.

[0050] The above detailed the preferred embodiments of the application. It should be understood that those skilled in the art can make many modifications and changes to the embodiments of the application without creative labor based on the concept of the application. Therefore, any technical solution obtained by logical analysis, reasoning or limited experiments based on the prior art according to the concept of the application shall be within the protection scope defined by the claims.

Claims

1. A method for quadrotor random obstacle racing based on curriculum reinforcement learning, characterized in that, The method includes the following steps: S1: Construct a quadcopter unmanned aerial vehicle system that includes a flight platform, sensing unit, computing unit and underlying controller; S2: Define the observation space and action space of the UAV. The observation space includes UAV status observation and visual observation, and the action space is the control commands of the UAV. S3: An end-to-end visual control strategy is trained using a multi-stage course reinforcement learning framework, which includes training stages with increasing difficulty and a multi-scenario parallel update mechanism. S4: A composite reward function is used to balance the objectives of obstacle avoidance and gate-crossing speed race. The expression for the reward function is: in, For advancement rewards, To target rewards, To control the smoothing of rewards, Speed-constrained rewards For obstacle avoidance rewards, Rewards for passing through doors As a reward for collision penalties; S5: Extract visual features and state features through a lightweight network architecture, fuse them for decision-making, and output the control command; S6: Export the trained policy network weights in ONNX format and deploy them in the computing unit to achieve zero-sample transfer from simulation to the real environment and complete the high-speed obstacle avoidance race.

2. The quadcopter random obstacle racing method based on curriculum reinforcement learning as described in claim 1, characterized in that, The hardware parameters of the quadcopter unmanned aerial vehicle system are as follows: The flight platform is a quadcopter drone with a mass of 0.58 kg, a moment of inertia of [1.01, 1.53, 2.03] g·m², a maximum thrust of 14 N, and an arm length of 0.075 m. The sensing unit is an Intel D435i depth camera, used to acquire environmental depth images in real time. The computing unit is a Raspberry Pi single-board computer used for real-time policy inference. The underlying controller is equipped with STM32F7 flight controller firmware supported by BetaFlight, which is used to convert high-level control commands into low-level commands and stabilize the drone's flight.

3. The quadrotor random obstacle racing method based on curriculum reinforcement learning as described in claim 1, characterized in that, The specific definition of the observation space is: the observation space Including state observation and visual observation : in, and This refers to the relative position of the drone to the centers of the two nearest door frames. For the drone's own linear velocity, For the desired speed, The pose is represented by a quaternion. Angular velocity; It is a 64x64 pixel downsampled depth image from a depth camera, which has been subjected to inverse depth transformation and Gaussian noise processing.

4. The quadrotor random obstacle racing method based on curriculum reinforcement learning as described in claim 1, characterized in that, The specific definition of the action space is: action space ,in, For collective thrust, These are the angular velocities of the machine along the x, y, and z axes, respectively, corresponding to the control commands.

5. The quadcopter random obstacle racing method based on curriculum reinforcement learning as described in claim 1, characterized in that, The training phases of the multi-stage course reinforcement learning are set up in ascending order of difficulty, including: Phase 1: In an obstacle-free track environment, train basic drone flight and doorway crossing skills; Phase 2: Randomly generate obstacles with a safety margin in the track using an obstacle generator; Phase 3: Set the initial position of the drone to randomize and the position of the door frame to be disturbed. The disturbance range is ±1m on the x and y axes and ±0.3m on the z axis. The drone is required to fly at high speed.

6. The quadrotor random obstacle racing method based on curriculum reinforcement learning as described in claim 1, characterized in that, Each reward component in the reward function is expressed by the following formula: in, Let be the distance from the center of the UAV to the target gate frame at time t. This is the current yaw angle of the drone. Align the door frame with the yaw angle. The control command at time t. The distance from the drone to the nearest collision point. This is a constant offset. - These are the weighting coefficients.

7. The quadrotor random obstacle racing method based on curriculum reinforcement learning as described in claim 1, characterized in that, The lightweight network architecture includes: Visual encoder: A three-layer CNN is used to extract features from the 64×64 pixel depth image acquired by the perception unit and output a visual feature vector; State Processor: Employs a two-layer MLP to process the UAV state observation vector. Perform feature processing and output the state feature vector; Feature fusion and decision module: The visual feature vector and the state feature vector are concatenated and input into a GRU module with a hidden layer dimension of 256, and then mapped to the action space output by a policy network. The policy network adopts a two-layer MLP. Value Network: The value network has the same structure as the policy network, and also adopts a two-layer MLP. The input is the concatenation result of the visual feature vector and the state feature vector, and the output is the value estimate, which is used to assist in the training and optimization of the end-to-end visual control policy.

8. The quadrotor random obstacle racing method based on curriculum reinforcement learning as described in claim 7, characterized in that, The multi-scenario parallel update mechanism is as follows: multiple differentiated track simulation environments are run in parallel and grouped, each group conducts training synchronously, training data is collected in real time, and the parameters of the policy network and the value network are jointly updated to prevent the policy from overfitting to a single scenario and improve generalization ability.

9. The quadrotor random obstacle racing method based on curriculum reinforcement learning as described in claim 1, characterized in that, The end-to-end visual control strategy is trained using a near-end policy optimization algorithm, and the training objects are the policy network and the value network in the lightweight network architecture.

10. The quadrotor random obstacle racing method based on curriculum reinforcement learning as described in claim 9, characterized in that, The parameters of the near-end policy optimization algorithm are: learning rate decaying from 1e-4 to 1e-5, discount factor of 0.99, pruning range of 0.2, GAE-λ coefficient of 0.95, and batch size of 51200.

Images