A small-rotor unmanned aerial vehicle reinforcement learning autonomous obstacle avoidance method

Abstract

The application discloses a kind of small rotor unmanned vehicle reinforcement learning autonomous obstacle avoidance methods, comprising: obtaining the multimodal visual input information and self state information of unmanned vehicle, the multimodal visual input information includes depth image and semantic segmentation image;According to the semantic segmentation image, obtain road area coverage rate;According to the road area coverage rate, obtain priori knowledge experience reward;According to the priori knowledge experience reward and reinforcement learning reward, the autonomous obstacle avoidance strategy function of unmanned vehicle is trained;According to the autonomous obstacle avoidance strategy function, the control instruction of unmanned vehicle is output, and unmanned vehicle autonomous obstacle avoidance control is carried out.The application introduces priori knowledge experience reward, effectively improves the training efficiency and data utilization rate of reinforcement learning, enhances the decision robustness and obstacle avoidance success rate of unmanned vehicle in complex environment.

Description

Technical Field

[0001] This invention belongs to the field of autonomous navigation and intelligent control technology for unmanned aerial vehicles (UAVs), and particularly relates to a reinforcement learning-based autonomous obstacle avoidance method for small rotary-wing UAVs. Background Technology

[0002] When drones perform missions in complex and dynamic environments (such as densely populated urban areas, unstructured scenarios like jungles and mountains), they urgently need real-time, autonomous obstacle avoidance capabilities. Traditional path planning methods based on global maps (such as A / B) are insufficient. Due to limitations in computing resources and sensor performance, obstacle avoidance methods (such as RRT) struggle to achieve efficient and flexible real-time obstacle avoidance in unknown or dynamically changing environments. In recent years, obstacle avoidance methods based on reinforcement learning, which autonomously learn and optimize strategies through interaction with the environment, have gradually become a research hotspot. These methods typically use the UAV's visual input (such as depth images) and its own state information as the state space, and design reward functions to guide the agent to learn end-to-end control strategies from perception to action, aiming to achieve autonomous obstacle avoidance and navigation in unknown environments.

[0003] However, existing UAV obstacle avoidance methods based on reinforcement learning still have the following technical problems: (1) Low data efficiency. When the state space and action space dimensions of pure reinforcement learning methods are high, a large number of simulation interaction steps are required to learn an effective strategy. The training time is long and the computational resources are consumed. (2) High exploration difficulty. In complex environments with sparse rewards, the agent is difficult to obtain effective positive feedback within a limited number of training steps. The learning process is prone to getting stuck in local optima, resulting in a low obstacle avoidance success rate. (3) Lack of effective use of human prior knowledge (such as driving experience such as driving along the road and avoiding obstacles in advance). Relying only on the design of a general reward function, it is difficult to achieve smooth and efficient obstacle avoidance behavior in dense obstacle scenarios. The decision robustness and environmental adaptability are insufficient. Summary of the Invention

[0004] This invention aims to provide a UAV reinforcement learning autonomous obstacle avoidance method that integrates prior knowledge and experience. By introducing prior human knowledge to guide the intelligent agent's exploration and extracting visual road features based on semantic segmentation to construct state input, it solves the problems of low data efficiency, high exploration difficulty, and long training time of existing pure reinforcement learning methods in complex dynamic environments. This significantly improves the learning efficiency, decision robustness, and obstacle avoidance success rate of UAVs in unknown environments.

[0005] To achieve the above objectives, this invention provides a reinforcement learning-based autonomous obstacle avoidance method for small rotary-wing unmanned aerial vehicles, comprising: The system acquires multimodal visual input information and its own state information of the UAV, wherein the multimodal visual input information includes depth images and semantic segmentation images; Based on the semantic segmentation image, the road area coverage is obtained; Based on the road area coverage rate, a prior knowledge experience reward is obtained; Based on the prior knowledge experience reward and reinforcement learning reward, the autonomous obstacle avoidance strategy function of the UAV is trained. Based on the autonomous obstacle avoidance strategy function, the control commands for the UAV are output to perform autonomous obstacle avoidance control.

[0006] Preferably, the process of acquiring the semantic segmentation image includes: The depth image is input into a semantic segmentation network, which outputs pixel-level segmentation masks for road and non-road regions to generate the semantic segmentation image.

[0007] Preferably, the process of obtaining the road area coverage includes: The semantic segmentation image is divided into multiple regions of interest, and the coverage rate of road pixels in each region of interest is calculated.

[0008] Preferably, the process of obtaining prior knowledge experience rewards includes: A reward for an obstacle warning is given based on the coverage of the area ahead in multiple attention areas. A reward for lateral approach warning is given based on the coverage of the lateral area in multiple areas of interest; The prior knowledge experience reward is obtained based on the forward obstacle warning reward and the side approach warning reward.

[0009] Preferably, the process of obtaining the obstacle warning reward includes: When the coverage of the area in front is lower than a first preset threshold, a negative reward is given. The process of obtaining the lateral approach warning reward includes: When the coverage of the lateral region is lower than the second preset threshold, a negative reward is given.

[0010] Preferably, the process of training the autonomous obstacle avoidance strategy function includes: The prior knowledge and experience rewards are linearly superimposed with the reinforcement learning rewards to construct a total reward function; Based on the total reward function, the policy network is optimized to obtain the autonomous obstacle avoidance policy function.

[0011] Preferably, the reinforcement learning reward includes sparse rewards and dense rewards; The sparse rewards include rewards for successfully reaching the target point, penalties for collisions, and penalties for flying out of the working state. The dense reward includes a reward for the change in distance between the UAV and the target point, a penalty for smooth motion, and a penalty for yaw error.

[0012] Preferably, the method further includes: A state vector is constructed based on the self-state information and the multimodal visual input information; The self-state information includes the drone's position and attitude information.

[0013] Preferably, the control commands include horizontal speed commands, vertical speed commands, and yaw rate commands; The process of outputting control commands for the UAV based on the autonomous obstacle avoidance strategy function includes: The state vector is input into the Actor network of the autonomous obstacle avoidance strategy function, and the horizontal velocity command, vertical velocity command, and yaw rate command are output.

[0014] Preferably, the depth image in the multimodal visual input information is acquired by the forward-facing camera of the UAV; The semantic segmentation images were captured by the drone's downward-facing camera.

[0015] Compared with the prior art, the present invention has the following advantages and technical effects: This invention achieves the following technical effects by introducing a joint optimization mechanism that combines prior knowledge experience rewards with reinforcement learning rewards: (1) Significantly improves training efficiency and data utilization. By guiding the agent with prior knowledge experience based on road coverage, it provides a reasonable exploration direction for the agent in the early stage of training, effectively alleviating the problem of slow training caused by sparse rewards and blind exploration in pure reinforcement learning methods, and greatly shortening the number of training steps required to achieve an effective strategy; (2) Enhances decision robustness and environmental adaptability. By extracting road features through semantic segmentation and designing differentiated rewards based on regional coverage, the UAV can perceive the road structure in front and to the side, and achieve smooth and efficient obstacle avoidance behavior in dense obstacle scenarios, thereby improving the obstacle avoidance success rate and path smoothness; (3) Effectively utilizes human prior knowledge. It encodes driving experience such as driving along the road and avoiding obstacles in advance into computable experience reward signals, making up for the problem of insufficient decision-making ability of pure reinforcement learning in complex dynamic environments, and improving the autonomous navigation capability of UAVs in unknown environments. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of a reinforcement learning network architecture based on prior knowledge and experience enhancement using multimodal visual input, according to an embodiment of the present invention. Figure 2 This is a schematic diagram comparing the obstacle avoidance accuracy of the models in this embodiment of the invention; Figure 3 This is a schematic diagram of image segmentation in a drone downward-facing camera scene according to an embodiment of the present invention; Figure 4 This is a comparison chart of model training losses in embodiments of the present invention; Figure 5 This is a top view of the simulated route of the UAV in an embodiment of the present invention; Figure 6 This is a cumulative reward graph for drone flights according to an embodiment of the present invention; Figure 7 This is a distance map between the drone and the destination according to an embodiment of the present invention; Figure 8 This is a diagram showing the change in motion state of the drone according to an embodiment of the present invention. Detailed Implementation

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0019] This embodiment provides a reinforcement learning-based autonomous obstacle avoidance method for small rotary-wing unmanned aerial vehicles, including: Acquire multimodal visual input information and self-state information of the UAV. The multimodal visual input information includes depth images and semantic segmentation images. Based on semantically segmented images, obtain road area coverage; Based on road area coverage, prior knowledge and experience rewards are obtained. Based on prior knowledge and experience rewards and reinforcement learning rewards, the autonomous obstacle avoidance strategy function of the UAV is trained. Based on the autonomous obstacle avoidance strategy function, the control commands for the UAV are output to perform autonomous obstacle avoidance control.

[0020] Furthermore, the process of obtaining semantic segmentation images includes: The depth image is input into the semantic segmentation network, which outputs pixel-level segmentation masks for road and non-road regions to generate a semantic segmentation image.

[0021] Furthermore, the process of obtaining road area coverage includes: The semantic segmentation image is divided into multiple regions of interest, and the coverage of road pixels in each region of interest is calculated.

[0022] Furthermore, the process of acquiring prior knowledge and experience rewards includes: A reward for an obstacle warning is given based on the coverage of the area ahead in multiple attention areas. A reward for lateral approach warning is given based on the coverage of the lateral area in multiple areas of interest; Prior knowledge and experience rewards are obtained based on the rewards for warnings of obstacles ahead and warnings of approaching from the side.

[0023] Furthermore, the process of obtaining a reward for an obstacle warning includes: When the coverage of the area in front is lower than the first preset threshold, a negative reward is given. The process of obtaining a side approach warning reward includes: When the coverage of the lateral area is lower than the second preset threshold, a negative reward is given.

[0024] Furthermore, the process of training to obtain the autonomous obstacle avoidance policy function includes: The total reward function is constructed by linearly superimposing prior knowledge and experience rewards with reinforcement learning rewards. Based on the total reward function, the policy network is optimized to obtain the autonomous obstacle avoidance policy function.

[0025] Furthermore, reinforcement learning rewards include sparse rewards and dense rewards; Sparse rewards include rewards for successfully reaching the target point, penalties for collisions, and penalties for flying out of the working state. Dense rewards include rewards for changes in distance between the drone and the target point, penalties for smooth motion, and penalties for yaw error.

[0026] Furthermore, the method also includes: Construct a state vector based on its own state information and multimodal visual input information; The drone's own status information includes its position and attitude information.

[0027] Furthermore, the control commands include horizontal speed commands, vertical speed commands, and yaw rate commands; The process of outputting control commands for the UAV based on the autonomous obstacle avoidance strategy function includes: The state vector is input into the Actor network of the autonomous obstacle avoidance strategy function, which outputs horizontal velocity command, vertical velocity command, and yaw rate command.

[0028] Furthermore, the depth image in the multimodal visual input information is acquired through the forward-facing camera of the drone; Semantic segmentation images are acquired using a drone's downward-facing camera.

[0029] Specifically, in the above embodiments, the technical solution is further optimized. This embodiment's autonomous obstacle avoidance method uses a quadcopter drone as the research object, with the drone equipped with forward-looking and downward-looking cameras for environmental perception. The drone... The input at any given time consists of images and the drone's own state information. The images include depth images captured by the forward-facing camera and semantically segmented images captured by the downward-facing camera; the state information includes the drone's location. Attitude information The output is the horizontal speed of the drone. , Vertical velocity and yaw rate With the enhancement of prior knowledge and experience, a strategy function for autonomous obstacle avoidance of the UAV is obtained through reinforcement learning methods, ultimately enabling the UAV to achieve obstacle avoidance from the starting point. The goal is to achieve collision-free flight from a point within a circle with a radius of 300m in an unknown environment.

[0030] For the control unit input by the UAV, the kinematic model of the obstacle avoidance problem can be described as follows: in, The coordinates of the drone's position in three-dimensional space. Indicates the yaw angle of the drone. This represents the resultant velocity in the horizontal plane. Represents the velocity in the vertical direction. This indicates the yaw rate.

[0031] Therefore, the input space of the UAV control unit is modeled as a three-dimensional space: And the range of the action space is defined as: ; For modeling obstacle avoidance learning for UAVs, the state transition equation can be expressed as: ; Where the state vector The state space containing the drone's position and attitude information can be represented as: ; Finally, this embodiment incorporates prior knowledge and experience into reinforcement learning to achieve autonomous obstacle avoidance for the UAV. Semantic segmentation is used to incorporate human prior knowledge into network training. The full model structure is as follows: Figure 1 As shown, the input on the left is a depth image, and the input at the top is a semantic segmentation image.

[0032] Furthermore, reward function design is crucial for reinforcement learning. This embodiment employs a combination of sparse and dense rewards. The sparse reward is defined as having three parts: a reward for successfully reaching the target point. Collision penalty And penalties for flying out of work mode The round ends when the drone successfully reaches the target point, collides with the target, flies out of the workspace, or exceeds the maximum number of steps.

[0033] Furthermore, during normal flight, dense rewards are used to guide the drone towards the target point. This article discusses dense rewards... It is divided into three parts: in The reward is calculated based on the change in distance between the drone and the target point, using the following formula: in, The distance from the current point to the target point is in three dimensions. The distance from the target point at the previous moment. The target distance threshold, This is the distance reward coefficient. As the drone approaches the target point... If the value is positive, positive feedback is given to the drone; conversely, if the drone moves away from the target point, A negative value results in a penalty for the drone. To encourage smooth control, a definition is provided. To penalize actions and prevent excessive changes in the instantaneous angular velocity of the drone, thus ensuring its safe and stable flight: in This represents the penalty coefficient for the action.

[0034] at last, To penalize yaw error, guide the drone to fly towards the target direction: in This is the yaw error penalty coefficient. This represents the yaw angle error (degrees).

[0035] Finally, the general formula for the reward function is: The formula for the dense reward function after incorporating prior knowledge and experience is: Dense reward function enhanced by prior knowledge and experience It consists of four parts. This represents the distance reward between the drone and the destination, calculated as the difference between the previous distance and the current distance. A positive reward is given if the drone gets closer to the target; a negative penalty is given if the drone moves further away from the target. It indicates the cost of each movement of the drone, penalizes excessive or frequent yaw movements, encourages smooth flight, and reduces unnecessary turns. This indicates the yaw error of the drone relative to the target, penalizing misalignment and encouraging flight toward the target. This represents a reward based on prior knowledge and experience from semantic segmentation. When a drone is detected traveling on the road, a positive reward is given; when a large area of ​​obstacles is detected ahead, a negative penalty is given.

[0036] Furthermore, this embodiment also incorporates prior knowledge and experience rewards. This part is designed based on semantic segmentation results, aiming to encode human driving experience into a computable reward signal. It considers two aspects in a complex urban environment: driving along roads and avoiding obstacles. In simulation experiments, semantic segmentation images are acquired using a drone's downward-facing camera. Then, the coverage ratio of the road area is extracted based on the semantic segmentation results, and this is used as the basis for the reward.

[0037] When the road occupies more than 50% of the area on both sides of the drone's field of view directly below it, it is considered to be traveling on the road and the drone is safe from collision, and a positive reward is given.

[0038] When the road occupies less than 60% of the area directly below the drone's field of view, there is a high probability of an obstacle ahead. When the drone is not traveling on the road, a negative reward is given to help it avoid the obstacle.

[0039] Final Prior Knowledge Experience Reward The sum of the two: Therefore, the formula for the total reward function of reinforcement learning based on prior knowledge and experience is: The prior knowledge-enhanced design in this embodiment not only guides the drone along the road but also encourages it to maintain a clear view ahead, reflecting the behavioral habits of human drivers in complex environments—driving along roads and avoiding obstacles in advance. This method integrates human driving experience with reinforcement learning, compensating for the shortcomings of traditional reinforcement learning in complex environments.

[0040] This embodiment achieves the following significant technical effects by integrating prior knowledge and experience into the UAV reinforcement learning obstacle avoidance framework and combining it with semantic segmentation technology to extract structured environmental features: (1) Significantly improves training efficiency and data utilization: By introducing a prior knowledge and experience guidance mechanism based on road coverage, a reasonable exploration direction is provided to the agent in the early stage of training, effectively alleviating the slow training problem caused by reward sparsity and blind exploration in complex environments caused by pure reinforcement learning methods. Experiments show that pure reinforcement learning... Even after initial training, some students still struggle to learn strategies effectively and require extended training periods; whereas reinforcement learning methods that enhance prior knowledge and experience shorten the learning time to... Learn successful strategies in one step, significantly reduce training time, and significantly improve learning efficiency.

[0041] Experiments show that the method in this embodiment significantly improves the efficiency of obstacle avoidance for UAVs in unknown environments and shortens training time. The training accuracy of models trained using pure reinforcement learning and those enhanced with prior knowledge is compared to... Figure 2 As shown in the figure, the success rate of obstacle avoidance by drones with enhanced prior knowledge and experience through reinforcement learning is steadily increasing, and in a short period of time... A good model with an accuracy of over 65% can be obtained within 21379 steps, while simple reinforcement learning... A suitable model still cannot be learned within (45006s).

[0042] (2) Enhance decision robustness and environmental adaptability: Combine road features extracted by semantic segmentation to construct state input, enabling UAVs to understand scene structure more precisely and improve perception and understanding capabilities in unknown dynamic environments. The introduction of prior knowledge and experience further enhances the agent's rapid learning of common obstacle avoidance patterns (such as driving along roads and side obstacle avoidance), thereby exhibiting higher decision stability and path smoothness in dense obstacle scenarios.

[0043] (3) Reduced dependence on global maps and high-precision sensors: This method only relies on local depth images obtained by UAV onboard vision sensors and extracts key road features through semantic segmentation. It does not require the construction of a global map or reliance on a high-cost positioning system, making it suitable for lightweight UAV platforms with limited computing resources. This improves the practicality and deployment flexibility of the method in real-world complex scenarios.

[0044] (4) Achieve end-to-end autonomous obstacle avoidance and navigation: The entire system integrates perception, decision-making and control, and achieves optimized output of continuous action space based on the SAC framework. There is no need to manually design complex motion primitives or segmented controllers, so that the UAV can achieve end-to-end autonomous obstacle avoidance and target navigation from visual input to control commands in simulation and real scenarios.

[0045] As an alternative implementation method, this embodiment takes the autonomous obstacle avoidance of a drone in a simulated urban scene as an example. The specific steps are as follows: 1. System hardware and simulation environment configuration; (1) Simulation environment: A three-dimensional city scene is built based on Unreal Engine and AirSim, which includes dense buildings, trees, roads and other obstacles.

[0046] (2) Computing platform: GPU servers are used during the training phase, and can be ported to onboard computing units during the deployment phase.

[0047] (3) Unmanned aerial vehicle platform: equipped with a downward-looking depth camera (for semantic segmentation) and a forward-looking camera (for acquiring depth images).

[0048] 2. Implementation of the semantic segmentation module A pre-trained UNet++ model is used as the semantic segmentation network. The input is a single-frame depth image, and the output is a pixel-level road / non-road binary classification mask. The output mask is divided into three regions of interest: the top 1 / 4 region of the image (front view), the left 1 / 4 region, and the right 1 / 4 region. The coverage of road pixels in each region is calculated. If the top area coverage is less than 60%, a "proximate obstacle warning" signal is generated; if the coverage of any side area is less than 50%, a "side approach warning" signal is generated. The image segmentation principle is as follows: Figure 3 As shown. Among them, Figure 3 (a) in the image represents the top 1 / 4 of the field of vision. Figure 3 (b) in the diagram represents the 1 / 4 area on either side of the field of view. Road coverage within the top and side fields of view is identified to facilitate obstacle avoidance in advance.

[0049] 3. Implementation of a priori knowledge and experience-guided mechanism; The total reward function is the sum of the task reward, collision penalty, and experience reward: The training loss of the Actor and Critic in model training is as follows: Figure 4 As shown in the figure, the training loss of both the Actor and Critic parts of reinforcement learning with prior knowledge and experience enhancement gradually decreases, which is in line with training expectations; while simple reinforcement learning fails to converge for a long time and cannot obtain a good model in a short period of time.

[0050] Here we only select the coordinates in the scene. The finish line is then displayed. Drones start from the beginning. The flight path simulation results for the drone are as follows: Figure 5As shown, during the drone's flight, it needs to avoid urban obstacles (such as tall buildings, trees, etc.) from the starting point in order to reach the target point. The condition for success is within the specified range. As shown in the diagram, the drone is gradually approaching the target point, and its path is relatively smooth, allowing it to avoid obstacles and achieve safe flight.

[0051] Cumulative rewards for drone flights, such as Figure 6 As shown, during flight, the drone's cumulative reward gradually increases, indicating that the drone has learned a core strategy that simultaneously satisfies safety and efficiency: continuously moving towards the target point while avoiding obstacles as much as possible. This is crucial for the success of the obstacle avoidance navigation mission. This demonstrates that the reward function is correctly oriented, and the drone understands the strong correlation between approaching the target and obtaining high rewards. The reward function provides a stable and continuous positive incentive to move towards the target.

[0052] The distance between the drone and the finish line is as follows Figure 7 As shown in the figure, during the training / testing process, the distance between the UAV and the target point decreases significantly with the number of steps, showing a clear convergence trend, which intuitively proves the effectiveness of the navigation strategy.

[0053] like Figure 8 The figures show the changes in the drone's motion state over 400 steps and 20 steps, respectively. Figure 8 (a) shows the drone's motion status and trends over a long period of time; from Figure 8 Figure (b) shows the motion state of the UAV in the short period after it begins flight, including its planar velocity. Vertical velocity and yaw rate As can be seen from the image, the drone's horizontal speed remains relatively stable. When an obstacle appears in front or to the side, the drone's speed decreases rapidly, but it adjusts its flight direction and continues to fly stably. Its vertical speed remains relatively stable at... Because the starting and ending points are at the same height, the drone does not move much in the vertical plane. The yaw rate fluctuates greatly at the beginning because the drone turns quickly toward the target point from the starting point. Afterward, it maintains a small angle with the target point. When it encounters an obstacle, it will deviate slightly.

[0054] 4. Deployment and Real-time Inference: During online execution, the following process is performed for each frame: (1) Acquire depth image → Semantic segmentation → Calculate region coverage; (2) Construct state vector → input Actor network → output action command; (3) Send speed and steering commands to the flight control system to achieve real-time obstacle avoidance.

[0055] Therefore, this embodiment integrates human prior knowledge and semantic segmentation into a UAV reinforcement learning obstacle avoidance system architecture. Specifically, it uses prior knowledge and experience in the form of structured rules (such as road coverage threshold) and visual features extracted by semantic segmentation as the input to the reinforcement learning state in a composite perception-decision framework. This includes a complete technical chain: visual input → semantic segmentation → road coverage calculation → state construction → policy output.

[0056] This embodiment designs an experience-guided mechanism based on semantic segmentation image regional coverage, which divides the UAV's downward view image into different regions of interest (such as the forward 1 / 4 region and the 1 / 4 regions on each side), and sets a differentiated reward / penalty function based on the road pixel coverage of each region. Specifically, it includes: triggering a collision risk penalty when the coverage of the forward region is less than 60%, triggering a deviation penalty when the coverage of the side region is less than 50%, and a feedback mechanism in which the coverage threshold can be adaptively adjusted.

[0057] This embodiment designs a training method that integrates prior knowledge and reinforcement learning. Specifically, it embeds a joint optimization method of prior knowledge and experience reward terms into the SAC (Soft Actor-Critic) algorithm, including: integrating prior knowledge and experience rewards... A design that linearly superimposes reinforcement learning rewards; a real-time reward calculation method based on coverage.

[0058] The solution in this embodiment is applicable to autonomous obstacle avoidance strategies for UAVs in dense, dynamic urban scenarios. In densely built-up environments, the UAV, based on the aforementioned system, implements a comprehensive control strategy for smooth, efficient, and high-success-rate obstacle avoidance within a continuous operational space, including the joint optimized output of speed, altitude, and steering.

[0059] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A reinforcement learning-based autonomous obstacle avoidance method for a small rotary-wing unmanned aerial vehicle (UAV), characterized in that, include: The system acquires multimodal visual input information and its own state information of the UAV, wherein the multimodal visual input information includes depth images and semantic segmentation images; Based on the semantic segmentation image, the road area coverage is obtained; Based on the road area coverage rate, a prior knowledge experience reward is obtained; Based on the prior knowledge experience reward and reinforcement learning reward, the autonomous obstacle avoidance strategy function of the UAV is trained. Based on the autonomous obstacle avoidance strategy function, the control commands for the UAV are output to perform autonomous obstacle avoidance control.

2. The method according to claim 1, characterized in that, The process of obtaining the semantic segmentation image includes: The depth image is input into a semantic segmentation network, which outputs pixel-level segmentation masks for road and non-road regions to generate the semantic segmentation image.

3. The method according to claim 1, characterized in that, The process of obtaining the road area coverage includes: The semantic segmentation image is divided into multiple regions of interest, and the coverage rate of road pixels in each region of interest is calculated.

4. The method according to claim 1, characterized in that, The process of gaining prior knowledge and experience rewards includes: A reward for an obstacle warning is given based on the coverage of the area ahead in multiple attention areas. A reward for lateral approach warning is given based on the coverage of the lateral area in multiple areas of interest; The prior knowledge experience reward is obtained based on the forward obstacle warning reward and the side approach warning reward.

5. The method according to claim 4, characterized in that, The process of obtaining the aforementioned obstacle warning reward includes: When the coverage of the area in front is lower than a first preset threshold, a negative reward is given. The process of obtaining the lateral approach warning reward includes: When the coverage of the lateral region is lower than the second preset threshold, a negative reward is given.

6. The method according to claim 1, characterized in that, The process of training to obtain the autonomous obstacle avoidance policy function includes: The prior knowledge and experience rewards are linearly superimposed with the reinforcement learning rewards to construct a total reward function; Based on the total reward function, the policy network is optimized to obtain the autonomous obstacle avoidance policy function.

7. The method according to claim 1, characterized in that, The reinforcement learning rewards include sparse rewards and dense rewards; The sparse rewards include rewards for successfully reaching the target point, penalties for collisions, and penalties for flying out of the working state. The dense reward includes a reward for the change in distance between the UAV and the target point, a penalty for smooth motion, and a penalty for yaw error.

8. The method according to claim 1, characterized in that, The method further includes: A state vector is constructed based on the self-state information and the multimodal visual input information; The self-state information includes the drone's position and attitude information.

9. The method according to claim 1, characterized in that, The control commands include horizontal speed commands, vertical speed commands, and yaw rate commands; The process of outputting control commands for the UAV based on the autonomous obstacle avoidance strategy function includes: The state vector is input into the Actor network of the autonomous obstacle avoidance strategy function, and the horizontal velocity command, vertical velocity command, and yaw rate command are output.

10. The method according to claim 1, characterized in that, The depth image in the multimodal visual input information is acquired by the forward-facing camera of the drone; The semantic segmentation images were captured by the drone's downward-facing camera.

Images