Method and system for controlling unmanned aerial vehicle to pass through complex obstacle based on visual perception

Abstract

The invention discloses a control method and system for an unmanned aerial vehicle to pass through a complex obstacle based on visual perception. The method comprises the following steps: synchronously acquiring an RGB image and a depth image of an obstacle in front of the unmanned aerial vehicle; inputting the acquired RGB image into a pre-trained deep neural network model, and performing semantic segmentation on a cavity area or a mesh structure gap of an obstacle in the image to generate a corresponding binary mask image; removing the binaryzation masks which are discontinuous with the barrier masks in depth distribution, performing expansion operation on the residual binaryzation masks, and filling the background masks with flood; mask reverse selection is carried out, hole and gap masks of a non-obstacle area are extracted, and a maximum rectangular area larger than the forward section of the unmanned aerial vehicle is obtained based on the boundaries of the mask areas to serve as a passable area; and based on the position of the passable area, controlling the unmanned aerial vehicle to cross the obstacle. According to the invention, accurate identification, path planning and attitude control of the cavity area capable of being traversed in the obstacle can be realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for unmanned aerial vehicles (UAVs), and in particular to a control method and system for UAVs to navigate through complex obstacles based on visual perception. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Drones have been widely used in various fields such as aerial photography, agriculture, logistics, and rescue. However, drones may encounter various complex obstacles during flight. Existing autonomous flight control methods for drones mainly focus on obstacle avoidance, relying on lidar or ultrasonic ranging to detect obstacles ahead. They treat all detected objects ahead as impassable areas, resulting in the inability to identify effective passage paths through grid-like or hollow obstacles.

[0004] Existing visual algorithm-based detection technologies can detect and track various types of targets in real time by acquiring RGB images of obstacles and using image segmentation and other techniques. However, for grid-like or hollow obstacles, objects behind the grid or holes are easily misjudged as obstacle information, thus failing to guide UAVs in detailed path planning and flight control. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a control method and system for UAVs traversing complex obstacles based on visual perception. By introducing visual image analysis and target area localization methods, combined with UAV attitude and position estimation, it achieves accurate identification, path planning, and attitude control of traversable cavity areas within obstacles.

[0006] In some implementations, the following technical solutions are adopted: A vision-based control method for unmanned aerial vehicles (UAVs) navigating complex obstacles includes: Simultaneously acquire RGB and depth images of obstacles in front of the drone; The acquired RGB images are input into a pre-trained deep neural network model to perform semantic segmentation on the hole regions or gaps in the mesh structure of obstacles in the image, and generate corresponding binary mask images. Remove binarized masks that are discontinuous with the obstacle mask in depth distribution, perform dilation operation on the remaining binarized masks, and fill in the background mask with flood filling. Perform mask inversion to extract the holes and gaps in the non-obstacle areas, and based on the boundaries of these mask areas, obtain a maximum rectangular area larger than the forward cross section of the UAV as the passable area; Control the drone to navigate through obstacles based on the location of the passable area.

[0007] As a further solution, the deep neural network model is the YOLO-Seg semantic segmentation network, which includes a backbone network, a neck network, and a head network. The backbone network is used to extract the edge and texture features of the hollow regions or gaps in the mesh structure, the neck network is used to fuse multi-scale feature information, and the head network uses the prototype mask and weight combination to generate the segmentation result, thereby distinguishing the hollow regions or gaps in the mesh structure, and finally outputting the binary mask of the hollow regions or gaps in the mesh structure of the obstacle.

[0008] As a further solution, the binarized mask that is discontinuous with the obstacle mask in terms of depth distribution is removed, specifically: Sort the pixel depth values ​​corresponding to all binarized masks from smallest to largest. When the difference between two adjacent pixel depth values ​​is greater than a set threshold, truncate the binarized mask sequence at this position and remove the part with the larger depth value after truncation.

[0009] As a further solution, the background mask is filled in by flooding, specifically as follows: Let the binarized mask image be a two-dimensional region. Starting seed points are given at the top, bottom, left, and right edges of the image, respectively. The set of mask pixels that are connected to p0 and have the same value is marked by connectivity rules. : ; Where I(p) represents the pixel value of p, γ is the path from p0 to p, and q refers to the elements of the region to be searched. Represents the pixel value at position q; The pixel value representing the starting seed position; The set minimum threshold; All points on the path satisfy the similarity condition. The region is filled by iteratively growing and expanding the region: ; in, Indicates the current filling area. Indicates the filling region for the next time step. The region represents the neighboring pixels of the current region boundary. When Rt+1=Rt, the region will no longer expand.

[0010] As a further solution, a maximum rectangular region larger than the forward cross-section of the UAV is obtained based on the boundaries of these areas as the passable area, specifically: Based on the holes and gaps in the non-obstacle areas, the approximate polygonal boundaries of each mask area are obtained; A binary mask image is constructed based on the approximate polygon boundary, and a height histogram H=[h1,h2,…,h] is constructed by scanning column by column. n ]; For each pillar h i Find the first position l on the left that is smaller than it. i and the position r of the first smaller than it on the right. i The area of ​​the rectangle that each pillar can form is: ; Find the N pillars with the largest rectangular area and select them as candidate rectangles; Compare the candidate rectangles with the width and height of the drone. If there are candidate rectangles that meet the passage conditions, select the candidate rectangle with the largest area as the passable area. At the same time, extract the pixel coordinates of the top left corner, bottom right corner, top right corner and center point of the candidate rectangle, and combine them with the depth image and camera intrinsic parameters to obtain its three-dimensional spatial coordinates in the camera coordinate system.

[0011] As a further solution, based on the location of the passable area, the drone is controlled to traverse obstacles, specifically: Flight alignment is performed based on the center point of the obstacle to ensure global visibility of the passable area within the drone's field of vision; Control the drone to smoothly approach a predetermined distance in front of the center point of the passable area; generate a planned path from the current position to the center point, and control the drone to fly along the planned path; during flight, the drone will correct the flight path based on real-time visual detection information, and determine whether the planned path has been completed by judging the distance between the current position and the center point; When the distance to the center point is set, path planning stops and a blind flight strategy is adopted. After crossing the center point, the drone will briefly stay near the center point to confirm that its attitude and position are stable.

[0012] In other embodiments, the following technical solutions are adopted: A vision-based control system for unmanned aerial vehicles (UAVs) navigating complex obstacles includes: The data acquisition module is configured to simultaneously acquire RGB and depth images of obstacles in front of the drone; The image segmentation module is configured to input the acquired RGB image into a pre-trained deep neural network model, perform semantic segmentation on the hole regions or mesh structure gaps of obstacles in the image, and generate the corresponding binarized mask image. The mask processing module is configured to remove binarized masks that are discontinuous with the obstacle mask in depth distribution, perform dilation operation on the remaining binarized mask, and fill in the background mask by flood filling. The passable area filtering module is configured to perform mask inversion, extract holes and gaps in non-obstacle areas, and obtain a maximum rectangular area larger than the forward cross section of the UAV as the passable area based on the boundaries of these mask areas; The obstacle crossing control module is configured to control the drone to cross obstacles based on the location of the passable area.

[0013] As a further solution, the mask processing module removes binarized masks that are discontinuous with the obstacle mask in terms of depth distribution, specifically: Sort the pixel depth values ​​corresponding to all binarized masks from smallest to largest. When the difference between two adjacent pixel depth values ​​is greater than a set threshold, truncate the binarized mask sequence at this position and remove the part with the larger depth value after truncation.

[0014] In other embodiments, the following technical solutions are adopted: A terminal device includes a processor and a memory, the processor being used to implement instructions; the memory being used to store multiple instructions adapted to be loaded and executed by the processor to control the above-described vision-based drone traversing complex obstacles.

[0015] In other embodiments, the following technical solutions are adopted: A computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the above-described vision-based control method for unmanned aerial vehicles navigating complex obstacles.

[0016] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention acquires RGB images and depth images simultaneously. By introducing a deep neural network model to perform semantic recognition on RGB images, it can accurately extract traversable areas such as holes or gaps. Compared with traditional geometric rule-based or edge detection methods, it has high recognition accuracy and is not easily affected by other factors. This invention combines the depth information corresponding to the binary mask obtained from segmentation, removes mask values ​​with large depth spans, and avoids identifying objects behind holes or gaps as current obstacle information, thus achieving accurate identification of hole-type or mesh-type obstacle information.

[0017] (2) The present invention combines depth map and camera parameters to accurately map the target area in the mask map to the camera coordinate system and further convert it into global coordinates to achieve precise spatial positioning of the target point, providing a reliable basis for subsequent flight control.

[0018] (3) The present invention adopts the “point-and-fly” control strategy, which avoids the high latency and high computational overhead of complex path planning algorithms. It can quickly respond to obstacle positions and achieve stable crossing. It is suitable for small flight platforms with limited resources. It does not require pre-built maps or reliance on structured scenes. It can work directly in unknown or semi-structured environments. It has good versatility and deployment flexibility. It is suitable for various mission scenarios such as rescue, security, and indoor navigation.

[0019] Other features and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] Figure 1 This is a flowchart of a vision-based control method for unmanned aerial vehicles (UAVs) to navigate through complex obstacles, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of an obstacle containing a cavity in an embodiment of the present invention; Figure 3 This is a schematic diagram of the mesh obstacle in an embodiment of the present invention; Figure 4 This is a schematic diagram of the mesh obstacle mask extracted in an embodiment of the present invention; Figure 5 This is a schematic diagram of the passable area extracted from the cavity obstacle in an embodiment of the present invention; Figure 6 This is a schematic diagram of the passable area extracted from the mesh obstacle in an embodiment of the present invention. Detailed Implementation

[0021] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0022] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0023] Example 1 In one or more embodiments, a vision-based method for controlling a drone to navigate through complex obstacles is disclosed, combined with... Figure 1 Specifically, it includes the following processes: S101: Simultaneously acquire RGB and depth images of obstacles in front of the drone.

[0024] This embodiment uses a compact and stable quadcopter drone as the flight platform, with an RGB-D depth camera (such as Intel Realsense D435i) mounted in front of it to simultaneously acquire RGB and depth images. The image data is processed in real time through an onboard embedded computing platform (such as NVIDIA Jetson series).

[0025] Depth images record the distance information from each pixel in the scene in front of the drone to the camera. They are not affected by color and texture. This embodiment combines RGB images and depth images to achieve three-dimensional stereoscopic perception of obstacles in front of the drone.

[0026] This embodiment is mainly aimed at obstacles containing holes or mesh-like obstacles. Figure 2 An example of an obstacle containing holes is given. Figure 3 An example of a mesh obstacle is given.

[0027] S102: Input the acquired RGB image into a pre-trained deep neural network model to perform semantic segmentation on the hole regions or gaps in the mesh structure of obstacles in the image, and generate the corresponding binarized mask image.

[0028] In this embodiment, the acquired RGB image is input into a pre-trained lightweight semantic segmentation network (this embodiment uses the improved YOLO-Seg model), and the semantic segmentation network identifies the hollow areas or gaps in the mesh structure of the obstacle. Figure 4 An example of the extracted binarized mask image of the mesh obstacle is given.

[0029] Specifically, the improved YOLO-Seg model includes a backbone network, a neck network, a head network, and a post-processing part connected in sequence. The head network includes a detection branch and a segmentation branch, and the post-processing part uses Soft-NMS. The backbone network is used to extract the edge and texture features of void regions or gaps in the mesh structure. The shallow network captures the fine lines and void boundaries in the void regions or mesh structure, the middle network learns its periodicity and dense distribution features, and the deep network forms the semantic understanding of the "wall or mesh region". The neck network is used to fuse the extracted multi-scale feature information; the detection branch of the head network predicts bounding boxes and mask coefficients; the segmentation branch generates prototype masks; the prototype masks are combined with weights to generate segmentation results, enabling the differentiation of hole regions or mesh gap regions; the post-processing part uses Soft-NMS, which reduces the confidence score of highly overlapping detection boxes (actually adjacent different holes or meshes) instead of deleting them directly. This ensures that all holes or meshes are accurately identified, and avoids the accidental deletion of targets in dense mesh or gap scenes, thus maintaining the continuity and integrity of the mesh region and hole segmentation results.

[0030] The final output is a binary mask of the hole region or mesh structure of the obstacle.

[0031] S103: Remove the binarized mask that is discontinuous with the obstacle mask in the depth distribution, perform an expansion operation on the remaining binarized mask, and fill in the background mask by flood filling.

[0032] In this embodiment, for abnormal mask pixels caused by model misidentification (such as misidentified pixels located behind holes or grids), since their depth distribution is not continuous with the obstacle mask, a sorting and truncation method is used to remove all mask values ​​with large depth value spans.

[0033] Specifically, a threshold Δ is set, and the pixel depth values ​​corresponding to all binary masks are sorted from smallest to largest. When the difference between two adjacent pixel depth values ​​is greater than the threshold Δ, the sequence is truncated at this position, and the portion of the binary mask sequence with the larger depth value is removed. For example, if the depth values ​​are sorted as {1.2, 1.3, 1.5, 1.7, 2.0, 5.0, 5.2, 5.5}, ​​since the difference between 2.0 and 5.0 is significantly greater than the threshold, truncation is performed at this point, and only {1.2, 1.3, 1.5, 1.7, 2.0} is retained.

[0034] Cut-off position Specifically: ; in, Let be the depth value of the i-th binary mask in the sequence. Let be the depth value of the (i+1)th binarized mask in the sequence. This represents the number of binarization masks.

[0035] In this embodiment, the center position of the obstacle is saved based on the finally extracted obstacle mask information, and the obstacle mask is expanded to reserve a safe space for the drone to fly through.

[0036] Additionally, by performing a flood fill operation to complete the background mask, the drone is ensured to ultimately pass through the holes or meshes, rather than bypassing them by passing through from the side. The specific process is as follows: Let the binarized mask image be a two-dimensional region. Starting seed points are given at the top, bottom, left, and right edges of the image, respectively. This means filling the image simultaneously from all four directions.

[0037] The starting seed point is marked using connectivity rules (such as 4-connectivity). A set of connected mask pixels with equal mask values : ; Where I(p) represents the pixel value of p, γ is the path from p0 to p, and q refers to the elements of the region to be searched. Represents the pixel value at position q; The pixel value representing the starting seed position; The set minimum threshold (close to 0); All points on the path satisfy the similarity condition. The region is filled by iteratively growing and expanding the region: ; in, Indicates the current filling area. Indicates the filling region for the next time step. The region is the neighboring pixels of the current region boundary. When Rt+1=Rt, the region no longer expands, meaning that the region to be filled has been completely filled.

[0038] To ensure flight safety, the area connecting to the ground will not be filled in.

[0039] S104: Perform mask inversion to extract the holes and gaps in the non-obstacle areas. Based on the boundaries of these mask areas, obtain a maximum rectangular area larger than the forward cross-section of the UAV as the passable area.

[0040] In this embodiment, the holes and gaps in the non-obstacle areas are extracted by mask inversion. These mask areas are traversed to obtain their approximate polygonal boundaries. Since the UAV flies horizontally when crossing, a rectangular area larger than the forward cross section of the UAV needs to be calculated in this boundary area as the passable area. The specific process is as follows: First, a binary mask image is constructed based on the input polygonal contour. Then, a height histogram H=[h1,h2,…,h] is constructed by scanning column by column. nThen, in each row, a histogram algorithm is used to search for the largest rectangle that meets the size constraints: For each column h i Find the first position l to its left that is smaller than it. i and the first position r to its right that is smaller than it i The area of ​​the rectangle that each pillar can form is: ; Find the N pillars with the largest rectangular area among all pillars, and select them as candidate rectangles; Candidate rectangles are sorted from largest to smallest area. Each candidate rectangle is then compared with the width and height of the drone. If a candidate rectangle meets the passage conditions (both its width and length are larger than the drone's physical width and height), the candidate rectangle with the largest area is selected as the passable area. For example, if the first two candidate rectangles meet the passage conditions, the candidate rectangle with the largest area is selected as the passable area. At the same time, the pixel coordinates of the top left, bottom right, top right, and center points of the candidate rectangle are extracted and combined with the depth image and camera intrinsic parameters to obtain its three-dimensional spatial coordinates in the camera coordinate system. Figure 5 An example of a passable area extracted from a hollow obstacle is given; Figure 6 An example of a passable area extracted from a mesh obstacle is given.

[0041] If no candidate rectangle that meets the passage conditions is found, the traversal process stops.

[0042] S105: Control the drone to navigate through obstacles based on the location of the passable area.

[0043] In this embodiment, the drone's crossing process is divided into the following two stages: The first stage involves flight alignment based on the obstacle's center point, ensuring global visibility of obstacles and passable areas within the drone's field of vision. After acquiring the target point (the center point of the passable area in the camera coordinate system), the flight control system transforms the target point's position from the camera coordinate system to the drone's body coordinate system. Alignment control is then performed using a pointing pointer, allowing the drone to smoothly approach the crossing point (the target position in the drone's body coordinate system) to a position 1.5 to 1.7 meters in front. Errors are detected and the position is adjusted in real time during alignment to ensure precise drone flight alignment.

[0044] The second phase, after alignment, involves the system entering the crossing phase. Path planning generates a path from the current position to the target point, and the drone flies along this path. The drone employs a relative target strategy, selecting a point 0.7 meters directly in front of the current position as the relative target on the planned path to avoid the influence of accumulated errors from the visual odometry. During flight, the drone locates the target point and corrects its flight path based on real-time visual detection information. During crossing, the flight controller adjusts control parameters based on the shortest distance between the current position and the planned path, using distance judgment to monitor whether path planning is complete. When the distance to the target point is 0.5 meters, path planning stops, and a blind flight strategy is adopted to prevent the visual detection module from discovering new passable areas through holes, thus affecting the execution of the current crossing mission. After crossing, the drone briefly pauses near the target point to confirm attitude and position stability before landing or proceeding to the next phase of the mission.

[0045] Example 2 In one or more embodiments, a vision-based unmanned aerial vehicle (UAV) control system for navigating complex obstacles is disclosed, comprising: The data acquisition module is configured to simultaneously acquire RGB and depth images of obstacles in front of the drone; The image segmentation module is configured to input the acquired RGB image into a pre-trained deep neural network model, perform semantic segmentation on the hole regions or mesh structure gaps of obstacles in the image, and generate the corresponding binarized mask image. The mask processing module is configured to remove binarized masks that are discontinuous with the obstacle mask in depth distribution, perform dilation operation on the remaining binarized mask, and fill in the background mask by flood filling. The passable area filtering module is configured to perform mask inversion, extract holes and gaps in non-obstacle areas, and obtain a maximum rectangular area larger than the forward cross section of the UAV as the passable area based on the boundaries of these mask areas; The obstacle crossing control module is configured to control the drone to cross obstacles based on the location of the passable area.

[0046] Specifically, the mask processing module removes binarized masks that are discontinuous with the obstacle mask in terms of depth distribution, as follows: Sort the pixel depth values ​​corresponding to all binarized masks from smallest to largest. When the difference between two adjacent pixel depth values ​​is greater than a set threshold, truncate the binarized mask sequence at this position and remove the part with the larger depth value after truncation.

[0047] The specific implementation methods of the above modules are exactly the same as those in Example 1, and will not be described in detail again.

[0048] Example 3 In one or more embodiments, a terminal device is disclosed, comprising a processor and a memory, wherein the processor is used to implement instructions; and the memory is used to store multiple instructions adapted to be loaded by the processor and executed by the processor to control the UAV through complex obstacles based on vision perception as described in Embodiment 1.

[0049] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0050] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0051] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0052] Example 4 In one or more embodiments, a computer-readable storage medium is disclosed, wherein a plurality of instructions are stored, the instructions being adapted to be loaded by a processor of a terminal device and executed by the visual perception-based control method for unmanned aerial vehicles traversing complex obstacles as described in Embodiment 1.

[0053] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A control method for a visually-based unmanned aerial vehicle (UAV) to navigate complex obstacles, characterized in that, include: Simultaneously acquire RGB and depth images of obstacles in front of the drone; The acquired RGB images are input into a pre-trained deep neural network model to perform semantic segmentation on the hole regions or gaps in the mesh structure of obstacles in the image, and generate corresponding binary mask images. Remove binarized masks that are discontinuous with the obstacle mask in depth distribution, perform dilation operation on the remaining binarized masks, and fill in the background mask with flood filling. Perform mask inversion to extract the holes and gaps in the non-obstacle areas, and based on the boundaries of these mask areas, obtain a maximum rectangular area larger than the forward cross section of the UAV as the passable area; Control the drone to navigate through obstacles based on the location of the passable area.

2. The control method for a visually-based UAV traversing complex obstacles as described in claim 1, characterized in that, The deep neural network model is the YOLO-Seg semantic segmentation network, which includes a backbone network, a neck network, and a head network. The backbone network is used to extract the edge and texture features of the hollow regions or gaps in the mesh structure. The neck network is used to fuse multi-scale feature information. The head network uses prototype masks and weights to generate segmentation results, thereby distinguishing the hollow regions or gaps in the mesh structure. Finally, it outputs a binary mask of the hollow regions or gaps in the mesh structure of the obstacle.

3. The control method for a visually-based UAV traversing complex obstacles as described in claim 1, characterized in that, Remove binarized masks that are discontinuous with the obstacle mask in terms of depth distribution, specifically as follows: Sort the pixel depth values ​​corresponding to all binarized masks from smallest to largest. When the difference between two adjacent pixel depth values ​​is greater than a set threshold, truncate the binarized mask sequence at this position and remove the part with the larger depth value after truncation.

4. The control method for a visually-based UAV traversing complex obstacles as described in claim 1, characterized in that, The background mask is filled in by flooding, specifically as follows: Let the binarized mask image be a two-dimensional region. Starting seed points are given at the top, bottom, left, and right edges of the image, respectively. The set of mask pixels that are connected to p0 and have the same value is marked by connectivity rules. : ； Where I(p) represents the pixel value of p, γ is the path from p0 to p, and q refers to the element in the region to be searched. Represents the pixel value at position q; The pixel value representing the starting seed position; The set minimum threshold; All points on the path satisfy the similarity condition. The region is filled by iteratively growing and expanding the region: ； in, Indicates the current filling area. Indicates the filling region for the next time step. The region represents the neighboring pixels of the current region boundary. When Rt+1=Rt, the region will no longer expand.

5. The control method for a visually-based UAV traversing complex obstacles as described in claim 1, characterized in that, Based on the boundaries of these regions, a rectangular region larger than the forward cross-section of the UAV is obtained as the passable area, specifically: Based on the holes and gaps in the non-obstacle areas, the approximate polygonal boundaries of each mask area are obtained; A binary mask image is constructed based on the approximate polygon boundary, and a height histogram H=[h1,h2,…,h] is constructed by scanning column by column. n ]; For each pillar h i Find the first position l on the left that is smaller than it. i and the position r of the first smaller than it on the right. i The area of ​​the rectangle that each pillar can form is: ; Find the N pillars with the largest rectangular area and select them as candidate rectangles; Compare the candidate rectangles with the width and height of the drone. If there are candidate rectangles that meet the passage conditions, select the candidate rectangle with the largest area as the passable area. At the same time, extract the pixel coordinates of the top left corner, bottom right corner, top right corner and center point of the candidate rectangle, and combine them with the depth image and camera intrinsic parameters to obtain its three-dimensional spatial coordinates in the camera coordinate system.

6. The control method for a visually-based UAV traversing complex obstacles as described in claim 1, characterized in that, Based on the location of the passable area, the drone is controlled to navigate through obstacles, specifically: Flight alignment is performed based on the center point of the obstacle to ensure global visibility of the passable area within the drone's field of vision; Control the drone to smoothly approach the center point of the passable area at a predetermined distance in front of it; Generate a planned path from the current location to the center point, and control the drone to fly along the planned path; during flight, the drone will correct the flight path based on real-time visual detection information, and determine whether the planned path has been completed by judging the distance between the current location and the center point; When the distance to the center point is set, path planning stops and a blind flight strategy is adopted. After crossing the center point, the drone will briefly stay near the center point to confirm that its attitude and position are stable.

7. A control system for a visually-based unmanned aerial vehicle (UAV) navigating complex obstacles, characterized in that, include: The data acquisition module is configured to simultaneously acquire RGB and depth images of obstacles in front of the drone; The image segmentation module is configured to input the acquired RGB image into a pre-trained deep neural network model, perform semantic segmentation on the hole regions or mesh structure gaps of obstacles in the image, and generate the corresponding binarized mask image. The mask processing module is configured to remove binarized masks that are discontinuous with the obstacle mask in depth distribution, perform dilation operation on the remaining binarized mask, and fill in the background mask by flood filling. The passable area filtering module is configured to perform mask inversion, extract holes and gaps in non-obstacle areas, and obtain a maximum rectangular area larger than the forward cross section of the UAV as the passable area based on the boundaries of these mask areas; The obstacle crossing control module is configured to control the drone to cross obstacles based on the location of the passable area.

8. A control system for a visually-based unmanned aerial vehicle (UAV) navigating complex obstacles as described in claim 7, characterized in that, The mask processing module removes binarized masks that are discontinuous with the obstacle mask in terms of depth distribution, specifically: Sort the pixel depth values ​​corresponding to all binarized masks from smallest to largest. When the difference between two adjacent pixel depth values ​​is greater than a set threshold, truncate the binarized mask sequence at this position and remove the part with the larger depth value after truncation.

9. A terminal device comprising a processor and a memory, the processor for implementing instructions; the memory for storing multiple instructions, characterized in that, The instructions are adapted to be loaded by a processor and executed by the control method for a vision-based unmanned aerial vehicle (UAV) traversing complex obstacles according to any one of claims 1-7.

10. A computer-readable storage medium storing a plurality of instructions, characterized in that, The instructions are adapted to be loaded by the processor of the terminal device and executed by the control method for a visually-based unmanned aerial vehicle (UAV) traversing complex obstacles according to any one of claims 1-7.

Images