A method for unmanned aerial vehicle landing area intelligent obstacle avoidance based on monocular camera and related equipment

Abstract

This application discloses an intelligent obstacle avoidance method and related equipment for UAV landing areas based on a monocular camera. The method includes: determining the coordinates of the warning area in the northeast-east coordinate system based on the warning distance and the landing point coordinates, and converting them to image warning area coordinates in the image coordinate system; processing the real-time monitoring image through a target detection model to obtain the positional and category features of obstacles, thereby obtaining the positional and category information of the obstacles; updating the obstacle count value of the warning area based on the obstacle positional and category information and the image warning area coordinates, and then controlling the UAV to execute an obstacle avoidance strategy. This application's embodiment achieves accurate identification and intelligent avoidance of obstacles in the landing area with low hardware cost and low computational burden by mapping the warning area to the image coordinate system and combining it with a lightweight target detection model. This application can be widely applied in the field of UAV control technology.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to an intelligent obstacle avoidance method and related equipment for UAV landing areas based on a monocular camera. Background Technology

[0002] With the development of the low-altitude economy, drones are increasingly being used in more and more industries due to their advantages of small size, low cost, ease of operation, and flight capability. However, if there are obstacles such as people or moving targets below the landing area during the drone's return mission, safety accidents can easily occur.

[0003] Existing intelligent obstacle avoidance solutions for drones mostly employ depth perception in path planning to detect unknown obstacles in the drone's direction of travel. However, single-point ranging sensors in existing technologies are prone to misjudgment in complex terrain. While binocular vision solutions offer higher accuracy, the sensors are expensive and require significant computing power from the drone.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The main objective of this application is to propose an intelligent obstacle avoidance method and related equipment for UAV landing areas based on a monocular camera. By dynamically mapping the physical warning area to the image coordinate system and combining it with a lightweight target detection model, it achieves accurate identification and intelligent avoidance of obstacles in the landing area with low hardware cost and low computing power burden.

[0006] To achieve the above objectives, one aspect of this application proposes an intelligent obstacle avoidance method for UAV landing areas based on a monocular camera, the method comprising: The system obtains the drone's body position coordinates, landing point position coordinates, and pose parameters in the northeast-east coordinate system during the landing process. It also obtains real-time monitoring images of the landing area through the drone's monocular camera and acquires the camera parameters of the monocular camera. The first warning area coordinates in the northeast coordinate system are determined based on the preset warning distance and the landing point coordinates. The first warning area coordinates are then converted into image warning area coordinates in the image coordinate system of the real-time monitoring image based on the aircraft position coordinates, the pose parameters, and the camera parameters. The real-time monitoring image is processed by a pre-trained target detection model to obtain obstacle location features and obstacle category features at different scales. The obstacle location features and obstacle category features are then post-processed to obtain obstacle location information and obstacle category information in the image coordinate system of the real-time monitoring image. The obstacle count in the warning area is updated based on the obstacle location information, the obstacle category information, and the coordinates of the image warning area. The drone is then controlled to execute a preset obstacle avoidance strategy based on the obstacle count.

[0007] In some embodiments, determining the coordinates of the first warning area in the northeast coordinate system based on a preset warning distance and the landing point coordinates includes: Based on the warning distance, a rectangular area centered on the landing point location coordinates is constructed in the northeast coordinate system, and the rectangular area is used as the warning area. Obtain the vertex coordinates of the warning area in the northeast coordinate system, and use the vertex coordinates as the coordinates of the first warning area.

[0008] In some embodiments, converting the coordinates of the first warning area into image warning area coordinates in the image coordinate system of the real-time monitoring image based on the body position coordinates, the pose parameters, and the camera parameters includes: The first rotation matrix from the northeast coordinate system to the body coordinate system is solved based on the pose parameters. The first warning area coordinates are converted into the second warning area coordinates in the body coordinate system of the UAV based on the first rotation matrix and the body position coordinates. Based on the camera parameters, the coordinates of the second warning area are converted into the image warning area coordinates in the image coordinate system of the real-time monitoring image.

[0009] In some embodiments, the camera parameters include a second rotation matrix from the body coordinate system to the camera coordinate system of the monocular camera, a translation vector from the body coordinate system to the camera coordinate system, and camera intrinsic parameters. The step of converting the coordinates of the second warning area into an image warning area in the image coordinate system of the real-time monitoring image based on the camera parameters includes: The coordinates of the second warning area are converted into the coordinates of the third warning area in the camera coordinate system based on the second rotation matrix and the translation vector. Based on the single-aperture imaging principle of the camera and the camera intrinsic parameters, the coordinates of the third warning area are converted into the coordinates of the image warning area.

[0010] In some embodiments, the target detection model includes a feature extraction layer, a feature fusion layer, and several detection heads at different scales. The process of processing the real-time monitoring image using the pre-trained target detection model to obtain obstacle location features and obstacle category features at different scales includes: The real-time monitoring image is input into the feature extraction layer for feature extraction to obtain image features; The image features are input into the feature fusion layer for fusion processing to obtain multi-scale fused features; The multi-scale fusion features are input into the corresponding scale detection heads to obtain the obstacle location features and obstacle category features at different scales.

[0011] In some embodiments, updating the obstacle count value of the warning area based on the obstacle location information, the obstacle category information, and the image warning area coordinates includes: The presence of an obstacle in the warning area is determined based on the obstacle location information, the obstacle category information, and the coordinates of the image warning area. When the obstacle is present in the warning area, increment the obstacle count by one; When there is no obstacle in the warning area, the obstacle count is set to zero.

[0012] In some embodiments, controlling the drone to execute a preset obstacle avoidance strategy based on the obstacle count value includes: Determine whether the obstacle count value exceeds a preset first threshold; When the obstacle count exceeds the first threshold, the drone is controlled to hover and a warning message is output.

[0013] To achieve the above objectives, another aspect of this application proposes an intelligent obstacle avoidance device for drone landing areas based on a monocular camera, the device comprising: The parameter acquisition module is used to acquire the drone's body position coordinates, landing point position coordinates, and pose parameters in the northeast-east coordinate system during the landing process. It also acquires real-time monitoring images of the landing area through the drone's monocular camera and obtains the camera parameters of the monocular camera. The coordinate system transformation module is used to determine the first warning area coordinates of the warning area in the northeast coordinate system according to the preset warning distance and the landing point position coordinates, and to convert the first warning area coordinates into the image warning area coordinates in the image coordinate system of the real-time monitoring image according to the body position coordinates, the pose parameters and the camera parameters. The target detection module is used to process the real-time monitoring image using a pre-trained target detection model to obtain obstacle location features and obstacle category features at different scales. The obstacle location features and obstacle category features are then post-processed to obtain obstacle location information and obstacle category information in the image coordinate system of the real-time monitoring image. The obstacle avoidance decision module is used to update the obstacle count value of the warning area based on the obstacle location information, the obstacle category information and the coordinates of the image warning area, and control the UAV to execute a preset obstacle avoidance strategy based on the obstacle count value.

[0014] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the methods described above.

[0015] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0016] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the methods described above.

[0017] The embodiments of this application include at least the following beneficial effects: This application provides an intelligent obstacle avoidance method and related equipment for UAV landing areas based on a monocular camera. This solution dynamically converts the physical warning area in the northeast-east coordinate system into the image warning area in the image coordinate system by acquiring relevant parameters of the UAV, landing point, and monocular camera. This eliminates the interference caused by factors such as UAV shaking, altitude changes, and field of view shift during UAV landing, and achieves strict alignment of the warning area between the physical ground and the monitoring screen. By performing multi-scale feature extraction and post-processing on the real-time monitoring image through a target detection model, high-precision obstacle perception can be achieved using only a monocular camera and limited computing power on the edge side, significantly reducing hardware costs and improving the real-time performance of obstacle perception. By introducing an obstacle avoidance mechanism based on obstacle count values, it can filter out occasional noise in the target detection process, avoid the UAV from performing unnecessary obstacle avoidance actions, and effectively improve the robustness of obstacle avoidance decisions. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the steps of an intelligent obstacle avoidance method for UAV landing areas based on a monocular camera, as provided in an embodiment of this application. Figure 2 This is a schematic diagram of a warning area in the northeast coordinate system from a top-down perspective, provided in an embodiment of this application. Figure 3 This is a schematic diagram of a warning area in a northeast coordinate system from a side view, provided in an embodiment of this application. Figure 4This is a schematic diagram of the network structure of a target detection model provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an intelligent obstacle avoidance device for drone landing area based on a monocular camera, provided in an embodiment of this application. Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] The concept of the present invention will now be explained in conjunction with the background art.

[0022] With the development of the low-altitude economy, drones are increasingly being used in more and more industries due to their advantages of small size, low cost, ease of operation, and flight capability. However, if there are obstacles such as people or moving targets below the landing area during the drone's return mission, safety accidents can easily occur.

[0023] Common intelligent obstacle avoidance solutions for drones often employ depth perception during path planning to detect unknown obstacles in the drone's direction of travel. This requires combining multiple sensors, such as laser rangefinders, ultrasonic rangefinders, and infrared rangefinders, for single-point depth perception. These sensors only acquire depth information at a single point, making them prone to misjudging the depth of the drone's direction of travel in complex terrain. Another type of solution utilizes binocular vision triangulation to obtain sparse or dense depth information, which is more adaptable to complex terrain compared to single-point ranging. However, these solutions suffer from the drawbacks of expensive sensors or limited operating environments.

[0024] Therefore, this application provides a method and related equipment for intelligent obstacle avoidance in the landing area of ​​a drone based on a monocular camera. The monocular camera is mounted on the underside of the drone, and the intelligent obstacle avoidance problem during the landing process of the drone is solved by setting the landing area and detecting the target. If the downward-view camera detects an obstacle in the set safety area during the descent of the drone, the drone generates an alarm event to notify the operator, which can avoid accidents as much as possible. It has the advantages of low cost, low operating computing power, and can avoid common obstacles that are likely to cause safety accidents, such as people and vehicles.

[0025] This application provides an intelligent obstacle avoidance method for drone landing areas based on a monocular camera, relating to the field of information technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle-mounted terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the intelligent obstacle avoidance method for drone landing areas based on a monocular camera, but is not limited to the above forms.

[0026] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0027] Figure 1This is an optional flowchart of an intelligent obstacle avoidance method for UAV landing areas based on a monocular camera, provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S104.

[0028] S101. Obtain the body position coordinates, landing point position coordinates, and attitude parameters of the UAV in the northeast-east coordinate system during the landing process. Obtain real-time monitoring images of the landing area through the UAV's monocular camera and obtain the camera parameters of the monocular camera. Specifically, in this embodiment, after the landing process begins, the three-dimensional body position coordinates of the UAV in the North East Down (NED) coordinate system, the preset landing point position coordinates, and parameters characterizing the UAV's attitude (such as quaternions obtained from the inertial measurement unit) can be directly obtained from the UAV flight control system. At the same time, real-time monitoring images of the landing area are collected by a monocular camera installed under the UAV body, and the camera parameters of the monocular camera are obtained, including camera intrinsic parameters and other parameters related to the installation relationship of the body. Among them, the NED coordinates and pose parameters can be directly obtained from the flight control system, which can characterize the spatial positional relationship between the UAV and the landing point. They are the basic data for the subsequent transformation of the warning area in the NED coordinate system to the image coordinate system. The real-time monitoring images provided by the monocular camera are used to identify whether there are obstacles such as personnel and vehicles in the landing area, and can be used as input data for the subsequent target detection model. The camera parameters are used to further eliminate the influence of factors such as field of view shift caused by fuselage sway and changes in field of view coverage caused by altitude reduction on the coordinate transformation process of the warning area, ensuring that the warning area in the image coordinate system of the real-time monitoring image corresponds to the warning position in the NED coordinate system of the actual geographical location of the aircraft.

[0029] For example, when a drone is preparing to land, its current altitude can be obtained in real time, and it can be determined whether the current altitude is lower than the set algorithm activation altitude. If it is lower than the set altitude, the intelligent obstacle avoidance method provided in this embodiment will be executed.

[0030] It can be recognized that, compared with existing drone landing obstacle avoidance solutions, this embodiment does not require the addition of additional sensors. It can provide sufficient input data for intelligent obstacle avoidance during landing by relying solely on the information provided by the flight control system, making it suitable for drone application scenarios where cost and computing power are limited.

[0031] S102. Determine the coordinates of the first warning area in the northeast coordinate system based on the preset warning distance and landing point coordinates. Convert the coordinates of the first warning area into the image warning area coordinates in the image coordinate system of the real-time monitoring image based on the aircraft position coordinates, pose parameters and camera parameters. In some embodiments, determining the coordinates of a first warning area in the northeast coordinate system based on a preset warning distance and landing point location coordinates includes: S1021. Based on the warning distance, construct a rectangular area centered on the landing point coordinates in the northeast coordinate system, and use the rectangular area as the warning area. S1022. Obtain the vertex coordinates of the warning area in the northeast coordinate system, and use the vertex coordinates as the coordinates of the first warning area.

[0032] Specifically, in this embodiment, after acquiring the basic status data of the drone, landing point, and monocular camera, a warning distance is first preset according to the actual operation requirements. Based on this warning distance, the range of the warning area in the NED coordinate system is determined. This warning area is the ROI (Region of Interest) in the obstacle avoidance process of this embodiment.

[0033] Traditional Regions of Interest (ROIs) are typically defined as fixed areas within image pixel coordinates. However, during a drone's descent, due to the drone's body swaying, the area seen by the camera may not be the landing point. Furthermore, as the drone descends, the area seen by the camera gradually increases. Setting the ROI using pixel coordinates would be inflexible and fail to reflect the actual landing area. Therefore, this embodiment utilizes information inherent in the drone's flight controller, such as the drone's current NED coordinates, the landing point's NED coordinates, the quaternions from the drone's body coordinate system (hereinafter referred to as the BODY coordinate system) to the NED coordinate system (i.e., the aforementioned pose parameters), and camera intrinsic and extrinsic parameters, to set the ROI within the drone's NED coordinate system. This is a real physical area, equivalent to setting a warning zone with a real physical distance in the real physical world, which will not change with the drone's swaying angle or current altitude.

[0034] For example, refer to Figure 2 and Figure 3 In this embodiment, the ROI area setting takes the takeoff point as the origin, north as the y-axis, east as the x-axis, and the direction towards the center of the earth as the z-axis. The warning distance is assumed to be 2m, and a 4x4m area is selected as the descent warning area.

[0035] It can be recognized that this embodiment uses physical coordinates to map the position of the warning area in the image, which solves the problem of detection area failure caused by factors such as changes in aircraft attitude and altitude in traditional visual obstacle avoidance methods.

[0036] In some embodiments, converting the coordinates of the first warning area into image warning area coordinates in the image coordinate system of the real-time monitoring image based on the body position coordinates, pose parameters, and camera parameters includes: S1023. Solve for the first rotation matrix from the northeast-east coordinate system to the body coordinate system based on the pose parameters; S1024. Convert the coordinates of the first warning area into the coordinates of the second warning area in the body coordinate system of the UAV based on the first rotation matrix and the body position coordinates. S1025. Convert the coordinates of the second warning area into the image warning area coordinates in the image coordinate system of the real-time monitoring image according to the camera parameters.

[0037] In some embodiments, the camera parameters include a second rotation matrix from the body coordinate system to the camera coordinate system of the monocular camera, a translation vector from the body coordinate system to the camera coordinate system, and camera intrinsic parameters. Converting the coordinates of the second warning area to image warning area coordinates in the image coordinate system of the real-time monitoring image based on the camera parameters includes: S10251. Based on the second rotation matrix and translation vector, convert the coordinates of the second warning area to the coordinates of the third warning area in the camera coordinate system. S10251. Based on the single-aperture imaging principle of the camera and the camera intrinsic parameters, the coordinates of the third warning area are converted into the coordinates of the image warning area.

[0038] Specifically, in this embodiment, after obtaining the warning area in the NED coordinate system, the coordinates of the warning area can be gradually transformed to the image warning area coordinates in the image coordinate system of the monitoring image based on the body position coordinates, pose parameters, and camera parameters.

[0039] For example, let the current position of the drone in the NED coordinate system be... The landing point is located in the NED coordinate system as follows: The quaternion from the NED coordinate system to the BODY coordinate system at the current moment for the UAV is: The second rotation matrix from the BODY coordinate system to the camera coordinate system is: The translation vector is The camera intrinsic parameter is K. Then, based on the obtained landing point in the NED coordinate system... Adding a warning distance d (assuming 2m), the coordinates of the four vertices p1, p2, p3, p4 of the rectangular ROI region in the NED coordinate system can be obtained using the following formula:

[0040] The following formula can be used to calculate the quaternions from the known NED coordinate system to the BODY coordinate system. Solve for the corresponding first rotation matrix. :

[0041]

[0042] Here, w is the scalar part of the quaternion, and x, y, and z are the vector parts.

[0043] Then, transform the four vertices of the ROI region from the NED coordinate system to the BODY coordinate system, as follows: (i.e., the coordinates of the first alert zone) replace the four vertices p1, p2, p3, p4 mentioned above. This represents the position after rotation to the body coordinate system (i.e., the coordinates of the second warning zone). The coordinates of the second warning zone in the body coordinate system can be obtained using the following formula:

[0044] Then, using the following formula... Rotate and translate to the camera coordinate system to obtain the coordinates of the third alert zone. :

[0045] Finally, based on the principle of single-aperture imaging, the points in the camera coordinate system are projected onto the image coordinate system, and u and v represent the horizontal and vertical coordinates in the image coordinate system (i.e., the coordinates of the image warning area):

[0046]

[0047] Based on the above calculation process, the four vertices p1, p2, p3, p4 of the ROI region in the NED coordinate system can be projected onto the image coordinate system. Then, based on the obstacle position information obtained from obstacle detection, it can be determined whether the obstacle is within the ROI region, thus confirming the presence of an obstacle within the landing point warning area.

[0048] It can be recognized that, compared with the traditional method of directly setting the ROI region on the image, this embodiment eliminates the interference of the dynamic changes in the UAV's pose on the determination of the warning area through rigorous transformation of multi-level coordinate systems. It can obtain the image warning area coordinates that are consistent with the real geographical warning area in the real monitoring image. It can provide more real-time feedback on the real coordinates of the warning area in the real physical world on the image. It does not require manual setting and does not change with the camera angle and flight altitude. It is more in line with the landing area setting scheme during the descent of the UAV and provides an accurate judgment benchmark for subsequent obstacle determination based on target detection.

[0049] S103. Process the real-time monitoring image using a pre-trained target detection model to obtain obstacle location features and obstacle category features at different scales. Perform post-processing on the obstacle location features and obstacle category features to obtain obstacle location information and obstacle category information in the image coordinate system of the real-time monitoring image. In some embodiments, the target detection model includes a feature extraction layer, a feature fusion layer, and several detection heads at different scales. The pre-trained target detection model processes real-time monitoring images to obtain obstacle location features and obstacle category features at different scales, including: S1031. Input the real-time monitoring image into the feature extraction layer for feature extraction to obtain image features; S1032. Input the image features into the feature fusion layer for fusion processing to obtain multi-scale fused features; S1033. Input the multi-scale fusion features into the corresponding scale detection head to obtain obstacle location features and obstacle category features at different scales.

[0050] Specifically, in this embodiment, due to the cost and size limitations of the UAV, the target detection model is deployed on the UAV's computing module. Therefore, computational load and memory consumption must be considered. Thus, a lightweight deep learning target detection model is needed to implement obstacle detection. For example, this embodiment selects the improved YOLOv8 (You Only Look Once Version 8, a single-stage target detection network) as the target detection model. It mainly consists of a feature extraction layer (Backbone), a feature fusion layer (Neck), and a detection head. Its simplified network structure... Figure 4 As shown below, when exporting a model in ONNX format from the original YOLOv8, the features output by the Head are concatenated along the channel dimension. However, this leads to a decrease in quantization accuracy when the model is deployed on edge devices. Based on experiments, this embodiment makes the following improvements: the Head output is decomposed into classification and detection features for output. There are three feature heads at different scales, resulting in a total of six features output, including three obstacle location features and three obstacle category features. Then, post-processing is performed on the feature maps at different scales to decode the predicted bounding box position and category of the target. For example, this embodiment can define common obstacles under drones, such as people and vehicles, as targets. The drone's downward-view camera image is input into the detection model to obtain the category and location information of the obstacles in the image. Combined with the image region obtained from the ROI region, it is determined whether the obstacle is within the region. If it is, a warning is generated and the drone is controlled to hover.

[0051] It can be recognized that this embodiment avoids the accumulation of quantization errors caused by complex feature fusion by decoupling the feature fusion at the output end of the target detection model, so that the model can still maintain high perception accuracy on embedded platforms with lower computing power. While ensuring the real-time processing speed of the UAV, it significantly improves the accuracy of the target detection model after quantization deployment.

[0052] S104. Update the obstacle count in the warning area based on obstacle location information, obstacle category information, and image warning area coordinates, and control the UAV to execute the preset obstacle avoidance strategy based on the obstacle count.

[0053] In some embodiments, updating the obstacle count value of the warning area based on obstacle location information, obstacle category information, and image warning area coordinates includes: S1041. Determine whether there are obstacles in the warning area based on the obstacle location information, obstacle category information, and image warning area coordinates; S1042. When there are obstacles in the warning area, increment the obstacle count by one; S1043. When there are no obstacles in the warning area, set the obstacle count to zero.

[0054] In some embodiments, controlling the drone to execute a preset obstacle avoidance strategy based on the obstacle count value includes: S1044. Determine whether the obstacle count value exceeds a preset first threshold; S1045. When the obstacle count exceeds the first threshold, control the drone to hover and output a warning message.

[0055] Specifically, in this embodiment, after obtaining the coordinates and types of obstacles in the image (such as the location boxes of pedestrians and vehicles), the spatial overlap is calculated between these coordinates and the coordinates of the aforementioned image warning area. This determines whether there are obstacles in the warning area that could affect the drone's landing, and intelligent obstacle avoidance is achieved by combining the obstacle count value. For example, the real-time monitoring image can be scaled proportionally and padded with pixels to obtain the image width and height required by the detection model. After algorithm preprocessing, the image is input into the network to obtain the location and category information of the obstacles on the input image. It is then determined whether the center coordinates of the currently detected obstacle are within the image ROI area calculated in step 3. If so, the number of detected obstacles is incremented by 1; otherwise, the number of detected obstacles is reset to zero. It is then determined whether the currently recorded number of detected obstacles is greater than the set warning count. If it is, a warning message is output, and the drone is made to hover, waiting for user input.

[0056] Compared to the direct triggering mechanism of traditional visual obstacle avoidance (i.e., immediate obstacle avoidance upon target detection), this embodiment introduces an obstacle avoidance mechanism based on obstacle count values. By setting count values ​​and trigger thresholds, occasional detection noise such as changes in lighting, object occlusion, or algorithm misjudgments can be filtered out, avoiding frequent unnecessary stuttering and shaking of the drone. Only when an obstacle is stably and continuously detected within the warning zone is it determined to be a real threat. It can be recognized that this embodiment achieves a dual obstacle avoidance judgment mechanism combining spatial and temporal factors through obstacle detection and obstacle count values ​​within the warning area, effectively improving the robustness of obstacle avoidance decisions without increasing hardware costs.

[0057] Please see Figure 5 This application also provides an intelligent obstacle avoidance device for drone landing areas based on a monocular camera, which can implement the above-mentioned method. The device includes: The parameter acquisition module is used to acquire the drone's body position coordinates, landing point position coordinates, and pose parameters in the northeast-east coordinate system during the landing process. It also acquires real-time monitoring images of the landing area through the drone's monocular camera and obtains the camera parameters of the monocular camera. The coordinate system transformation module is used to determine the first warning area coordinates of the warning area in the northeast coordinate system according to the preset warning distance and the landing point position coordinates, and to convert the first warning area coordinates into the image warning area coordinates in the image coordinate system of the real-time monitoring image according to the body position coordinates, the pose parameters and the camera parameters. The target detection module is used to process the real-time monitoring image using a pre-trained target detection model to obtain obstacle location features and obstacle category features at different scales. The obstacle location features and obstacle category features are then post-processed to obtain obstacle location information and obstacle category information in the image coordinate system of the real-time monitoring image. The obstacle avoidance decision module is used to update the obstacle count value of the warning area based on the obstacle location information, the obstacle category information and the coordinates of the image warning area, and control the UAV to execute a preset obstacle avoidance strategy based on the obstacle count value.

[0058] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0059] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0060] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0061] Please see Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the methods described in the embodiments of this application. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0062] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0063] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0064] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0065] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0066] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0067] This application provides an intelligent obstacle avoidance method and related equipment for UAV landing areas based on a monocular camera. The method acquires relevant parameters of the UAV, landing point, and monocular camera, dynamically converting the physical warning area in the NE-G coordinate system into an image warning area in the image coordinate system. This eliminates interference from factors such as UAV swaying, altitude changes, and field-of-view shift during landing, achieving strict alignment of the warning area between the physical ground and the monitoring screen. A target detection model performs multi-scale feature extraction and post-processing on the real-time monitoring image, achieving high-precision obstacle perception with only the monocular camera and limited computing power at the edge, significantly reducing hardware costs and improving the real-time performance of obstacle perception. By introducing an obstacle avoidance mechanism based on obstacle counts, occasional noise during target detection is filtered, preventing the UAV from performing unnecessary obstacle avoidance actions and effectively improving the robustness of obstacle avoidance decisions.

[0068] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0069] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0070] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0071] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0072] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0073] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0074] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0075] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0077] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for intelligent obstacle avoidance in the landing area of ​​a drone based on a monocular camera, characterized in that, include: The system obtains the drone's body position coordinates, landing point position coordinates, and pose parameters in the northeast-east coordinate system during the landing process. It also obtains real-time monitoring images of the landing area through the drone's monocular camera and acquires the camera parameters of the monocular camera. The first warning area coordinates in the northeast coordinate system are determined based on the preset warning distance and the landing point coordinates. The first warning area coordinates are then converted into image warning area coordinates in the image coordinate system of the real-time monitoring image based on the aircraft position coordinates, the pose parameters, and the camera parameters. The real-time monitoring image is processed by a pre-trained target detection model to obtain obstacle location features and obstacle category features at different scales. The obstacle location features and obstacle category features are then post-processed to obtain obstacle location information and obstacle category information in the image coordinate system of the real-time monitoring image. The obstacle count in the warning area is updated based on the obstacle location information, the obstacle category information, and the coordinates of the image warning area. The drone is then controlled to execute a preset obstacle avoidance strategy based on the obstacle count.

2. The intelligent obstacle avoidance method for UAV landing area based on a monocular camera according to claim 1, characterized in that, The step of determining the coordinates of the first warning area in the northeast coordinate system based on the preset warning distance and the landing point coordinates includes: Based on the warning distance, a rectangular area centered on the landing point location coordinates is constructed in the northeast coordinate system, and the rectangular area is used as the warning area. Obtain the vertex coordinates of the warning area in the northeast coordinate system, and use the vertex coordinates as the coordinates of the first warning area.

3. The intelligent obstacle avoidance method for UAV landing area based on a monocular camera according to claim 1, characterized in that, The step of converting the coordinates of the first warning area into the image warning area coordinates in the image coordinate system of the real-time monitoring image based on the body position coordinates, the pose parameters, and the camera parameters includes: The first rotation matrix from the northeast coordinate system to the body coordinate system is solved based on the pose parameters. The first warning area coordinates are converted into the second warning area coordinates in the body coordinate system of the UAV based on the first rotation matrix and the body position coordinates. Based on the camera parameters, the coordinates of the second warning area are converted into the image warning area coordinates in the image coordinate system of the real-time monitoring image.

4. The intelligent obstacle avoidance method for UAV landing area based on a monocular camera according to claim 3, characterized in that, The camera parameters include a second rotation matrix from the body coordinate system to the camera coordinate system of the monocular camera, a translation vector from the body coordinate system to the camera coordinate system, and camera intrinsic parameters. The step of converting the coordinates of the second warning area into an image warning area in the image coordinate system of the real-time monitoring image based on the camera parameters includes: The coordinates of the second warning area are converted into the coordinates of the third warning area in the camera coordinate system based on the second rotation matrix and the translation vector. Based on the single-aperture imaging principle of the camera and the camera intrinsic parameters, the coordinates of the third warning area are converted into the coordinates of the image warning area.

5. The intelligent obstacle avoidance method for UAV landing area based on a monocular camera according to claim 1, characterized in that, The target detection model includes a feature extraction layer, a feature fusion layer, and several detection heads at different scales. The real-time monitoring image is processed using the pre-trained target detection model to obtain obstacle location features and obstacle category features at different scales, including: The real-time monitoring image is input into the feature extraction layer for feature extraction to obtain image features; The image features are input into the feature fusion layer for fusion processing to obtain multi-scale fused features; The multi-scale fusion features are input into the corresponding scale detection heads to obtain the obstacle location features and obstacle category features at different scales.

6. The intelligent obstacle avoidance method for UAV landing area based on a monocular camera according to claim 1, characterized in that, The step of updating the obstacle count value of the warning area based on the obstacle location information, the obstacle category information, and the coordinates of the image warning area includes: The presence of an obstacle in the warning area is determined based on the obstacle location information, the obstacle category information, and the coordinates of the image warning area. When the obstacle is present in the warning area, increment the obstacle count by one; When there is no obstacle in the warning area, the obstacle count is set to zero.

7. The intelligent obstacle avoidance method for UAV landing area based on a monocular camera according to claim 1, characterized in that, The step of controlling the drone to execute a preset obstacle avoidance strategy based on the obstacle count value includes: Determine whether the obstacle count value exceeds a preset first threshold; When the obstacle count exceeds the first threshold, the drone is controlled to hover and a warning message is output.

8. A smart obstacle avoidance device for drone landing areas based on a monocular camera, characterized in that, The device includes: The parameter acquisition module is used to acquire the drone's body position coordinates, landing point position coordinates, and pose parameters in the northeast-east coordinate system during the landing process. It also acquires real-time monitoring images of the landing area through the drone's monocular camera and obtains the camera parameters of the monocular camera. The coordinate system transformation module is used to determine the first warning area coordinates of the warning area in the northeast coordinate system according to the preset warning distance and the landing point position coordinates, and to convert the first warning area coordinates into the image warning area coordinates in the image coordinate system of the real-time monitoring image according to the body position coordinates, the pose parameters and the camera parameters. The target detection module is used to process the real-time monitoring image using a pre-trained target detection model to obtain obstacle location features and obstacle category features at different scales. The obstacle location features and obstacle category features are then post-processed to obtain obstacle location information and obstacle category information in the image coordinate system of the real-time monitoring image. The obstacle avoidance decision module is used to update the obstacle count value of the warning area based on the obstacle location information, the obstacle category information and the coordinates of the image warning area, and control the UAV to execute a preset obstacle avoidance strategy based on the obstacle count value.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images