An unmanned aerial vehicle holder focusing and zooming automatic control method based on an intelligent agent

Abstract

The application discloses a kind of unmanned aerial vehicle holder focusing and zooming automatic control method based on agent, belong to unmanned aerial vehicle intelligent inspection technical field.The application carries out target detection to real-time image in air by lightweight small model with low confidence threshold, and extracts three-dimensional GIS coordinates by frame selecting suspected inspection target object;Agent is based on the three-dimensional GIS coordinates, flight height and imaging field of view angle, and the spatial position relationship is solved, and the holder pitch angle, yaw angle and zoom ratio are dynamically regulated by iterative correction, until target object occupies not less than one-third in imaging picture, and high-definition inspection photo is collected;Again, the visual large model carries out hidden danger accurate identification, and outputs the standardized hidden danger event report of binding hidden danger target coordinate information.The application constructs full-process automation closed loop by large and small model hierarchical cooperation and agent adaptive control, solves the problem of target imaging blur, high miss rate and low automation in air inspection.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection technology for unmanned aerial vehicles (UAVs), specifically to an automatic control method for UAV gimbal focusing and zooming based on an intelligent agent. Background Technology

[0002] Currently, drone inspections are widely used in fields such as power poles, high-rise buildings, bridge engineering, and outdoor equipment security inspection. While conventional low-altitude inspection technology is relatively mature, existing technologies have significant shortcomings in low-altitude inspection scenarios above 70 meters. In low-altitude environments, factors such as air turbulence, light scattering, and long-distance image attenuation occur. When drones equipped with gimbal cameras capture images, the effective image pixels of the inspected target are low, leading to insufficient image coverage and the inability to effectively present key details such as safety hazards, severely impacting the accuracy of subsequent hazard identification.

[0003] Current drone gimbal control largely relies on manual pre-setting of flight paths, waypoints, and fixed operating parameters, executing operations according to programmed instructions. Upon target detection during flight, the camera can only be adjusted based on preset parameters, resulting in an inability to adapt to targets at different altitudes and locations. This leads to issues such as target misalignment, insufficient image coverage, and inadequate effective pixel counts for the inspected target. Furthermore, traditional inspection solutions often employ a single recognition model, which is adequate for low-altitude, conventional scenarios, but prone to missed or false detections of targets in low-quality images above 70 meters, resulting in low accuracy in hazard identification and reliance on manual verification. Current technology has not yet achieved fully automated linkage between gimbal control, high-definition imaging, and hazard identification.

[0004] In summary, existing technologies lack dedicated intelligent control solutions for low-altitude inspection scenarios above 70 meters. They cannot achieve integrated automatic control of precise target positioning, gimbal adaptive focusing and zooming, high-definition data acquisition, and intelligent hazard identification. As a result, they suffer from low operational efficiency, poor imaging quality, low accuracy in hazard identification, and high reliance on manual labor, making it difficult to meet the operational needs of refined aerial inspections. Summary of the Invention

[0005] The technical problem this invention aims to solve is that existing technologies, which rely heavily on manual preset routes, waypoints, and other fixed tasks and parameters for shooting, cannot adapt to different altitudes and inspection targets, and are prone to problems such as target offset, insufficient image coverage, and insufficient effective pixels in target imaging. This invention provides an intelligent agent-based method for automatic control of UAV gimbal focusing and zooming, addressing the technical problems of blurry imaging, low gimbal control accuracy, insufficient target image coverage, low automation, and poor hazard identification accuracy in aerial inspections using UAVs above 70 meters. This method achieves intelligent and automated operation throughout the entire aerial inspection process, improving inspection imaging quality and the reliability of hazard detection.

[0006] To solve the above problems, the present invention adopts the following technical solution:

[0007] An agent-based automatic control method for focusing and zooming of a drone gimbal includes the following steps:

[0008] A lightweight small model is used to perform target detection on real-time images collected by UAVs during aerial inspections, select suspected inspection targets, and extract the 3D geographic information system (GIS) coordinates corresponding to the suspected inspection targets.

[0009] The drone is controlled to maintain a fixed hovering state. The intelligent agent calculates the spatial positional relationship between the drone and the suspected inspection target based on the three-dimensional GIS coordinates, the drone's real-time flight altitude, and the imaging field of view. It also adjusts the pitch angle, yaw angle, and zoom of the gimbal until the suspected inspection target occupies no less than a preset proportion in the imaging image. Then, a high-definition photo capture command is automatically triggered to collect high-definition inspection photos.

[0010] The high-definition inspection photos are pushed to a large visual model for hazard identification to obtain hazard identification results.

[0011] Output a standardized hazard event report that includes the coordinates of the hazard target.

[0012] Preferably, the lightweight small model performs target detection on the real-time image with a threshold lower than the confidence level of conventional target detection, and the real-time image is an aerial long-distance blurred image with weakened target features.

[0013] Preferably, the intelligent agent controls the pitch angle, yaw angle, and zoom magnification of the gimbal, specifically including the following steps:

[0014] Based on the GIS coordinates of the UAV, historical GIS coordinates, and the three-dimensional GIS coordinates of the suspected inspection target, the horizontal distance, vertical height difference, and spatial slant distance between the UAV and the suspected inspection target are calculated.

[0015] Based on the horizontal distance, vertical height difference, spatial slant distance, and optical parameters of the gimbal camera, the required pitch angle, yaw angle, and zoom magnification of the gimbal are calculated.

[0016] Preferably, the calculation of the required pitch angle, yaw angle, and zoom magnification of the gimbal further includes the following steps:

[0017] Obtain the center coordinates of the region of interest (ROI) of the suspected inspection target in the image;

[0018] Calculate the center offset between the center coordinates of the ROI and the center coordinates of the image;

[0019] When the center offset exceeds the preset target centering determination threshold, the agent iteratively corrects the pitch and yaw angles of the gimbal based on the center offset and the preset mapping coefficient until the center offset is within the target centering determination threshold.

[0020] Preferably, the mapping coefficient is predetermined based on the field of view, focal length parameters, and image resolution of the gimbal camera.

[0021] Preferably, the pitch angle, yaw angle, and zoom of the gimbal are adjusted until the suspected inspection target occupies no less than a preset proportion in the image, wherein the preset proportion is one-third.

[0022] Preferably, the step of automatically triggering the high-definition photo capture command and acquiring high-definition inspection photos specifically involves: automatically triggering the photo capture command and acquiring two high-definition inspection photos within a continuous time sequence.

[0023] Preferably, after the visual large model completes the hazard identification, the method further includes: comparing the clarity, signal-to-noise ratio and target integrity of the two high-definition inspection photos, and selecting the best photo; integrating the best photo, the hazard identification result, and the bound hazard target latitude and longitude coordinates and image acquisition time to generate and output the standardized hazard event report.

[0024] Preferably, the three-dimensional GIS coordinates include the relative coordinates of the suspected inspection target relative to the UAV, absolute latitude and longitude coordinates, target size range, and spatial height information.

[0025] Preferably, the real-time flight altitude is any altitude between 70 meters and 120 meters.

[0026] The present invention, by adopting the above technical solution, has the following beneficial effects:

[0027] 1. This invention specifically addresses the pain points of low-altitude inspection above 70 meters, effectively overcoming the problems of insufficient effective pixels of the inspection target caused by light scattering in the air, long-distance imaging attenuation, and airflow interference. Through intelligent agent adaptive parameter adjustment, it stably ensures that the target object occupies no less than 1 / 3 of the image, greatly improving the image quality of aerial inspection.

[0028] 2. This invention adopts a hierarchical collaborative architecture of small and large models. The small model is adapted to low-quality aerial images and low-confidence weak target detection to avoid missing targets; the large model focuses on high-precision hazard identification, taking into account both detection comprehensiveness and identification accuracy, and is adapted to complex aerial inspection scenarios.

[0029] 3. This invention automates the entire process of drone hovering control, gimbal alignment, automatic zoom, high-definition photography, hazard identification, and information output, eliminating the need for manual operation of the gimbal and equipment parameters, significantly reducing labor costs and improving inspection efficiency.

[0030] 4. This invention adopts a dual-image acquisition and optimal output mechanism, which effectively avoids the problems of single-image shooting error and noise interference. Combined with latitude, longitude and time information binding, it realizes accurate tracing of hidden danger information and standardized output, and the inspection results are more reliable.

[0031] 5. The intelligent agent of this invention can dynamically and iteratively adjust parameters according to different low-altitude altitudes and target positions, making it highly adaptable and versatile. It can be adapted to various low-altitude drone inspection scenarios such as power, construction, bridges, and security. Attached Figure Description

[0032] Figure 1 This is a hierarchical diagram of the method involved in the present invention.

[0033] Figure 2 This is a schematic diagram illustrating the geometric relationship between the UAV and the inspection target during gimbal parameter control in the embodiment. Detailed Implementation

[0034] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0035] This embodiment is a specific implementation of an agent-based automatic control method for UAV gimbal focusing and zooming, adapted for aerial inspection scenarios of UAVs at altitudes above 70 meters. The core mechanism utilizes a linkage mechanism of small-model low-confidence target detection (C∈[0,0.5]), agent-adaptive gimbal parameter adjustment, and large-model accurate hazard identification to complete automated inspection operations. (Refer to...) Figure 1 The specific implementation steps are as follows:

[0036] 1. Target Recognition and Information Acquisition: A lightweight small model is used to detect low-confidence targets within the UAV's field of view, accurately selecting suspected targets and extracting and outputting the corresponding GIS (Geographic Information System) information. This small model is adapted for long-distance, blurred image recognition scenarios. Addressing the characteristics of low-altitude imaging (above 70 meters) with low resolution, lack of detail, and weakened target features, the model's confidence threshold is optimized to reduce the probability of missing valid targets, achieving effective selection and localization of low-confidence suspected targets. The target's GIS information includes its relative coordinates to the UAV, absolute latitude and longitude coordinates, target size range, and spatial altitude information, providing data support for agent parameter calculation.

[0037] 2. Intelligent agent adaptive gimbal control and high-definition photography: Control the drone to maintain a fixed hovering state, with a preset altitude threshold. Furthermore, the operating height meets the inspection requirements for heights above 70 meters and below 120 meters. The intelligent agent calculates the current height in real time based on a preset control formula. Corresponding gimbal pitch angle Yaw angle and zoom magnification The system dynamically adjusts the gimbal attitude and zoom parameters to ensure the target object occupies at least 1 / 3 of the image. After parameter adjustment, it automatically triggers a photo-taking command, capturing two high-definition inspection photos. The intelligent agent is a dedicated autonomous decision-making and control agent for UAV inspection, with built-in attitude control algorithms and zoom adaptation models. It requires no manual intervention and can dynamically and iteratively correct the gimbal control parameters based on real-time collected GIS information and the UAV's flight altitude. The control formula is constructed based on the UAV's flight altitude, the target object's spatial position, optical parameters, and the camera's imaging field of view, achieving precise matching of pitch angle, orientation angle, and zoom magnification to ensure the target's image clarity and image occupancy meet standards at long distances in the air. The two high-definition photos are captured sequentially for image selection in subsequent large-scale model recognition, avoiding recognition errors caused by noise or slight blur in a single image.

[0038] Known current GIS coordinates of the drone The GIS coordinates of the drone during the last shooting GIS coordinates of the inspection target It can calculate the horizontal distance between the drone and the inspection target. Vertical height difference Slant distance :

[0039] ;

[0040] ;

[0041] ;

[0042] Based on the proportion of inspection targets in historical photos Zoom ratio during shooting Camera reference wide-angle focal length Sensor width It can calculate the spatial slant distance between the drone's GIS coordinates and the inspection target's GIS coordinates at the time of the last shooting. And calculate the width of the inspection target accordingly. :

[0043] ;

[0044] ;

[0045] refer to Figure 2 The camera gimbal parameters can be adjusted according to the distance between the drone and the inspection target to automatically align the gimbal with the target's center point and zoom so that the inspection target occupies more than 1 / 3 of the frame. The default roll angle is [not specified]. To keep the image level, the gimbal pitch angle can be calculated using the following formula. Horizontal yaw angle Zoom magnification The zoom ranges from 1x to the camera's maximum zoom. between:

[0046] ;

[0047] ;

[0048] ;

[0049] ;

[0050] ;

[0051] ;

[0052] To improve target centering accuracy, the agent uses the center coordinates of the target's region of interest (ROI) obtained through real-time detection. , ) and image center coordinates ( , Calculate the center offset:

[0053] ;

[0054] ;

[0055] when or At that time, the tilt angle of the gimbal Gimbal yaw angle Iterative corrections were performed, with correction magnitudes of: and :

[0056] ;

[0057] ;

[0058] in, , This is the mapping coefficient between image pixel offset and gimbal angle offset, which can be determined based on camera field of view, focal length parameters, and image resolution. , The target centering threshold can be preset based on image resolution, target ROI size, or system accuracy requirements.

[0059] The corrected gimbal control angle is:

[0060] ;

[0061] ;

[0062] Until the target center offset satisfies:

[0063] ;

[0064] ;

[0065] 3. Intelligent Identification of Hidden Dangers from High-Definition Images: Two high-definition inspection photos are simultaneously pushed to a large-scale visual model. The model performs semantic understanding to accurately identify and extract features of various defects, faults, and abnormal hazards in the images. Simultaneously, the two high-definition inspection photos are also pushed to a knowledge base (RAG enhancement) to retrieve industry characteristic standards, defect patterns, or historical inspection data for objects such as bridges, tunnels, and slopes. This supplements the large-scale model with industry background information, preventing "illusions" generated by the general-purpose model. Next, the high-definition images and the retrieved professional knowledge are input into the general-purpose model, where its multimodal cognitive functions are used for semantic understanding (e.g., assessing the severity of cracks, determining if a slope has landslide precursors, analyzing the causes of abnormal phenomena, etc.), to accurately identify and extract features of various defects, faults, and abnormal hazards in the images.

[0066] 4. Standardized Hazard Information Output: After the large-scale model completes hazard identification, it compares the clarity, signal-to-noise ratio, and target integrity of two original photos, selecting the best single photo. Simultaneously, it binds the precise latitude and longitude coordinates of the hazard target, image acquisition time, hazard type, and other information to generate and output a standardized hazard event report, completing a single automated aerial inspection operation. The original photo selection rule is that the large-scale model compares the clarity, signal-to-noise ratio, and target integrity of two photos, selecting the best single image and binding it with hazard information for output.

[0067] This embodiment is applied to the drone-based intelligent inspection service project of the Hangzhou section of the Hangchunkai Expressway. The entire process of target recognition, gimbal control, high-definition imaging, and engineering safety hazard identification is fully automated. The specific steps are as follows:

[0068] 1. Low-confidence target identification and GIS information collection: When the UAV flies to an inspection height of 80 meters, a lightweight small model is launched. The model is adapted to the features of long-distance blurred images in the air, and the inherent confidence threshold is lowered to reduce the false negative rate. The UAV performs full-area detection on inspection targets such as bridges, piers, tunnels, slopes, and retaining walls, accurately selects low-confidence suspected hidden danger targets, and extracts GIS information such as the absolute latitude and longitude of the target, relative UAV spatial coordinates, and target size in real time to complete the accurate positioning of the target.

[0069] 2. Intelligent Agent Gimbal Adaptive Control and High-Definition Photography: The intelligent agent receives the target GIS information output in step 1, sends a hovering command, and controls the drone to hover stably at the current altitude. The intelligent agent calls the built-in control formula, combining the current flight altitude H=80m, target spatial position, gimbal camera field of view, and zoom parameters, to calculate the optimal gimbal pitch angle, horizontal orientation angle, and zoom magnification in real time. It dynamically drives the gimbal to complete attitude deflection and zoom adjustment, and verifies the image quality in real time to ensure that the potential target occupies no less than 1 / 3 of the image and is clearly visible. After the parameters are stabilized, two high-definition inspection photos are automatically taken continuously at 0.5-second intervals to complete image acquisition.

[0070] 3. Intelligent identification of hidden dangers in large models: The two high-definition inspection photos of hidden dangers are transmitted in real time to the trained inspection vision large model. The large model is specially optimized for safety hazard samples, accurately identifying various hidden dangers such as slope landslides and cracks, and extracting core feature information such as the location of the hidden danger and the type of defect.

[0071] 4. Standardized Hazard Information Output: After the large model completes hazard identification, it compares the clarity, signal-to-noise ratio, and target integrity of two original photos, selects the best single photo, and binds the precise latitude and longitude coordinates of the hazard target, image acquisition time, hazard type, and other information to generate and output a standardized hazard event report, completing a single automated aerial inspection operation.

[0072] Repeat the above steps to complete the full-area aerial inspection operation. No manual intervention is required throughout the process. The imaging quality is stable and the accuracy of hazard identification is high, which can effectively meet the needs of fine-grained inspection at heights of 70 meters and above.

[0073] In a preferred embodiment, this system builds an automatic zoom intelligent agent for drones based on a large language model (LLM), an industry business knowledge base, and an action rule base. It integrates two modes: high-throughput parallel data processing and deep serial cognitive decision-making, to construct a fully closed-loop drone inspection system from environmental perception, intelligent analysis, decision-making and judgment to equipment control.

[0074] In a preferred embodiment, the lightweight small model adopts a multi-module parallel computing architecture, which includes multiple lightweight dedicated target detection modules, each used to identify different categories of inspection objects. Each detection module runs independently and synchronously outputs the target's geographical latitude and longitude, image pixel coordinates, and bounding box information to improve the target detection response speed.

[0075] In a preferred embodiment, before or simultaneously with inputting high-definition inspection photos into the visual big model, the business knowledge base can be searched to retrieve the industry standards, defect characteristics, and historical inspection records of the corresponding inspection object. The background information and high-definition images are then input into the big model together to supplement professional industry knowledge and improve the accuracy of hazard identification.

[0076] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0077] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A method for automatic control of focusing and zooming of a UAV gimbal based on an intelligent agent, characterized in that, Includes the following steps: A lightweight small model is used to perform target detection on real-time images collected by UAVs during aerial inspections, select suspected inspection targets, and extract the 3D geographic information system (GIS) coordinates corresponding to the suspected inspection targets. The drone is controlled to maintain a fixed hovering state. The intelligent agent calculates the spatial positional relationship between the drone and the suspected inspection target based on the three-dimensional GIS coordinates, the drone's real-time flight altitude, and the imaging field of view. It also adjusts the pitch angle, yaw angle, and zoom of the gimbal until the suspected inspection target occupies no less than a preset proportion in the imaging image. Then, a high-definition photo capture command is automatically triggered to collect high-definition inspection photos. The high-definition inspection photos are pushed to a large visual model for hazard identification to obtain hazard identification results. Output a standardized hazard event report that includes the coordinates of the hazard target.

2. The method according to claim 1, characterized in that, The lightweight small model performs target detection on the real-time image with a threshold lower than the confidence level of conventional target detection. The real-time image is a long-range blurred aerial image with weakened target features.

3. The method according to claim 1, characterized in that, The intelligent agent controls the pitch angle, yaw angle, and zoom magnification of the gimbal, specifically including the following steps: Based on the GIS coordinates of the UAV, historical GIS coordinates, and the three-dimensional GIS coordinates of the suspected inspection target, the horizontal distance, vertical height difference, and spatial slant distance between the UAV and the suspected inspection target are calculated. Based on the horizontal distance, vertical height difference, spatial slant distance, and optical parameters of the gimbal camera, the required pitch angle, yaw angle, and zoom magnification of the gimbal are calculated.

4. The method according to claim 3, characterized in that, The calculation of the required pitch angle, yaw angle, and zoom magnification for the gimbal further includes the following steps: Obtain the center coordinates of the region of interest (ROI) of the suspected inspection target in the image; Calculate the center offset between the center coordinates of the ROI and the center coordinates of the image; When the center offset exceeds the preset target centering determination threshold, the agent iteratively corrects the pitch and yaw angles of the gimbal based on the center offset and the preset mapping coefficient until the center offset is within the target centering determination threshold.

5. The method according to claim 4, characterized in that, The mapping coefficients are predetermined based on the field of view, focal length parameters, and image resolution of the gimbal camera.

6. The method according to claim 1, characterized in that, Adjust the pitch angle, yaw angle, and zoom of the gimbal until the suspected inspection target occupies no less than a preset proportion in the image, where the preset proportion is one-third.

7. The method according to claim 1, characterized in that, The step of automatically triggering the high-definition photo capture command and acquiring high-definition inspection photos specifically involves: automatically triggering the photo capture command and acquiring two high-definition inspection photos within a continuous time sequence.

8. The method according to claim 7, characterized in that, After the visual big data model completes the hazard identification, the method further includes: comparing the clarity, signal-to-noise ratio and target integrity of the two high-definition inspection photos, and selecting the best photo; integrating the best photo, the hazard identification result, and the bound latitude and longitude coordinates of the hazard target and the image acquisition time to generate the standardized hazard event report and output it.

9. The method according to claim 1, characterized in that, The three-dimensional GIS coordinates include the relative coordinates of the suspected inspection target relative to the UAV, the absolute latitude and longitude coordinates, the target size range, and spatial height information.

10. The method according to claim 1, characterized in that, The real-time flight altitude is any altitude between 70 meters and 120 meters.

Images