High-precision holder control method and system

Abstract

The invention relates to the technical field of remote monitoring, and discloses a high-precision holder control method and system. Comprising the steps of collecting a target scene image, performing denoising, distortion correction and resolution adjustment, obtaining a target pixel coordinate and confidence through a lightweight target detection model, and calling a corresponding image enhancement model to perform preprocessing when an abnormal environment is detected; transmitting a part of detection results as key frames to an edge end based on a frame extraction strategy, caching the rest of the detection results locally, and switching to a local tracking mode and reducing control frequency when network delay exceeds a threshold value; fusing visual detection data and inertial measurement data, and predicting a target position through extended Kalman filtering; carrying out sub-pixel-level refinement on the target position, and calculating a three-dimensional orientation in combination with a participated holder attitude in a camera; and adjusting the attitude of the holder in real time based on a prediction result and an anti-saturation integral separation PID controller.

Description

Technical Field

[0001] The present invention relates to the field of remote monitoring technology, and in particular to a high-precision pan / tilt control method and system. Background Art

[0002] Remote monitoring and automated inspections of power facilities are crucial for ensuring the safe operation of power grids. Traditional power inspections rely primarily on manual on-site inspections, which are inefficient and costly, and pose safety risks in adverse weather conditions. With the continuous expansion and increasing intelligence of power systems, remote monitoring systems based on vision technology are becoming a growing trend in the power industry. These systems typically utilize high-precision pan-tilt cameras and automatic target tracking technology to achieve real-time monitoring of critical infrastructure such as substation equipment and transmission lines.

[0003] Existing pan-tilt control technology faces numerous challenges in power inspection applications. The first is positioning accuracy. Traditional target detection algorithms are susceptible to background interference, lighting variations, and other factors in complex power environments, resulting in inaccurate target positioning. Second, response latency is a problem. Limited by network transmission and computing resources, existing systems often experience significant control delays, making it difficult to achieve real-time tracking of fast-moving targets. Furthermore, abnormal weather conditions such as rain, snow, fog, and haze can severely affect image quality, further reducing the reliability and robustness of the system. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a high-precision pan-tilt control method and system, the purpose of which is to take into account the power consumption limitations and cost control requirements of the power site, while achieving high-performance target tracking functions under the constraints of limited hardware resources.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: A high-precision pan-tilt control method, comprising:

[0006] Collect and preprocess the image data of the target scene, and determine the pixel coordinates and confidence of the target in the image through a lightweight target detection model. When a preset abnormal environment is detected, the image enhancement model is called to perform image enhancement;

[0007] Based on the frame extraction strategy, some detection results are transmitted to the edge as key frames, and the remaining detection results are cached locally. When it is detected that the network transmission delay exceeds the first threshold, the fallback mechanism is triggered, switching to the local tracking algorithm and reducing the update frequency of the gimbal control instructions;

[0008] The real-time attitude information of the gimbal is obtained and compared with the trajectory data of the edge end. When the attitude difference exceeds the preset angle threshold, attitude resynchronization is triggered. During the attitude calculation process, the visual detection data and inertial measurement data are integrated to predict the target motion to improve the attitude synchronization accuracy and anti-jitter capability.

[0009] The target pixel coordinates are refined at the sub-pixel level and the target's three-dimensional spatial orientation is calculated based on the gimbal attitude parameters.

[0010] Feedforward prediction is performed based on the target motion prediction model, and the gimbal actuator is adjusted in real time in combination with the feedback controller to achieve low-latency, high-precision target tracking.

[0011] Furthermore, the step of preprocessing the image data includes:

[0012] Perform adaptive filtering on the captured image and dynamically adjust the filtering parameters according to the image noise characteristics to suppress random noise and retain edge details;

[0013] Perform geometric transformation on the image based on the camera calibration parameters to compensate for the radial and tangential distortion introduced by the lens to restore the real scene scale;

[0014] The image is scaled according to the preset target resolution to ensure that the detailed information required for detection is retained while reducing the amount of data.

[0015] Furthermore, the abnormal environment includes rain, snow, fog, haze, strong light and low illumination at night; the implementation logic of the image enhancement model is:

[0016] When a rainy or snowy environment is detected, the image is processed using a generative adversarial network to remove precipitation marks to eliminate blur and light spots caused by raindrops and snowflakes.

[0017] When a hazy environment is detected, the image is dehazed using a dark channel prior combined with color constancy enhancement to restore the contrast and color of the target area.

[0018] When a bright light environment is detected, the image is processed with a high dynamic range through a multi-exposure image fusion algorithm to preserve details in the highlight area and suppress overexposure;

[0019] When a low-light environment is detected at night, the image brightness is enhanced and noise is suppressed by combining infrared fill light acquisition with a denoising enhancement network to improve the signal-to-noise ratio and visibility of the image.

[0020] Furthermore, the frame extraction strategy is as follows: key frames are selected at a preset frame interval from the continuously acquired multiple frames of images, and after downsampling the key frames to a target resolution lower than the original resolution, they are sent to the edge end via a transmission protocol with forward error correction function, while the remaining non-key frames are cached locally;

[0021] The fallback mechanism is: when it is detected that the network transmission delay exceeds a preset threshold, it switches to the locally running target tracking algorithm to predict the target position, and reduces the update frequency of the gimbal control instructions to a second preset value to maintain continuous tracking of the target under communication-restricted conditions.

[0022] Furthermore, the target motion prediction step includes:

[0023] Obtain the target position coordinates and their confidence levels output by the visual detection model, and simultaneously collect angular velocity and acceleration data provided by the inertial measurement unit;

[0024] Taking the target position coordinates and inertial measurement data as input, the extended Kalman filter algorithm that fuses visual and inertial information is used to predict the target's next position at a certain moment;

[0025] The observation noise covariance is dynamically adjusted according to the target detection confidence to increase the weight of inertial data when the confidence decreases, thereby improving the stability and anti-interference ability of the prediction.

[0026] Furthermore, the step of calculating the three-dimensional spatial orientation of the target includes:

[0027] Preprocessing the collected images, including denoising and contrast enhancement, to obtain clear images for precise positioning;

[0028] The center point position is refined using edge detection based on the Sobel operator combined with a sub-pixel interpolation method using a second-order Taylor expansion to improve the positioning accuracy to within ±0.1 pixels.

[0029] The refined target pixel coordinates are combined with the current gimbal posture parameters and the camera intrinsic parameter matrix, and the azimuth and pitch angles of the target relative to the gimbal are calculated through the three-dimensional space coordinate solution model.

[0030] Furthermore, the step of adjusting the pan / tilt actuator in real time includes:

[0031] Based on the target motion prediction model, the target's expected position at the next moment is calculated under the assumption of uniform motion to form a feedforward control instruction;

[0032] Combining the feedforward control instructions with visual detection and posture feedback data, and performing error correction through a PID controller with anti-saturation integral separation function;

[0033] The corrected control quantity is output to the pan-tilt actuator according to the preset sampling frequency to achieve real-time response and precise tracking of target position changes.

[0034] Furthermore, the real-time adjustment further includes abnormality detection and relocation, and the abnormality detection and relocation steps include:

[0035] Monitor the confidence level and response peak value changes of target detection. When the confidence level of multiple consecutive frames is lower than the preset threshold or the response peak value deviates from the historical mean by more than the preset standard deviation, the target tracking is determined to be lost.

[0036] When the target is determined to be lost, the gimbal is controlled to enter spiral scanning mode, performing a three-dimensional space search at a preset angular velocity and increasing radius;

[0037] The cached high-confidence target template is used for feature matching. When the number of matching points and the ratio of inliers exceed the preset threshold, the target is re-locked and the normal tracking mode is restored.

[0038] Furthermore, the anomaly detection and relocation also includes a long-term robustness guarantee mechanism, and the implementation strategy of the guarantee mechanism is:

[0039] A gated recurrent unit is used to determine the update timing of the target template. The template is updated only when the target appearance changes slowly and the detection confidence is higher than a preset value, preventing template aging and avoiding the erroneous introduction of background information.

[0040] When relocation attempts continue to fail for longer than a preset time, the system automatically switches to low-power standby mode, reduces the image acquisition frame rate and continuously monitors trigger signals, and resumes normal tracking mode after the target re-enters the detection range.

[0041] The present invention also provides a high-precision pan / tilt control system, comprising:

[0042] An image preprocessing module is used to collect and preprocess the image data of the target scene, and determine the pixel coordinates and confidence level of the target in the image through a lightweight target detection model. When a preset abnormal environment is detected, an image enhancement model is called to perform image enhancement;

[0043] The edge collaboration module is used to transmit some detection results as key frames to the edge based on the frame extraction strategy, and the remaining detection results are cached locally. When it is detected that the network transmission delay exceeds a first threshold, the fallback mechanism is triggered, switching to the local tracking algorithm and reducing the update frequency of the gimbal control instructions;

[0044] The attitude synchronization module is used to obtain the real-time attitude information of the gimbal and compare it with the trajectory data of the edge end. When the attitude difference exceeds the preset angle threshold, attitude resynchronization is triggered. During the attitude calculation process, visual detection data and inertial measurement data are integrated to predict the target motion to improve attitude synchronization accuracy and anti-jitter capability.

[0045] The target orientation acquisition module is used to refine the target pixel coordinates at the sub-pixel level and calculate the target's three-dimensional spatial orientation based on the gimbal attitude parameters;

[0046] The gimbal control module is used to perform feedforward prediction based on the target motion prediction model and to adjust the gimbal actuator in real time in combination with the feedback controller to achieve low-latency, high-precision target tracking.

[0047] The present invention provides a high-precision pan / tilt control method and system, which has the following beneficial effects:

[0048] 1. By deploying a lightweight target detection model combined with sub-pixel refinement technology, we achieved a positioning accuracy of ±0.1 pixels, equivalent to a spatial accuracy of ±4.4 cm at a distance of 50 meters. We also adopted a composite control strategy combining feedforward prediction with feedback control, using prediction and compensation based on the target motion model to control system response latency to less than 80ms. End-to-end processing on the RK3568 edge computing platform avoids additional latency in cloud transmission and significantly improves the real-time and accuracy of target tracking.

[0049] 2. A comprehensive abnormal weather processing mechanism has been established to address the complex environmental conditions of power inspections. By integrating multiple image enhancement methods, including a generative adversarial network rain removal model, a dark channel prior dehazing algorithm, HDR fusion technology, and infrared fill illumination, the system effectively handles harsh conditions such as rain, snow, fog, haze, strong sunlight, and low illumination at night. Actual measurement data shows that even in heavy rain, the target detection recall rate remains above 91%, ensuring the system's reliable operation in various climates.

[0050] 3. This invention utilizes a three-stage pipeline architecture of "lightweight perception, edge collaboration, and dynamic feedback," optimizing network resource utilization through frame extraction strategies and forward error correction transmission. When network latency exceeds a threshold, the system automatically triggers the local KCF tracking algorithm as a backup to ensure continuous target tracking. By integrating visual detection with IMU inertial data, the extended Kalman filter prediction method dynamically adjusts sensor weights based on detection confidence, achieving sub-second response and robust tracking around the clock within a power consumption constraint of <5W, meeting the practical needs of unmanned monitoring of power facilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flow chart of a high-precision pan / tilt control method provided by the present invention;

[0052] Figure 2 A flowchart of the steps for calculating the three-dimensional spatial orientation of a target provided by the present invention;

[0053] Figure 3 A flowchart of the steps for real-time adjustment of the pan / tilt actuator provided by the present invention;

[0054] Figure 4 A flowchart of the steps of anomaly detection and relocation provided by the present invention;

[0055] Figure 5 This is a structural diagram of a high-precision pan-tilt control system provided by the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0057] The embodiment of the present invention provides a high-precision pan / tilt control method, such as Figure 1 Shown, including:

[0058] S100: collecting and preprocessing image data of the target scene, and determining the pixel coordinates and confidence of the target in the image through a lightweight target detection model. When a preset abnormal environment is detected, calling an image enhancement model to perform image enhancement;

[0059] Furthermore, the step of preprocessing the image data includes:

[0060] Perform adaptive filtering on the captured image and dynamically adjust the filtering parameters according to the image noise characteristics to suppress random noise and retain edge details;

[0061] Perform geometric transformation on the image based on the camera calibration parameters to compensate for the radial and tangential distortion introduced by the lens to restore the real scene scale;

[0062] The image is scaled according to the preset target resolution to ensure that the detailed information required for detection is retained while reducing the amount of data.

[0063] Specifically, a 4K@30fps camera is connected to the RK3568 front-end system via the MIPI CSI2 interface. Immediately after image acquisition, the hardware pipeline begins processing. The RK3568's built-in image signal processor (ISP) hardware 3DNR module first denoises the original image. The 3DNR module uses combined temporal and spatial filtering to effectively suppress random noise while preserving image detail. The RGA (raster graphics acceleration) engine then performs distortion correction and scaling, scaling the 4K image to 1080P resolution. Single-frame processing takes less than 4ms.

[0064] Calculate the local variance of the image ,in, Represents a point on the image The pixel value of Indicates that the image is at point The local mean of . The filter parameters are dynamically adjusted using adaptive median filtering, and the window size is dynamically adjusted according to the noise variance. In a feasible implementation, when When the window size is set to ,when When the window size is set to ,when When the window size is set to The camera's intrinsic parameter matrix and distortion parameters are obtained in advance through the checkerboard calibration method. Distortion correction is performed on each pixel using radial and tangential distortion. The corrected image is scaled from the original resolution to the target resolution (e.g., 1080P) using a bilinear interpolation algorithm.

[0065] This step effectively suppresses random noise, corrects lens distortion, and optimizes resolution while ensuring the integrity of image details, thereby significantly improving image clarity and geometric accuracy, providing stable and reliable input data for subsequent target detection and positioning, while reducing data volume to improve processing and transmission efficiency.

[0066] Furthermore, the abnormal environment includes rain, snow, fog, haze, strong light and low illumination at night; the implementation logic of the image enhancement model is:

[0067] When a rainy or snowy environment is detected, the image is processed using a generative adversarial network to remove precipitation marks to eliminate blur and light spots caused by raindrops and snowflakes.

[0068] When a hazy environment is detected, the image is dehazed using a dark channel prior combined with color constancy enhancement to restore the contrast and color of the target area.

[0069] When a bright light environment is detected, the image is processed with a high dynamic range through a multi-exposure image fusion algorithm to preserve details in the highlight area and suppress overexposure;

[0070] When a low-light environment is detected at night, the image brightness is enhanced and noise is suppressed by combining infrared fill light acquisition with a denoising enhancement network to improve the signal-to-noise ratio and visibility of the image.

[0071] Specifically, environmental condition detection determines the type of environment through image statistical features. In one feasible implementation, rain and snow detection is achieved by detecting high-frequency noise and streak interference; haze detection is achieved by calculating the dark channel mean, and when the mean is greater than a threshold, it is determined to be haze; strong light detection is achieved by counting the proportion of overexposed pixels, and when it is greater than 15%, it is determined to be strong light; low light detection is achieved by calculating the average brightness of the image, and when it is less than 30, it is determined to be low light. When in rain and snow, a lightweight GAN deraining model (such as the 0.5MB MobileGAN) is called to remove precipitation traces through a generative adversarial network; when in haze, a dark channel prior dehazing algorithm is used. The implementation process can be expressed as: ,in represents the atmospheric light value, represents a foggy image, Indicates transmittance. In bright light environments, images with different exposure times are collected and weighted fused based on saturation and contrast. The specific weighting coefficients can be set manually and are not restricted here. In low-light environments, 850nm infrared fill light is enabled, denoised using BM3D, and enhanced using a low-light enhancement network (such as LLNet).

[0072] This image enhancement strategy can adaptively process a variety of abnormal environments such as rain, snow, haze, strong light and low illumination at night, effectively eliminating traces of precipitation, improving contrast, suppressing overexposure and enhancing the brightness and details of low-light images, thereby significantly improving the clarity and recognizability of the target area, ensuring that high-quality image input can still be provided for subsequent detection and tracking under complex weather conditions.

[0073] It should be added that the pixel coordinates and confidence of the target in the image are determined by a lightweight target detection model. The implementation steps of this process are implemented by the YOLONano model (0.8MB INT8 format) deployed on the NPU of RK3568. The pre-trained model is loaded through RKNNToolkit, and the target bounding box and confidence are finally output.

[0074] S200: Based on the frame extraction strategy, some detection results are transmitted as key frames to the edge end, and the remaining detection results are cached locally. When it is detected that the network transmission delay exceeds a first threshold, a fallback mechanism is triggered, switching to the local tracking algorithm and reducing the update frequency of the gimbal control command;

[0075] Furthermore, the frame extraction strategy is as follows: key frames are selected at a preset frame interval from the continuously acquired multiple frames of images, and after downsampling the key frames to a target resolution lower than the original resolution, they are sent to the edge end via a transmission protocol with forward error correction function, while the remaining non-key frames are cached locally;

[0076] The fallback mechanism is: when it is detected that the network transmission delay exceeds a preset threshold, it switches to the locally running target tracking algorithm to predict the target position, and reduces the update frequency of the gimbal control instructions to a second preset value to maintain continuous tracking of the target under communication-restricted conditions.

[0077] Specifically, the front-end image acquisition module captures multiple frames of image data of the target scene at a fixed frame rate, which in this embodiment is 30 frames per second. The system selects key frames according to a preset frame extraction interval. In a feasible embodiment, one frame is selected as a key frame for every five frames of image captured, and the key frame is downsampled to the target resolution, for example, from the original resolution to 1280×720 pixels (720P) to reduce the data transmission bandwidth requirement. The downsampled key frames are sent to the edge via a high-speed transmission protocol with forward error correction (FEC) function. In one embodiment, a lightweight transmission protocol based on UDP can be used, and forward error correction coding is added to the protocol layer to reduce picture loss caused by data packet loss. The remaining non-key frames that are not selected are cached locally and used as reference data for local tracking when network communication is limited.

[0078] The implementation process of the fallback mechanism is as follows: the system continuously monitors the network transmission delay between the front end and the edge end, and when it detects that the delay value exceeds a first preset threshold (such as 50 milliseconds), it automatically switches to the locally running target tracking algorithm. In a feasible embodiment, the local tracking algorithm is a lightweight visual tracking method based on kernel correlation filtering (KCF), which can achieve real-time tracking while maintaining low computing resource usage. In fallback mode, the system reduces the update frequency of the gimbal control instructions from 30Hz in normal mode to a second preset value (such as 10Hz) to reduce the bandwidth usage of the control link and improve the stability of the system under communication-restricted conditions.

[0079] By combining the above-mentioned frame extraction strategy with the fallback mechanism, high-precision remote control is achieved when network conditions are good, and local autonomous tracking is achieved when network latency is high or unstable, thereby maintaining continuous and stable tracking of the target under different communication conditions.

[0080] S300: Acquires the real-time attitude information of the gimbal and compares it with the trajectory data of the edge end. When the attitude difference exceeds the preset angle threshold, attitude resynchronization is triggered. During the attitude calculation process, the visual detection data and inertial measurement data are integrated to predict the target motion to improve the attitude synchronization accuracy and anti-shake capability.

[0081] Furthermore, the target motion prediction step includes:

[0082] Obtain the target position coordinates and their confidence levels output by the visual detection model, and simultaneously collect angular velocity and acceleration data provided by the inertial measurement unit;

[0083] Taking the target position coordinates and inertial measurement data as input, the extended Kalman filter algorithm that fuses visual and inertial information is used to predict the target's next position at a certain moment;

[0084] The observation noise covariance is dynamically adjusted according to the target detection confidence to increase the weight of inertial data when the confidence decreases, thereby improving the stability and anti-interference ability of the prediction.

[0085] Specifically, the visual detection module processes the collected images and outputs the target's coordinates in the image coordinate system. At the same time, the inertial measurement unit (IMU) collects the angular velocity of the gimbal at a sampling frequency of 200Hz. With acceleration , and is zero-bias corrected and low-pass filtered to reduce noise.

[0086] The system takes the visual detection results and IMU data as input to construct a six-dimensional state vector ,in Indicates the target center position, represents the velocity component, Represents the acceleration component. The state prediction equation is:

[0087] ;

[0088] in, represents the state transition matrix, represents the control input matrix, Represents the control input consisting of angular velocity and acceleration provided by the IMU. Set the observation vector ,in Represents the center point position obtained by visual detection. The observation update equation is:

[0089] ;

[0090] ;

[0091] ;

[0092] in, represents the Kalman gain, represents the state covariance matrix, represents the observation matrix, represents the observation noise covariance matrix. In the filtering process, According to the detection confidence Adaptive adjustment, when When the visual observation noise component is increased, the inertial observation noise component is reduced, thereby increasing the weight of inertial information in the prediction results and improving the prediction stability and anti-interference ability when vision fails.

[0093] Through the above-mentioned extended Kalman filtering method based on the fusion of visual detection and inertial measurement, the system can maintain continuous and stable target motion prediction in cases of partial occlusion, insufficient lighting or motion blur, and provide high-precision position information input for the gimbal's feedforward and feedback control.

[0094] S400: Sub-pixel refinement of the target pixel coordinates is performed, and the target's three-dimensional spatial orientation is calculated in combination with the gimbal attitude parameters;

[0095] Furthermore, the steps for calculating the three-dimensional spatial orientation of the target are as follows: Figure 2 Shown, including:

[0096] Preprocessing the collected images, including denoising and contrast enhancement, to obtain clear images for precise positioning;

[0097] The center point position is refined using edge detection based on the Sobel operator combined with a sub-pixel interpolation method using a second-order Taylor expansion to improve the positioning accuracy to within ±0.1 pixels.

[0098] The refined target pixel coordinates are combined with the current gimbal posture parameters and the camera intrinsic parameter matrix, and the azimuth and pitch angles of the target relative to the gimbal are calculated through the three-dimensional space coordinate solution model.

[0099] Specifically, the target scene image acquired by the front-end system is first subjected to denoising and contrast enhancement to improve the accuracy and robustness of subsequent positioning. An adaptive median filtering method is first used to dynamically adjust the filter window size based on the image noise variance. When the noise variance is large, the window size is increased to enhance the smoothing effect, and when the noise variance is small, the window size is reduced to retain more edge details. A contrast-constrained adaptive histogram equalization method is then used to adjust the brightness distribution within a set contrast threshold, thereby improving the contrast between the target and the background while avoiding noise amplification caused by over-enhancement. After preprocessing, a clear image with well-preserved details is obtained, providing stable input for high-precision positioning.

[0100] Based on the center point position output by the target detection module, it is refined with sub-pixel accuracy. First, the Sobel operator is used to calculate the horizontal and vertical gradient components of the target center area, thereby obtaining edge direction information and gradient strength. Second, a second-order Taylor expansion is performed near the gradient extreme point, and sub-pixel interpolation of pixel-level coordinates is achieved by solving the extreme position of the fitting curve. This method improves the positioning accuracy of the target center point to within ±0.1 pixels, providing high-precision two-dimensional input for subsequent 3D solution.

[0101] The refined target pixel coordinates The camera's internal parameter matrix K and the gimbal's current attitude parameters Combined with the three-dimensional space coordinate solution model, the spatial orientation of the target is calculated. The three-dimensional space coordinate solution model can be described as:

[0102] ;

[0103] Solving this equation, we can get the azimuth angle of the gimbal and pitch angle .

[0104] High-quality images are obtained through denoising and contrast enhancement, and sub-pixel center point refinement is achieved by combining Sobel edge detection and second-order Taylor expansion. On the basis of improving positioning accuracy to ±0.1 pixel, three-dimensional coordinate solution is performed using camera intrinsic parameters and gimbal attitude parameters. The azimuth and pitch angles of the target relative to the gimbal can be accurately obtained, providing reliable spatial position information for subsequent high-precision tracking control.

[0105] S500: Performs feedforward prediction based on the target motion prediction model and combines it with a feedback controller to adjust the gimbal actuator in real time to achieve low-latency, high-precision target tracking.

[0106] Furthermore, the steps for real-time adjustment of the PTZ actuator are as follows: Figure 3 Shown, including:

[0107] Based on the target motion prediction model, the target's expected position at the next moment is calculated under the assumption of uniform motion to form a feedforward control instruction;

[0108] Combining the feedforward control instructions with visual detection and posture feedback data, and performing error correction through a PID controller with anti-saturation integral separation function;

[0109] The corrected control quantity is output to the pan-tilt actuator according to the preset sampling frequency to achieve real-time response and precise tracking of target position changes.

[0110] Specifically, the system first uses the target motion prediction model to estimate its expected position at the next moment using the current target position and velocity under the assumption of uniform motion. The prediction formula is: ,in, Indicates the current target position. represents the velocity vector of the target, The calculated predicted position is used to generate feedforward control commands, allowing the gimbal to make attitude adjustments in advance and reduce tracking delay.

[0111] The feedforward control instructions are combined with the real-time visual detection results and the gimbal attitude encoder feedback data to calculate the control error: ,in represents the predicted target position, Indicates the actual aiming position of the gimbal. Error correction is achieved through a PID controller, which has an anti-saturation integral separation function to prevent integral accumulation and overshoot problems caused by actuator saturation. The control quantity calculation formula is:

[0112] ;

[0113] in, 、 and These coefficients are the proportional, integral, and differential coefficients, respectively. These coefficients are determined using the Ziegler-Nichols tuning method to strike a balance between fast response and system stability. Control commands are updated at a 500Hz sampling frequency and sent to the gimbal actuator in real time via the drive interface, ensuring instant response to changes in target position.

[0114] Through the above-mentioned adjustment method that combines feedforward prediction with feedback control, the system can maintain high-precision tracking performance even when the target moves rapidly or changes direction suddenly. According to actual measurements, the delay from target movement to gimbal response is less than 80 milliseconds.

[0115] Furthermore, the real-time adjustment also includes abnormality detection and relocation, and the steps of abnormality detection and relocation are as follows: Figure 4 Shown, including:

[0116] Monitor the confidence level and response peak value changes of target detection. When the confidence level of multiple consecutive frames is lower than the preset threshold or the response peak value deviates from the historical mean by more than the preset standard deviation, the target tracking is determined to be lost.

[0117] When the target is determined to be lost, the gimbal is controlled to enter spiral scanning mode, performing a three-dimensional space search at a preset angular velocity and increasing radius;

[0118] The cached high-confidence target template is used for feature matching. When the number of matching points and the ratio of inliers exceed the preset threshold, the target is re-locked and the normal tracking mode is restored.

[0119] Specifically, during the target tracking process, the system continuously monitors the confidence level output by the target monitoring module and the peak amplitude in the tracking response graph. ), or when the deviation of the response peak value relative to its historical mean exceeds three times the standard deviation, the system determines that the current tracking state is abnormal, that is, the target may be blocked, lost, or the image is severely blurred.

[0120] When an anomaly detection result triggers a loss determination, the gimbal is controlled to execute a spiral scanning mode, searching within three-dimensional space. The scan starts at the last aiming position before the target was lost. The gimbal's azimuth rotates continuously at a preset angular velocity (e.g., 10° / s), and the scanning radius increases by a fixed step size after each cycle. Simultaneously, the pitch angle is gradually adjusted within a set range to cover the search space at different altitude levels. This scanning method effectively covers the gimbal's observable range in a shorter time, increasing the probability of target reacquisition.

[0121] During the scanning process, the system calls the most confident frame from the last 10 frames from the local cache as the target template. It uses a scale-invariant feature transform to extract key feature points from the template and the current frame, and establishes a correspondence through nearest neighbor matching. To ensure matching reliability, the system performs a random sampling consensus algorithm (RANSAC) on the matching results to remove incorrect matching points and calculate the inlier ratio. When the number of matching points exceeds 15 and the RANSAC inlier ratio is greater than 60%, the target is considered to have been re-identified. When the target is successfully recaptured, the system immediately switches back to normal target detection and feedforward + feedback control mode, stops spiral scanning, and updates the target tracking status.

[0122] This anomaly detection and relocalization mechanism can promptly determine tracking loss when the target confidence continues to decline or the response peak is abnormal, and quickly cover the search area through spiral scanning, combined with high-confidence template feature matching to achieve target recapture, thereby quickly restoring lock after tracking interruption, significantly improving the system's continuity in situations such as target occlusion and loss.

[0123] Furthermore, the anomaly detection and relocation also includes a long-term robustness guarantee mechanism, and the implementation strategy of the guarantee mechanism is:

[0124] A gated recurrent unit is used to determine the update timing of the target template. The template is updated only when the target appearance changes slowly and the detection confidence is higher than a preset value, preventing template aging and avoiding the erroneous introduction of background information.

[0125] When relocation attempts continue to fail for longer than a preset time, the system automatically switches to low-power standby mode, reduces the image acquisition frame rate and continuously monitors trigger signals, and resumes normal tracking mode after the target re-enters the detection range.

[0126] Specifically, during the target tracking process, the system uses a gated recurrent unit (GRU) neural network to determine whether to update the locally stored target template. The input to the GRU includes the currently detected target appearance feature vector, the target appearance feature vector at the previous moment, and the corresponding detection confidence. The GRU network has been trained to identify the rate and stability of target appearance changes, and triggers template updates only when the following two conditions are met simultaneously: the target appearance changes smoothly, that is, the cosine similarity of adjacent feature vectors is higher than the set threshold (such as 0.9); the current detection confidence is higher than the preset value (such as 0.8). When the above conditions are met, the system crops and normalizes the target area in the current frame image and replaces the original target template.

[0127] When the anomaly detection and relocalization module runs continuously and recapture failures persist for longer than a preset duration (e.g., 15 seconds), the system automatically switches to low-power standby mode. In low-power mode, the image acquisition frame rate is reduced from 30fps in normal mode to 5fps to reduce processing and transmission bandwidth. High-power inference modules (such as the target detection model on the NPU) are disabled, leaving only lightweight monitoring algorithms or sensor event monitoring modules active. External trigger signals (such as radar detection, infrared thermal imaging triggering, and acoustic sensor triggering) are used to determine whether the target has re-entered the detection range. Once an external trigger signal confirms that the target has re-entered the detection range, the system immediately returns to normal operating mode, re-enabling the full-precision target detection and tracking algorithm and adjusting the gimbal's attitude to the target's direction to quickly re-establish lock.

[0128] The present invention also proposes a high-precision PTZ control system, the structure of which is as follows: Figure 5 This system is used for UAV transmission line inspections in heavy rain. Equipped with this system, drones can achieve real-time, high-precision tracking and monitoring of transmission line components under complex weather conditions.

[0129] The drone cruised along the transmission line at an altitude of 50 meters, capturing 4K@30fps image streams of the transmission towers and insulators using its onboard camera. These images were then fed into the front-end processing unit via an image preprocessing module. In heavy rain (20 mm / h), the camera's field of view was obscured by numerous raindrops and mist. Upon detecting that the environment met the abnormal "rain and snow" conditions, the system automatically invoked a lightweight generative adversarial network model to remove precipitation artifacts from the image, significantly reducing the blur and flare caused by raindrops.

[0130] After preprocessing and target detection, the system identified the location of the insulator atop the transmission tower. The edge collaboration module downsampled key frames to 720P resolution, extracting one frame every five. These frames were then transmitted in real time to the ground edge processing station via the UDP protocol with forward error correction. The remaining non-critical frames were cached locally on the drone. During this inspection, the average network transmission latency fell below the first threshold (50ms), allowing the system to maintain normal edge collaboration mode without triggering the fallback mechanism.

[0131] The attitude synchronization module acquires attitude angle data from the gimbal encoder in real time and compares it with the global trajectory transmitted by the ground edge. If a deviation exceeding 2° is detected, attitude resynchronization is immediately performed. During attitude calculation, the system integrates visual position detection with angular velocity and acceleration data output by the IMU (200Hz). Using an extended Kalman filter, the system predicts the target's next position, maintaining stable tracking even when the drone is disturbed by wind and rain.

[0132] The target orientation acquisition module performs Sobel edge detection and second-order Taylor expansion sub-pixel interpolation on the insulator center point, improving positioning accuracy to ±0.1 pixel. Combining the camera's intrinsic parameter matrix with the gimbal's attitude parameters, the system calculates the insulator's azimuth and pitch angles in three-dimensional space.

[0133] The gimbal control module generates feedforward control commands based on a uniform motion prediction model and uses a PID controller with anti-windup integral separation to correct errors. The corrected control variables are output to the gimbal actuator at a frequency of 500Hz, achieving a latency of less than 80 milliseconds from target motion to gimbal response.

[0134] The measured results show that in static scenarios, the system positioning accuracy reaches ±0.05° (equivalent to ±4.4cm at a distance of 50 meters); in a rainstorm environment, the insulator target detection recall rate increases from 88% before processing to 91% after processing with the rain removal model, significantly improving the reliability of detection and tracking, and meeting the needs of UAVs for transmission line inspections in harsh weather conditions.

[0135] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A high-precision pan-tilt control method, characterized in that: include: Collect and preprocess the image data of the target scene, and determine the pixel coordinates and confidence of the target in the image through a lightweight target detection model. When a preset abnormal environment is detected, the image enhancement model is called to perform image enhancement; Based on the frame extraction strategy, some detection results are transmitted to the edge as key frames, and the remaining detection results are cached locally. When it is detected that the network transmission delay exceeds the first threshold, the fallback mechanism is triggered, switching to the local tracking algorithm and reducing the update frequency of the gimbal control instructions; Obtain the real-time attitude information of the gimbal and compare it with the trajectory data of the edge end. When the attitude difference exceeds the preset angle threshold, attitude resynchronization is triggered; During the attitude calculation process, visual detection data and inertial measurement data are integrated to predict target motion to improve attitude synchronization accuracy and anti-jitter capability; The target pixel coordinates are refined at the sub-pixel level and the target's three-dimensional spatial orientation is calculated based on the gimbal attitude parameters. Feedforward prediction is performed based on the target motion prediction model, and the gimbal actuator is adjusted in real time in combination with the feedback controller to achieve low-latency, high-precision target tracking.

2. A high-precision PTZ control method according to claim 1, characterized in that: The steps for preprocessing image data include: Perform adaptive filtering on the captured image and dynamically adjust the filtering parameters according to the image noise characteristics to suppress random noise and retain edge details; Perform geometric transformation on the image based on the camera calibration parameters to compensate for the radial and tangential distortion introduced by the lens to restore the real scene scale; The image is scaled according to the preset target resolution to ensure that the detailed information required for detection is retained while reducing the amount of data.

3. A high-precision PTZ control method according to claim 1, characterized in that: The abnormal environments include rain, snow, fog, haze, strong light and low illumination at night. The implementation logic of the image enhancement model is: When a rainy or snowy environment is detected, the image is processed using a generative adversarial network to remove precipitation marks to eliminate blur and light spots caused by raindrops and snowflakes. When a hazy environment is detected, the image is dehazed using a dark channel prior combined with color constancy enhancement to restore the contrast and color of the target area. When a bright light environment is detected, the image is processed with a high dynamic range through a multi-exposure image fusion algorithm to preserve details in the highlight area and suppress overexposure; When a low-light environment is detected at night, the image brightness is enhanced and noise is suppressed by combining infrared fill light acquisition with a denoising enhancement network to improve the signal-to-noise ratio and visibility of the image.

4. A high-precision PTZ control method according to claim 1, characterized in that: The frame extraction strategy is as follows: key frames are selected at a preset frame interval from multiple frames of continuously acquired images, and after downsampling the key frames to a target resolution lower than the original resolution, they are sent to the edge end via a transmission protocol with forward error correction function, while the remaining non-key frames are cached locally; The fallback mechanism is: when it is detected that the network transmission delay exceeds a preset threshold, it switches to the locally running target tracking algorithm to predict the target position, and reduces the update frequency of the gimbal control instructions to a second preset value to maintain continuous tracking of the target under communication-restricted conditions.

5. The high-precision PTZ control method according to claim 1, characterized in that: The target motion prediction step includes: Obtain the target position coordinates and their confidence levels output by the visual detection model, and simultaneously collect angular velocity and acceleration data provided by the inertial measurement unit; Taking the target position coordinates and inertial measurement data as input, the extended Kalman filter algorithm that fuses visual and inertial information is used to predict the target's next position at a certain moment; The observation noise covariance is dynamically adjusted according to the target detection confidence to increase the weight of inertial data when the confidence decreases, thereby improving the stability and anti-interference ability of the prediction.

6. A high-precision PTZ control method according to claim 1, characterized in that: The steps for calculating the three-dimensional spatial orientation of the target include: Preprocessing the collected images, including denoising and contrast enhancement, to obtain clear images for precise positioning; The center point position is refined using edge detection based on the Sobel operator combined with a sub-pixel interpolation method using a second-order Taylor expansion to improve the positioning accuracy to within ±0.1 pixels. The refined target pixel coordinates are combined with the current gimbal posture parameters and the camera intrinsic parameter matrix, and the azimuth and pitch angles of the target relative to the gimbal are calculated through the three-dimensional space coordinate solution model.

7. The high-precision PTZ control method according to claim 1, characterized in that: The steps for real-time adjustment of the PTZ actuator include: Based on the target motion prediction model, the target's expected position at the next moment is calculated under the assumption of uniform motion to form a feedforward control instruction; Combining the feedforward control instructions with visual detection and posture feedback data, and performing error correction through a PID controller with anti-saturation integral separation function; The corrected control quantity is output to the pan-tilt actuator according to the preset sampling frequency to achieve real-time response and precise tracking of target position changes.

8. A high-precision PTZ control method according to claim 7, characterized in that: The real-time adjustment further includes anomaly detection and relocation, and the steps of anomaly detection and relocation include: Monitor the confidence level and response peak value changes of target detection. When the confidence level of multiple consecutive frames is lower than the preset threshold or the response peak value deviates from the historical mean by more than the preset standard deviation, the target tracking is determined to be lost. When the target is determined to be lost, the gimbal is controlled to enter spiral scanning mode, performing a three-dimensional space search at a preset angular velocity and increasing radius; The cached high-confidence target template is used for feature matching. When the number of matching points and the ratio of inliers exceed the preset threshold, the target is re-locked and the normal tracking mode is restored.

9. A high-precision PTZ control method according to claim 8, characterized in that: The anomaly detection and relocation also includes a long-term robustness guarantee mechanism, the implementation strategy of which is: A gated recurrent unit is used to determine the update timing of the target template. The template is updated only when the target appearance changes slowly and the detection confidence is higher than a preset value, preventing template aging and avoiding the erroneous introduction of background information. When relocation attempts continue to fail for longer than a preset time, the system automatically switches to low-power standby mode, reduces the image acquisition frame rate and continuously monitors trigger signals, and resumes normal tracking mode after the target re-enters the detection range.

10. A high-precision pan-tilt control system, characterized in that: include: An image preprocessing module is used to collect and preprocess the image data of the target scene, and determine the pixel coordinates and confidence level of the target in the image through a lightweight target detection model. When a preset abnormal environment is detected, an image enhancement model is called to perform image enhancement; The edge collaboration module is used to transmit some detection results as key frames to the edge based on the frame extraction strategy, and the remaining detection results are cached locally. When it is detected that the network transmission delay exceeds a first threshold, the fallback mechanism is triggered, switching to the local tracking algorithm and reducing the update frequency of the gimbal control instructions; The attitude synchronization module is used to obtain the real-time attitude information of the gimbal and compare it with the trajectory data of the edge end. When the attitude difference exceeds the preset angle threshold, attitude resynchronization is triggered; During the attitude calculation process, visual detection data and inertial measurement data are integrated to predict target motion to improve attitude synchronization accuracy and anti-jitter capability; The target orientation acquisition module is used to refine the target pixel coordinates at the sub-pixel level and calculate the target's three-dimensional spatial orientation based on the gimbal attitude parameters; The gimbal control module is used to perform feedforward prediction based on the target motion prediction model and to adjust the gimbal actuator in real time in combination with the feedback controller to achieve low-latency, high-precision target tracking.

Images