Electronic image stabilization method based on closed-loop stable state of a pan-tilt head

Abstract

The application discloses an electronic image stabilization method based on a closed-loop stable state of a holder, and comprises the following steps: acquiring image frames and holder control state data and judging whether the holder is in a stable locking state; when the holder is in the stable locking state, detecting whether there is residual small-arc oscillation and estimating parameters thereof; performing delay compensation according to the estimated parameters, converting the residual angle into image compensation quantity; performing smoothing processing, compensation limitation and weight control on the image compensation quantity and outputting final compensation quantity; establishing an affine compensation matrix with a center interesting region as a reference, performing affine transformation and center cutting, and outputting the image frames. The application takes a PID stable state as a starting reference of EIS micro compensation and performs image end compensation with a center region as a priority only for the closed-loop residual small-arc oscillation, so that the problem of slight shaking of a picture after the holder is stably locked can be improved without adjusting hardware and PID parameters, and EIS and PID are prevented from interfering with each other.

Description

Technical Field

[0001] This invention belongs to the field of image stabilization technology, and in particular relates to an electronic image stabilization method based on the closed-loop stabilization state of a gimbal. Background Technology

[0002] With the development of applications such as drone aerial photography, handheld gimbal photography, and robot vision, the requirements for the stability of camera modules are increasing. As a mechanical stabilization device, the gimbal typically uses a PID controller to drive a motor to maintain the camera module in a predetermined posture. Theoretically, the image should be stable once the gimbal approaches the target angle and enters a stable state. However, in practice, small-radius residual oscillations still exist due to factors such as PID convergence behavior, control loop delay, motor resolution, mechanical friction, structural resonance, or micro-vibrations from the camera body. These oscillations have small amplitudes in the gimbal control angle, but when reflected in the image, they may cause slight center drift, back-and-forth shaking, or micro-jitter. Traditional solutions attempt to adjust PID parameters to reduce residual oscillations, but this may lead to problems such as controller oversensitivity, increased high-frequency oscillations, and increased power consumption. On the other hand, electronic image stabilization (EIS) technology performs global anti-shake compensation based on inter-frame motion estimation or inertial measurement unit (IMU) data, but it may misjudge normal camera movement as shaking, resulting in incorrect compensation. Furthermore, EIS and PID mechanical control may interfere with each other if applied to the same motion simultaneously. Summary of the Invention

[0003] To address the shortcomings of the prior art, the technical problem to be solved by the present invention is to provide an electronic image stabilization method based on the closed-loop stabilization state of a gimbal.

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution: An electronic image stabilization method based on gimbal closed-loop stabilization, characterized by the following steps: S100: Obtain image frames and gimbal control status data, and determine whether the gimbal mounted on the camera module is in a stable locked state based on the gimbal control status data. S200. When the gimbal is in the stable locked state, detect whether there is residual small arc oscillation of the gimbal and estimate its parameters. S300. When the gimbal has residual small arc oscillation, delay compensation is performed based on the estimated parameters of the residual small arc oscillation, and the residual angle of the residual small arc oscillation is converted into an image compensation amount. S400: After smoothing, limiting and controlling the compensation amount of the image, output the final compensation amount; S500. Based on the final compensation amount, establish an affine compensation matrix with the center region of interest of the image frame as the reference, and perform affine transformation and center cropping on the image frame to output a stable image frame. S600 automatically optimizes EIS compensation parameters.

[0005] Furthermore, step S100 includes: S110. Acquire the current image frame through the image sensor and record the exposure center time of the image frame; S120. Obtain gimbal control status data through gimbal controller, PID controller, inertial measurement unit and encoder; S130. Determine whether the gimbal meets multiple steady-state conditions based on the gimbal control status data, and confirm that the gimbal is in a stable locked state when at least two of the multiple steady-state conditions are simultaneously met and the duration of the met conditions is greater than a preset steady-state time threshold. In the process of determining whether the gimbal meets the multiple steady-state conditions, if an active camera movement command is detected or the gimbal is in the process of attitude change, it is determined that the gimbal is in normal camera movement operation and the image compensation is stopped or reduced. The image compensation is only used for post-processing at the image end and does not adjust the gimbal hardware or modify the control parameters of the PID controller.

[0006] Furthermore, step S200 includes: S210. Perform time-domain and frequency-domain analysis on at least one of the angular velocity signal, angle error signal, PID output signal or motor current command signal in the gimbal control status data, and confirm the existence of residual small-radian oscillation when both the time-domain analysis and frequency-domain analysis indicate the presence of periodic components. S220. The residual small-radian oscillation is modeled as a periodic signal, and the amplitude, frequency, phase and attenuation coefficient of the periodic signal are estimated. S230 independently performs residual vibration detection and parameter estimation for the pitch axis, yaw axis and roll axis respectively.

[0007] Furthermore, step S300 includes: S310. Calculate the phase delay based on the exposure center time of the image sensor and the total image processing delay, and predict the actual residual angle at the exposure center time based on the phase delay to compensate for the phase lag in the image processing link. S320. Bandwidth limitation is performed based on the estimated frequency of the residual small arc oscillation. Compensation is enabled when the frequency falls within a preset compensable frequency range, and compensation is stopped or reduced when the frequency falls outside the compensable frequency range. S330. Calculate the horizontal compensation, vertical compensation, and rotational compensation based on the yaw residual angle, pitch residual angle, and roll residual angle, respectively. The horizontal and vertical compensation are calculated based on the focal length parameters obtained from the camera's intrinsic parameter calibration.

[0008] Furthermore, the total image processing delay in step S310 is obtained by pre-measuring the processing pipeline delay of the image acquisition system or by real-time estimation based on the current processing load of the system.

[0009] Furthermore, step S400 includes: S410. The image compensation amount is filtered and smoothed using a preset smoothing coefficient, and the difference between the smoothed compensation amount and the original compensation amount is calculated as the compensation difference. S420: Apply maximum compensation limits in the corresponding directions to the horizontal compensation difference, vertical compensation difference, and rotational compensation difference, respectively. S430. Set an EIS compensation weight. Based on whether the gimbal is in the stable locked state and whether there is residual small arc oscillation, control the EIS compensation weight to gradually change between 0 and 1. The compensation difference is weighted by the EIS compensation weight to generate the final compensation amount.

[0010] Furthermore, step S500 includes: S510. Define the region centered on the geometric center of the image frame as the central region of interest, set the geometric center point of the central region of interest as the center of the affine transformation, and establish the affine transformation matrix according to the horizontal compensation amount, vertical compensation amount and rotation compensation amount in the final compensation amount. S520. Perform an affine transformation on the image frame according to the affine transformation matrix, and perform center cropping on the transformed image to eliminate black borders. S530 outputs the stabilized image frame.

[0011] Furthermore, in step S510, the size of the central region of interest is 50% of the width and 50% of the height of the image frame.

[0012] Furthermore, step S600 includes: S610. Obtain the gimbal operating conditions and EIS compensation results; S620. Calculate the compensation effect index based on the gimbal working conditions and the EIS compensation result, and determine that the EIS parameters need to be adjusted when the compensation effect index exceeds a preset threshold. S630. Based on the gimbal working conditions and the compensation effect index, adjust at least one of the EIS judgment threshold or compensation parameters to adapt to different working conditions, maintain the compensation effect, and form a closed-loop optimization. S640. Update the adjusted parameters in a smooth iterative manner and apply upper and lower limits to the updated parameters.

[0013] Furthermore, the EIS judgment threshold or the compensation parameter in step S630 includes at least one of the following: steady-state angle error threshold, angular velocity threshold, PID output judgment threshold, residual vibration amplitude threshold, residual vibration frequency range, maximum horizontal compensation amount, maximum vertical compensation amount, maximum rotational compensation amount, EIS compensation weight rise rate, and EIS compensation weight fall rate.

[0014] In this invention, using PID steady-state as the starting benchmark for EIS micro-compensation avoids erroneous compensation during normal camera movements. Furthermore, image compensation is treated solely as post-processing at the image end without adjusting PID parameters, thus decoupling EIS and PID to prevent mutual interference. Through comprehensive analysis in the time and frequency domains and axis-specific determination, precise micro-compensation can be performed for small-radius oscillations in the closed loop, avoiding unnecessary image distortion along axes without oscillation. Affine compensation based on the central region of interest prioritizes stabilizing the central area of ​​the image while reducing computational load and processing latency. Predicting the residual angle based on the exposure center time and total processing latency effectively reduces phase lag, bandwidth limitation avoids erroneous compensation, and progressive weight control prevents abrupt image changes. Through automatic optimization of EIS compensation parameters, the system can adapt to different operating conditions to maintain optimal compensation results. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of an embodiment of the electronic image stabilization method based on the gimbal closed-loop stable state of the present invention.

[0016] Figure 2 A flowchart for determining whether the gimbal is in a stable locked state.

[0017] Figure 3 A flowchart for detecting residual small-radian oscillations of the gimbal and estimating its parameters.

[0018] Figure 4 A flowchart for performing delay compensation and calculating image compensation amount.

[0019] Figure 5 This is a flowchart for calculating the final compensation amount.

[0020] Figure 6 A flowchart for outputting stabilized image frames.

[0021] Figure 7 This is a flowchart for automatically optimizing EIS compensation parameters. Detailed Implementation

[0022] The following specific examples illustrate the implementation of the present invention. The illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0023] Please see Figure 1 , Figure 1 This is a flowchart of an embodiment of the electronic image stabilization method based on gimbal closed-loop stabilization state according to the present invention. The electronic image stabilization method based on gimbal closed-loop stabilization state in this embodiment includes the following steps: S100: Obtain image frames and gimbal control status data, and determine whether the gimbal is in a stable locked state based on the gimbal control status data. In this step, the system can obtain the current image frame from the image sensor, and simultaneously obtain the gimbal control status data at the current or previous moment from related components such as the gimbal controller, PID controller, IMU, and encoder. The system is a gimbal image processing system that executes the electronic image stabilization method of this embodiment, which includes at least a portion of the aforementioned image sensor, camera lens, gimbal motor, gimbal controller, PID controller, inertial measurement unit, angle sensor or encoder, EIS micro-compensation module, image processor, and image output module, and is configured to execute the functions described in each step of this application.

[0024] The simultaneous acquisition of image frames and control status data in this step is necessary because subsequent steady-state determination and residual vibration detection both rely on this control status information. Since the final object of image compensation is the image frame, the two must correspond precisely in time to ensure the accuracy and effectiveness of the compensation. The control status data should at least include information sufficient to reflect the current control and motion states of the gimbal. In this embodiment, the control status data includes, but is not limited to: target angle, actual angle, angle control error, angular velocity, control command speed, PID output value, motor current command, and encoder feedback angle. Specifically, this step may include the following sub-steps: S110. Acquire the current image frame using the image sensor and record the exposure center time of the image frame. The image sensor can continuously acquire image frames according to a preset frame rate, and the acquired image frames are transmitted to the input buffer of the image processor. When the image processor receives the current frame, it records the reception time of the current frame and its corresponding exposure center time. The exposure center time refers to the midpoint of the exposure period for the image sensor, which can be calculated using the driving timing parameters of the image sensor (exposure start time and exposure duration) or read from the timestamp information provided by the sensor itself. The purpose of recording the exposure center time is to ensure that the actual exposure time of the image is accurately aligned with the sampling time of the control state data during subsequent delay compensation, thereby avoiding compensation errors caused by time misalignment.

[0025] S120. Gimbal control status data is acquired through the gimbal controller, PID controller, inertial measurement unit, and encoder. Synchronously or quasi-synchronously with the acquisition of image frames, the gimbal controller reads the target angle and actual angle of the current control cycle through the PID controller and calculates the difference as the angle control error. Simultaneously, the current angular velocity of the gimbal is read through the IMU (i.e., gyroscope output), the actual rotation angle of each axis is read through the encoder or angle sensor, the control quantity output by the PID controller (i.e., PID output value) is read, and the current attitude control command speed (i.e., control command speed) is acquired through the flight control system or user remote control command receiving module. All the above control status data is associated with the frame number or timestamp of the current image frame for use in subsequent steps. By binding the control status data to the image frame in time, the system can ensure that the control status data used in subsequent steady-state determination and residual vibration detection is time-aligned with the currently processed image frame, thereby improving the accuracy of determination and compensation.

[0026] S130. Based on the gimbal control status data, determine whether the gimbal meets multiple steady-state conditions, and when at least two of the multiple steady-state conditions are simultaneously met and the duration of the met conditions is greater than the preset steady-state time threshold, confirm that the gimbal is in a stable locked state.

[0027] The PTZ controller or EIS compensation module compares the control state data obtained in step S120 with preset steady-state judgment thresholds one by one. In this embodiment, the steady-state judgment is based on several steady-state conditions, including: the absolute difference between the target angle and the actual angle is less than a preset angle error threshold; the absolute value of the angular velocity is less than a preset angular velocity threshold; the absolute value of the control command speed is less than a preset control command speed threshold; and the absolute value of the PID output is less than a preset PID output judgment threshold.

[0028] The system confirms that the gimbal is currently in a stable locked state only when at least two of the above conditions are met simultaneously, and the duration of these conditions is greater than a preset steady-state time threshold. The purpose of introducing the duration condition is to prevent misjudgments caused by instantaneous noise or accidental measurement fluctuations. Only when all steady-state indicators remain stable over a continuous period of time can it be confidently concluded that the gimbal has indeed entered a steady state. In this embodiment, the steady-state time threshold can be set to the duration of 10 to 15 image frames.

[0029] In determining whether the gimbal meets multiple steady-state conditions, if an active camera movement command (i.e., a command to manually control the gimbal or drone to perform attitude changes such as rotation, translation, and pitch) is detected, or if the gimbal is undergoing an attitude change, the system determines that the gimbal is in normal camera movement operation and stops or reduces image compensation. In this case, image compensation is only used for post-processing at the image end without adjusting the gimbal hardware or modifying the control parameters of the PID controller. In this embodiment, when any of the following conditions are met, the system determines that the gimbal is in normal camera movement, target tracking, or a non-steady-state state, and accordingly disables or immediately stops residual vibration EIS micro-compensation: First, if the control command speed exceeds the aforementioned speed threshold, it indicates that the user or the autopilot system is actively operating the gimbal to perform panning, tilting, or rotating camera movements. Second, if the target angle continues to change, it indicates that the gimbal is tracking a moving target or scanning along a predetermined path. Third, if the error between the target angle and the actual angle exceeds the aforementioned angle threshold, it indicates that the gimbal is converging towards a new target position. Fourth, if the gimbal angular velocity exceeds the aforementioned angular velocity threshold, it indicates that the gimbal is rotating. Fifth, if the PID output value exceeds the upper limit of the preset small-amplitude control range, it indicates that the controller is outputting a large driving force to change the gimbal's attitude. Once any of the above five conditions is triggered, the system will directly skip the subsequent EIS micro-compensation process, or set the EIS compensation weight to zero in the weight control step described later, to ensure that image stabilization processing does not interfere with the photographer's creative intent. By adopting a normal camera movement exclusion mechanism, a balance can be achieved between the photographer's creative intent and the automatic stabilization function, with compensation only activated when the gimbal is in a passively locked state, thus avoiding undue interference with normal camera movements. The aforementioned image compensation is only used for post-processing at the image end without adjusting the gimbal hardware or modifying the control parameters of the PID controller. This achieves decoupling of EIS and PID at the control level, allowing the mechanical control end and the image processing end to operate independently without affecting each other.

[0030] S200. When the gimbal is in a stable locked state, detect whether there is residual small-arc oscillation of the gimbal, and estimate the parameters of the residual small-arc oscillation when it is detected. This step can further detect whether there is residual small-arc oscillation caused by the closed-loop control characteristics under steady-state conditions, and accurately estimate the mathematical characteristics of the oscillation. Detecting residual oscillation after steady-state determination is because residual oscillation is essentially a small disturbance superimposed on the steady state. If detection is performed under non-steady-state conditions, the normal large-amplitude attitude changes of the gimbal will completely mask the residual oscillation signal, making the detection results meaningless. Specifically, this step may include the following sub-steps: S210. Perform time-domain and frequency-domain analysis on at least one of the angular velocity signal, angle error signal, PID output signal, or motor current command signal in the gimbal control status data. If both the time-domain and frequency-domain analyses indicate the presence of a periodic component, confirm the existence of residual small-radian oscillations. In this step, perform time-domain and frequency-domain analysis on at least one of the angular velocity signal, angle error signal, PID output signal, or motor current command signal obtained in step S120 to determine whether small-amplitude periodic back-and-forth vibrations exist.

[0031] Time-domain analysis provides a direct view of signal amplitude characteristics, while frequency-domain analysis reveals hidden periodic components. The two complement each other, improving detection accuracy and robustness. In time-domain analysis, angle errors or encoder feedback angles are calculated, extracting peak values, root mean square values, or zero-crossing rates within a certain time window. If the signal exhibits regular zero-crossing alternations and the peak value exceeds a preset residual amplitude threshold, oscillation is preliminarily identified. The advantages of time-domain analysis are its simplicity, low latency, and suitability for real-time processing. In frequency-domain analysis, the IMU angular velocity signal undergoes a Fast Fourier Transform or other known spectral analysis methods (e.g., power spectral density estimation) to detect significant isolated peaks in the spectrum. If the energy at a certain frequency is significantly higher than the surrounding background noise level, the component corresponding to that frequency is identified as residual oscillation. Frequency-domain analysis effectively eliminates interference from non-periodic noise and accurately identifies the frequency components of oscillations. When both time-domain and frequency-domain analyses indicate the presence of periodic components, the system confirms the existence of residual small-radian oscillations. Through comprehensive judgment, the system can effectively reduce the false detection rate and avoid misjudging random noise as residual vibration that needs compensation.

[0032] S220. The residual small-radian oscillation is modeled as a periodic signal, and the amplitude, frequency, phase, and attenuation coefficient of the periodic signal are estimated. The estimated amplitude, frequency, and phase are used in subsequent steps to predict the actual residual angle at the exposure center moment. After confirming the existence of residual oscillation, the system models the residual oscillation as a mathematical model and estimates its parameters. In this embodiment, if there is no particular convergence trend, the residual angle signal is modeled as a sinusoidal form as follows:

[0033] Where θres(t) represents the residual angle at time t; A represents the amplitude of the residual angle signal; f represents the frequency of the residual angle signal; and φ represents the initial phase of the residual angle signal.

[0034] If the residual oscillations exhibit a gradually decaying characteristic (e.g., caused by a sudden disturbance), they can be modeled as having an exponentially decaying envelope:

[0035] Where β represents the attenuation coefficient. The purpose of parametric modeling of residual vibration is to, once A, f, With these parameters, the system can predict the residual angle at any future moment, thus providing a mathematical basis for forward-looking compensation. Based on the aforementioned spectral analysis results, the frequency corresponding to the peak value in the spectrum can be used as an estimate of the residual frequency f, thereby completing the frequency estimation; the accuracy of the frequency estimation directly affects the calculation accuracy of the phase delay in subsequent delay compensation. Based on the obtained frequency f, least-squares fitting or correlation analysis can be performed on the time-domain signal to separate the amplitude A from the phase. Specifically, the amplitude and phase can be calculated by correlating the time-domain signal with sine and cosine reference signals of the same frequency. If the model is in a decaying form, the decay coefficient β can be estimated by performing a logarithmic transformation and linear regression on the envelope of the oscillating signal. The estimation of the decay coefficient helps to determine the persistence characteristics of the residual oscillation. If the decay is too rapid, EIS compensation may not be necessary.

[0036] S230: Residual vibration detection and parameter estimation are performed independently for the pitch, yaw, and roll axes. If no residual vibration is detected on a particular axis, no image compensation is generated for that axis. In this embodiment, the system independently executes the detection and estimation processes of S210 and S220 for the pitch, yaw, and roll axes respectively. If the pitch axis is determined to be in a steady state and residual vibration exists, a subsequent compensation amount is generated in the vertical direction. If the yaw axis is determined to be in a steady state and residual vibration exists, a subsequent compensation amount is generated in the horizontal direction. If the roll axis is determined to be in a steady state and residual vibration exists, a subsequent compensation amount is generated in the rotational direction. If no residual vibration is detected on a particular axis, no EIS compensation amount is generated for that axis. This split-axis processing mechanism effectively avoids unnecessary image distortion on axes without oscillation, thereby minimizing the impact on the overall image quality while ensuring the stability of the central region.

[0037] S300. When the gimbal exhibits residual small-radian oscillations, delay compensation is performed based on the estimated parameters of the residual small-radian oscillations, and the residual angle of the residual small-radian oscillations is converted into an image compensation amount. The purpose of this step is to address the delay problem in the image processing link and convert the residual angle in physical space into a compensation amount in image pixel space. Specifically, this step may include the following sub-steps: S310. Calculate the phase lag based on the exposure center time of the image sensor and the total image processing delay, and predict the actual residual angle at the exposure center time based on the phase lag to compensate for the phase lag in the image processing chain. Since there is an inherent processing delay between the start of exposure and final processing by the EIS module, directly using the estimated residual angle at the current time for compensation will result in a phase lag in the compensated image relative to the actual exposure time, thus reducing the compensation effect or even producing the opposite effect.

[0038] To avoid the aforementioned problems, this step utilizes the exposure center time recorded in step S110 for forward prediction, ensuring that the compensation amount is precisely aligned with the actual exposure time of the image frame. Specifically, the system calculates the phase delay based on the exposure center time of the image sensor and the total image processing delay. The formula is as follows:

[0039] in, The residual frequency is estimated in step S220; Tdelay represents the total image processing delay. The actual residual angle at the exposure center time is then predicted based on the phase delay, as shown in the following formula:

[0040] in, This represents the residual vibration amplitude estimated in step S220. The residual phase estimated in step S220 is represented by t, where t is the current time, to compensate for the phase lag in the image processing pipeline. This prediction ensures that the calculated compensation corresponds to the actual image displacement at the exposure center moment of the image frame, thus avoiding the problem of phase lag compensation and significantly improving the accuracy of the compensation. The total image processing delay Tdelay is obtained either by pre-measuring the processing pipeline delay of the image acquisition system or by real-time estimation based on the current processing load of the system.

[0041] S320. Bandwidth limiting is applied based on the estimated frequency of the residual small-radian oscillation. Compensation is activated when the frequency falls within a preset compensable frequency range, and stopped or reduced when the frequency falls outside the compensable frequency range. To prevent EIS compensation from acting on non-target motion (e.g., extremely low-frequency slow drift or extremely high-frequency mechanical noise), this step further adjusts the compensation based on the estimated residual frequency of the residual small-radian oscillation. Perform gating control.

[0042] If the estimated residual frequency If the motion is less than the preset low-frequency threshold f_min (e.g., 1Hz), it is determined to be normal slow camera movement or extremely low-frequency posture change. The system classifies this part of the motion as uncompensable or should not be compensated, and forces the corresponding compensation weight to be set to zero or significantly attenuated. This is because extremely low-frequency motion usually belongs to intentional posture adjustment or slow environmental change, and compensating for it will destroy the naturalness of the picture.

[0043] If the estimated residual frequency If the frequency range is between f_min and the preset high-frequency threshold f_max (e.g., 20Hz), it is determined to be a compensable closed-loop residual oscillation. The system then initiates the normal EIS micro-compensation process. The compensable frequency range defined by f_min and f_max usually covers most of the residual oscillations caused by PID control characteristics and mechanical resonance.

[0044] If the estimated residual frequency If the frequency exceeds f_max, it is determined to be a high-frequency, uncompensable vibration (usually exceeding the gimbal control bandwidth). Such vibrations are often difficult to compensate effectively through image affine transformation and easily lead to image blurring. The system will reduce the compensation weight or stop compensation completely. Through this frequency band limitation mechanism, it is possible to effectively distinguish between compensable target residual vibrations and other motions that should not be compensated, thereby improving the targeting and effectiveness of compensation.

[0045] S330. Based on the yaw residual angle, pitch residual angle, and roll residual angle, calculate the horizontal compensation, vertical compensation, and rotation compensation, respectively. The horizontal and vertical compensation are calculated based on the focal length parameters obtained from the camera intrinsic parameter calibration. After the above delay compensation and bandwidth limitation, the system converts the predicted residual angles into pixel compensation amounts in the image frame. Let the obtained predicted residual angles for each axis be the yaw residual angle (yaw_res), pitch residual angle (pitch_res), and roll residual angle (roll_res), then this step converts them into horizontal translation (dx), vertical translation (dy), and rotation (da) on the image plane. The conversion relationship is as follows:

[0046]

[0047]

[0048] The negative sign in the above formula indicates that the compensation direction is opposite to the residual motion direction. That is, if the gimbal yaws clockwise, the image should be translated counterclockwise to counteract the residual motion, thereby achieving a visually stable center image. In the above formula, dx represents the horizontal compensation amount; dy represents the vertical compensation amount; da represents the rotation compensation amount; fx represents the focal length parameter in the horizontal direction (in pixels); and fy represents the focal length parameter in the vertical direction (in pixels).

[0049] The above parameters reflect the lens projection geometry, that is, the linear ratio between angle change and pixel displacement under small-angle approximation conditions. The focal length parameters fx and fy can be obtained in advance and stored in the system through offline camera intrinsic calibration (e.g., Zhang Zhengyou checkerboard calibration method); or, given the lens optical focal length (in millimeters) and sensor pixel size (micrometers / pixel), they can be calculated using the following formula:

[0050]

[0051] Where f_actual represents the actual optical focal length of the lens; pixel_size_x represents the physical size of the pixel in the horizontal direction; and pixel_size_y represents the physical size of the pixel in the vertical direction.

[0052] S400: After smoothing, limiting, and weighting the image compensation amount, the final compensation amount is output. After obtaining the original image compensation amounts dx, dy, and da, this step performs post-processing on the compensation amounts to prevent the compensation action itself from introducing new image abrupt changes or exceeding image boundaries. Specifically, this step may include the following sub-steps: S410. The image compensation amount is filtered and smoothed using a preset smoothing coefficient, and the difference between the smoothed compensation amount and the original compensation amount is calculated as the compensation difference. To avoid abrupt changes in the compensation amount between frames that could cause a jumpy feel in the output video, the system performs low-pass filtering on the compensation amount using a preset smoothing coefficient. In this embodiment, a first-order infinite impulse response (IIR) filter is used to process the original compensation amount for each axis, and the filtering formula is as follows:

[0053] Where raw[n] represents the original compensation amount of the current frame (i.e., dx, dy, or da); smooth[n-1] represents the compensation amount after smoothing the previous frame; smooth[n] represents the compensation amount after smoothing the current frame; α is the smoothing coefficient (0 < α < 1). The larger the value of α, the stronger the smoothing effect and the smoother the trajectory, but the slower the response speed, which may introduce some lag; the smaller the value of α, the faster the response, but high-frequency jitter may remain, affecting the smoothness of the compensation. In this embodiment, the optimal value range of α is 0.3 to 0.9, and the specific value can be adjusted according to the actual shaking characteristics of the image.

[0054] After this smoothing process, the system can effectively filter out high-frequency noise components in the compensation amount, making the compensation trajectory smoother and more natural. Subsequently, the system calculates the difference between the smoothed compensation amount smooth[n] and the original compensation amount raw[n] as the compensation difference corr[n]. The compensation difference corr[n] represents the compensation increment that needs to be gradually introduced in order to achieve a smooth trajectory. In this way, the system can achieve a smooth compensation transition without sacrificing response speed.

[0055] S420. Apply maximum compensation limits in the corresponding directions to the horizontal compensation difference, vertical compensation difference, and rotational compensation difference, respectively. To prevent excessive black edges or severe geometric distortion in the image due to large-scale compensation, the system further applies maximum compensation limits in the corresponding directions to the horizontal compensation difference, vertical compensation difference, and rotational compensation difference, as follows:

[0056]

[0057]

[0058] Here, corr_x, corr_y, and corr_a represent the compensation differences in the horizontal, vertical, and rotational directions, respectively. The clamp function is defined as limiting the input value (first parameter) to a specified lower limit (second parameter) and upper limit (third parameter), i.e., if it is less than the lower limit, the lower limit is used; if it is greater than the upper limit, the upper limit is used; if it is in between, the input value itself is used. max_x represents the maximum compensation amount in the horizontal direction, max_y represents the maximum compensation amount in the vertical direction, and max_a represents the maximum compensation amount in the rotational direction. By setting the maximum compensation limit, the system can control the black border ratio of the compensated image within an acceptable range, avoiding severe cropping or obvious distortion of the image due to overcompensation. In this embodiment, max_x and max_y can be set to a certain proportion of the image width and height (e.g., 2% to 8%), and max_a can be set to 0.5 degrees to 1.5 degrees, the specific values ​​depending on the preset allowable black border ratio and the lens field of view.

[0059] S430. Set an EIS compensation weight. Based on whether the gimbal is in a stable locked state and whether there is residual small arc oscillation, control the EIS compensation weight to gradually change between 0 and 1. The compensation difference is weighted by the EIS compensation weight to generate the final compensation amount.

[0060] To avoid screen flickering during the activation or deactivation of the EIS function, this step introduces a smoothly varying EIS compensation weight w, with a value ranging from 0 to 1. The dynamic change logic of the compensation weight w is as follows: When a stable locking state is determined, and residual small-radian oscillations are confirmed with frequencies within the compensable range, the weight w gradually increases from 0 to 1 at a preset rate. When normal camera movement is determined, residual small-radian oscillations disappear, or frequencies exceed the compensable range, the weight w gradually decreases from its current value to 0 at a preset rate. The final compensation amount applied to the image frame is...

[0061]

[0062]

[0063] Wherein, final_corr_x, final_corr_y, and final_corr_a represent the final compensation amounts in the horizontal, vertical, and rotational directions, respectively. This embodiment employs a weighted control mechanism, with the intervention and withdrawal of EIS micro-compensation proceeding gradually, thereby significantly improving the smoothness of the visual experience and making the start and stop of the stabilization function almost imperceptible to the viewer.

[0064] S500. Based on the final compensation amount, establish an affine compensation matrix using the central region of interest (ROI) of the image frame as a reference, and perform affine transformation and center cropping on the image frame to output a stabilized image frame. This step focuses on stabilizing the central region of the image, rather than pursuing global optimization across the entire frame. This step prioritizes stabilizing the central region because in most photographic compositions, the viewer's visual focus is usually concentrated in or near the center of the image. Compensating based on the central ROI achieves the best visual stabilization effect with limited computational resources, while minor deformations or cropping at the edges have a relatively small impact on the overall visual experience. Specifically, this step may include the following sub-steps: S510. Define the region centered on the geometric center of the image frame as the central region of interest, set the geometric center point of the central region of interest as the center of the affine transformation, and establish the affine transformation matrix based on the horizontal compensation, vertical compensation and rotation compensation in the final compensation amount.

[0065] The system first defines the region centered on the geometric center of the image frame as the central region of interest (ROI). For example, the ROI can be set as a rectangular region equal to 50% of the image frame's width and 50% of its height. This step sets the geometric center of the ROI as the center of the affine transformation. Then, based on the horizontal compensation values ​​`final_corr_x`, vertical compensation values ​​`final_corr_y`, and rotational compensation values ​​`final_corr_a` calculated in step S430, a 2×3 affine transformation matrix M is constructed, as follows:

[0066] The affine transformation matrix M defines the mapping relationship of the image frame in the horizontal, vertical, and rotational directions. By setting the center of the affine transformation at the center of the region of interest rather than the geometric center of the image frame, the image offset within the region of interest can be minimized when compensating for rotation, thereby ensuring the stability of the central region.

[0067] S520. Perform an affine transformation on the image frame according to the affine transformation matrix, and center-crop the transformed image to eliminate black borders. The image processor performs an affine transformation on the original input image frame according to the above affine transformation matrix (e.g., through the warpAffine function in the OpenCV library or an equivalent hardware acceleration unit). After the transformation, blank areas (black borders) will appear at the edges of the image due to compensation rotation and translation. In this embodiment, in order to eliminate black borders, the system centers-crops the transformed image. The cropping size is predetermined based on the maximum compensation amount and a preset retention ratio to ensure that the cropped image does not contain any black borders. The center-cropping operation and the affine transformation can be combined into a one-time coordinate mapping calculation to reduce processing latency and improve real-time performance.

[0068] S530, Output the stabilized image frame. The image frame obtained after affine transformation and cropping is transmitted to the image output module. The image output module encodes the stabilized image frame into a standard video stream (e.g., H.264 or HEVC format) or directly outputs it to a display device or storage medium, thus completing the entire image stabilization process. Through the above processing, the final output image has significantly improved stability in the central area, while the impact of slight cropping or distortion in the edge areas on the overall visual experience is controlled within an acceptable range.

[0069] In another preferred embodiment of this application, the following optional steps may also be included: S600: Automatically optimize EIS compensation parameters. This step is independent of the gimbal mechanical control and only affects parameter adjustments at the image end. Therefore, it will not affect the mechanical stability of the gimbal or change the response characteristics of the PID controller. Its purpose is to enable the system to adapt to different operating conditions (such as different lens focal lengths, load weights, or ambient temperatures), thereby maintaining optimal compensation effects under various conditions. Specifically, this step may include the following sub-steps: S610. Obtain the gimbal operating conditions and EIS compensation results. The operating conditions include, but are not limited to: current camera load weight, lens focal length or field of view, gimbal operating mode, motor temperature, motor current, and historical residual vibration statistics. The EIS compensation results include, but are not limited to: residual shake in the center of the image after compensation, black border ratio, image jitter, and changes in residual vibration frequency and amplitude; the system can calculate whether parameter adjustments are needed based on this information. By continuously monitoring the above operating conditions and compensation results, the system can promptly detect the impact of changes in operating conditions on the compensation effect, thereby triggering the parameter optimization process when necessary.

[0070] S620. Calculate the compensation effect index based on the PTZ operating conditions and EIS compensation results, and determine that the EIS parameters need to be adjusted when the compensation effect index exceeds a preset threshold. The system establishes a comprehensive scoring mechanism to quantify the current compensation effect. Relevant indicators for calculating the comprehensive score include the residual shaking amount in the center of the image after compensation, the black border ratio, the degree of image jitter, and the degree of overcompensation. The comprehensive score is defined as:

[0071]

[0072] Among them, `residual_shake` represents the residual shaking amplitude in the center of the image after compensation, i.e., the residual shaking amount after compensation. The larger the value, the less effective the compensation. `w1` represents the weight coefficient of the residual shaking amount after compensation. `black_border_ratio` represents the proportion of the black border area appearing at the edge of the image after compensation to the entire image frame area, i.e., the black border ratio after compensation. The larger the value, the more invalid pixel areas are generated due to compensation translation or rotation. `w2` represents the weight coefficient of the black border ratio after compensation. `jump_score` represents the degree of image jump caused by a sudden change in compensation amount between adjacent image frames. The larger the value, the more discontinuous the image appears. `w3` represents the weight coefficient of the image jump degree. `over_comp_score` represents the degree to which the compensation amount exceeds the actual required compensation amount, resulting in reverse shaking of the image, i.e., the overcompensation degree. The larger the value, the more severe the overcompensation phenomenon. `w4` represents the weight coefficient of the overcompensation degree.

[0073] The four weighting coefficients mentioned above are all preset positive numbers used to adjust the relative importance of each indicator in the overall score. Specific values ​​can be adjusted based on the visual evaluation results. For example: when the evaluation results show that the center image still has significant shaking after compensation, the weight w1 corresponding to `residual_shake` can be increased; when the evaluation results show obvious black borders, cropping, or edge distortion, the weight w2 corresponding to `black_border_ratio` can be increased; when the evaluation results show obvious image jumps between adjacent image frames, the weight w3 corresponding to `jump_score` can be increased; when the evaluation results show that the compensated image is unnatural, has reverse shaking, or appears to be overcompensated, the weight w4 corresponding to `over_comp_score` can be increased.

[0074] When the score exceeds the preset scoring threshold `score_th`, or when any single metric among `residual_shake`, `black_border_ratio`, `jump_score`, and `over_comp_score` exceeds its corresponding abnormal threshold, the system determines that the current EIS parameters need adjustment. By introducing this comprehensive scoring mechanism, the current compensation effect can be evaluated from multiple dimensions, avoiding the potential bias that may result from relying solely on a single metric. The calculation methods for these parameters are illustrated below.

[0075] The residual sway after compensation can be calculated as follows: Using the most recent N image frames as the statistical window and the central region of interest as the primary evaluation region, the ORB feature point method is used to calculate the compensated residual shake. Specifically, in the compensated adjacent image frames, ORB feature points and their descriptors are extracted from the central region of interest, and feature point matching is performed based on the Hamming distance between the descriptors. The matching results can be further processed through cross-alignment, distance ratio testing, or RANSAC outlier removal to obtain a set of effectively matched ORB feature points.

[0076] Let the k-th frame and the (k)-th frame be... 1) The set of ORB feature points (hereinafter referred to as matching points) that are effectively matched between frames is M[k]. Let the i-th matching point be in the (k)th frame. 1) The coordinates p_i[k] in the frame 1] is (x_i[k 1],y_i[k 1]); The corresponding coordinate p_i[k] in the k-th frame is (x_i[k], y_i[k]). Then the residual displacement of the i-th matching point can be expressed as: Δx_i[k]=x_i[k] x_i[k 1] Δy_i[k]=y_i[k] y_i[k 1] Where Δx_i[k] represents the residual horizontal displacement of the i-th matching point, and Δy_i[k] represents the residual vertical displacement of the i-th matching point. To reduce the impact of moving objects, noisy matching, or local mismatches on the evaluation results, the median of the residual horizontal displacements of the k matching points in the matching point set M[k] can be taken to obtain the residual horizontal displacement δx_orb[k] of the central region of the k-th frame; the median of the residual vertical displacements of the k matching points in the matching point set M[k] can be taken to obtain the residual vertical displacement δy_orb[k] of the central region of the k-th frame. Then, the residual sway r_orb[k] of the central region of the k-th frame can be defined as:

[0077] Therefore, within the statistical window of the most recent N image frames, the original residual shake amount (residual_shake_raw) of the center image after compensation can be defined as:

[0078] Where N represents the number of image frames in the statistical window, and k represents the image frame number.

[0079] To avoid the impact of dimensional differences between different indicators on the overall score, `residual_shake_raw` can be normalized to obtain the compensated residual sway `residual_shake`, which can be used for weighted calculation. For example, the calculation method can be as follows:

[0080] Where R_th represents the normalization threshold of the residual sway after compensation, which can be obtained through experimental calibration. In the above formula, the clamp function can limit the output value to the range of 0 to 1, that is: output 0 when the value of the first parameter is less than 0, output 1 when the value of the first parameter is greater than 1, and directly output the value of the first parameter when the value of the first parameter is between 0 and 1.

[0081] The original black border ratio (black_border_ratio_raw) of the compensated image edge can be calculated based on the area of ​​invalid pixels after affine transformation and the total area of ​​the output image. The calculation formula is as follows:

[0082] Where A_invalid represents the area of ​​the black border region or invalid pixel region caused by insufficient translation, rotation or cropping after affine transformation; A_frame represents the total area of ​​the output image frame; black_border_ratio_raw represents the original black border ratio of the image edge after compensation.

[0083] To avoid the impact of differences in the units of measurement of different indicators on the overall score, black_border_ratio_raw can be normalized to obtain the compensated black border ratio black_border_ratio that can be used for weighted calculation. For example, the calculation method can be as follows:

[0084] Wherein, B_th represents the normalization threshold of the black border ratio after compensation, which can be obtained through experimental calibration.

[0085] The jump score is calculated based on the change in final compensation between adjacent frames. Let the final horizontal compensation of the k-th frame be fin_x[k], and the final vertical compensation be fin_y[k]; 1) The final horizontal compensation amount of a frame is fin_x[k-1], and the final vertical compensation amount is fin_y[k-1]. Then, the compensation change between adjacent frames can be defined as:

[0086] Where ΔC[k] represents the relationship between the k-th frame and the (k-th)-th frame. 1) The combined change of the final horizontal compensation amount and the final vertical compensation amount between frames.

[0087] The raw image jitter level (jump_score_raw) can be defined as:

[0088] To avoid the impact of differences in the units of measurement of different indicators on the overall score, jump_score_raw can be normalized to obtain the screen bounce level jump_score that can be used for weighted calculation. For example, the calculation method can be as follows:

[0089] Here, J_th represents the normalized threshold for the degree of screen jitter, which can be obtained through experimental calibration.

[0090] The overcompensation level, over_comp_score, can be calculated based on the change in center frame shake before and after compensation. Let the amount of center frame shake before compensation be shake_before, and the residual center frame shake after compensation be shake_after. Then, the formula for calculating the original overcompensation level, over_comp_score_raw, can be as follows:

[0091] The function `max` takes the maximum of the two parameters as the output value. `ε` is a very small positive number set to prevent the denominator from being zero; for example, `ε = 0.001`. Of course, the value of `ε` can also be set to other values ​​according to actual needs. When `shake_after` is greater than `shake_before`, it indicates that EIS compensation actually increases image shake, possibly due to overcompensation or reverse compensation. When `shake_after` is less than or equal to `shake_before`, `over_comp_score_raw` is zero or close to zero.

[0092] The calculation method for the center image shake amount before compensation is as follows: First, obtain multiple consecutive image frames before compensation as statistical windows, and then use the same calculation method as the original residual shake amount residual_shake_raw to calculate the center image shake amount before compensation, shake_before.

[0093] The calculation method for the residual shake amount (shake_after) in the center image after compensation is as follows: First, obtain multiple consecutive image frames after compensation as statistical windows, and then calculate the residual shake amount (shake_after) in the center image after compensation using the same calculation method as the original residual shake amount (residual_shake_raw).

[0094] For example: Assume that the sway of the center ROI before compensation is shake_before = 5 pixels; After compensation, the center ROI sway is shake_after = 2 pixels; Then over_comp_score_raw=max(0,2 5) / 5=0; The representative stated that no compensation had been paid, therefore the compensation was valid.

[0095] If the sway of the center ROI after compensation is shake_after=8 pixels; Then over_comp_score_raw=max(0,8 5) / 5 = 0.6; The fact that the water is more shaky after compensation compared to before compensation suggests that compensation may have been paid previously.

[0096] To avoid the influence of differences in the dimensions of different indicators on the overall score, over_comp_score_raw can be normalized to obtain the overcompensation degree over_comp_score that can be used for weighted calculation. For example, the calculation method can be as follows:

[0097] Where O_th represents the normalization threshold of the overcompensation level, which can be obtained through experimental calibration.

[0098] S630. Based on the gimbal's operating conditions and compensation effect indicators, adjust at least one of the EIS judgment threshold or compensation parameters to adaptively maintain optimal compensation effects under different operating conditions and form a closed-loop optimization. The purpose of this step is: Based on the real-time monitoring of the PTZ operating conditions and the quantitative evaluation results of the current compensation effect, the judgment threshold or compensation parameters at the EIS end are adaptively adjusted to ensure that the system can maintain a good compensation effect under different operating conditions. By introducing operating conditions as the basis for adjustment, the EIS parameters can be matched with the current operating conditions, avoiding the problem of insufficient or excessive compensation in different scenarios when using fixed parameters. At the same time, a closed-loop optimization mechanism of "evaluation-adjustment-re-evaluation" is formed based on the compensation effect index, so that the compensation effect continuously approaches the optimal state. All the above adjustments are limited to the EIS end parameters and do not involve hardware or PID modifications, in order to maintain the integrity of the EIS and PID decoupling architecture. According to the judgment result of step S620 and the operating condition information obtained in step S610, the system executes the corresponding parameter adjustment logic.

[0099] When the system is determined to be undercompensated (excessive residual sway), it can lower the residual vibration amplitude threshold, expand the residual vibration frequency range (i.e., lower the low-frequency threshold or raise the high-frequency threshold), increase the maximum horizontal / vertical / rotational compensation, or accelerate the EIS weight rise rate. These adjustments are designed to enhance the compensation strength of the EIS to more effectively suppress residual sway.

[0100] When the system determines that the compensation is too strong (the black border ratio or the screen jitter is too large), it can raise the threshold for the residual vibration amplitude, or narrow the residual vibration frequency range (i.e., raise the low frequency threshold or lower the high frequency threshold), or reduce the maximum compensation amount, or reduce the EIS weight rise speed and accelerate the weight fall speed. These adjustments are intended to weaken the compensation intensity of EIS to reduce the generation of black borders and screen jitter.

[0101] When a longer focal length (narrower field of view) is detected, the system automatically lowers the residual vibration trigger threshold and increases the smoothness of compensation (increasing the smoothness coefficient α). Since residual vibration at the same angle under a longer focal length will produce a larger pixel displacement, a more sensitive trigger condition and a smoother compensation trajectory are required.

[0102] When a wide-angle lens (large field of view) is detected, the system raises the residual vibration threshold and limits the amount of rotation compensation. Since residual vibration at the same angle under a wide-angle lens has a smaller impact on the center image, but rotation compensation can easily cause obvious distortion at the edges, the compensation intensity should be appropriately reduced and rotation compensation should be limited first.

[0103] When an abnormal increase in motor temperature or current is detected, the system relaxes the steady-state angle error judgment threshold, increases the steady-state duration requirement, and actively reduces the EIS weight rise rate and maximum compensation amount to avoid excessive interference with the image when the mechanical load is abnormal, thus protecting the hardware while maintaining basic image stabilization functions.

[0104] The aforementioned EIS judgment thresholds or compensation parameters include at least one of the following: steady-state angle error threshold, angular velocity threshold, PID output judgment threshold, residual vibration amplitude threshold, residual vibration frequency range, maximum horizontal compensation amount, maximum vertical compensation amount, maximum rotational compensation amount, EIS compensation weight rise rate, and EIS compensation weight fall rate.

[0105] S640. Update the adjusted parameters using a smooth iterative approach and apply upper and lower limits to the updated parameters. To prevent frequent parameter mutations from causing oscillations in the compensation behavior, the system does not directly apply the new parameter param_est estimated in step S630, but instead uses a smooth iterative approach to update it, as shown in the following formula:

[0106] Here, `param_old` represents the current parameter, `param_new` represents the updated parameter, and `λ` represents the preset smoothing coefficient (0 < λ < 1). This smoothing update method ensures that the parameter changes gradually rather than abruptly, thus avoiding drastic changes in compensation effect caused by parameter mutations. Furthermore, the system presets engineering safety upper and lower limits for all parameters (e.g., minimum / maximum values ​​for angle error thresholds, minimum / maximum values ​​for maximum compensation, etc.) to ensure that the updated parameters do not exceed these limits. The smoothly updated parameters are used in the EIS processing flow of subsequent frames to achieve adaptability to changing working conditions, thereby enabling the system to maintain stable and effective image compensation performance under different usage environments. The above-mentioned automatic optimization of EIS compensation parameters does not adjust the gimbal hardware, motor, or gimbal mechanism, nor does it modify the control parameters of the PID controller; it only adjusts the EIS judgment threshold or compensation parameters.

[0107] By performing the above steps, it is possible to accurately identify and compensate for minor residual oscillations caused by mechanical and control characteristics when the gimbal closed-loop control is in a steady state. At the same time, through rigorous steady-state determination, camera movement exclusion, delay compensation, and progressive control, the naturalness and reliability of the image stabilization effect are ensured.

[0108] This embodiment has at least the following technical effects: (1) In this embodiment, the steady-state judgment of PID is used as the starting basis for EIS micro-compensation. Residual vibration compensation is only started after the gimbal enters a steady state (as defined by step S130 and active camera movement judgment). Therefore, EIS compensation can be avoided for normal gimbal camera movement. By treating image compensation only as post-processing at the image end without adjusting the gimbal hardware and without modifying the control parameters of the PID controller, the decoupling of EIS and gimbal PID control can be achieved, avoiding mutual interference between the two.

[0109] (2) In this embodiment, by performing time-domain and frequency-domain comprehensive analysis and parameter estimation on the residual oscillation under steady state, it can accurately perform micro-compensation for the closed-loop residual small arc oscillation under steady state of PID, rather than performing general anti-shake compensation for all image motions. At the same time, by independently determining the axis, unnecessary image distortion can be avoided for the axis without oscillation.

[0110] (3) In this embodiment, affine compensation is performed based on the central region of interest, prioritizing the stabilization of the central region of the image, which conforms to the attention characteristics of human vision. Furthermore, compensation is only performed for residual small arc oscillations under steady state rather than performing complex motion estimation and compensation for the entire image, which can reduce the amount of computation and image processing delay.

[0111] (4) This embodiment predicts the residual angle based on the exposure center time of the image frame and the total delay of image processing, which can effectively reduce the compensation phase lag problem caused by processing delay; by limiting the compensation frequency band, it can avoid false compensation for normal slow camera movement or high frequency mechanical noise; by using progressive weight control, it can avoid sudden changes in compensation amount that cause sudden image jumps.

[0112] (5) In this embodiment, the system can adapt to different gimbal working conditions (such as different lens focal lengths, load weights, motor temperatures, etc.) through automatic optimization of EIS compensation parameters, thereby maintaining a better compensation effect under various working conditions. Moreover, during automatic optimization, only the judgment threshold and compensation parameters of the EIS end are adjusted without any hardware adjustment or PID parameter modification, ensuring the integrity of the architecture of EIS and PID decoupled control.

[0113] The above embodiments merely illustrate preferred embodiments of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention should be determined by the appended claims.

Claims

1. An electronic image stabilization method based on gimbal closed-loop stabilization, characterized in that, Includes the following steps: S100: Obtain image frames and gimbal control status data, and determine whether the gimbal mounted on the camera module is in a stable locked state based on the gimbal control status data. S200. When the gimbal is in the stable locked state, detect whether there is residual small arc oscillation of the gimbal and estimate its parameters. S300. When the gimbal has residual small arc oscillation, delay compensation is performed based on the estimated parameters of the residual small arc oscillation, and the residual angle of the residual small arc oscillation is converted into an image compensation amount. S400: After smoothing, limiting and controlling the compensation amount of the image, output the final compensation amount; S500. Based on the final compensation amount, establish an affine compensation matrix with the center region of interest of the image frame as the reference, and perform affine transformation and center cropping on the image frame to output a stable image frame. S600: Automatically optimize EIS compensation parameters; The S200 step includes: S210. Perform time-domain and frequency-domain analysis on at least one of the angular velocity signal, angle error signal, PID output signal or motor current command signal in the gimbal control status data, and confirm the existence of residual small-radian oscillation when both the time-domain analysis and frequency-domain analysis indicate the presence of periodic components. S220. The residual small-radian oscillation is modeled as a periodic signal, and the amplitude, frequency, phase and attenuation coefficient of the periodic signal are estimated. S230 independently performs residual vibration detection and parameter estimation for the pitch axis, yaw axis and roll axis respectively.

2. The electronic image stabilization method based on gimbal closed-loop stabilization state according to claim 1, characterized in that, The S100 step includes: S110. Acquire the current image frame through the image sensor and record the exposure center time of the image frame; S120. Obtain gimbal control status data through gimbal controller, PID controller, inertial measurement unit and encoder; S130. Determine whether the gimbal meets multiple steady-state conditions based on the gimbal control status data, and confirm that the gimbal is in a stable locked state when at least two of the multiple steady-state conditions are simultaneously met and the duration of the met conditions is greater than a preset steady-state time threshold. The multiple steady-state conditions include: the absolute difference between the target angle and the actual angle is less than a preset angle error threshold; the absolute value of the angular velocity is less than a preset angular velocity threshold; the absolute value of the control command velocity is less than a preset control command velocity threshold; and the absolute value of the PID output is less than a preset PID output judgment threshold. In the process of determining whether the gimbal meets the multiple steady-state conditions, if an active camera movement command is detected or the gimbal is in the process of attitude change, it is determined that the gimbal is in normal camera movement operation and the image compensation is stopped or reduced. The image compensation is only used for post-processing at the image end and does not adjust the gimbal hardware or modify the control parameters of the PID controller.

3. The electronic image stabilization method based on gimbal closed-loop stabilization state according to claim 1, characterized in that, The S300 step includes: S310. Calculate the phase delay based on the exposure center time of the image sensor and the total image processing delay, and predict the actual residual angle at the exposure center time based on the phase delay to compensate for the phase lag in the image processing link. S320. Bandwidth limitation is performed based on the estimated frequency of the residual small arc oscillation. Compensation is enabled when the frequency falls within a preset compensable frequency range, and compensation is stopped or reduced when the frequency falls outside the compensable frequency range. S330. Calculate the horizontal compensation, vertical compensation, and rotational compensation based on the yaw residual angle, pitch residual angle, and roll residual angle, respectively. The horizontal and vertical compensation are calculated based on the focal length parameters obtained from the camera's intrinsic parameter calibration.

4. The electronic image stabilization method based on gimbal closed-loop stabilization state according to claim 3, characterized in that, The total image processing delay mentioned in step S310 is obtained by pre-measuring the processing pipeline delay of the image acquisition system or by real-time estimation based on the current processing load of the system.

5. The electronic image stabilization method based on gimbal closed-loop stabilization state according to claim 1, characterized in that, The S400 step includes: S410. The image compensation amount is filtered and smoothed using a preset smoothing coefficient, and the difference between the smoothed compensation amount and the original compensation amount is calculated as the compensation difference. S420: Apply maximum compensation limits in the corresponding directions to the horizontal compensation difference, vertical compensation difference, and rotational compensation difference, respectively. S430. Set an EIS compensation weight. Based on whether the gimbal is in the stable locked state and whether there is residual small arc oscillation, control the EIS compensation weight to gradually change between 0 and 1. The compensation difference is weighted by the EIS compensation weight to generate the final compensation amount.

6. The electronic image stabilization method based on gimbal closed-loop stabilization state according to claim 1, characterized in that, The S500 step includes: S510. Define the region centered on the geometric center of the image frame as the central region of interest, set the geometric center point of the central region of interest as the center of the affine transformation, and establish the affine transformation matrix according to the horizontal compensation amount, vertical compensation amount and rotation compensation amount in the final compensation amount. S520. Perform an affine transformation on the image frame according to the affine transformation matrix, and perform center cropping on the transformed image to eliminate black borders. S530 outputs the stabilized image frame.

7. The electronic image stabilization method based on gimbal closed-loop stabilization state according to claim 6, characterized in that, In step S510, the size of the central region of interest is 50% of the width and 50% of the height of the image frame.

8. The electronic image stabilization method based on gimbal closed-loop stabilization state according to claim 1, characterized in that, The S600 step includes: S610. Obtain the gimbal operating conditions and EIS compensation results; S620. Calculate the compensation effect index based on the gimbal working conditions and the EIS compensation result, and determine that the EIS parameters need to be adjusted when the compensation effect index exceeds a preset threshold. S630. Based on the gimbal working conditions and the compensation effect index, adjust at least one of the EIS judgment threshold or compensation parameters to adapt to different working conditions, maintain the compensation effect, and form a closed-loop optimization. S640. Update the adjusted parameters in a smooth iterative manner and apply upper and lower limits to the updated parameters.

9. The electronic image stabilization method based on gimbal closed-loop stabilization state according to claim 8, characterized in that, The EIS judgment threshold or compensation parameter in step S630 includes at least one of the following: steady-state angle error threshold, angular velocity threshold, PID output judgment threshold, residual vibration amplitude threshold, residual vibration frequency range, maximum horizontal compensation amount, maximum vertical compensation amount, maximum rotational compensation amount, EIS compensation weight rise rate, and EIS compensation weight fall rate.

Images