Method and system for eliminating jitter of image video, storage medium and equipment

Abstract

The invention discloses an image video jitter elimination method and system, a storage medium and equipment, and relates to the technical field of video processing. The method comprises the following steps: performing enhancement processing on an original video image sequence to obtain an enhanced image sequence; calculating a global motion vector based on the attitude sensor data and the enhanced image sequence; performing motion filtering on each global motion vector to obtain a jitter component and an effective motion component; calculating a jitter compensation parameter according to the jitter component, determining a motion holding parameter in combination with the effective motion component, determining a compensation coordinate position of a to-be-compensated pixel point based on the jitter compensation parameter and the motion holding parameter, and performing weighted interpolation calculation on the neighborhood pixel point set to obtain an optimized gray value of the to-be-compensated pixel point; and performing pixel compensation processing according to the optimized gray value pair to generate a target video image sequence. By implementing the technical scheme provided by the invention, the influence of jitter on the video image quality can be reduced, and the target identification and tracking accuracy is further improved.

Description

Technical Field

[0001] This application relates to the field of video processing technology, specifically to a method, system, storage medium, and device for eliminating image and video jitter. Background Technology

[0002] With the rapid development of carrier platform technologies such as UAVs, carrier platforms equipped with electro-optical pods have been widely used in military reconnaissance, emergency rescue, and other fields. However, in actual operation, environmental disturbances and platform vibrations often cause varying degrees of attitude fluctuations in the carrier platform, leading to quality issues such as jitter and blurring in the video images acquired by the electro-optical pods, severely impacting the accuracy of target recognition and tracking.

[0003] Currently, common video jitter reduction methods mainly employ digital filtering to suppress image jitter. However, in complex motion environments, due to the variable motion state of the carrier platform, this method is prone to loss of image details and struggles to adapt to drastic motion changes, resulting in the inability to obtain clear and stable video image sequences, thus affecting the accuracy of target recognition and tracking. Summary of the Invention

[0004] This application provides a method, system, storage medium, and device for image and video jitter reduction, which can reduce the impact of jitter on video image quality, thereby improving the accuracy of target recognition and tracking.

[0005] In a first aspect, this application provides a method for eliminating image and video jitter, the method comprising: The original video image sequence and corresponding attitude sensor data collected by the carrier platform are acquired, and the original video image sequence is enhanced to obtain an enhanced image sequence. Based on the attitude sensor data and the enhanced image sequence, calculate the global motion vector between adjacent image frames; Motion filtering is performed on each of the global motion vectors to obtain the separated jitter components and effective motion components; The jitter compensation parameters are calculated based on the jitter components, and the motion holding parameters are determined in combination with the effective motion components, wherein the jitter compensation parameters include translation compensation parameters and rotation compensation parameters; Based on the jitter compensation parameters and the motion preservation parameters, the compensation coordinate positions of the pixels to be compensated in the enhanced image sequence are determined, and the set of neighboring pixels of the compensation coordinate positions is obtained. Weighted interpolation is performed on the set of neighboring pixels to obtain the optimized grayscale value of the pixel to be compensated. Pixel compensation processing is performed on the corresponding pixels to be compensated in the enhanced image sequence based on the optimized grayscale value to generate the target video image sequence after eliminating jitter.

[0006] By employing the above technical solution, and acquiring the original video image sequence of the carrier platform and the corresponding attitude sensor data, and calculating the global motion vector between adjacent image frames based on both, the actual motion state of the platform can be reflected more accurately. Furthermore, by performing motion filtering on the global motion vector, jitter components and effective motion components can be effectively separated, thereby enabling targeted calculation of jitter compensation parameters and motion-maintaining parameters. Based on these parameters, the compensation coordinate positions of the pixels to be compensated are determined, and combined with weighted interpolation calculations using the neighboring pixel set, accurate pixel compensation can be achieved while preserving image details. This ultimately generates a jitter-free and detail-clear target video image sequence, improving the accuracy of target recognition and tracking. This solution not only adapts to the drastic motion changes of UAVs in complex flight environments but also effectively avoids the image detail loss problem caused by traditional digital filtering methods.

[0007] A second aspect of this application provides a system for image and video jigging removal, the system comprising: The image enhancement processing module is used to acquire the original video image sequence and corresponding attitude sensor data collected by the carrier platform, and to enhance the original video image sequence to obtain an enhanced image sequence. The motion vector calculation module is used to calculate the global motion vector between adjacent image frames based on the attitude sensor data and the enhanced image sequence; The motion filtering module is used to perform motion filtering on each of the global motion vectors to obtain the separated jitter components and effective motion components. The compensation parameter determination module is used to calculate jitter compensation parameters based on the jitter components and determine motion holding parameters in combination with the effective motion components, wherein the jitter compensation parameters include translation compensation parameters and rotation compensation parameters; The compensation position determination module is used to determine the compensation coordinate position of the pixel to be compensated in the enhanced image sequence based on the jitter compensation parameters and the motion preservation parameters, and to obtain the set of neighboring pixels of the compensation coordinate position; The pixel interpolation calculation module is used to perform weighted interpolation calculation on the set of neighboring pixels to obtain the optimized grayscale value of the pixel to be compensated. The pixel compensation processing module is used to perform pixel compensation processing on the corresponding pixels to be compensated in the enhanced image sequence according to the optimized grayscale value, so as to generate the target video image sequence after eliminating jitter.

[0008] A third aspect of this application provides a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps described above.

[0009] A fourth aspect of this application provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0010] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: This application acquires the original video image sequence and corresponding attitude sensor data of the carrier platform, and calculates the global motion vector between adjacent image frames based on both, which can more accurately reflect the actual motion state of the platform. Furthermore, by performing motion filtering on the global motion vector, jitter components and effective motion components can be effectively separated, thereby enabling targeted calculation of jitter compensation parameters and motion-maintaining parameters. Based on these parameters, the compensation coordinate positions of the pixels to be compensated are determined, and combined with weighted interpolation calculations using the neighboring pixel set, accurate pixel compensation can be achieved while preserving image details. Ultimately, a jitter-free and detailed target video image sequence is generated, improving the accuracy of target recognition and tracking. This scheme not only adapts to the drastic motion changes of UAVs in complex flight environments but also effectively avoids the image detail loss problem caused by traditional digital filtering methods. Attached Figure Description

[0011] Figure 1 This is a schematic flowchart of a method for eliminating image and video jitter provided in an embodiment of this application; Figure 2 This is a comparative schematic diagram of video frame image jitter removal provided in an embodiment of this application; Figure 3 This is a schematic diagram of a system for eliminating image and video jitter provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0012] Explanation of reference numerals in the attached drawings: 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. Detailed Implementation

[0013] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0014] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0015] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0016] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0017] The image and video shake reduction method provided in this application can be applied to a variety of practical scenarios. For example, in handheld shooting devices, it can be used to eliminate video shake caused by hand tremors; in vehicle monitoring systems, it can be used to stabilize image shaking caused by road bumps; and in robot vision systems, it can be used to eliminate image shake during robot movement.

[0018] As an optional scenario, this approach can be applied to video acquisition scenarios for UAV electro-optical pods. Specifically, UAV electro-optical pods need to acquire stable and clear video images for target identification and tracking when performing reconnaissance missions. In this application scenario, due to the influence of atmospheric disturbances and body vibrations on the UAV, coupled with attitude changes during high-speed flight and maneuvers, the acquired video images may exhibit varying degrees of jitter and blurring. For example, when the UAV performs missions in strong winds, atmospheric disturbances cause continuous platform shaking, and engine operation also generates mechanical vibrations. The combination of these factors significantly degrades the video image quality. In this specific application scenario, the image and video jitter elimination method provided in this application embodiment, by combining attitude sensor data and image information, separating jitter components and effective motion components, and employing precise pixel compensation techniques, can effectively address various jitter situations encountered by UAV electro-optical pods during video acquisition, ensuring the clarity and stability of the output video. Subsequent embodiments will use this scenario as an example to detail the specific implementation methods of each technical feature of this application.

[0019] Please refer to Figure 1 A flowchart illustrating a method for image and video jitter reduction is presented. This method can be implemented using a computer program, a microcontroller, or run on an image and video jitter reduction system. The computer program can be integrated into a computer device or run as a standalone utility application. Specifically, the method includes steps 10 to 70, as follows: Step 10: Obtain the original video image sequence and corresponding attitude sensor data collected by the carrier platform, and perform enhancement processing on the original video image sequence to obtain the enhanced image sequence.

[0020] In this embodiment of the application, the carrier platform refers to the body of the UAV carrying the optoelectronic pod equipment. Due to factors such as airflow disturbance and mechanical vibration during the flight of the UAV, the carrier platform will experience irregular attitude changes and positional jitter.

[0021] Attitude sensors refer to a combination of sensors installed on a carrier platform to measure the platform's motion state in real time. These include, but are not limited to, inertial measurement units such as gyroscopes and accelerometers, which can accurately acquire motion parameter data such as the angular velocity, acceleration, and attitude angle changes of the carrier platform.

[0022] Image enhancement refers to image data obtained after preprocessing the original video image sequence. Since original images often suffer from noise interference, insufficient contrast, and uneven brightness, these factors can affect the accuracy of subsequent motion vector calculations and jitter detection. Therefore, it is necessary to perform denoising, contrast enhancement, and brightness correction on the original images to improve image quality and the reliability of feature point extraction, thereby providing a better data foundation for accurate jitter analysis and compensation.

[0023] Specifically, the carrier platform continuously acquires raw video image sequences at a rate of 30 frames per second using an onboard high-resolution CCD or CMOS image sensor. The image resolution can be set to 1920×1080 pixels. Simultaneously, the attitude sensor module on the carrier platform, including a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer, acquires the carrier platform's angular velocity, linear acceleration, and magnetic field strength data in real time at the same sampling frequency. A timestamp mechanism ensures strict time synchronization between image data and attitude data. Because the carrier platform is inevitably affected by various interference factors such as atmospheric turbulence, propeller vibration, and mechanical structure resonance in complex flight environments, the raw video image sequences generally contain noise and image quality issues such as uneven brightness and low contrast due to changes in lighting conditions. These defects severely affect the accuracy and stability of subsequent image feature extraction and motion estimation algorithms.

[0024] Furthermore, to effectively improve the quality of the original image, the system first uses a Gaussian filter with a 5×5 convolution kernel to smooth each frame of the image. The standard deviation of the filter is set to 1.2. This convolution operation effectively suppresses high-frequency noise components in the image while maintaining the integrity of edge information. Next, a global histogram equalization algorithm is used to redistribute the gray-level distribution of the image. By calculating the cumulative distribution function and constructing a gray-level mapping table, the gray-level range of the original image is stretched to the full dynamic range of 0-255, thereby improving the overall contrast of the image. To further optimize local image quality, the system employs a contrast-limited adaptive histogram equalization algorithm. The image is divided into 8×8 sub-blocks, and histogram equalization is performed on each sub-block separately. A contrast limit threshold of 3.0 is set to prevent noise amplification caused by over-enhancement. Finally, the system applies a gamma correction function with a gamma value of 0.8 to perform non-linear brightness adjustment on the image, optimizing the visual effect and detail representation of the image through power function transformation. After the above complete enhancement process, the resulting enhanced image sequence significantly improves the signal-to-noise ratio, contrast, and brightness uniformity of the image while maintaining the original spatial resolution. This provides high-quality input data for subsequent key steps such as feature point detection, optical flow calculation, and motion parameter estimation, thereby ensuring that the entire image stabilization algorithm can accurately identify and compensate for the jitter of the carrier platform.

[0025] Step 20: Calculate the global motion vector between adjacent image frames based on attitude sensor data and enhanced image sequences.

[0026] In this embodiment, the global motion vector refers to the motion parameter vector describing the overall displacement and rotation of the entire image frame relative to the reference frame. It is typically represented as a three-dimensional vector containing horizontal translation components, vertical translation components, and a rotation angle around the image center. The global motion vector reflects the overall motion relationship between adjacent frames in the image sequence caused by platform jitter and is a key parameter for subsequent motion compensation and image stabilization processing. Unlike local motion vectors, the global motion vector describes the overall motion trend of the image, rather than the independent motion of a specific region or target within the image.

[0027] Specifically, using angular velocity and Euler angle data collected by the carrier platform's attitude sensors, the global motion vector between adjacent image frames is directly calculated by establishing a three-dimensional rotation matrix and translation vector. The system first reads the attitude sensor data corresponding to two frames with timestamps t and t+1, including changes in pitch, yaw, and roll angles. Then, based on the composite operation rules of the rotation matrix, a three-dimensional transformation matrix is ​​constructed from frame t to frame t+1. Since the minor jitter of the carrier platform mainly manifests as small-angle rotations around its axes and limited translational motion, the system uses a small-angle approximation to simplify the computational complexity, projecting the three-dimensional motion onto a two-dimensional image plane to obtain a global motion vector containing a horizontal translation component Δx, a vertical translation component Δy, and a rotation angle Δθ. This method can quickly respond to changes in the carrier platform's motion, has high computational efficiency and good real-time performance, and is particularly suitable for stable flight conditions with small jitter amplitudes, providing accurate initial estimates for subsequent motion compensation.

[0028] Based on the above embodiments, as an optional embodiment, step 20: calculating the global motion vector between adjacent image frames based on attitude sensor data and enhanced image sequences, further includes the following steps: Step 101: Extract the angle change and displacement change of the carrier platform from the attitude sensor data.

[0029] Specifically, motion parameters are extracted from the attitude sensor data stream of the carrier platform. The angular velocity data output by the three-axis gyroscope within a sampling interval Δt is integrated over time to calculate the angular changes of the carrier platform around the X, Y, and Z axes. The specific formulas are Δφ = ωx × Δt, Δθ = ωy × Δt, and Δψ = ωz × Δt, where ωx, ωy, and ωz are the three-axis angular velocity components, respectively. Simultaneously, the system utilizes the output data from the three-axis accelerometer to first remove the influence of gravitational acceleration, and then performs a second integration on the net acceleration to obtain the displacement changes of the carrier platform in the three directions. The calculation formulas are Δx = ½ax × (Δt)², Δy = ½ay × (Δt)², and Δz = ½az × (Δt)², where ax, ay, and az are the three-axis acceleration components after gravity removal. Since inertial sensors are prone to accumulating errors during integration, the system uses a Kalman filter to filter the extracted angle and displacement changes, effectively suppressing the influence of sensor noise and drift errors, ensuring the accuracy and stability of motion parameter extraction, and providing reliable basic data for subsequent coordinate system transformation and motion compensation.

[0030] Step 102: Convert the angle change and displacement change into motion components in the image coordinate system.

[0031] Specifically, the system establishes a transformation relationship from the three-dimensional coordinate system of the carrier platform to the two-dimensional image coordinate system. First, a perspective projection model is constructed based on the intrinsic and extrinsic parameter matrices of the camera in the optoelectronic pod. The extracted three-dimensional angular changes are projected onto the image plane through rotation matrix transformation. Specifically, basic rotation matrices Rx(Δφ), Ry(Δθ), and Rz(Δψ) around the X, Y, and Z axes are constructed, and matrix composition operations are performed according to the ZYX Euler angle sequence to obtain the total rotation matrix R = Rz(Δψ) × Ry(Δθ) × Rx(Δφ). For the conversion of displacement changes, the system considers the influence of the carrier platform height and camera focal length, converting the three-dimensional displacement into pixel displacement on the image plane through scaling. The conversion formulas are: horizontal motion component ux = f × Δx / H and vertical motion component uy = f × Δy / H in the image coordinate system, where f is the camera focal length and H is the flight altitude. This coordinate system transformation method can accurately map the complex motion of the carrier platform in three-dimensional space onto the two-dimensional image plane, providing a unified coordinate reference for subsequent comparative analysis of the motion of image feature points and ensuring effective fusion between data from different sensors.

[0032] Step 103: Select uniformly distributed feature points in the enhanced image sequence as motion reference points.

[0033] Specifically, in each frame of the enhanced image sequence, an improved Harris corner detection algorithm is used to automatically select motion reference points. First, the image is divided into 8×8 grid regions to ensure the spatial uniformity of feature point distribution. Within each grid region, the Harris response function H = det(M) - k×trace²(M) is calculated, where M is the structure tensor matrix of the image gradient, and k is an empirical constant set to 0.04. The system sets a Harris response threshold of 0.01 and selects the pixel with the largest response value within each grid region as a candidate feature point. Simultaneously, a non-maximum suppression algorithm is used to ensure that the minimum distance between adjacent feature points is no less than 15 pixels, avoiding excessive clustering of feature points. To improve the stability and trackability of the motion reference points, the system further calculates the gradient variance within a 3×3 neighborhood around each candidate feature point. Feature points with a gradient variance greater than a set threshold are selected as the final motion reference points, ensuring that these reference points have rich texture information and good distinguishability.

[0034] Step 104: Calculate the actual displacement vector of the motion reference point between adjacent image frames.

[0035] Specifically, the system uses an optical flow method based on template matching to calculate the actual displacement vector of the motion reference point between adjacent image frames. For each motion reference point selected in frame t, the system extracts a 15×15 pixel template window centered on that point. Then, in frame t+1, a 31×31 pixel search region is established centered on that point. The system uses a normalized cross-correlation matching algorithm to find the position most similar to the template window within the search region. The matching similarity is calculated using the formula: Where I1 and I2 are the pixel values ​​of the two template windows, and μ1 and μ2 are the pixel averages of the corresponding windows. The system selects the position with the largest NCC value as the matching point and calculates the displacement vector of this motion reference point from frame t to frame t+1, denoted as (Δu, Δv), where Δu and Δv are the pixel displacements in the horizontal and vertical directions, respectively.

[0036] To improve the accuracy of displacement calculation, the system uses a subpixel-level interpolation algorithm to optimize the matching results. By fitting a quadratic surface around the matching peak, subpixel-precision displacement estimation is obtained, thus yielding the actual displacement vector.

[0037] Step 105: Compare the difference between the motion component and the actual displacement vector and correct the deviation to obtain the target deviation value.

[0038] Specifically, the system performs point-by-point comparison and analysis between the motion components and the actual displacement vectors in the image coordinate system. For each motion reference point i, the difference between its theoretical motion components (uxi, uyi) and the actual displacement vector (Δui, Δvi) is calculated to obtain the deviation vector (εxi, εyi) for that point, where εxi = Δui - uxi and εyi = Δvi - uyi. Since different reference points may have varying degrees of measurement noise and local motion interference, a weighted least squares method is used to statistically analyze the deviation vectors of all reference points. The weights are set proportional to the Harris response intensity and matching correlation of each point to improve the contribution of stable feature points in the deviation calculation. The system calculates the weighted average deviation as the target deviation value, specifically using the formula: [Target Deviation Value]. Where εx and εy are the target deviation values ​​(pixels) in the horizontal and vertical directions, respectively, w i is the weight coefficient for the i-th motion reference point.

[0039] Step 106: Determine the global motion vector between adjacent image frames based on the target deviation value.

[0040] Specifically, the theoretical motion components are synthesized with the target deviation value to obtain the final global motion vector between adjacent image frames. The system uses vector superposition to calculate the global motion components in the horizontal direction. The global motion components in the vertical direction are calculated as follows: Where ux and uy are the theoretical motion components obtained from the attitude sensor data conversion, and εx and εy are the target deviation values. For the rotation component, the system estimates the overall rotation angle θ_global of the image by analyzing the displacement vector field distribution pattern of all motion reference points and using the least squares fitting method. The fitting model is as follows: u and v are the fitted motion components (pixels), θ_global is the overall rotation angle of the image (radians), and (x,y) are the image coordinates of the motion reference point. Finally, the system outputs a global motion vector containing three components: This vector accurately describes the overall motion relationship between adjacent image frames caused by carrier platform jitter.

[0041] Step 30: Perform motion filtering on each global motion vector to obtain the separated jitter components and effective motion components.

[0042] In this embodiment, the jitter component refers to a global motion vector identified by the system as having jitter characteristics. Specifically, it is a motion vector with a rate of change greater than a preset rate of change and a duration less than a preset duration. This type of motion vector reflects the unexpected random motion generated when the carrier platform is affected by external interference factors such as atmospheric turbulence, engine vibration, and mechanical resonance. Its characteristics include large amplitude of change, short duration, and irregular direction of motion. The jitter component directly causes image jumps, oscillations, and instability in images captured by the photoelectric pod camera. It is a harmful motion type that electronic image stabilization systems need to detect, identify, compensate for, and eliminate to improve the visual quality and stability of the images.

[0043] Effective motion components refer to global motion vectors identified by the system as having normal motion characteristics. Specifically, these are motion vectors whose rate of change is no greater than a preset rate of change or whose duration is no less than a preset duration. These motion vectors reflect the expected motion behavior of the carrier platform according to the flight plan or operational instructions, including purposeful motion processes such as normal maneuvering, heading adjustments, altitude changes, and target tracking. Their characteristics include relatively smooth changes, a certain degree of continuity, and compliance with the flight dynamics constraints of the carrier platform. Effective motion components represent the carrier platform's true motion intentions and actual flight trajectory. They are useful motion information that the electronic image stabilization system needs to retain and reflect in the stabilized image, ensuring that image stabilization processing does not affect the authenticity of the carrier platform's normal maneuvering.

[0044] Specifically, the system performs motion filtering on each continuously acquired global motion vector, separating jitter components from effective motion components by calculating the motion change rate and duration of each global motion vector. The system first calculates the amplitude of change between adjacent global motion vectors as the motion change rate, and simultaneously counts the duration of vector sequences with similar motion characteristics. For global motion vectors with a motion change rate greater than a preset change rate threshold (e.g., 5 pixels / frame) and a duration less than a preset duration threshold (e.g., 3 frames), the system marks them as separated jitter components; these vectors reflect random disturbances and high-frequency vibrations of the carrier platform. For global motion vectors with a motion change rate not greater than the preset change rate threshold or a duration not less than the preset duration threshold, the system marks them as separated effective motion components; these vectors represent the normal maneuvering motion of the carrier platform. Through this classification method based on dual judgment conditions, the system can accurately identify motion vectors of different properties, providing a precise classification basis for subsequent motion compensation.

[0045] Based on the above embodiments, as an optional embodiment, step 30: performing motion filtering on each global motion vector to obtain the separated jitter components and effective motion components further includes the following steps: Step 201: Establish a time-series change trajectory diagram based on each global motion vector.

[0046] Specifically, the system arranges the continuously acquired global motion vectors in chronological order, using time as the horizontal axis and the horizontal displacement component, vertical displacement component, and rotation angle component as the vertical axes to create three two-dimensional sub-plots. For each time t, the global motion vector V(t) = (ux(t), uy(t), θ(t)), the system marks the coordinate points in the corresponding sub-plot and connects them using linear interpolation to form a continuous curve. The system normalizes the trajectory plot and uses different colors to distinguish the three motion components, generating a complete time-series trajectory plot, providing a visualization foundation for subsequent motion analysis.

[0047] Step 202: Identify the turning points of the motion direction and the points of discontinuity in the time-series trajectory diagram.

[0048] Specifically, the system identifies inflection points by calculating the second derivative of the time-series trajectory curve. A valid inflection point is confirmed when the second derivative changes from a positive value to a negative value or from a negative value to a positive value with a change exceeding a preset threshold. For discontinuities in the trajectory, the system calculates the Euclidean distance between the global motion vectors of adjacent time points. When the distance exceeds a continuity threshold, it is determined to be a discontinuity. The identified inflection points and discontinuities are compiled into a timestamp list, which serves as boundary markers for dividing motion segments.

[0049] Step 203: Based on the turning points and trajectory discontinuities, divide the time series of each global motion vector into multiple motion segments.

[0050] Specifically, the system sorts the turning points and discontinuities in chronological order to form a sequence of segmented time points, and divides the global motion vector into multiple motion segments using adjacent segmentation points as boundaries. The system assigns a unique identifier to each segment and records its start time, end time, and the number of vectors it contains. When a segment's length is less than a minimum threshold, it is merged with adjacent segments to ensure the rationality and continuity of the segmentation results.

[0051] Step 204: Calculate the rate of change and duration of motion of the global motion vector within each segment.

[0052] Specifically, the system calculates the rate of change of motion by dividing the Euclidean distance between adjacent global motion vectors within a segment by the time interval, and then takes the average of all rates of change within the segment as a representative value. The duration is calculated by multiplying the number of vectors contained in the segment by the sampling interval. The system associates and stores the rate of change of motion and duration parameters with the corresponding segments, establishing a motion segment parameter database.

[0053] Step 205: Use global motion vectors with a motion rate of change greater than a preset rate of change and a duration of duration less than a preset duration as jitter components, and use global motion vectors with a motion rate of change not greater than a preset rate of change or a duration of duration not less than a preset duration as effective motion components.

[0054] Specifically, the system sets preset change rate thresholds and preset duration thresholds as classification benchmarks. For segments that meet both the conditions of a motion change rate greater than the preset change rate and a duration less than the preset duration, all global motion vectors within them are marked as jitter components. For segments that do not meet the above conditions, all global motion vectors within them are marked as valid motion components. The system establishes a classification result database to record the classification label of each vector, providing accurate classification information for subsequent motion compensation.

[0055] Step 40: Calculate the jitter compensation parameters based on the jitter components, and determine the motion holding parameters in combination with the effective motion components. The jitter compensation parameters include translation compensation parameters and rotation compensation parameters.

[0056] In this embodiment of the application, the jitter compensation parameter refers to a numerical compensation amount used to eliminate or reduce the influence of jitter components. For example, it can be a horizontal displacement compensation amount, a vertical displacement compensation amount, and a rotation angle compensation amount that are equal in magnitude and opposite in direction to the jitter components.

[0057] Motion retention parameters refer to the numerical retention quantities used to retain and represent the effective motion components, including the horizontal displacement retention quantity, vertical displacement retention quantity, and rotation angle retention quantity corresponding to the effective motion components.

[0058] Specifically, the system calculates corresponding jitter compensation parameters based on the numerical characteristics of the jitter components, eliminating the impact of jitter on image stability through a reverse compensation principle. For the identified jitter components, the system extracts their horizontal and vertical displacement components, inverts their values ​​to obtain translation compensation parameters, i.e., the translation compensation parameters are equal to the negative translation value of the jitter component, ensuring that the compensation direction is completely opposite to the jitter direction. Simultaneously, the system extracts the rotation angle component of the jitter component and inverts its value as a rotation compensation parameter, achieving precise cancellation of jitter rotation. For effective motion components, the system directly extracts their motion values ​​as motion-holding parameters, maintaining the authenticity of the carrier platform's normal maneuvering motion. The system superimposes the jitter compensation parameters and motion-holding parameters to generate the final image transformation parameters. This reverse compensation method can accurately cancel the effects of jitter while retaining effective motion information, ensuring that the stabilized image eliminates jitter interference and accurately reflects the actual motion state of the carrier platform, effectively improving the stability and comfort of image observation.

[0059] Based on the above embodiments, as another optional embodiment, the step of calculating jitter compensation parameters based on jitter components further includes the following steps: Step 301: Perform frequency domain transformation on the jitter components to obtain the jitter spectrum distribution map.

[0060] Specifically, the system performs frequency domain analysis on the identified jitter components, converting the time-domain jitter motion signal into frequency-domain spectral distribution information. The system processes the jitter components separately into three components: horizontal displacement, vertical displacement, and rotation angle. For each component, the time-series data undergoes preprocessing, including data detrending and Hanning window weighting to eliminate DC components and reduce spectral leakage. The system performs a Fast Fourier Transform (FFT) operation, converting each jitter component from a time-domain signal into a frequency-domain complex sequence, calculating the amplitude and phase information of the complex number to generate amplitude and phase spectra. The system integrates the spectral information of the three motion components into a unified jitter spectral distribution map, displaying the distribution relationship of frequency, amplitude, and phase in a three-dimensional visualization. This frequency domain transformation method reveals the energy distribution characteristics of jitter motion at different frequency components, providing an accurate frequency domain basis for the system to identify the main frequency components of the jitter.

[0061] Step 302: Identify the primary and secondary frequency components in the jitter spectrum distribution map.

[0062] Specifically, the system automatically identifies primary and secondary frequency components in the jitter spectrum distribution map through amplitude threshold analysis and energy proportion calculation. The system calculates the total energy of the entire spectrum, sets the energy proportion threshold for primary frequency components to 75% of the total energy, sorts all frequency components from largest to smallest amplitude, and accumulates the energy contribution of each frequency component until a preset threshold is reached. These frequency components are then marked as primary frequency components. The system uses a peak detection algorithm to identify significant peaks in the spectrum, avoids noise interference by calculating local maxima and setting minimum inter-peak distances, and cross-validates the detected peak frequencies with the energy proportion results. The remaining frequency components are classified as secondary frequency components. The system establishes a database of primary and secondary frequency components, recording the center frequency, amplitude, and phase information of each frequency component. This classification and identification method can accurately distinguish between dominant and secondary frequency characteristics in jitter motion, providing a scientific basis for adopting differentiated compensation strategies for different frequency components.

[0063] Step 303: Calculate the intensity coefficient of the main jitter direction based on the amplitude and phase information of the main frequency components.

[0064] Specifically, the system calculates the intensity coefficient of the jitter principal direction using the amplitude and phase information of the main frequency components, quantifying the intensity characteristics of the jitter motion in the dominant direction. First, the system performs vector synthesis on the horizontal and vertical displacement components of the main frequency components. By calculating the vector sum of the horizontal and vertical amplitudes corresponding to each main frequency point, the system obtains the synthesized amplitude and phase for that frequency point. The system uses a weighted average method to calculate the combined motion direction of all main frequency components, using the energy proportion of each frequency component as the weight to perform a weighted average of the synthesized phases to obtain the jitter principal direction angle. The system calculates the intensity coefficient of the jitter principal direction by superimposing the projected amplitudes of all main frequency components in the principal direction to obtain the overall intensity value of the principal direction. To eliminate mutual interference between frequencies, the system employs phase compensation technology, correcting the amplitude based on the phase differences of each frequency component to ensure the accuracy of the intensity coefficient calculation.

[0065] Step 304: Determine the jitter coupling direction and coupling strength based on the distribution characteristics of the secondary frequency components.

[0066] Specifically, the system constructs a complex coupling mechanism in jitter motion by analyzing the spatial distribution characteristics and frequency correlations of secondary frequency components to determine the jitter coupling direction and strength. First, the system calculates the correlation coefficients of secondary frequency components in the three motion dimensions of horizontal, vertical, and rotation, and identifies the coupling relationships between different motion components through cross-correlation analysis. The system uses principal component analysis to reduce the dimensionality of the secondary frequency components and extracts the main coupling direction vectors, which represent the main distribution trend and coupling direction of the secondary frequency components. The system calculates the coupling strength index by statistically analyzing the projection amplitudes of the secondary frequency components in the coupling direction to obtain a quantitative value of the coupling strength. To identify rotational coupling characteristics, the system also analyzes the phase relationship between the rotational and translational components in the secondary frequency components, determining the directionality and temporal characteristics of rotational coupling through phase difference calculation.

[0067] Step 305: Convert the intensity coefficient of the jitter main direction into translation compensation parameters, and convert the jitter coupling direction and coupling intensity into rotation compensation parameters.

[0068] Specifically, based on the frequency domain analysis results, jitter characteristic parameters are converted into specific compensation parameters, achieving a precise mapping from frequency domain features to spatial domain compensation. The system decomposes the intensity coefficient of the jitter principal direction into horizontal and vertical translation compensation parameters according to the principal direction angle using trigonometric function transformation. The calculated translation component is negatively evaluated using the reverse compensation principle as the final translation compensation parameter. For rotation compensation parameters, the system comprehensively processes the jitter coupling direction and coupling strength, determining the rotation compensation direction based on the coupling direction angle and the rotation compensation amplitude based on the coupling strength coefficient. The system considers the geometric relationship between translation and rotation, correcting the spatial coordinates of the rotation compensation parameters based on the positional relationship between the geometric center of the carrier platform and the image center. The system performs low-pass filtering on the calculated compensation parameters to eliminate high-frequency noise and maintain the smoothness of the compensation action. This frequency domain to spatial domain parameter conversion method can accurately convert complex frequency domain jitter characteristics into directly applicable image compensation parameters, ensuring precise jitter compensation based on scientific frequency domain analysis results.

[0069] Based on the above embodiments, as another optional embodiment, the step of determining the motion holding parameters in conjunction with the effective motion components further includes the following steps: Step 401: Analyze the vector correlation and directional consistency between adjacent effective motion components.

[0070] Specifically, the system first establishes a time-series data structure for the effective motion components, representing each effective motion component at any given time as a three-dimensional vector containing horizontal displacement, vertical displacement, and rotation angle. For vector correlation analysis, the system calculates the dot product of effective motion component vectors at adjacent time points and uses the vector dot product formula to calculate the dot product of two adjacent vectors. This dot product reflects the similarity of the two vectors in direction and magnitude. The system also calculates the magnitude ratio of adjacent vectors and assesses the continuity of motion intensity by comparing the rate of change of vector magnitude. For directional consistency analysis, the system calculates the angle between adjacent effective motion component vectors using the inverse cosine function. This angle directly reflects the degree of change in motion direction. The system performs directional consistency analysis separately for translational and rotational components. For translational components, the system uses a two-dimensional vector angle calculation, while for rotational components, it uses the absolute value of the angle difference. The system establishes a sliding window mechanism to perform statistical analysis of vector correlation and directional consistency at multiple consecutive time points, and assesses the stability of motion characteristics by calculating the mean and variance within the time window.

[0071] Step 402: Calculate the vector correlation coefficient based on vector correlation, and calculate the directional deviation angle based on directional consistency.

[0072] Specifically, based on the analysis results of adjacent effective motion components, it is necessary to quantitatively calculate characteristic parameters. The vector correlation coefficient is calculated by dividing the dot product of adjacent motion vectors by the product of the magnitudes of the two vectors to obtain the normalized correlation coefficient, which ranges from -1 to 1. The closer the value is to 1, the stronger the correlation. The directional deviation angle is obtained by calculating the angle between two adjacent motion vectors. Specifically, the arccos function is used to calculate the normalized vector dot product to obtain the radian value, which is then converted into an angle value. These parameters provide a quantitative basis for subsequent interval division.

[0073] Step 403: Determine the continuity interval and the jump interval based on the vector correlation coefficient and the direction deviation angle.

[0074] Specifically, based on the calculated feature parameters, motion intervals with different characteristics can be identified. When the vector correlation coefficient is greater than a preset correlation threshold (e.g., 0.8) and the directional deviation angle is less than a preset angle threshold (e.g., 30 degrees), the corresponding motion segment is determined to be a continuous interval, indicating that the motion within this interval is relatively smooth and continuous. Conversely, when the vector correlation coefficient is not greater than the preset correlation threshold or the directional deviation angle is not less than the preset angle threshold, the corresponding motion segment is determined to be a jump interval, indicating that the motion within this interval exhibits obvious turning or velocity changes. This interval division method can distinguish sequence segments of different motion patterns.

[0075] Step 404: Calculate the motion inertia retention strength for the effective motion components within the continuous interval, and calculate the motion transition retention strength for the effective motion components within the jump interval.

[0076] Specifically, for different types of intervals, corresponding motion-maintaining characteristics need to be calculated. For continuous intervals, the strength of motion inertia maintenance is characterized by calculating the average correlation coefficient of adjacent motion vectors within the interval, reflecting the degree of inertial continuity of the motion. Specifically, the correlation coefficients of all adjacent vector pairs within the interval are summed and then divided by the number of vector pairs. For jump intervals, the strength of motion transition maintenance is characterized by calculating the cumulative value of the direction deviation angle within the interval, reflecting the degree of turning of the motion. Specifically, the direction deviation angles of all adjacent vector pairs within the interval are summed and then normalized. These characteristic parameters can quantitatively describe the maintenance characteristics of different types of motion.

[0077] Step 405: Combine the motion inertia holding strength and motion conversion holding strength to form motion holding parameters.

[0078] Specifically, the calculated motion inertia retention strength and motion transition retention strength are used as two components of the motion retention parameter. The motion inertia retention strength characterizes the smooth continuation of motion within a continuous interval, while the motion transition retention strength characterizes the turning and changing characteristics of motion within a jump interval. These two parameters describe the retention characteristics under different motion modes, together constituting the motion retention parameter, which can comprehensively characterize the retention characteristics of the entire motion sequence, providing a basis for subsequent motion compensation.

[0079] Step 50: Determine the compensation coordinates of the pixels to be compensated in the enhanced image sequence based on the jitter compensation parameters and motion preservation parameters, and obtain the set of neighboring pixels of the compensation coordinates.

[0080] In this embodiment, the pixel to be compensated refers to the target pixel that needs to undergo position correction and grayscale value recalculation during the image stabilization process. The pixel to be compensated typically refers to a pixel whose position coordinates in the output stabilized image are non-integer coordinates after transformation by jitter compensation parameters and motion-keeping parameters, or a pixel whose grayscale value needs to be redefined due to geometric transformation during the transformation process.

[0081] The neighborhood pixel set refers to the set of original image pixels within a certain range surrounding the pixel to be compensated. These pixels have known coordinate positions and definite gray values. The neighborhood pixel set typically includes 4, 8, or 16 pixels near the pixel to be compensated, depending on the type of interpolation algorithm used.

[0082] Specifically, a geometric transformation matrix is ​​constructed based on jitter compensation parameters and motion-preserving parameters. Matrix operations are then used to determine the compensation coordinates of the pixels to be compensated in the enhanced image sequence. The system superimposes the translation and rotation compensation parameters from the jitter compensation parameters with the motion-preserving parameters to generate comprehensive transformation parameters, including horizontal displacement, vertical displacement, and rotation angle. The system constructs a 2×3 affine transformation matrix and fills the comprehensive transformation parameters into their corresponding positions, forming a complete geometric transformation description. For each pixel in the enhanced image sequence, the system uses its original coordinates as an input vector and obtains the transformed compensation coordinates through matrix multiplication. Since the transformed coordinates are usually non-integer values, the system uses bilinear interpolation to determine the set of neighboring pixels, selecting the four nearest integer coordinates around the compensation coordinate position as the neighboring pixel set. The system calculates the distance weights between the compensation coordinate position and each neighboring pixel, providing a weighting basis for subsequent grayscale interpolation calculations.

[0083] Based on the above embodiments, as another optional embodiment, step 50: determining the compensation coordinate positions of the pixels to be compensated in the enhanced image sequence based on the jitter compensation parameters and motion preservation parameters may further include the following steps: Step 501: Identify the pixels to be compensated in the enhanced image sequence.

[0084] Specifically, since not all pixels are affected by jitter, the system needs to accurately identify pixels requiring compensation to improve computational efficiency. First, the system identifies candidate pixels affected by jitter in the enhanced image sequence based on jitter components. Specifically, it calculates the projection distribution of the jitter components onto the image plane and marks pixels within the projection area as candidate pixels. Then, the system calculates the jitter impact intensity of each candidate pixel using translation and rotation compensation parameters. The translation impact intensity is related to the pixel's position coordinates and the translation compensation parameters, while the rotation impact intensity is related to the pixel's distance from the image center and the rotation compensation parameters. A weighted combination yields the final jitter impact intensity. Finally, the system compares the jitter impact intensity with a preset impact threshold, identifying candidate pixels with an impact intensity greater than the preset threshold as pixels requiring compensation. This multi-step screening method accurately identifies pixels significantly affected by jitter, avoiding unnecessary compensation processing for areas with minimal jitter impact and improving algorithm efficiency.

[0085] Based on the above embodiments, as an optional embodiment, step 501: determining the pixels to be compensated in the enhanced image sequence may further include the following steps: Step 5011: Identify candidate pixels affected by jitter in the enhanced image sequence based on jitter components.

[0086] Specifically, the system first needs to identify candidate pixels that may be affected by jitter in the enhanced image sequence to narrow down the computational scope of subsequent processing. The system utilizes the separated jitter components and determines candidate regions by calculating the spatial distribution characteristics of these components. First, a jitter impact distribution map D(x,y) is constructed, calculated using the following formula: Where Jx and Jy are the projection values ​​of the jitter component in the horizontal and vertical directions, respectively. Then, the adaptive threshold Ta is calculated: Where μD and σD are the mean and standard deviation of the distribution map, respectively, and k is the adjustment coefficient. When D(x,y)>Ta, pixel (x,y) is marked as a candidate pixel. To ensure the continuity of the recognition results, the system also performs morphological processing on the marking results, including dilation and smoothing operations, to eliminate discrete noise and expand the boundaries of the candidate region. This dithering component-based pre-screening method can quickly locate potentially affected areas, providing an initial range for subsequent accurate calculations.

[0087] Step 5012: Calculate the jitter impact intensity of each candidate pixel based on the jitter compensation parameters.

[0088] Specifically, the system needs to quantitatively evaluate the identified candidate pixels and calculate the degree to which they are affected by jitter. For each candidate pixel P(x,y), the translation effect intensity St is first calculated: Where Tx and Ty are translation compensation parameters, and W and H are the image width and height, respectively. Then, the rotation effect intensity Sr is calculated: Where θ is the rotation compensation parameter, and R is half the length of the image diagonal. The final jitter effect intensity S is obtained through weighted fusion: Where wp and wr are weighting coefficients and wp+wr=1. This step-by-step calculation method can comprehensively evaluate the jitter impact on candidate pixels, providing a reliable quantitative basis for subsequent threshold selection.

[0089] Step 5013: Select candidate pixels whose jitter intensity is greater than the preset threshold as pixels to be compensated.

[0090] Specifically, the system needs to filter out the pixels that truly require compensation from the candidate pixels. First, the system determines a preset impact threshold based on image statistical features, taking into account both the overall jitter level and local variation characteristics of the current image. The system calculates the distribution characteristics of the jitter impact intensity of all candidate pixels, including the mean and standard deviation, and sets an appropriate preset impact threshold based on practical application requirements. Then, the jitter impact intensity of each candidate pixel is compared with the preset impact threshold; when the impact intensity exceeds the threshold, the pixel is marked as a pixel to be compensated.

[0091] Step 502: Calculate the compensation displacement vector of the pixel to be compensated based on the jitter compensation parameters.

[0092] Specifically, the system calculates a compensation displacement vector for each pixel to be compensated, representing the amount of spatial displacement required to counteract the jitter effect. The system extracts translation compensation parameters. and rotational compensation parameters Calculate the position vector of the pixel relative to the rotation center. Then, the rotational transformation matrix is ​​applied to calculate the rotational compensation displacement components: , ;in, The displacement component (pixel) caused by rotation compensation. Here are the rotation compensation parameters (in radians). The system performs vector superposition of translation and rotation compensation to obtain the final compensated displacement vector: ,in, To ensure the physical rationality and mathematical accuracy of the compensation effect, the final compensation displacement vector (pixel) is calculated.

[0093] Step 503: Calculate the motion displacement vector of the pixel to be compensated based on the motion retention parameters.

[0094] Specifically, the system calculates the motion displacement vector based on motion-keeping parameters to preserve the normal maneuvering information of the carrier platform. Motion-keeping parameters include the motion inertia retention strength. Maintain strength during motion transition Two components. The system determines the current motion state type using the formula for pixels within a continuous interval: For pixels within a jump range, use the formula: ,in, It is the motion displacement vector (pixel). The velocity vector (pixels / frame) of the current effective motion component. The conversion smoothing factor (dimensionless) is dynamically adjusted based on the drastic change in motion direction, with a value ranging from 0.3 to 1.0. This differentiation strategy ensures appropriate motion realism across different motion modes.

[0095] Step 504: Combine the compensation displacement vector with the motion displacement vector to obtain the comprehensive displacement vector.

[0096] Specifically, the compensation displacement vector With the direction of motion displacement Vector synthesis is performed. A weighted vector synthesis method is adopted, introducing a balancing weight factor w to adjust the relative contributions of the two vectors: ,in, The system defines a composite displacement vector (in pixels), where w is a dimensionless balancing weight factor dynamically determined based on the relative magnitudes of the current jitter and motion intensities. When the jitter intensity is high, the w value approaches 1.0; when the motion intensity is high and the jitter is weak, the w value decreases appropriately. The system also incorporates a vector magnitude limiting mechanism, where the composite displacement vector magnitude... Exceeding the preset maximum displacement threshold Scale proportionally to avoid overcompensation.

[0097] Step 505: Determine the compensation coordinate position of the pixel to be compensated based on the comprehensive displacement vector.

[0098] Specifically, based on the comprehensive displacement vector The compensation coordinates are calculated using the original coordinates (x, y) of the pixel to be compensated, employing the principle of inverse mapping: .

[0099] in, The compensation coordinates (in pixels) are the positions of the pixels to be compensated. Since target coordinates are usually non-integer values, the system employs a sub-pixel precision preservation strategy, retaining non-integer coordinates to two decimal places. For pixels whose target coordinates exceed the image boundary, boundary reflection or boundary extension methods are used to constrain them to the effective image area. The system establishes a coordinate mapping table to record the original coordinates and corresponding compensation coordinate positions of each pixel to be compensated, providing an accurate geometric basis for subsequent acquisition of neighboring pixel sets and weighted interpolation calculations.

[0100] Step 60: Perform weighted interpolation calculation on the set of neighboring pixels to obtain the optimized grayscale value of the pixel to be compensated.

[0101] Specifically, to achieve high-quality compensation, this embodiment employs cubic polynomial interpolation to optimize the calculation of the grayscale values ​​of the pixels to be compensated. Compared to bilinear interpolation, cubic polynomial interpolation uses a 4×4 neighborhood, totaling 16 pixels, to obtain the grayscale values ​​of the final interpolation points, resulting in a smoother effect. The system first constructs matrix B: ; where (x,y) represents the coordinate position of the pixel to be compensated.

[0102] Simultaneously construct coefficient matrices A and C: ; where the offset parameters δx and δy are: δx and δy represent the offset of the point to be compensated relative to integer coordinates. These offsets are directly related to the jitter compensation parameters. δx is obtained from the translation compensation parameter Tx, and δy is obtained from the translation compensation parameter Ty. The values ​​are all in the range of [0, 1].

[0103] The cubic function S(ω) is: ; In S(ω), ω is a distance parameter representing the relative distance between the interpolation point and the reference pixel. This distance is a normalized distance, with pixel spacing as the unit parameter. 'a' is a coefficient controlling the smoothness of the interpolation, ranging from -0.5 to -0.75. ω represents the relative distance, which can be δx or δy. When |ω| < 1, it represents the distance between the interpolation point and its nearest neighbor pixel; when 1 ≤ |ω| < 2, it represents the distance to the second nearest neighbor pixel; when |ω| ≥ 2, it represents a distant pixel whose influence can be ignored. For example, in matrix A, ω corresponds to (1+δx), δx, (1-δx), and (2-δx); while in matrix C, ω corresponds to (1+δy), δy, (1-δy), and (2-δy). That is, the elements of matrices A and C are calculated based on the cubic function S(ω). S(ω) is the core function in the entire interpolation calculation process, used to generate weight coefficients. These weight coefficients participate in the final interpolation calculation through matrices A and C. Finally, the optimized grayscale value of the pixel to be compensated is obtained through matrix operations A·B·C. This method based on polynomial cubic interpolation can effectively preserve image details, reduce jagged edges and blurring effects during the interpolation process, and improve the visual quality of the compensated image.

[0104] Step 70: Perform pixel compensation processing on the corresponding pixels to be compensated in the enhanced image sequence based on the optimized grayscale values ​​to generate the target video image sequence after eliminating jitter.

[0105] Specifically, after obtaining the optimized grayscale values ​​of the pixels to be compensated, the system needs to perform compensation processing on the corresponding pixels in the enhanced image sequence to effectively eliminate jitter. For each pixel to be compensated, the system directly replaces its original grayscale value with the optimized grayscale value calculated through polynomial cubic interpolation. This direct replacement method is simple and effective, ensuring the accuracy of the compensation effect. To ensure the continuity and naturalness of the compensation effect, the system also needs to optimize the compensation result. First, edge detection is performed on the compensated image, focusing on the edge continuity of the area surrounding the pixel to be compensated. If any breaks or discontinuities are found in the edges, the compensation result in the corresponding area needs to be locally adjusted to ensure a smooth transition of the image edges.

[0106] Secondly, the system also needs to consider the continuity in the temporal dimension. By comparing the grayscale value changes of corresponding pixels in adjacent frames, abrupt regions caused by compensation are identified. For these regions, the system adopts a local smoothing process, comprehensively considering the grayscale values ​​of the current frame and adjacent frames to ensure a smooth transition of the video sequence in the temporal dimension. This processing method can effectively avoid flickering or jump phenomena that occur during the compensation process.

[0107] To further improve the compensation effect, the system will also perform an overall performance evaluation of the compensation results. This mainly includes the following aspects: First, checking whether the transition between the compensated and uncompensated areas is natural, ensuring that there are no obvious compensation traces; second, analyzing the contrast and detail retention of local areas to avoid detail loss or blurring caused by the compensation process; and finally, evaluating the overall visual effect to ensure that the compensated image is both stable and clear. If the compensation effect in some areas is found to be unsatisfactory during the evaluation process, the system will perform targeted optimization. For example, for areas with obvious compensation traces, the compensation range can be expanded and a gradual transition can be used to improve the image; for areas with lost details, high-frequency information of the original image can be appropriately preserved; and for areas with blurred edges, edge enhancement processing can be introduced to improve clarity.

[0108] This pixel compensation method, based on optimized grayscale values ​​and combined with multi-level optimization strategies, achieves precise compensation for the effects of jitter. The resulting target video image sequence not only effectively eliminates the effects of jitter but also maintains natural image transitions and detail, significantly improving overall visual quality. The compensated video sequence exhibits stable image quality.

[0109] Please refer to Figure 2 This is a comparative schematic diagram of video frame image jitter removal provided in an embodiment of this application.

[0110] Combination Figure 2 As can be seen, the image on the left is the original video frame image acquired from the carrier platform (such as a drone). It is obvious that there are slight misalignments and ghosting on both sides of the crosshair for the two cars and the road sign. These are due to the image frame misalignment caused by the shaking of the carrier platform. The image on the right is the target video frame image after the shaking has been eliminated by the solution of this application. The outlines of the vehicles and the road sign are aligned with the height of the crosshair, the ghosting is eliminated, and the image is stable and clear, indicating that the shaking has been compensated and the target position has been restored to normal.

[0111] Please see Figure 3 This is a schematic diagram of a system for eliminating image and video jitter according to an embodiment of this application, wherein the system includes: The image enhancement processing module is used to acquire the original video image sequence and corresponding attitude sensor data collected by the carrier platform, and to enhance the original video image sequence to obtain an enhanced image sequence. The motion vector calculation module is used to calculate the global motion vector between adjacent image frames based on the attitude sensor data and the enhanced image sequence; The motion filtering module is used to perform motion filtering on each of the global motion vectors to obtain the separated jitter components and effective motion components. The compensation parameter determination module is used to calculate jitter compensation parameters based on the jitter components and determine motion holding parameters in combination with the effective motion components, wherein the jitter compensation parameters include translation compensation parameters and rotation compensation parameters; The compensation position determination module is used to determine the compensation coordinate position of the pixel to be compensated in the enhanced image sequence based on the jitter compensation parameters and the motion preservation parameters, and to obtain the set of neighboring pixels of the compensation coordinate position; The pixel interpolation calculation module is used to perform weighted interpolation calculation on the set of neighboring pixels to obtain the optimized grayscale value of the pixel to be compensated. The pixel compensation processing module is used to perform pixel compensation processing on the corresponding pixels to be compensated in the enhanced image sequence according to the optimized grayscale value, so as to generate the target video image sequence after eliminating jitter.

[0112] Optionally, the motion vector calculation module is also used to extract the angle change and displacement change of the carrier platform from the attitude sensor data; The angle change and displacement change are converted into motion components in the image coordinate system; In the enhanced image sequence, uniformly distributed feature points are selected as motion reference points; Calculate the actual displacement vector of the motion reference point between adjacent image frames; The target deviation value is obtained by comparing the difference between the motion component and the actual displacement vector and correcting the deviation. The global motion vector between adjacent image frames is determined based on the target deviation value.

[0113] Optionally, the motion filtering processing module is also used to establish a time-series change trajectory diagram based on each of the global motion vectors; Identify the turning points of the motion direction and the points of trajectory discontinuity in the time-series trajectory diagram; Based on the turning points and trajectory discontinuities, the time series of each global motion vector is divided into multiple motion segments; Calculate the rate of change and duration of the global motion vector within each motion segment; The global motion vector whose motion change rate is greater than a preset change rate and whose duration is less than a preset duration is taken as the jitter component, and the global motion vector whose motion change rate is not greater than a preset change rate or whose duration is not less than a preset duration is taken as the effective motion component.

[0114] Optionally, the compensation parameter determination module is also used to perform frequency domain transformation on the jitter component to obtain a jitter spectrum distribution map; Identify the primary and secondary frequency components in the jitter spectrum distribution diagram; Based on the amplitude and phase information of the main frequency components, the intensity coefficient of the jitter main direction is calculated; Based on the distribution characteristics of the secondary frequency components, determine the jitter coupling direction and coupling strength; The intensity coefficient of the jitter main direction is converted into translation compensation parameters, and the jitter coupling direction and coupling intensity are converted into rotation compensation parameters.

[0115] Optionally, the compensation parameter determination module is also used to analyze the vector correlation and directional consistency between adjacent effective motion components; The vector correlation coefficient is calculated based on the vector correlation, and the directional deviation angle is calculated based on the directional consistency. The continuity interval and the jump interval are determined based on the vector correlation coefficient and the direction deviation angle; The motion inertia retention strength is calculated for the effective motion components within the continuous interval, and the motion transition retention strength is calculated for the effective motion components within the jump interval. The motion inertia retention strength and motion transition retention strength are combined to form the motion retention parameter.

[0116] Optionally, the compensation location determination module is also used to determine the pixels to be compensated in the enhanced image sequence; Calculate the compensation displacement vector of the pixel to be compensated based on the jitter compensation parameters; Calculate the motion displacement vector of the pixel to be compensated based on the motion retention parameters; The compensation displacement vector and the motion displacement vector are vector-synthesized to obtain a comprehensive displacement vector. The compensation coordinates of the pixel to be compensated are determined based on the comprehensive displacement vector.

[0117] Optionally, the compensation location determination module is further configured to identify candidate pixels affected by jitter in the enhanced image sequence based on the jitter component; The jitter impact intensity of each candidate pixel is calculated based on the jitter compensation parameters. Candidate pixels whose jitter intensity is greater than a preset threshold are identified as pixels to be compensated.

[0118] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0119] This application also provides a computer storage medium that can store multiple instructions. The instructions are adapted to be loaded by a processor and executed by the above-described method for eliminating image and video jitter. For details of the execution process, please refer to the specific description of the above-described embodiments, which will not be repeated here.

[0120] Please refer to Figure 4 This application also discloses an electronic device. Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 300 may include: at least one processor 301, at least one network interface 304, a user interface 303, a memory 305, and at least one communication bus 302.

[0121] The communication bus 302 is used to enable communication between these components.

[0122] The user interface 303 may include a display screen and a camera. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0123] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0124] The processor 301 may include one or more processing cores. The processor 301 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 305, and by calling data stored in memory 305. Optionally, the processor 301 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 301 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 301 and may be implemented as a separate chip.

[0125] The memory 305 may include random access memory (RAM) or read-only memory. Optionally, the memory 305 may include a non-transitory computer-readable storage medium. The memory 305 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 305 may also be at least one storage device located remotely from the aforementioned processor 301. (Refer to...) Figure 4 The memory 305, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for a method of eliminating image and video jitter.

[0126] exist Figure 4 In the illustrated electronic device 300, the user interface 303 is mainly used to provide an input interface for the user and acquire user input data; while the processor 301 can be used to call an application program stored in the memory 305 for a method of image and video anti-shake. When executed by one or more processors 301, the electronic device 300 performs one or more methods as described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0127] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0128] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0129] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0131] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0132] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will readily conceive of those skilled in the art upon consideration of the specification and the disclosure of practical truths.

[0133] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art that are not described in this disclosure.

Claims

1. A method for eliminating image / video jitter, characterized in that, The method includes: The original video image sequence and corresponding attitude sensor data collected by the carrier platform are acquired, and the original video image sequence is enhanced to obtain an enhanced image sequence. Based on the attitude sensor data and the enhanced image sequence, calculate the global motion vector between adjacent image frames; Motion filtering is performed on each of the global motion vectors to obtain the separated jitter components and effective motion components; The jitter compensation parameters are calculated based on the jitter components, and the motion holding parameters are determined in combination with the effective motion components, wherein the jitter compensation parameters include translation compensation parameters and rotation compensation parameters; Based on the jitter compensation parameters and the motion preservation parameters, the compensation coordinate positions of the pixels to be compensated in the enhanced image sequence are determined, and the set of neighboring pixels of the compensation coordinate positions is obtained. Weighted interpolation is performed on the set of neighboring pixels to obtain the optimized grayscale value of the pixel to be compensated. Pixel compensation processing is performed on the corresponding pixels to be compensated in the enhanced image sequence based on the optimized grayscale value to generate the target video image sequence after eliminating jitter.

2. The method for eliminating image and video jitter according to claim 1, characterized in that, The step of calculating the global motion vector between adjacent image frames based on the attitude sensor data and the enhanced image sequence includes: Extract the angle change and displacement change of the carrier platform from the attitude sensor data; The angle change and displacement change are converted into motion components in the image coordinate system; In the enhanced image sequence, uniformly distributed feature points are selected as motion reference points; Calculate the actual displacement vector of the motion reference point between adjacent image frames; The target deviation value is obtained by comparing the difference between the motion component and the actual displacement vector and correcting the deviation. The global motion vector between adjacent image frames is determined based on the target deviation value.

3. The method for eliminating image and video jitter according to claim 1, characterized in that, The step of performing motion filtering on each of the global motion vectors to obtain the separated jitter components and effective motion components includes: A time-series trajectory diagram is established based on each of the aforementioned global motion vectors; Identify the turning points of the motion direction and the points of trajectory discontinuity in the time-series trajectory diagram; Based on the turning points and trajectory discontinuities, the time series of each global motion vector is divided into multiple motion segments; Calculate the rate of change and duration of the global motion vector within each motion segment; The global motion vector whose motion change rate is greater than a preset change rate and whose duration is less than a preset duration is taken as the jitter component, and the global motion vector whose motion change rate is not greater than a preset change rate or whose duration is not less than a preset duration is taken as the effective motion component.

4. The method for eliminating image and video jitter according to claim 1, characterized in that, The step of calculating jitter compensation parameters based on the jitter components includes: The jitter components are subjected to frequency domain transformation to obtain the jitter spectrum distribution map; Identify the primary and secondary frequency components in the jitter spectrum distribution diagram; Based on the amplitude and phase information of the main frequency components, the intensity coefficient of the jitter main direction is calculated; Based on the distribution characteristics of the secondary frequency components, determine the jitter coupling direction and coupling strength; The intensity coefficient of the jitter main direction is converted into translation compensation parameters, and the jitter coupling direction and coupling intensity are converted into rotation compensation parameters.

5. The method for eliminating image and video jitter according to claim 1, characterized in that, The determination of motion maintenance parameters by combining the effective motion components includes: Analyze the vector correlation and directional consistency between adjacent effective motion components; The vector correlation coefficient is calculated based on the vector correlation, and the directional deviation angle is calculated based on the directional consistency. The continuity interval and the jump interval are determined based on the vector correlation coefficient and the direction deviation angle; The motion inertia retention strength is calculated for the effective motion components within the continuous interval, and the motion transition retention strength is calculated for the effective motion components within the jump interval. The motion inertia retention strength and motion transition retention strength are combined to form the motion retention parameter.

6. The method for eliminating image and video jitter according to claim 1, characterized in that, Determining the compensation coordinates of the pixels to be compensated in the enhanced image sequence based on the jitter compensation parameters and the motion preservation parameters includes: Identify the pixels to be compensated in the enhanced image sequence; Calculate the compensation displacement vector of the pixel to be compensated based on the jitter compensation parameters; Calculate the motion displacement vector of the pixel to be compensated based on the motion retention parameters; The compensation displacement vector and the motion displacement vector are vector-synthesized to obtain a comprehensive displacement vector. The compensation coordinates of the pixel to be compensated are determined based on the comprehensive displacement vector.

7. The method for eliminating image and video jitter according to claim 6, characterized in that, Determining the pixels to be compensated in the enhanced image sequence includes: Based on the jitter component, candidate pixels with jitter effects are identified in the enhanced image sequence; The jitter impact intensity of each candidate pixel is calculated based on the jitter compensation parameters. Candidate pixels whose jitter intensity is greater than a preset threshold are identified as pixels to be compensated.

8. A system for eliminating image and video jitter, characterized in that, The system includes: The image enhancement processing module is used to acquire the original video image sequence and corresponding attitude sensor data collected by the carrier platform, and to enhance the original video image sequence to obtain an enhanced image sequence. The motion vector calculation module is used to calculate the global motion vector between adjacent image frames based on the attitude sensor data and the enhanced image sequence; The motion filtering module is used to perform motion filtering on each of the global motion vectors to obtain the separated jitter components and effective motion components. The compensation parameter determination module is used to calculate jitter compensation parameters based on the jitter components and determine motion holding parameters in combination with the effective motion components, wherein the jitter compensation parameters include translation compensation parameters and rotation compensation parameters; The compensation position determination module is used to determine the compensation coordinate position of the pixel to be compensated in the enhanced image sequence based on the jitter compensation parameters and the motion preservation parameters, and to obtain the set of neighboring pixels of the compensation coordinate position; The pixel interpolation calculation module is used to perform weighted interpolation calculation on the set of neighboring pixels to obtain the optimized grayscale value of the pixel to be compensated. The pixel compensation processing module is used to perform pixel compensation processing on the corresponding pixels to be compensated in the enhanced image sequence according to the optimized grayscale value, so as to generate the target video image sequence after eliminating jitter.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores multiple instructions suitable for being loaded by a processor and executed as described in any one of claims 1-7.

10. An electronic device, characterized in that, It includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

Images