Real-time video anti-shake method based on adaptive motion modeling and artificial intelligence prediction

Abstract

The present application relates to the field of image processing and computer vision, and particularly relates to a real-time video anti-shake method based on adaptive motion modeling and artificial intelligence prediction, comprising: establishing a grid coordinate system mapping relationship; extracting feature points from a video frame and calculating preliminary displacement of grid vertices; constructing a sliding time window, dynamically adjusting and optimizing parameters according to scene characteristics, establishing a time domain smooth optimization objective function, and introducing a frequency prediction constraint when periodic motion is detected; using a parallel iterative algorithm to solve the objective function to obtain steady-state displacement; and generating a stable frame based on the steady-state displacement. The present application realizes real-time processing of a video on a general CPU through a pre-computation caching strategy and vectorized parallel solving; and realizes adaptive prediction compensation of periodic motion and intelligent identification of a scene through FFT frequency detection and a lightweight machine learning model.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, and in particular to a real-time video stabilization method based on adaptive motion modeling and artificial intelligence prediction. Background Technology

[0002] With the widespread use of mobile imaging devices, video stabilization technology has become crucial for improving image quality. Existing video stabilization technologies mainly include global stabilization methods based on homography or affine transformation, and local stabilization methods based on mesh-based warping.

[0003] In existing image stabilization technologies, especially in applications involving motion devices such as drones, quadruped robots, or wearable cameras, video footage is often susceptible to periodic or regular shaking. Conventional methods typically employ time-domain smoothing filters to eliminate shaking, but when dealing with non-random shaking with a fixed rhythm (such as walking gait or mechanical vibration), there is often a significant hysteresis effect, resulting in "ghosting" or rhythm mismatch in the stabilized image.

[0004] Furthermore, existing high-precision local image stabilization methods often come with enormous computational overhead. To achieve good local correction results, it is usually necessary to solve large-scale objective functions, making the algorithms highly dependent on high-performance graphics processing units (GPUs) or dedicated acceleration hardware. On low-power computing platforms with only general-purpose central processing units (CPUs), existing image stabilization schemes struggle to meet real-time processing requirements (e.g., 30 frames per second) while maintaining high resolution (e.g., 720p and above). At the same time, most existing technologies employ fixed-parameter strategies, lacking the ability to adaptively identify and dynamically schedule motion scenes (e.g., stationary, uniform motion, severe vibration), making them prone to image stabilization failure or resource waste during complex scene transitions.

[0005] Therefore, there is an urgent need for a technical solution that can address the aforementioned issues of real-time bottlenecks, lack of periodic predictions, and weak scene adaptability. Summary of the Invention

[0006] The purpose of this invention is to provide a real-time video stabilization method based on adaptive motion modeling and artificial intelligence prediction, in order to solve the problems of high algorithm computation latency that makes it difficult to meet the real-time requirements of CPU, insufficient ability to predict and compensate for periodic jitter, and poor scene adaptability in the existing technology.

[0007] To achieve the above-mentioned objectives, the technical solution provided by this invention includes: A real-time video stabilization method based on adaptive motion modeling and artificial intelligence prediction, the method comprising the following steps: Establish a mapping relationship for the grid coordinate system, wherein the mapping relationship includes the initial coordinates of the grid vertices and a pixel mapping table; Feature points are extracted from the video frame to be processed, and the preliminary displacement of the grid vertices is calculated based on the extracted feature points. A sliding time window containing the current frame and historical frames is constructed. The optimization parameters are dynamically adjusted according to the processing delay or scene characteristics of the current frame. An objective function for temporal smoothing optimization is established. When periodic motion is detected, the objective function introduces frequency prediction constraints. The objective function is solved based on the optimization parameters using a parallel iterative algorithm, and the initial displacement of the grid vertex within the sliding time window is optimized by temporal smoothing to obtain the steady-state displacement; Based on the steady-state displacement and the pixel mapping table, the video frame to be processed is remapped to generate a stable video frame.

[0008] Preferably, the step of using a parallel iterative algorithm to perform temporal smoothing optimization on the initial displacement of the mesh vertices within the sliding time window includes: Construct a system of linear equations with the target displacements of all grid vertices within the sliding time window as unknowns. The system of linear equations includes data fidelity terms and time-domain smoothing terms. The Jacobi iterative method is used to solve the linear equations in batches in parallel. On a general-purpose processor including a CPU, the displacement values ​​of all the mesh vertices within the sliding time window are updated simultaneously through vectorized operations until the convergence condition is met or the preset number of iterations is reached.

[0009] Preferably, the introduction of frequency prediction constraints when periodic motion is detected includes: Perform a short-time Fourier transform (FFT) on the historical motion trajectories of the grid vertices to detect the dominant frequency of the motion signal; If the significance of the dominant frequency is higher than a preset threshold, it is determined that there is periodic motion in the current scene; Based on the detected dominant frequency and historical phase information, the grid vertex displacement of future frames is predicted, and the prediction result is added as an additional constraint to the objective function of the temporal smoothing optimization.

[0010] Preferably, the step of dynamically adjusting the optimization parameters based on the processing delay or scene characteristics of the current frame includes: A lightweight machine learning model is used to extract the mean of feature point motion vectors, the variance of feature point motion vectors, and texture features of the current frame; Identify the scene type to which the current frame belongs, wherein the scene type includes at least a static scene, a steady motion scene, and a violent vibration scene; Based on the identified scene type and the confidence level output by the lightweight machine learning model, the smoothing weights of the temporal smoothing optimization and the length of the sliding time window are adaptively adjusted.

[0011] Preferably, the dynamically adjusted optimization parameters further include: Real-time monitoring of the processing latency of the current frame; If the calculated average processing latency exceeds the preset frame rate threshold, a degradation strategy will be triggered. The degradation strategy includes lowering the maximum number of feature points extracted, reducing the number of iterations of the parallel iterative algorithm, or reducing the mesh density.

[0012] Preferably, the mapping relationship for establishing the grid coordinate system includes: The vertex coordinates of the grid cells and the pixel index range within each grid cell are pre-calculated and cached. When the resolution of the video frame to be processed is consistent with the preset working resolution, the cached mapping relationship is directly invoked.

[0013] Preferably, calculating the preliminary displacement of the mesh vertices includes: The video frame to be processed is divided into multiple sub-frame regions; The Random Sample Consensus (RANSAC) algorithm is run independently within each subframe region to eliminate outliers and calculate the local homography transformation. Merge the interior points of all subframe regions and calculate the global transformation displacement of the grid vertices.

[0014] Preferably, calculating the preliminary displacement of the mesh vertices further includes: Calculate the residual vector between the actual tracking displacement of the feature point and the global transformation displacement; Map the residual vector to the grid vertices in the neighborhood; The residual vectors collected from each of the mesh vertices are vectorized and statistically aggregated to obtain the local compensation displacement.

[0015] Beneficial effects 1. This invention solves the objective function for temporal smoothing optimization using a parallel iterative algorithm. It leverages the processor's vectorized computational capabilities to simultaneously update the displacement values ​​of all grid vertices within the sliding time window, significantly improving numerical solution efficiency. Combined with a pre-computation strategy for grid coordinate system mapping, it greatly reduces the computational latency of single-frame processing, making real-time image stabilization of high-resolution video possible without GPU acceleration.

[0016] 2. This invention, through frequency domain analysis (such as FFT detection) of the grid vertex trajectory, can automatically extract the inherent motion frequency of the device, thereby introducing targeted frequency prediction constraints into the objective function. Compared with traditional hysteresis smoothing, this prediction mechanism can provide feedforward compensation, effectively counteracting periodic shaking caused by gait, mechanical vibration, etc., making the output image more stable and natural.

[0017] 3. This invention introduces a lightweight machine learning model for scene recognition and adjusts optimization parameters (such as smoothing weights and sampling density) in real time based on processing latency. This dynamic scheduling strategy enables the system to automatically find the optimal balance between mass and speed under different motion intensities and hardware loads, providing both smoothing for intense motion and effective suppression of local outliers (such as occluders and dynamic objects). Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a preferred embodiment of the real-time video stabilization method based on adaptive motion modeling and artificial intelligence prediction provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings.

[0020] The real-time video stabilization method based on adaptive motion modeling and artificial intelligence prediction described in this invention can be implemented in a control system including a computer processor, memory, and communication interface. In a typical embodiment, the system runs on an embedded device or mobile terminal equipped with a general-purpose CPU (such as an ARM Cortex-A series or Intel / AMD x86 processor), without requiring a GPU or dedicated AI acceleration chip. The system acquires video frame data (e.g., 1280×720 resolution, 30FPS) in real time through a camera interface, and the processed stable video frames can be output to a display device, storage device, or network transmission interface.

[0021] Example 1 like Figure 1 As shown, this embodiment provides a real-time video stabilization method based on adaptive motion modeling and artificial intelligence prediction, specifically including: S1. Establish the mapping relationship of the grid coordinate system, wherein the mapping relationship includes the initial coordinates of the grid vertices and the pixel mapping table.

[0022] Traditional video stabilization methods require independent processing of each pixel, which is computationally intensive for 720p resolution (approximately 920,000 pixels at 1280×720), making real-time processing on general-purpose CPUs difficult. This invention employs a gridded representation, discretizing the continuous pixel space into a finite number of control points (grid vertices). For example, an 8×8 grid corresponds to 9×9, resulting in 81 vertices, reducing the scale of the subsequent optimization problem by four orders of magnitude.

[0023] In the specific implementation, the number of grid rows and columns is set (default 8×8 units, corresponding to 9×9 vertices), and the vertex coordinates are represented as integer pixel arrays and stored as a template. The entire frame's pixel coordinate grid is pre-calculated, generating two float 32 matrices (x and y) with dimensions of height×width. For each grid unit, the bounding box and pixel coordinate array within that unit are pre-calculated for fast indexing during subsequent local homography transformations. If the goal is a higher frame rate, the grid density can be reduced to 6×6; if the goal is higher image stabilization quality, it can be increased to 10×10.

[0024] In practice, rebuilding the aforementioned mesh structure every time a new frame is processed introduces unnecessary computational overhead. However, when the video resolution remains constant, the mesh vertex coordinates and pixel-cell index table are essentially fixed. Therefore, in some preferred embodiments, the above mapping relationships are pre-calculated and cached in memory during system initialization or when the resolution changes. Subsequent frame processing directly calls the cached data, avoiding redundant calculations. Pre-calculation uses the float32 data type to balance precision and memory usage.

[0025] S2. Extract feature points from the video frame to be processed, and calculate the preliminary displacement of the grid vertices based on the extracted feature points.

[0026] First, receive the input frames, ensuring they are the same size. If the input resolution differs from the working resolution, use fast linear interpolation `cv2.resize` to scale them to the target resolution (1280×720), employing `cv2.INTER_LINEAR` as the interpolation mode to minimize overhead. Add the frames to a sliding queue or buffer (capacity equal to the time window size plus output delay), and record the frame timestamp and frame index for time window processing and prediction. Use a circular buffer to avoid frequent memory allocations.

[0027] Detect stable, trackable feature points in the current frame. Keypoints are detected using the FAST (Features from AcceleratedSegment Test) algorithm, and a detector is created using OpenCV's cv2.FastFeatureDetector_create. Parameters are set as follows: threshold=30 (higher values ​​result in fewer and more stable keypoints), nonmaxSuppression=True, and TYPE_9_16 type is used. If the number of detected keypoints exceeds the upper limit max_features (default 500), they are sorted by response value and truncated.

[0028] Conventional methods detect feature points uniformly across the entire frame, but this can easily lead to over-concentration of feature points in texture-rich areas and lack of coverage in sparsely textured areas. To reduce the impact of local occlusion and ensure balanced sampling across the entire image, in some preferred embodiments, feature detection is performed region-by-region on the image according to a preset subframe grid (e.g., 2×2 subframes), with at least 10 features retained in each subframe. If there are insufficient features, the detection threshold is lowered or the neighborhood is expanded. In low-texture scenes, ORB features can be used in conjunction with the BRIEF descriptor to enhance matching stability.

[0029] The optical flow is calculated by determining the corresponding positions of feature points between two consecutive frames. Classical pyramid Lucas-Kanade optical flow tracing (PyrLK) is employed, using the `cv2.calcOpticalFlowPyrLK` function. The input consists of the feature points detected in the previous frame and the grayscale image of the current frame. The LK parameters are set as follows: `winSize=(13,13)`, `maxLevel=1`, `criteria=(cv2.TERM_CRITERIA_EPS|cv2.TERM_CRITERIA_COUNT,10,0.03)`, and `minEigThreshold=1e-3`. Matching point pairs are obtained, and points that failed to be tracked or had low confidence are filtered out using the returned status mask. If CPU resources are limited, `winSize` can be reduced to (9,9) to decrease computational load.

[0030] During feature point tracking, the original matching results may contain a large number of outliers due to the presence of moving objects in the foreground or mismatches. Conventional methods use a single global RANSAC algorithm to remove outliers, but this method is prone to failure when there are multiple independent moving objects or large-scale non-uniform motion in the image, because global RANSAC assumes that most matching points conform to the same transformation model.

[0031] To address this issue, in some preferred embodiments, the present invention divides the entire frame into several subframes (default 2×2). Within each subframe, cv2.findHomography is used in conjunction with RANSAC mode to independently eliminate outliers, retaining the set of inliers. Since the motion within each region is relatively simple, RANSAC exhibits higher convergence and accuracy. If a subframe lacks sufficient features, it is skipped or supplemented from surrounding regions. Subsequently, the inlier pairs of all subframes are merged as feature pairs for the entire frame, and cv2.findHomography or cv2.estimateAffine2D is used to calculate the global transformation matrix. The RANSAC reprojection threshold is set to 3.0 pixels; if local deformation is strong, it can be relaxed to 5.0 pixels. Homography is preferred when the scene is close to a plane; if the motion is mainly translation, rotation, and scaling, affine transformation is more stable.

[0032] Apply the global transformation to the mesh vertices to obtain the geometrically predicted displacements of the vertices (the global motion part). Calculate the new coordinates of all vertices using cv2.perspectiveTransform (or Homography) or affine transformation, and store them as the global displacement field of the vertices (a float32 array of 2 times the number of vertices), storing them sequentially for vectorization in subsequent calculations.

[0033] Global transformation models assume that the entire image undergoes the same rigid motion, but in real-world scenes, local deformations often exist, such as the rolling shutter effect or the relative motion between the foreground and background. Conventional methods, which only use global transformations, cannot effectively handle these local distortions, resulting in local jitter or geometric distortion even after image stabilization.

[0034] Therefore, in some preferred embodiments, the present invention calculates the residual between the actual tracked displacement and the global projection, and uses visual information to compensate for the local vertex velocity. Specifically: First, the residual vector of each matched feature is calculated, that is, the actual tracked position minus the position predicted by the global transformation; then, the spatial coordinates of the feature are mapped to the grid index (normalized by the image width and height and multiplied by the number of grid columns and rows, rounded to the nearest grid point index); for each feature, it is extended to its nearest grid point into a k×k neighborhood (default k=3, radius 1), and the residual value is assigned to these neighborhood vertices; the median of the residuals collected for each vertex is obtained using a vectorized grouping statistical method (e.g., scipy.stats.binned_statistic or NumPy binning aggregation) to obtain the vertex-level residual velocity; the residual results are padded with NaN to 0.

[0035] The median is used instead of the mean because the median is more robust to outliers. If the texture is sparse, k can be increased to 5 to expand the search range, and the minimum number of samples per vertex threshold can be set to 1 or 2.

[0036] The global displacement and residual compensation are added together, and then neighborhood space smoothing and denoising are performed. For each vertex, the velocity is synthesized along the coordinate axes: vertex_velocity = global_velocity + residual_velocity. The resulting vertex velocity field is then subjected to a 3×3 median filter (using cv2.medianBlur or a custom-implemented neighborhood median filter) on both the x and y components to smooth local anomalies. A 5×5 filter kernel can be used for stronger smoothing, but this will reduce the local response.

[0037] S3. Construct a sliding time window that includes the current frame and historical frames, dynamically adjust the optimization parameters according to the processing delay or scene characteristics of the current frame, and establish a temporal smoothing optimization objective function. The objective function introduces frequency prediction constraints when periodic motion is detected.

[0038] Several frames of unstable vertex displacements are stored in a sliding time window for subsequent time-domain optimization. By jointly optimizing vertex displacements within the time series, a smoother trajectory can be achieved in the time domain, thereby reducing short-term jitter and noise.

[0039] A fixed-size circular buffer is used to store the sequence of unstable vertex displacements and corresponding global transformation information for the most recent N frames. The default window length is approximately 1.0 to 1.5 seconds (1.34 seconds recommended), corresponding to a window size of approximately 40 frames at 30 FPS. When the window is not full, it is filled cumulatively; when the window is full, it is updated in a sliding manner (discarding the earliest frame). This can be shortened to 0.6 to 0.8 seconds when resources are limited.

[0040] Construct an objective function with vertex displacement per frame as the variable, including a data fidelity term (to maintain consistency with unstable displacements as much as possible) and a temporal smoothing regularization term (to encourage continuous temporal smoothing). The objective function E is defined as: ; in, For data fidelity, the constraint is that the steady-state displacement does not deviate from the initial displacement estimate. It is defined as the sum of squares of the differences between the steady-state displacement and the initial estimated displacement of all vertices across all frames. For time-domain smoothing, constraining the continuous change of displacement between adjacent frames, it is defined as the sum of squares of the vertex displacement differences between adjacent frames; For periodic prediction terms, when periodic motion is detected, the constrained displacement moves closer to the predicted value, and is defined as the sum of the squares of the differences between the steady-state displacement and the predicted displacement; The time-domain smoothing weight coefficient is used to balance the relative importance of data fidelity and time-domain smoothing. The larger the value, the smoother the trajectory, but the more detailed the response may be lost. These are periodic prediction weighting coefficients used to control the strength of prediction constraints, and they only take effect when periodic motion is detected.

[0041] Among them, the data fidelity term constrains the steady-state displacement from deviating from the initial displacement estimate, the time-domain smoothing term constrains the continuous change of displacement between adjacent frames, and the periodic prediction term (condition enabled) constrains the displacement to approach the predicted value.

[0042] In scenarios involving walking, running, or robotic gait, camera shake exhibits a distinct periodicity. Existing smoothing filtering methods (such as Kalman filtering and Gaussian filtering) have an inherent lag in their response to periodic motion, resulting in a trailing effect in the image. Some existing technologies attempt to fit periodic motion using preset sine wave models, but this method cannot adaptively detect the vibration frequency of the actual device and performs poorly in non-standard or dynamically changing gait modes.

[0043] Therefore, in some preferred embodiments, the present invention automatically detects periodic features and performs feedforward compensation through frequency domain analysis. The periodic signal can be represented by the dominant frequency in the frequency domain from the historical vertex displacement sequence. Specifically, this includes: collecting vertex or global centroid motion sequences from the most recent several frames (e.g., within 1 to 2 seconds, i.e., 30 to 60 frames); performing a short-time Fourier transform (FFT) on the sequence and identifying the dominant frequency and its significance (peak height divided by the noise baseline). The significance threshold for the dominant peak is set to require the peak amplitude to be 2 to 3 times higher than the neighborhood average; if the peak is significant, periodic motion is considered to exist. Based on the detected dominant frequency and historical phase information, the expected velocity for the next 1 to 3 frames is predicted using sine function extrapolation. The predicted velocity is added as an additional constraint (with adjustable weight) to the time domain optimization, so that the final steady-state trajectory satisfies both smoothness and the predicted trend.

[0044] Compared to methods using a pre-set fixed sine wave model, this FFT-based method can automatically adapt to different gait frequencies (e.g., walking at approximately 1 to 2 Hz, running at approximately 2 to 3 Hz, robotic gait, etc.), exhibiting stronger adaptability. In other embodiments, lightweight temporal models (e.g., single-layer LSTM or small 1D-CNN) can be used for short-term prediction of historical sequences, with the AI ​​model inference budget controlled within 5ms (CPU).

[0045] Different shooting scenarios have significantly different requirements for image stabilization strategies: static scenes require strong smoothing to eliminate minor shakes, while fast-moving scenes require weaker smoothing to preserve motion fluidity. Existing methods typically use fixed parameter configurations, making it difficult to strike a balance between stability and responsiveness; hard-coded rules cannot cover all possible scene combinations; and end-to-end deep learning models are computationally too demanding to meet the real-time requirements of CPUs.

[0046] Therefore, in some preferred embodiments, this invention utilizes a lightweight machine learning model to achieve scene adaptation. A lightweight classifier (e.g., a single-layer hidden layer MLP or a small CNN) is designed, using low-dimensional input features: mean inter-frame optical flow, optical flow variance, image texture entropy, maximum displacement amplitude, detected dominant frequency, etc., with the input feature vector length not exceeding 32 dimensions. The classifier outputs a scene label (e.g., stationary, steady movement, periodic gait, strong oscillation) and a confidence value. The classification result is mapped to a parameter strategy: for example, prediction is enabled and the prediction weight is appropriately increased for periodic scenes, while the window is shortened for strong oscillation scenes. The scene classifier inference time budget is controlled within 5ms (CPU). A confidence threshold is used to enable prediction, and a value of no less than 0.7 is recommended.

[0047] Simultaneously, affine components and translations are extracted from the global transformation matrix and mapped to weight values ​​using a set of empirical formulas (mapping larger translations or strong affine distortions to smaller smoothing weights, and vice versa), and the weights are truncated from 0 to the upper limit (empirical values ​​range from 0 to 10). This reduces oversmoothing during intense motion to avoid motion blur.

[0048] S4. Using a parallel iterative algorithm, the objective function is solved based on the optimization parameters, and the initial displacement of the grid vertex within the sliding time window is optimized by temporal smoothing to obtain the steady-state displacement.

[0049] The objective function constructed in step S3 is a large-scale sparse linear least squares problem. Direct methods (such as Cholesky decomposition) are accurate but computationally complex, making them unsuitable for real-time requirements. Iterative methods, while requiring multiple iterations to converge, have low computational cost per iteration and naturally support parallelization, making them very suitable for the SIMD architecture of modern CPUs.

[0050] The normal equation corresponding to the objective function is expressed as a matrix form Ap = b, where A is a sparse symmetric positive definite matrix, p is the steady-state displacement vector to be solved, and b is the right-hand vector composed of the initial displacement and constraints. A time-domain weight matrix is ​​constructed (with Gaussian decay of time distance) and adaptive weights for each frame are incorporated. The system is organized into diagonal and off-diagonal matrix coefficients, and a batch parallelization method (using numpy.einsum) is used to simultaneously perform Jacobi iterative updates on all vertices, which significantly improves CPU vectorization efficiency.

[0051] The Jacobi iterative method is used to solve the problem. ; in, This is the steady-state displacement vector after the kth iteration, containing the displacement values ​​of all mesh vertices in all frames within the sliding time window; Let be the steady-state displacement vector after the (k+1)th iteration update; D is the diagonal matrix of the coefficient matrix A; This is the off-diagonal part of the coefficient matrix A, which contains the coupling relationships between adjacent frames; The right-hand vector is composed of a weighted combination of the preliminary displacement estimate and various constraints.

[0052] The default number of iterations is 50 (30 to 60 is recommended), and a lightweight convergence check (e.g., residual convergence threshold) is performed to exit early. To reduce latency, the number of iterations can be reduced to 10 to 20 with a shorter time window. Parallel computation is prioritized using NumPy vectorization; batch computation using numpy.einsum can complete a single iteration in approximately 0.5ms on a general-purpose CPU.

[0053] The parallelism of Jacobi iterations stems from its separate update characteristic: in the (k+1)th iteration, the new value of each unknown depends only on the values ​​of other unknowns in the kth iteration. This means that all unknowns can be updated simultaneously and independently, making it very suitable for SIMD vectorization and multi-core parallelism.

[0054] In other embodiments, more advanced iterative solvers (such as parallel Gaussian-Seidel, conjugate gradients) can be used, but a trade-off between implementation complexity and real-time performance is required.

[0055] The CPU performance of mobile devices fluctuates significantly due to factors such as temperature, battery status, and background tasks. Fixed parameter configurations may lead to frame drops under high load and waste computing power under low load.

[0056] To ensure real-time performance under varying load conditions, in some preferred embodiments, this invention monitors processing latency and dynamically adjusts parameters during runtime. Processing time is recorded at the beginning and end of each frame, and the average latency and variation are calculated. A threshold is set: if the processing time for a single frame exceeds 80% of the desired frame interval (e.g., exceeding 26ms at 30FPS), a lightweight mode is triggered, and the following degradation measures are executed sequentially: reducing grid density (e.g., from 8×8 to 6×6); reducing the upper limit of the number of features (e.g., from 500 to 300); reducing the number of time-domain iterations (e.g., from 50 to 15); temporarily disabling AI prediction or reducing the frequency of AI inference. If the processing time falls below the threshold and stabilizes, a high-quality parameter set is restored.

[0057] Use the exponential moving average (EMA) to track latency to avoid over-responding to transient jitter, with the switching threshold set to a value greater than 80% of the target frame interval for the EMA latency.

[0058] S5. Based on the steady-state displacement and the pixel mapping table, perform image remapping on the video frame to be processed to generate a stable video frame.

[0059] The deformation positions of each vertex to be applied in the current output frame are calculated from the time-domain optimization results. For each mesh cell, a local homography or affine transformation is calculated using the vertices of the stable and unstable cells (using cv2.findHomography or cv2.estimateAffine2D). The target coordinates of each pixel within the cell are mapped to the original unstable frame coordinates using the local transformation, filling the corresponding regions of frame_x_map and frame_y_map. A uniformity strategy (e.g., smooth vertex interpolation) is used for boundary pixels and cross-cell connections to avoid stitching marks. If the cell deformation is close to affine, the affine transformation can be solved first to save computation.

[0060] The calculated pixel map is used to upsample the original unstable frame to generate the final stable frame. The `cv2.remap` function is used with interpolation mode `cv2.INTER_LINEAR`, and boundary processing is performed using `cv2.BORDER_REPLICATE` or `cv2.BORDER_REFLECT` to avoid black borders. If cropping is enabled to avoid black borders, a center crop is performed after stabilization, and the image is scaled back to its original resolution. The cropping ratio can be set to 0.85 to 0.95 (adjusted according to the deformation amplitude).

[0061] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A real-time video stabilization method based on adaptive motion modeling and artificial intelligence prediction, characterized in that, The method includes the following steps: Establish a mapping relationship for the grid coordinate system, wherein the mapping relationship includes the initial coordinates of the grid vertices and a pixel mapping table; Feature points are extracted from the video frame to be processed, and the preliminary displacement of the grid vertices is calculated based on the extracted feature points. A sliding time window containing the current frame and historical frames is constructed. The optimization parameters are dynamically adjusted according to the processing delay or scene characteristics of the current frame. An objective function for temporal smoothing optimization is established. When periodic motion is detected, the objective function introduces frequency prediction constraints. The objective function is solved based on the optimization parameters using a parallel iterative algorithm, and the initial displacement of the grid vertex within the sliding time window is optimized by temporal smoothing to obtain the steady-state displacement; Based on the steady-state displacement and the pixel mapping table, the video frame to be processed is remapped to generate a stable video frame.

2. The method according to claim 1, characterized in that, The temporal smoothing optimization of the initial displacement of the mesh vertices within the sliding time window using a parallel iterative algorithm includes: Construct a system of linear equations with the target displacements of all grid vertices within the sliding time window as unknowns. The system of linear equations includes data fidelity terms and time-domain smoothing terms. The Jacobi iterative method is used to solve the linear equations in batches in parallel. On a general-purpose processor including a CPU, the displacement values ​​of all the mesh vertices within the sliding time window are updated simultaneously through vectorized operations until the convergence condition is met or the preset number of iterations is reached.

3. The method according to claim 1 or 2, characterized in that, The introduction of frequency prediction constraints when periodic motion is detected includes: Perform a short-time Fourier transform (FFT) on the historical motion trajectories of the grid vertices to detect the dominant frequency of the motion signal; If the significance of the dominant frequency is higher than a preset threshold, it is determined that there is periodic motion in the current scene; Based on the detected dominant frequency and historical phase information, the grid vertex displacement of future frames is predicted, and the prediction result is added as an additional constraint to the objective function of the temporal smoothing optimization.

4. The method according to claim 1, characterized in that, The dynamic adjustment and optimization parameters based on the processing delay or scene characteristics of the current frame include: A lightweight machine learning model is used to extract the mean of feature point motion vectors, the variance of feature point motion vectors, and texture features of the current frame; Identify the scene type to which the current frame belongs, wherein the scene type includes at least a static scene, a steady motion scene, and a violent vibration scene; Based on the identified scene type and the confidence level output by the lightweight machine learning model, the smoothing weights of the temporal smoothing optimization and the length of the sliding time window are adaptively adjusted.

5. The method according to claim 1, characterized in that, The dynamically adjusted optimization parameters also include: Real-time monitoring of the processing latency of the current frame; If the calculated average processing latency exceeds the preset frame rate threshold, a degradation strategy will be triggered. The degradation strategy includes lowering the maximum number of feature points extracted, reducing the number of iterations of the parallel iterative algorithm, or reducing the grid partitioning density.

6. The method according to claim 1, characterized in that, The mapping relationship for establishing the grid coordinate system includes: The vertex coordinates of the grid cells and the pixel index range within each grid cell are pre-calculated and cached. When the resolution of the video frame to be processed is consistent with the preset working resolution, the cached mapping relationship is directly invoked.

7. The method according to claim 1, characterized in that, The calculation of the preliminary displacement of the mesh vertices includes: The video frame to be processed is divided into multiple sub-frame regions; The Random Sample Consensus (RANSAC) algorithm is run independently within each subframe region to eliminate outliers and calculate the local homography transformation. Merge the interior points of all subframe regions and calculate the global transformation displacement of the grid vertices.

8. The method according to claim 7, characterized in that, The calculation of the preliminary displacement of the mesh vertices also includes: Calculate the residual vector between the actual tracking displacement of the feature point and the global transformation displacement; Map the residual vector to the grid vertices in the neighborhood; The residual vectors collected from each of the mesh vertices are vectorized and statistically aggregated to obtain the local compensation displacement.

Images