Video anti-shake method and system based on multi-scale fusion and adaptive smoothing

Abstract

The invention discloses a video anti-shake method and system based on multi-scale fusion and adaptive smoothing, relates to the technical field of video image processing, and aims to effectively solve the image quality problem caused by shake in a video shooting process. Gradient histograms and wavelet energy distribution characteristics of video frames are extracted through graying and normalization processing, and the gradient histograms and the wavelet energy distribution characteristics are input into a jitter type recognition network to recognize translation, rotation and Z-axis jitter probabilities. And further extracting motion, frequency domain and edge features, and generating multi-modal coupling features through combination of a dynamic feature interaction network and a dot product attention mechanism. And constructing a motion trajectory by using the features, optimizing the trajectory by using a texture perception double-layer smoothing strategy, introducing an adaptive penalty term into a dynamic planning cost function, and outputting a smooth motion compensation parameter. And finally, processing the boundary region through motion compensation and image extrapolation to generate an anti-shake video frame. Through multi-scale feature fusion and a self-adaptive smoothing strategy, the video anti-shake effect is effectively improved, and the method is suitable for complex scenes.

Description

Technical Field

[0001] This invention relates to the field of video image processing, and more specifically to a video stabilization method and system based on multi-scale fusion and adaptive smoothing. Background Technology

[0002] With the widespread use of mobile devices and portable cameras, video shakiness during recording has become an increasingly prominent issue. Video shakiness not only affects the viewing experience but also reduces video quality and can even render video content unusable. Therefore, developing efficient electronic image stabilization technology for video is of significant practical value.

[0003] Currently, video electronic image stabilization technology is mainly divided into two categories: hardware stabilization and software stabilization. Hardware stabilization compensates for camera shake through mechanical devices (such as optical image stabilization and sensor-based image stabilization), but it is costly and has limited applicability. Software stabilization, on the other hand, is achieved through digital image processing technology, and has advantages such as low cost and wide applicability, making it a current research hotspot.

[0004] Early electronic image stabilization methods for video were primarily based on traditional digital image processing techniques. For example, the paper "Real-Time Digital Image Stabilization Using Kalman Filter" proposed a motion trajectory smoothing method based on Kalman filtering. This method uses a Kalman filter to predict and correct motion trajectories between video frames, thereby achieving image stabilization. Furthermore, the paper "Block Matching Algorithms for Motion Estimation" proposed a block matching method that achieves fast image stabilization through block-based motion compensation. This method divides video frames into multiple small blocks, estimates motion by matching the positions of corresponding blocks in adjacent frames, and then performs compensation. However, these traditional methods have significant limitations in feature selection, making it difficult to fully capture the spatiotemporal dynamic characteristics of video.

[0005] In recent years, deep learning technology has provided new solutions for video stabilization. For example, the end-to-end convolutional neural network (CNN) model proposed in the paper "Deep Online Video Stabilization" generates stable videos through adversarial training. This model can automatically learn motion patterns in videos and stabilize them. Nevertheless, existing methods still face the following technical bottlenecks: 1. Limited motion estimation accuracy: Traditional methods have an error rate exceeding 42% in fast-moving scenes (>30° / s), while deep learning models experience a PSNR drop of up to 6.8dB in low-texture regions. 2. Insufficient adaptive adjustment capability: Fixed-parameter smoothing filters produce 12.6% oversmoothing artifacts in dynamic scenes.

[0006] Existing video stabilization methods mainly suffer from the following problems:

[0007] 1. Insufficient motion estimation accuracy: Traditional methods are prone to motion estimation errors in fast motion or complex scenes, resulting in unsatisfactory stabilization performance;

[0008] 2. Insufficient adaptability: Existing methods are often optimized for specific scenarios and are difficult to adapt to the video stabilization requirements under different shooting conditions.

[0009] The above research shows that existing technologies are difficult to balance motion estimation accuracy, feature fusion efficiency, and real-time processing performance. Therefore, it is necessary to develop a video stabilization method that can simultaneously balance motion estimation accuracy, feature extraction comprehensiveness, smoothing effect naturalness, and high computational efficiency. Summary of the Invention

[0010] The technical solution of this invention is to solve the contradiction between motion blur and detail preservation in complex jitter scenarios (such as translation / rotation / Z-axis mixed jitter) and the problem of insufficient adaptability to dynamic scenes.

[0011] This invention solves the above-mentioned technical problems through the following technical means: a video stabilization method based on multi-scale fusion and adaptive smoothing, the method comprising:

[0012] S1. Preprocess the input video frames, extract multi-dimensional feature vectors, and determine the jitter type probability distribution through a pre-trained jitter type recognition network;

[0013] S2. Extract the motion features, frequency domain features, and edge features of the video frame. Combine the jitter type probability distribution and physical criteria, dynamically weight and fuse the three types of features through a dynamic feature interaction network and dot product attention mechanism to generate fused features. Further enhance the fused features through residual connections and output multimodal fused features.

[0014] S3. Construct motion trajectories based on video frame motion information, optimize motion trajectories using a two-layer smoothing strategy, and output motion compensation parameters;

[0015] S4. Perform motion compensation on the video frame according to the motion compensation parameters, and combine the multimodal fusion features to fill the content of the compensated boundary region through a deep learning model, and output the stabilized video frame.

[0016] This invention significantly improves the accuracy and robustness of jitter type recognition by extracting gradient histogram features and wavelet energy distribution features, combining them with a pre-trained jitter type recognition network, and integrating physical criteria, providing a reliable basis for subsequent processing. Through a dynamic feature interaction network and dot product attention mechanism, motion features, frequency domain features, and edge features are dynamically weighted and fused to generate multimodal fusion features. This feature fusion method fully utilizes multi-source information, enhances the completeness and discriminative ability of feature representation, and improves the accuracy of subsequent image processing. A texture-aware two-layer smoothing strategy is adopted to optimize the motion trajectory. Kalman filtering is used to achieve global trajectory smoothing, and local offsets are introduced in textured areas for adaptive fine-tuning. This strategy effectively suppresses high-frequency jitter, maintains the overall stability of the video, and avoids excessive smoothing of textured areas, preserving image details. After motion compensation, combined with multimodal fusion features, a deep learning model is used to fill content in boundary areas. This effectively improves the realism and visual consistency of the filled content in boundary areas, thereby improving the accuracy, robustness, and visual quality of video stabilization.

[0017] Further, S1 includes:

[0018] S11. Perform grayscale processing on the input video frame to obtain a grayscale video frame, and perform normalization processing on the grayscale video frame.

[0019] S12. Calculate the gradient magnitude and direction of the normalized grayscale video frame using the gradient direction histogram method, and statistically analyze the gradient distribution in each direction to form gradient histogram features.

[0020] S13. Perform discrete wavelet transform on the normalized grayscale video frame, calculate the wavelet coefficients of each sub-band and statistically form the wavelet energy distribution characteristics.

[0021] S14. The gradient histogram features and the wavelet energy distribution features are concatenated into a multidimensional feature vector; the multidimensional feature vector is classified using the pre-trained jitter type recognition network to output the probability distributions of translation jitter, rotation jitter, and Z-axis jitter; the jitter type recognition network is a multilayer perceptron structure, including: input layer: receiving multidimensional feature vectors; hidden layer 1: 256 neurons, using the GELU activation function; hidden layer 2: 128 neurons, using the GELU activation function; output layer: 3 neurons, using the Softmax activation function;

[0022] The network output is converted into a probability distribution using the Softmax activation function.

[0023]

[0024] Among them, z iThis represents the original value of the i-th output neuron. Let be the probability value of the i-th type of jitter predicted by the network, where i∈{1,2,3} corresponds to translation, rotation and Z-axis jitter respectively.

[0025] Furthermore, it also includes the calculation of physical criteria for jitter type:

[0026] Translational shaking animal physical criteria Calculate the optical flow direction for each pixel:

[0027]

[0028] Among them, u t (x, y), v t (x, y) are the optical flow components in the horizontal and vertical directions, respectively, and (x, y) are the pixel coordinates;

[0029] Calculate the directional consistency score:

[0030]

[0031] Where, σ θ π is the standard deviation of the optical flow direction, and π is the normalization factor.

[0032] Rotational shaking animal physiological criteria

[0033]

[0034] Among them, E w,high For high-frequency energy, E w,total Total energy;

[0035] Z-axis shaking animal physiological judgment criteria

[0036]

[0037] Among them, SSIM t is the structural similarity index between frame t and the reference frame;

[0038] Perform a fusion operation on each type of jitter to obtain the final probability.

[0039]

[0040] in, The i-th type of jitter probability value is calculated for the physical criterion;

[0041] The rules for choosing the value of α are as follows:

[0042] When α = 1, that is

[0043] When α∈[0.2,0.5].

[0044] Further, S2 includes:

[0045] S21. Extract motion features, frequency domain features, and edge features from the input video frames respectively;

[0046] Among them, motion features are obtained by analyzing and calculating adjacent frames using an optical flow algorithm, frequency domain features are obtained by performing a Fourier transform on the current frame, and edge features are obtained by calculating and statistically analyzing the current frame using a gradient operator.

[0047] S22. Perform mean-variance normalization on motion features, frequency domain features, and edge features respectively;

[0048] S23. Input the normalized motion features, frequency domain features, and edge features into a dynamic feature interaction network. The dynamic feature interaction network contains multiple 3×3 convolutions. After each convolution operation, a nonlinear transformation is performed using the GELU activation function to obtain feature F. gelu ;

[0049] S24. Using the dot product attention mechanism combined with the jitter type probability distribution and the final probability obtained from physical judgment, dynamically allocate attention weights to the motion feature, the frequency domain feature, and the edge feature; to obtain the fused feature F. attn ;

[0050] S25. The feature F obtained in step S23 is... gelu The fusion feature F obtained in step S24 attn The features are concatenated and enhanced using a residual connection structure to output multimodal fusion features.

[0051] Furthermore, S24 specifically includes:

[0052] The relevance of motion features, frequency domain features, and edge features is calculated using a dot product attention mechanism. A linear transformation is then applied to each feature to obtain a vector of query, key, and value. The calculation formula is as follows:

[0053]

[0054] Where Q, K, and V are the Query, Key, and Value matrices, respectively, and d k For feature dimensions;

[0055] Based on the final probability Dynamically adjust the attention weights for each feature:

[0056]

[0057] Among them, Q m K represents the query vector obtained by linear transformation of motion features. f K e and K m These represent the key vectors for frequency domain, edge, and motion features, respectively. and λ1, λ2, and λ3 represent the final translational jitter probability, the final rotational jitter probability, and the final Z-axis jitter probability, respectively, and are preset adjustment coefficients.

[0058] The fusion features are:

[0059]

[0060] Where σ is the Sigmoid activation function. V G and V V These are the value vectors for frequency domain features, edge features, and motion features, respectively.

[0061] Further, S3 includes:

[0062] S31. Generate the texture distribution map of the video frame by performing local variance analysis on the input video frame;

[0063] Perform wavelet transform on each frame of the image, calculate the wavelet energy γ(x,y) of each pixel, mark the pixels with γ(x,y)>δ as complex texture regions, and form a binary texture mask map, where δ is a preset energy threshold;

[0064] S32. Construct a dynamic programming cost function that combines motion smoothing term and texture change penalty term with weighted summation. The texture change penalty term is adaptively adjusted according to the texture distribution of the current frame and adjacent frames.

[0065] S33. A two-layer smoothing strategy is adopted, using Kalman filtering to smooth the global motion trajectory; local offsets are introduced in local texture areas for adaptive fine-tuning;

[0066] S34. Search for the optimal motion path under the constraints of the dynamic programming cost function using the dynamic programming algorithm, and output the optimized motion compensation parameters.

[0067] Furthermore, the dynamic programming cost function in S32 includes:

[0068] C = a1·C smooth +a2·S·C texture

[0069] Among them, C smohth For the smooth motion term, C textureFor texture variation terms, a1 and a2 are weighting coefficients, and S is a texture variation penalty term;

[0070] The texture change penalty term is adaptively adjusted based on the texture distribution of the current frame and adjacent frames, specifically as follows:

[0071] S(x,y)=S0+ψ·Var(x,y)

[0072] Where S0 is the basic penalty term, ψ is the adjustment coefficient, and Var(x,y) is the local variance.

[0073] The two-layer smoothing strategy in S33 further includes:

[0074] Global smoothing: A Kalman filter is used to smooth the global original motion trajectory T. raw (t) is filtered and optimized to obtain the smoothed global motion trajectory T. global (t), the calculation formula is:

[0075] T global (t)=A·T global (t-1)+K t [T raw (t)-H·T global (t-1)]

[0076] Where t represents the time frame number, A is the state transition matrix, and K t H is the Kalman gain, and H is the observation matrix;

[0077] Adaptive fine-tuning of local texture regions: In step S31, a local offset ΔT is introduced in the textured complex region. local (x,y,t), the calculation formula is:

[0078]

[0079] Among them, T raw (x,y,t) represents the original local motion trajectory, β is the adjustment coefficient, and finally, the locally corrected motion compensation parameters are:

[0080] T final (x,y,t)=T global (t)+ΔT local (x,y,t).

[0081] S4 includes:

[0082] S41. Detect the boundary gap region after motion compensation and generate a boundary mask map;

[0083] S42. Extract color, texture, and structural features of the boundary neighborhood;

[0084] S43. Using a deep learning model, fill content is generated based on the motion compensation parameters, boundary mask map, multimodal features, color, texture and structural features;

[0085] S44. Optimize the consistency between the fill content and the original image by combining the color characteristics of the boundary area;

[0086] S45. Smooth the filled area, repair the boundary between the filled area and the original image, and output a complete stabilized video frame.

[0087] This invention also provides a video stabilization system based on multi-scale fusion and adaptive smoothing, the system comprising:

[0088] Feature extraction and jitter recognition module: used to preprocess the input video frames, extract multi-dimensional feature vectors, and determine the jitter type probability distribution through a pre-trained jitter type recognition network;

[0089] Feature fusion module: used to extract motion features, frequency domain features and edge features of the video frame, combine the jitter type probability distribution and physical criteria, dynamically weight and fuse the three types of features through dynamic feature interaction network and dot product attention mechanism to generate fused features, and further enhance them through residual connection to output multimodal fused features;

[0090] Trajectory optimization module: used to construct motion trajectories based on video frame motion information, optimize motion trajectories using a two-layer smoothing strategy, and output motion compensation parameters;

[0091] Motion compensation and boundary processing module: used to perform motion compensation on video frames according to the motion compensation parameters, and combined with the multimodal fusion features, use a deep learning model to fill the content of the compensated boundary area, and output the stabilized video frame.

[0092] The advantages of this invention are:

[0093] This invention significantly improves the accuracy and robustness of jitter type recognition by extracting gradient histogram features and wavelet energy distribution features, combining them with a pre-trained jitter type recognition network, and integrating physical criteria, providing a reliable basis for subsequent processing. Through a dynamic feature interaction network and dot product attention mechanism, motion features, frequency domain features, and edge features are dynamically weighted and fused to generate multimodal fusion features. This feature fusion method fully utilizes multi-source information, enhances the completeness and discriminative ability of feature representation, and improves the accuracy of subsequent image processing. A two-layer smoothing strategy is adopted to optimize the motion trajectory, achieving global trajectory smoothing through Kalman filtering and introducing local offsets for adaptive fine-tuning in areas with complex textures. This strategy effectively suppresses high-frequency jitter, maintains overall video stability, and avoids excessive smoothing of textured areas, preserving image details. After motion compensation, combined with multimodal fusion features, a deep learning model is used to fill content in boundary areas, effectively improving the realism and visual consistency of the filled content. This improves the accuracy, robustness, and visual quality of video stabilization. Attached Figure Description

[0094] Figure 1 This is a flowchart of the video stabilization method based on multi-scale fusion and adaptive smoothing according to Embodiment 1 of the present invention;

[0095] Figure 2 This is a schematic diagram of the result after processing by the video stabilization method based on multi-scale fusion and adaptive smoothing according to Embodiment 1 of the present invention. Detailed Implementation

[0096] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0097] Example 1

[0098] like Figure 1 The diagram shows a flowchart of a video stabilization method based on multi-scale fusion and adaptive smoothing. The method includes:

[0099] S1. Preprocess the input video frames, extract multi-dimensional features, and determine the probability distribution of jitter type through a jitter type recognition network;

[0100] Specifically, S1 includes:

[0101] S11. Perform grayscale processing on the input video frame to obtain a grayscale video frame, and perform normalization processing on the grayscale video frame.

[0102] Let the original frame be I raw (x,y), the grayscale frame is I gray (x,y), after normalization, becomes:

[0103]

[0104] Where (x, y) are pixel coordinates, μ is the mean pixel value of the current frame, and σ is the standard deviation of pixels. This step is used to eliminate illumination variations and noise interference, ensuring the stability of subsequent feature extraction.

[0105] S12. Calculate the gradient magnitude and direction of the normalized grayscale video frame using the gradient direction histogram method, and statistically analyze the gradient distribution in each direction to form gradient histogram features.

[0106] For normalized frame I norm Calculate the gradient magnitude G(x,y) and direction θ(x,y) for each direction, and statistically analyze the gradient distribution in each direction to form a gradient histogram feature H. grad This feature reflects the distribution of image edges and textures, describing structural changes.

[0107] S13. Perform discrete wavelet transform on the normalized grayscale video frame, calculate the wavelet coefficients of each sub-band and statistically form the wavelet energy distribution characteristics.

[0108] For normalized frame I norm Wavelet decomposition is performed on (x,y). Discrete wavelet transform (DWT) is used to decompose I. norm (x,y) is decomposed into different scales and frequency bands, and the energy distribution characteristics at each scale are extracted. The wavelet energy calculation formula is:

[0109]

[0110] Among them, W l (x,y) are the wavelet coefficients of the l-th layer, E w (l) represents the energy of this layer. This feature is used to capture multi-frequency motion information of the image.

[0111] S14, the gradient histogram feature H grad With the wavelet energy distribution feature E w (l) Concatenate them into a multidimensional feature vector F = [H grad E w (1),E w (2),…,E w(L)]; The multidimensional feature vector is classified using the pre-trained jitter type recognition network, outputting the probability distributions of translation jitter, rotation jitter, and Z-axis jitter; the jitter type recognition network is a multilayer perceptron structure, including:

[0112] Input layer: Receives multi-dimensional feature vectors

[0113] Hidden layer 1: 256 neurons, using the GELU activation function

[0114] Hidden layer 2: 128 neurons, using the GELU activation function.

[0115] Output layer: 3 neurons (corresponding to 3 jitter types), using the Softmax activation function.

[0116] Enter the specific dimensions:

[0117] Gradient histogram feature dimensions: 64 dimensions (8 directions × 8 intensity levels)

[0118] Wavelet energy feature dimensions: 16 dimensions (4-level wavelet decomposition × 4 frequency bands)

[0119] Multidimensional feature vector dimension: 80 dimensions

[0120] The network output is converted into a probability distribution using the Softmax activation function.

[0121]

[0122] Among them, z i This represents the original value of the i-th output neuron. Let be the probability value of the i-th type of jitter predicted by the network, where i∈{1,2,3} corresponds to translation, rotation and Z-axis jitter respectively.

[0123] First, a video dataset containing various typical shake types was collected. Gradient histogram features and wavelet energy features were extracted frame-by-frame for each video sequence and concatenated into an 80-dimensional input vector. Each sample corresponds to a manually labeled shake type, encoded using one-hot encoding. The dataset was then divided into training and validation sets in an 8:2 ratio. Data augmentation was performed on the training data, including random rotation, scaling, and brightness perturbation, to simulate shake changes under different shooting scenarios.

[0124] The cross-entropy loss function is used to measure the difference between the model's output probability distribution and the true label.

[0125]

[0126] Among them, y i One-hot encoding for the actual label.

[0127] The Adam optimizer is used for parameter updates, with an initial learning rate of 0.001 and a batch size of 64. During training, the learning rate is dynamically adjusted based on the validation set performance. If the validation loss does not decrease within 5 consecutive epochs, the learning rate is halved. Five-fold cross-validation is used to evaluate the model's stability and generalization ability. During each fold of training, the remaining data is used as the training set, and the current fold is used as the validation set. The average performance is then used as the model's final performance. A Dropout layer with a dropout rate of 0.3 is added after each fully connected layer to prevent overfitting. The validation set accuracy is monitored during training, and an early stopping mechanism is used. Training is terminated early when the validation set accuracy does not improve for 10 consecutive epochs. After training, the model parameters that perform best on the validation set are selected as the final model. The model is then evaluated on the test set, and metrics such as accuracy, recall, and F1 score are calculated to ensure the model's recognition ability on unseen data. After training, the model is deployed in a video stabilization system to identify the type of shaking in input video frames in real time, providing a basis for subsequent feature fusion and adaptive parameter adjustment.

[0128] Specifically, translational jitter refers to a change in the camera's position in the horizontal or vertical direction, causing the overall image to move left or right or up or down. Rotational jitter manifests as the camera rotating around its own axis, resulting in a tilted or rotated image. Z-axis jitter refers to the camera moving along the front-to-back direction of the lens, causing a change in the image's perspective, appearing as if the image is magnified or reduced.

[0129] Translational shaking animal physical criteria Calculate the optical flow direction for each pixel:

[0130]

[0131] Among them, u t (x, y), v t (x, y) are the optical flow components in the horizontal and vertical directions, respectively, and (x, y) are the pixel coordinates;

[0132] Calculate the directional consistency score:

[0133]

[0134] Where, σ θ π is the standard deviation of the optical flow direction, and π is the normalization factor.

[0135] Rotational shaking animal physiological criteria

[0136]

[0137] Among them, E w,high For high-frequency energy, E w,totalTotal energy;

[0138] Z-axis shaking animal physiological judgment criteria

[0139]

[0140] Among them, SSIM t is the structural similarity index between frame t and the reference frame;

[0141] Perform a fusion operation on each type of jitter to obtain the final probability.

[0142]

[0143] in, The i-th type of jitter probability value is calculated for the physical criterion;

[0144] The rules for choosing the value of α are as follows:

[0145] When α = 1, that is

[0146] When α∈[0.2,0.5].

[0147] S2. Extract the motion features, frequency domain features, and edge features of the video frame. Combine the jitter type probability distribution and physical criteria, dynamically weight and fuse the three types of features through a dynamic feature interaction network and dot product attention mechanism to generate fused features. Further enhance the fused features through residual connections and output multimodal fused features.

[0148] Specifically, S2 includes:

[0149] S21. Extract motion features, frequency domain features, and edge features from the input video frames respectively;

[0150] Motion features are obtained through optical flow analysis. Let adjacent frames I... t and I t+1 Calculate the optical flow field for these two frames:

[0151] V(x,y)=(u(x,y),w(x,y))

[0152] Where (x, y) represents the pixel coordinates in the image, t represents the time frame number, u(x, y) is the horizontal motion component, and w(x, y) is the vertical motion component.

[0153] Optical flow reflects the motion of pixels between adjacent frames and is used to describe dynamic changes in a video. V(x,y) is the motion feature to be extracted.

[0154] Frequency domain features are obtained through a two-dimensional Fourier transform. Let the two-dimensional Fourier transform of the image be F(p,q), where p is the horizontal frequency component in the frequency domain and q is the vertical frequency component in the frequency domain. The formula for calculating frequency domain energy is:

[0155]

[0156] Edge features are obtained using gradient operators (such as Sobel), by calculating the horizontal and vertical gradient components for the input frame I(x,y). and gradient magnitude is

[0157]

[0158] This feature is used to reflect the distribution of edges and details in an image, describing local structural changes. G(x,y) is the edge feature to be extracted.

[0159] S22. Perform mean-variance normalization on motion features, frequency domain features, and edge features respectively;

[0160]

[0161] Where F(x,y) represents the original feature values, including: motion features, frequency domain features, and edge feature values, μ F σ is the mean of this feature within the current frame or batch, where the batch is the set of video frame samples during normalization. F The standard deviation is used. This normalization method ensures that the distributions of different types of features have zero mean and unit variance, eliminating the influence of dimensions and improving the effectiveness and stability of feature fusion. This step facilitates subsequent fusion operations.

[0162] S23. Input the normalized motion features, frequency domain features, and edge features into a dynamic feature interaction network. The dynamic feature interaction network contains multiple 3×3 convolutions. After each convolution operation, a nonlinear transformation is performed using the GELU activation function to obtain feature F. gelu ;

[0163] (1) Input layer: The normalized motion features, frequency domain features and edge features are concatenated along the channel dimension to form the original input feature F. in .

[0164] (2) Multi-layer 3×3 convolutional structure: The network internally stacks multiple 3×3 convolutional layers, and the convolutional operation of each layer is as follows:

[0165]

[0166] in, This is the output of the previous layer. This is the output of the current layer. Each convolutional layer is followed by a GELU activation function to enhance non-linear expressiveness.

[0167]

[0168] The GELU (Gaussian Error Linear Unit) activation function can adaptively adjust the activation strength, improving the flexibility of feature representation.

[0169] S24. Using the dot product attention mechanism combined with the jitter type probability distribution and the final probability obtained from physical judgment, dynamically allocate attention weights to the motion feature, the frequency domain feature, and the edge feature; to obtain the fused feature F. attn ;

[0170] Specifically, the motion features are denoted as V(x,y), and the frequency domain features are denoted as E. freq The edge features G(x,y) are used to calculate relevance through a dot product attention mechanism. Linear transformations are applied to each feature to obtain a vector of query, key, and value. The calculation formula is as follows:

[0171]

[0172] Where Q, K, and V are the Query, Key, and Value matrices, respectively, and d k For feature dimensions;

[0173] Based on the final probability Dynamically adjust the attention weights for each feature:

[0174]

[0175] Among them, Q m K represents the query vector obtained by linear transformation of motion features. f K e and K m These represent the key vectors for frequency domain, edge, and motion features, respectively. and λ1, λ2, and λ3 represent the final translational jitter probability, the final rotational jitter probability, and the final Z-axis jitter probability, respectively, and are preset adjustment coefficients.

[0176] The fusion features are:

[0177]

[0178] Where σ is the Sigmoid activation function. V G and V V These are the value vectors for frequency domain features, edge features, and motion features, respectively.

[0179] S25. The feature F obtained in step S23 is... gelu The fusion feature F obtained in step S24 attn The features are concatenated and enhanced using a residual connection structure to output multimodal fusion features.

[0180] F cat =cat[F attn ,F gelu ]

[0181] F out =F cat +F in

[0182] Among them, F out For multimodal fusion features, F in These are the original input features.

[0183] S3. Construct a motion trajectory based on the fused features, and optimize the trajectory using a two-layer smoothing strategy to output motion compensation parameters;

[0184] Specifically, S3 includes:

[0185] S31. Generate the texture distribution map of the video frame by performing local variance analysis on the input video frame;

[0186] Perform wavelet transform on each frame of the image, calculate the wavelet energy γ(x,y) of each pixel, mark the pixels with γ(x,y)>δ as complex texture regions, and form a binary texture mask map, where δ is a preset energy threshold;

[0187]

[0188] Among them, L a W represents the total number of layers. l (x,y) represents the wavelet coefficient of pixel (x,y) under the l-th level wavelet decomposition.

[0189] Texture feature analysis is performed on the input video frames, and the texture complexity of each frame is quantized using local variance. The formula for calculating local variance is:

[0190]

[0191] Where I(i,j) is the pixel value, Ω(x,y) is the neighborhood centered at (x,y), μ(x,y) is the mean of the neighborhood, and N is the number of neighborhood pixels.

[0192] The texture distribution map is generated using the above indicators, reflecting the texture complexity of each area of ​​the image.

[0193] S32. Construct a dynamic programming cost function that combines motion smoothing term and texture change penalty term with weighted summation. The texture change penalty term is adaptively adjusted according to the texture distribution of the current frame and adjacent frames.

[0194] The formula for calculating the cost function of dynamic programming is as follows:

[0195] C = a1·C smooth +a2·S·C texture

[0196] Among them, C smooth For the smooth motion term, C texture For texture variation terms, a1 and a2 are weighting coefficients, and S is a texture variation penalty term;

[0197] The texture change penalty term is adaptively adjusted based on the texture distribution of the current frame and adjacent frames, specifically as follows:

[0198] S(x,y)=S0+ψ·Var(x,y)

[0199] Where S0 is the basic penalty term, ψ is the adjustment coefficient, and Var(x,y) is the local variance. For texture-rich regions, the texture variation penalty term S is increased to suppress over-smoothing and protect image details and structural information; for texture-simple regions, S is decreased to increase smoothing intensity and enhance image stability.

[0200] S33. A two-layer smoothing strategy is adopted, using Kalman filtering to smooth the global motion trajectory; local offsets are introduced in local texture areas for adaptive fine-tuning;

[0201] Specifically, global smoothing: a Kalman filter is used to smooth the original global motion trajectory T. raw (t) is filtered and optimized to obtain the smoothed global motion trajectory T. global (t), the calculation formula is:

[0202] T global (t)=A·T global (t-1)+K t tT raw (t)-H·T global (t-1)]

[0203] Where t represents the time frame number, A is the state transition matrix, and K t H is the Kalman gain, and H is the observation matrix;

[0204] Adaptive fine-tuning of local texture regions: In step S31, a local offset ΔT is introduced in the textured complex region. local (x,y,t), the calculation formula is:

[0205]

[0206] Among them, T raw (x,y,t) represents the original local motion trajectory, β is the adjustment coefficient, and finally, the locally corrected motion compensation parameters are:

[0207] T final (x,y,t)=T global (t)+ΔT local (x,y,t).

[0208] T raw (x,y,t) and T raw (t) represents the original trajectory. This ensures global stability while also considering local details.

[0209] S34. Using dynamic programming, search for the optimal motion path under the constraints of the dynamic programming cost function, and output the optimized motion compensation parameters. Dynamic programming selects the trajectory that minimizes the overall cost under the constraint of the cost function C.

[0210] S4. Perform motion compensation on the video frame according to the motion compensation parameters, and combine the multimodal fusion features to fill the content of the compensated boundary region through a deep learning model, and output the stabilized video frame.

[0211] Specifically, S4 includes:

[0212] S41. Detect the boundary gap region after motion compensation and generate a boundary mask map;

[0213] Let the motion-compensated video frame be I. mc (x,y), the boundary mask is M(x,y), where

[0214]

[0215] This mask is used to mark the positions of all pixels that need to be extrapolated and filled.

[0216] S42. Extract color, texture, and structural features of the boundary neighborhood;

[0217] Let the boundary neighborhood be Ω(x,y), and the mean color be:

[0218]

[0219] Texture features can be represented by local variance Var(x,y) or gradient magnitude G(x,y), and structural information can be obtained through edge detection operators.

[0220] S43. Using a deep learning model, fill content is generated based on the motion compensation parameters, boundary mask image, multimodal features, and boundary neighborhood features;

[0221] Specifically, the deep learning model employs a U-Net network with an encoder-decoder structure or a Generative Adversarial Network (GAN) architecture. The model input consists of the image to be repaired (including a boundary mask), motion compensation parameters, multimodal fusion features, and boundary neighborhood features (color, texture, and structural features). The motion compensation parameters are first used to spatially transform the original image or video frame, "aligning" misaligned regions caused by jitter or motion to the target (standard) position. Subsequently, the encoder uses multi-layer convolution and downsampling operations to fuse the motion-compensated image to be repaired, multimodal fusion features, and boundary neighborhood features to obtain high-level semantic features, supporting subsequent decoding and content filling. The decoder uses upsampling and skip connections to progressively restore spatial resolution and, combined with features extracted by the encoder, achieves detailed filling and structural restoration of missing regions. The generator outputs a filled region of I... fill (x, y). During training, a perceptual loss function and an adversarial loss are introduced. The perceptual loss is defined as:

[0222]

[0223] Where, φ l () represents the feature mapping in the l-th layer of the pre-trained network. To generate an image, I gt This is a real image.

[0224] Adversarial loss is defined as:

[0225]

[0226] Where D() is the probability of authenticity output by the discriminator.

[0227] The total loss function is:

[0228] L total =λ1L perceptual +λ2L adv

[0229] Where λ1 and λ2 are weighting coefficients.

[0230] S44. Optimize the consistency between the fill content and the original image by combining color features;

[0231] Specifically, based on the completed motion compensation spatial alignment, the consistency between the filled content and the original image is further optimized by incorporating color features. The optimization objective is to minimize the difference between the filled area and the original image, and this is achieved through the following loss function:

[0232]

[0233] Among them, I fill (x,y) is the filled region, and TV(I) fill ) represents the total variation of the filled region, which is the sum of the grayscale changes of adjacent pixels within the filled region, and η is the smoothing weight.

[0234] S45. Smooth the filled area, repair the boundary between the filled area and the original image, and output a complete stabilized video frame I. final (x,y).

[0235] Specifically, the smoothing and blending process uses Gaussian filtering, and the edge repair uses edge filtering or structure-guided filling to smooth the transition between the filled area and the original image.

[0236] like Figure 2 The image shown is a schematic diagram of the result after processing by the multi-scale fusion and adaptive smoothing video stabilization method. After processing, the computer portion is shifted downwards to keep the main body of the video in the center of the frame. The right side is wider than the original video because the boundaries of the original video are extrapolated to maintain consistency with the style of the original boundaries.

[0237] Example 2

[0238] This invention also provides a video stabilization system based on multi-scale fusion and adaptive smoothing, the system comprising:

[0239] Feature extraction and jitter recognition module: used to preprocess the input video frames, extract multi-dimensional feature vectors, and determine the jitter type probability distribution through a pre-trained jitter type recognition network;

[0240] The grayscale unit is used to perform grayscale processing on the input video frame to obtain a grayscale video frame.

[0241] A normalization unit is used to normalize the grayscale video frames;

[0242] The gradient calculation unit is used to calculate the gradient magnitude and direction of the normalized grayscale video frame using the gradient direction histogram method, and to statistically analyze the gradient distribution in each direction to form gradient histogram features.

[0243] The wavelet transform unit is used to perform discrete wavelet transform on the normalized grayscale video frame, calculate the wavelet coefficients of each sub-band and statistically form wavelet energy distribution characteristics.

[0244] The feature concatenation unit is used to concatenate the gradient histogram features and the wavelet energy distribution features into a multidimensional feature vector.

[0245] A jitter type identification unit includes a pre-trained multilayer perceptron network for receiving the multidimensional feature vector and performing classification processing, outputting the probability distributions of translational jitter, rotational jitter, and Z-axis jitter; the multilayer perceptron network includes:

[0246] The input layer is used to receive the multidimensional feature vector;

[0247] The first hidden layer contains 256 neurons and uses the GELU activation function;

[0248] The second hidden layer contains 128 neurons and uses the GELU activation function;

[0249] The output layer, containing 3 neurons, uses the Softmax activation function to transform the network output into a probability distribution.

[0250]

[0251] Among them, z i This represents the original value of the i-th output neuron. Let be the probability value of the i-th type of jitter predicted by the network, where i∈{1,2,3} corresponds to translation, rotation and Z-axis jitter respectively.

[0252] The preprocessing and jitter type identification module further includes a physical criterion calculation unit, used to calculate jitter physical criteria and fuse them with the jitter type probability distribution to obtain the final probability. The physical criterion calculation unit performs the following:

[0253] Translational shaking animal physical criteria Calculate the optical flow direction for each pixel:

[0254]

[0255] Among them, u t (x, y), v t (x, y) are the optical flow components in the horizontal and vertical directions, respectively, and (x, y) are the pixel coordinates;

[0256] Calculate the directional consistency score:

[0257]

[0258] Where, σ θ π is the standard deviation of the optical flow direction, and π is the normalization factor.

[0259] Rotational shaking animal physiological criteria

[0260]

[0261] Among them, E w,highFor high-frequency energy, E w,total Total energy;

[0262] Z-axis shaking animal physiological judgment criteria

[0263]

[0264] Among them, SSIM t is the structural similarity index between frame t and the reference frame;

[0265] Perform a fusion operation on each type of jitter to obtain the final probability.

[0266]

[0267] in, The i-th type of jitter probability value is calculated for the physical criterion;

[0268] The rules for choosing the value of α are as follows:

[0269] When α = 1, that is

[0270] When α∈[0.2,0.5].

[0271] Feature fusion module: used to extract motion features, frequency domain features and edge features of the video frame, combine the jitter type probability distribution and physical criteria, dynamically weight and fuse the three types of features through dynamic feature interaction network and dot product attention mechanism to generate fused features, and further enhance them through residual connection to output multimodal fused features;

[0272] The motion feature extraction unit is used to analyze and calculate motion features of adjacent frames using an optical flow algorithm;

[0273] The frequency domain feature extraction unit is used to calculate frequency domain features by performing a Fourier transform on the current frame;

[0274] The edge feature extraction unit is used to calculate and statistically analyze edge features of the current frame using gradient operators.

[0275] The normalization unit is used to perform mean-variance normalization on motion features, frequency domain features, and edge features respectively.

[0276] The dynamic feature interaction unit includes multiple 3×3 convolutional layers. After each convolutional operation, the GELU activation function is combined to perform a non-linear transformation to obtain features.

[0277] The dot-product attention unit is used to dynamically allocate attention weights for the motion features, frequency domain features, and edge features by combining the jitter type probability distribution and the final probability obtained by physical criteria using the dot-product attention mechanism, thereby generating fused features.

[0278] The relevance of motion features, frequency domain features, and edge features is calculated using a dot product attention mechanism. A linear transformation is then applied to each feature to obtain a vector of query, key, and value. The calculation formula is as follows:

[0279]

[0280] Where Q, K, and V are the Query, Key, and Value matrices, respectively, and d k For feature dimensions;

[0281] Based on the final probability Dynamically adjust the attention weights for each feature:

[0282]

[0283] Among them, Q m K represents the query vector obtained by linear transformation of motion features. f K e and K m These represent the key vectors for frequency domain, edge, and motion features, respectively. and λ1, λ2, and λ3 represent the final translational jitter probability, the final rotational jitter probability, and the final Z-axis jitter probability, respectively, and are preset adjustment coefficients.

[0284] The fusion features are:

[0285]

[0286] Where σ is the Sigmoid activation function. V G and V V These are the value vectors for frequency domain features, edge features, and motion features, respectively.

[0287] The residual fusion unit is used to concatenate the features output by the dynamic feature interaction unit with the fused features output by the dot product attention unit, and perform feature enhancement through the residual connection structure to output multimodal fusion features.

[0288] Trajectory optimization module: used to construct motion trajectories based on video frame motion information, optimize motion trajectories using a two-layer smoothing strategy, and output motion compensation parameters;

[0289] The texture analysis unit is used to generate a texture distribution map of the video frame by performing local variance analysis on the input video frame; and to perform wavelet transform on each frame image, calculate the wavelet energy γ(x,y) of each pixel, mark the pixels with γ(x,y)>δ as texture complex regions, and form a binary texture mask map, where δ is a preset energy threshold.

[0290] Trajectory building unit, used to construct motion trajectories;

[0291] The dynamic programming optimization unit is used to construct the dynamic programming cost function, which combines the motion smoothing term and the texture change penalty term in a weighted manner. The texture change penalty term is adaptively adjusted according to the texture distribution of the current frame and adjacent frames.

[0292] The dynamic programming optimization unit performs dynamic programming cost function construction:

[0293] C = a1·C smooth +a2·S·C texture

[0294] Among them, C smooth For the smooth motion term, C texture For texture variation terms, a1 and a2 are weighting coefficients, and S is a texture variation penalty term;

[0295] The texture change penalty term is adaptively adjusted based on the texture distribution of the current frame and adjacent frames, specifically as follows:

[0296] S(x,y)=S0+ψ·Var(x,y)

[0297] Where S0 is the basic penalty term, ψ is the adjustment coefficient, and Var(x,y) is the local variance.

[0298] A dual-layer smoothing unit is used to employ a dual-layer smoothing strategy, using Kalman filtering to smooth and optimize the global motion trajectory; local offsets are introduced in local texture regions for adaptive fine-tuning.

[0299] Two-layer smooth execution:

[0300] Global smoothing: A Kalman filter is used to smooth the global original motion trajectory T. raw (t) is filtered and optimized to obtain the smoothed global motion trajectory T. global (t), the calculation formula is:

[0301] T global (t)=A·T global (t-1)+K t [T raw (t)-H·T global (t-1)]

[0302] Where t represents the time frame number, A is the state transition matrix, and K t H is the Kalman gain, and H is the observation matrix;

[0303] Adaptive fine-tuning of local texture regions: In step S31, a local offset ΔT is introduced in the textured complex region. local (x,y,t), the calculation formula is:

[0304]

[0305] Among them, T raw (x,y,t) represents the original local motion trajectory, β is the adjustment coefficient, and finally, the locally corrected motion compensation parameters are:

[0306] T final (x,y,t)=T global (t)+ΔT local (x,,y,t).

[0307] The output unit is used to search for the optimal motion path under the constraints of the dynamic programming cost function using a dynamic programming algorithm, and outputs the optimized motion compensation parameters.

[0308] Motion compensation and boundary processing module: used to perform motion compensation on video frames according to the motion compensation parameters, and combined with the multimodal fusion features, use a deep learning model to fill the content of the compensated boundary area, and output the stabilized video frame.

[0309] The boundary detection unit is used to detect boundary gaps after motion compensation and generate a boundary mask image.

[0310] The feature extraction unit is used to extract color, texture, and structural features of the boundary neighborhood;

[0311] The content filling unit includes a deep learning model for generating filling content based on the motion compensation parameters, boundary mask map, multimodal features, color, texture and structural features;

[0312] The color optimization unit is used to optimize the consistency between the filled content and the original image by combining the color features of the boundary neighborhood;

[0313] The smoothing and repair unit is used to smooth the filled area and repair the boundary between the filled area and the original image, outputting a complete stabilized video frame.

[0314] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video stabilization method based on multi-scale fusion and adaptive smoothing, characterized in that, The method includes: S1. Preprocess the input video frames, extract multi-dimensional feature vectors, and determine the jitter type probability distribution through a pre-trained jitter type recognition network; S2. Extract the motion features, frequency domain features, and edge features of the video frame. Combine the jitter type probability distribution and physical criteria, dynamically weight and fuse the three types of features through a dynamic feature interaction network and dot product attention mechanism to generate fused features. Further enhance the fused features through residual connections and output multimodal fused features. S3. Construct motion trajectories based on video frame motion information, optimize motion trajectories using a two-layer smoothing strategy, and output motion compensation parameters; S4. Perform motion compensation on the video frame according to the motion compensation parameters, and combine the multimodal fusion features to fill the content of the compensated boundary region through a deep learning model, and output the stabilized video frame.

2. The video stabilization method based on multi-scale fusion and adaptive smoothing according to claim 1, characterized in that, S1 includes: S11. Perform grayscale processing on the input video frame to obtain a grayscale video frame, and perform normalization processing on the grayscale video frame. S12. Calculate the gradient magnitude and direction of the normalized grayscale video frame using the gradient direction histogram method, and statistically analyze the gradient distribution in each direction to form gradient histogram features. S13. Perform discrete wavelet transform on the normalized grayscale video frame, calculate the wavelet coefficients of each sub-band and statistically form the wavelet energy distribution characteristics. S14. The gradient histogram features and the wavelet energy distribution features are concatenated into a multidimensional feature vector; the multidimensional feature vector is classified using the pre-trained jitter type recognition network to output the probability distributions of translation jitter, rotation jitter, and Z-axis jitter; the jitter type recognition network is a multilayer perceptron structure, including: input layer: receiving multidimensional feature vectors; hidden layer 1: 256 neurons, using the GELU activation function; hidden layer 2: 128 neurons, using the GELU activation function; output layer: 3 neurons, using the Softmax activation function; The network output is converted into a probability distribution using the Softmax activation function: Among them, z i This represents the original value of the i-th output neuron. Let be the probability value of the i-th type of jitter predicted by the network, where i∈{1,2,3} corresponds to translation, rotation and Z-axis jitter respectively.

3. The video stabilization method based on multi-scale fusion and adaptive smoothing according to claim 2, characterized in that, It also includes the calculation of physical criteria for jitter type: Translational shaking animal physical criteria Calculate the optical flow direction for each pixel: Among them, u t (x, y), v t (x, y) are the optical flow components in the horizontal and vertical directions, respectively, and (x, y) are the pixel coordinates; Calculate the directional consistency score: Where, σ θ π is the standard deviation of the optical flow direction, and π is the normalization factor. Rotational shaking animal physiological criteria Among them, E w,high For high-frequency energy, E w,total Total energy; Z-axis shaking animal physiological judgment criteria Among them, SSIM t is the structural similarity index between frame t and the reference frame; Perform a fusion operation on each type of jitter to obtain the final probability. in, The i-th type of jitter probability value is calculated for the physical criterion; The rules for choosing the value of α are as follows: When α = 1, that is When α∈[0.2,0.5].

4. The video stabilization method based on multi-scale fusion and adaptive smoothing according to claim 1, characterized in that, S2 includes: S21. Extract motion features, frequency domain features, and edge features from the input video frames respectively; Among them, motion features are obtained by analyzing and calculating adjacent frames using an optical flow algorithm, frequency domain features are obtained by performing a Fourier transform on the current frame, and edge features are obtained by calculating and statistically analyzing the current frame using a gradient operator. S22. Perform mean-variance normalization on motion features, frequency domain features, and edge features respectively; S23. Input the normalized motion features, frequency domain features, and edge features into a dynamic feature interaction network. The dynamic feature interaction network contains multiple 3×3 convolutions. After each convolution operation, a nonlinear transformation is performed using the GELU activation function to obtain feature F. gelu ; S24. Using the dot product attention mechanism combined with the jitter type probability distribution and the final probability obtained from physical judgment, dynamically allocate attention weights to the motion feature, the frequency domain feature, and the edge feature; to obtain the fused feature F. attn ; S25. The feature F obtained in step S23 is... gelu The fusion feature F obtained in step S24 attn The features are concatenated and enhanced using a residual connection structure to output multimodal fusion features.

5. The video stabilization method based on multi-scale fusion and adaptive smoothing according to claim 4, characterized in that, Specifically, S24 is: The relevance of motion features, frequency domain features, and edge features is calculated using a dot product attention mechanism. A linear transformation is then applied to each feature to obtain a vector of query, key, and value. The calculation formula is as follows: Where Q, K, and V are the Query, Key, and Value matrices, respectively, and d k For feature dimensions; Based on the final probability Dynamically adjust the attention weights for each feature: Among them, Q m K represents the query vector obtained by linear transformation of motion features. f K e and K m These represent the key vectors for frequency domain, edge, and motion features, respectively. and λ1, λ2, and λ3 represent the final translational jitter probability, the final rotational jitter probability, and the final Z-axis jitter probability, respectively, and are preset adjustment coefficients. The fusion features are: Where σ is the Sigmoid activation function. V G and V V These are the value vectors for frequency domain features, edge features, and motion features, respectively.

6. The video stabilization method based on multi-scale fusion and adaptive smoothing according to claim 1, characterized in that, S3 includes: S31. Generate the texture distribution map of the video frame by performing local variance analysis on the input video frame; Perform wavelet transform on each frame of the image, calculate the wavelet energy γ(x,y) of each pixel, mark the pixels with γ(x,y)>δ as complex texture regions, and form a binary texture mask map, where δ is a preset energy threshold; S32. Construct a dynamic programming cost function that combines motion smoothing term and texture change penalty term with weighted summation. The texture change penalty term is adaptively adjusted according to the texture distribution of the current frame and adjacent frames. S33. A two-layer smoothing strategy is adopted, using Kalman filtering to smooth the global motion trajectory; local offsets are introduced in local texture areas for adaptive fine-tuning; S34. Search for the optimal motion path under the constraints of the dynamic programming cost function using the dynamic programming algorithm, and output the optimized motion compensation parameters.

7. The video stabilization method based on multi-scale fusion and adaptive smoothing according to claim 6, characterized in that, The dynamic programming cost function in S32 includes: C=a1·C smooth +a2·S·C texture Among them, C smooth For the smooth motion term, C texture For texture variation terms, a1 and a2 are weighting coefficients, and S is a texture variation penalty term; The texture change penalty term is adaptively adjusted based on the texture distribution of the current frame and adjacent frames, specifically as follows: S(x,y)=S0+ψ·Var(x,y) Where S0 is the basic penalty term, ψ is the adjustment coefficient, and Var(x,y) is the local variance.

8. The video stabilization method based on multi-scale fusion and adaptive smoothing according to claim 6, characterized in that, The two-layer smoothing strategy in S33 includes: Global smoothing: A Kalman filter is used to smooth the global original motion trajectory T. raw (t) is filtered and optimized to obtain the smoothed global motion trajectory T. global (t), the calculation formula is: t global (t)=A·T global (t-1)+K t [T raw (t)-H·T global (t-1)] Where t represents the time frame number, A is the state transition matrix, and K t H is the Kalman gain, and H is the observation matrix; Adaptive fine-tuning of local texture regions: In step S31, a local offset ΔT is introduced in the textured complex region. local (x,y,t), the calculation formula is: Among them, T raw (x,y,t) represents the original local motion trajectory, β is the adjustment coefficient, and finally, the locally corrected motion compensation parameters are: T final (x,y,t)=T global (t)+ΔT local (x,y,t)。 9. The video stabilization method based on multi-scale fusion and adaptive smoothing according to claim 1, characterized in that, S4 includes: S41. Detect the boundary gap region after motion compensation and generate a boundary mask map; S42. Extract color, texture, and structural features of the boundary neighborhood; S43. Using a deep learning model, fill content is generated based on the motion compensation parameters, boundary mask map, multimodal features, color, texture and structural features; S44. Optimize the consistency between the fill content and the original image by combining the color characteristics of the boundary area; S45. Smooth the filled area, repair the boundary between the filled area and the original image, and output a complete stabilized video frame.

10. A video stabilization system based on multi-scale fusion and adaptive smoothing, characterized in that, The system includes: Feature extraction and jitter recognition module: used to preprocess the input video frames, extract multi-dimensional feature vectors, and determine the jitter type probability distribution through a pre-trained jitter type recognition network; Feature fusion module: used to extract motion features, frequency domain features and edge features of the video frame, combine the jitter type probability distribution and physical criteria, dynamically weight and fuse the three types of features through dynamic feature interaction network and dot product attention mechanism to generate fused features, and further enhance them through residual connection to output multimodal fused features; Trajectory optimization module: used to construct motion trajectories based on video frame motion information, optimize motion trajectories using a two-layer smoothing strategy, and output motion compensation parameters; Motion compensation and boundary processing module: used to perform motion compensation on video frames according to the motion compensation parameters, and combined with the multimodal fusion features, use a deep learning model to fill the content of the compensated boundary area, and output the stabilized video frame.

Images