A decoupled space-time transformer end-to-end video stabilization method

Abstract

This invention discloses an end-to-end video stabilization method using a decoupled spatiotemporal Transformer, belonging to the field of computer vision. The method constructs a multimodal input by fusing RGB images and bidirectional optical flow, and directly regresses a pixel-level dense deformation field via a spatiotemporal Transformer-UNet network to achieve image stabilization. Its core lies in the decoupled spatiotemporal block embedded within the network, which alternately executes spatial self-attention and trajectory-aware temporal interactive attention based on a Lagrange perspective. The latter utilizes optical flow priors to distort and align features of adjacent frames along the motion trajectory, fundamentally solving the feature misalignment problem caused by severe camera shake. Furthermore, by designing an unsupervised training framework consisting of motion smoothing, photometric reconstruction, and geometric consistency loss, camera shake and foreground motion can be implicitly separated. This invention exhibits excellent performance in terms of cropping rate, distortion rate, and stability metrics, demonstrating strong robustness.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and in particular to a decoupled spatiotemporal Transformer end-to-end video stabilization method. Background Technology

[0002] With the widespread adoption of digital image sensors and portable shooting devices, video has become a primary medium for information recording and dissemination. However, in non-professional or dynamic scenarios such as handheld shooting, drone aerial photography, and vehicle monitoring, cameras are subject to severe and unexpected shaking due to human operation or vibration of the device. This shaking manifests as high-frequency, disordered motion of the camera's six degrees of freedom in the temporal domain, and in the spatial domain, it leads to problems such as random shaking, perspective distortion, and motion blur in the video sequence. Video shaking not only severely impairs visual perception but also poses a significant challenge to subsequent computer vision tasks, such as visual simultaneous localization and mapping (SLAM), object tracking, and 3D reconstruction, easily leading to feature tracking loss, increased estimation errors, and even algorithm failure. Therefore, video stabilization technology, as a crucial preprocessing step, is becoming increasingly important.

[0003] Existing video stabilization techniques primarily follow a geometry-driven cascaded processing paradigm, typically comprising three discrete steps: feature point tracking, motion path smoothing, and image mesh deformation. These existing methods have the following limitations: (1) Feature dependence and scene limitations: Two-dimensional image stabilization schemes, represented by MeshFlow and L1 norm-based path optimization methods, are highly dependent on the extraction and matching of sparse feature points such as SIFT and KLT. In weak texture areas (such as water surfaces and walls), repetitive texture environments, or low lighting conditions, reliable feature points are difficult to obtain or are prone to mismatch, causing the motion estimation module to fail, which in turn leads to severe geometric distortion or "jelly effect" in subsequent steps.

[0004] (2) Error accumulation in cascaded architecture: Traditional methods typically design and optimize each module independently, lacking a global feedback mechanism. For example, the optical flow estimation module only pursues optical flow accuracy without considering its suitability for the final smooth path. This fragmented design causes small errors in the upstream module (such as feature point drift) to be amplified step by step after path optimization and image deformation, and the downstream module cannot backpropagate the optimized signal to correct the front-end error.

[0005] (3) Insufficient 3D scene modeling capabilities: Most 2D methods are based on the assumption that the scene is planar or that the camera only rotates, using homography matrices to describe inter-frame transformations. However, the real world is 3D, and when the camera translates, parallax occurs between objects at different depths. 2D models cannot simultaneously align the foreground and background, resulting in a stable background but a shaky foreground in scenes with large parallax, or distortion of the overall scene structure. Although 3D methods based on motion reconstruction structures are theoretically more accurate, their computational complexity is extremely high, and they are exceptionally sensitive to noise, making them difficult to apply in practice.

[0006] With the development of deep learning technology, video stabilization solutions based on convolutional neural networks have begun to emerge, but they still face the following key challenges: (1) Data Dilemma of Supervised Learning: Supervised learning methods, represented by StabNet, attempt to learn a direct mapping from jittery frames to stable frames. However, training such networks requires a large amount of precisely paired jittery-stable video data. In the real world, it is extremely difficult to obtain perfectly paired data of the same scene at the same time, with only differences in camera motion state. Existing datasets are mostly generated by robotic arms, with simple scenes, and the synthetic data differs significantly from real videos in terms of physical characteristics such as lighting and motion blur, resulting in weak model generalization ability.

[0007] (2) Structural defects of unsupervised methods: Unsupervised methods, represented by DUTs, use physical constraints such as optical flow consistency to replace manual annotation, but their network architectures mostly still follow the cascade approach of "first estimating motion and then smoothing the path", which fails to fundamentally solve the problem of error accumulation. At the same time, the introduction of non-differentiable parameter optimization steps in the process hinders the end-to-end backpropagation of gradients and limits the overall optimization capability of the model.

[0008] (3) Limitations of Temporal Modeling Perspective: Existing methods often employ recurrent neural networks or 3D convolutions when fusing temporal information from videos. In recent years, Vision Transformer has been introduced into video processing, but its standard multi-head self-attention mechanism uses an Euler perspective, which calculates cross-frame associations on a fixed pixel coordinate grid. In videos with severe shaking, the pixel positions of the same physical object can shift significantly in adjacent frames. The Euler-based attention mechanism may incorrectly associate semantically unrelated content located at the same grid position in different frames (such as objects in the current frame and the background of the reference frame), leading to severe "feature misalignment" and greatly weakening the effectiveness of temporal feature aggregation.

[0009] In summary, existing video stabilization techniques still have significant shortcomings when handling scenes with large parallax, weak texture, and high dynamic range. Furthermore, they lack an end-to-end joint optimization framework that can avoid error accumulation, overcome feature misalignment, and eliminate dependence on paired data. Therefore, developing a novel and robust video stabilization method has significant theoretical and practical value. Summary of the Invention

[0010] To address the shortcomings of existing technologies, this invention provides a decoupled spatiotemporal Transformer end-to-end video stabilization method based on the perspective of Lagrange dynamics, aiming to solve the problems of cascaded error accumulation, feature loss in weak texture regions, and strong dependence on paired training data in existing video stabilization methods.

[0011] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: a decoupled spatiotemporal Transformer end-to-end video stabilization method based on the perspective of Lagrange dynamics, characterized in that the method includes the following steps: Step 1: Obtain the unstable video sequence to be processed. The video sequence consists of multiple consecutive frames arranged in chronological order. Select the current frame to be processed as the center frame, and use this center frame as a reference to select adjacent frames within a set time window before and after it as context input. Let the current frame to be processed be... And select adjacent frames within its time window range. As context input, where, For frame number, The radius of the time window; Step 2: Calculate the bidirectional optical flow between adjacent frames in the unstable video sequence, generating the forward optical flow field from the previous frame to the next frame and the reverse (backward) optical flow field from the next frame to the previous frame. For each frame (RGB 3 channels), the forward optical flow of that frame... (2 channels, H and W (Image height and width, respectively), and the backward optical flow of this frame. (2 channels) are spliced ​​together along the channel dimension to form The feature tensor, for containing 2 k A window of +1 frame is used to construct a multimodal input tensor with dimension . B represents the batch size. Step 3: Input the multimodal input tensor into the pre-built spatiotemporal Transformer-UNet (ST-UNet) neural network model; the ST-UNet model adopts an encoder-decoder architecture, and decoupled spatiotemporal blocks (DST-Block) are embedded in each stage of the encoder, bottleneck layer and decoder. Step 4: The input features are decoupled in time and space using the decoupled spatiotemporal block in the ST-UNet model. The decoupled spatiotemporal block alternately executes the spatial self-attention (SSA) mechanism and the trajectory-aware temporal interactive attention (TATA) mechanism. The TATA mechanism is based on the Lagrange perspective and uses the bidirectional optical flow field as motion guidance to distort and align the feature maps of adjacent frames along the motion trajectory to the coordinate system of the current frame. Attention weights and aggregated temporal features are then calculated in the aligned feature space. Step 5: Generate a dense deformation field corresponding to the resolution of the center frame through the output layer of the ST-UNet model; wherein, the dense deformation field has two channels, which respectively represent the horizontal displacement component and the vertical displacement component required for each pixel in the image to eliminate jitter and parallax effects under the original jitter frame coordinate system. Step 6: Based on the dense deformation field, for the target stable frame Each pixel in the image is determined based on the dense deformation field. The corresponding displacement component is determined in the original jitter center frame. The corresponding sampling coordinates are used, and a differentiable resampling warp operation is performed on the center frame using bilinear interpolation to generate the stable video frame. ; Step 7: During the model training phase, construct a composite loss function that includes motion smoothing loss, photometric reconstruction loss, and geometric consistency loss, and optimize the parameters of the ST-UNet model through backpropagation under unsupervised conditions.

[0012] As a further optimization of the above scheme, the spatial self-attention (SSA) mechanism in the decoupled spatiotemporal block (DST-Block) is configured as follows: the input feature map is divided into rectangular windows of fixed size, and multi-head self-attention is independently calculated within each window; between consecutively stacked DST-Blocks, a cyclic shift operation is used to offset the window grid to establish a cross-window spatial information interaction channel; the SSA mechanism is based on the Euler perspective and focuses on the extraction of texture geometric features within a fixed pixel grid, enhancing the structural representation capability within a single frame.

[0013] As a further optimization of the above scheme, the specific calculation steps of the trajectory-aware temporal interactive attention (TATA) mechanism in the decoupled spatiotemporal block (DST-Block) are as follows: Let... For the current frame Feature map, adjacent frames Feature map, From Frame to The optical flow field of the frame, and The coordinate systems are different, respectively the first... Frame and the Frame coordinate system; first, for each position Utilizing optical flow field Feature maps of adjacent frames Perform a space-warp Warp transform to obtain the first... Feature maps in frame coordinates, i.e., physically aligned. Then, based on the feature map of the current frame... and aligned feature maps Generate query vectors respectively Key vector Sum value vector : , , ; in The linear projection matrix is ​​used; finally, attention weights are calculated and features are aggregated. ; in The scaling factor is the feature dimension; the TATA mechanism, guided by optical flow, ensures that the attention mechanism focuses on the historical and future states of the same physical object, suppressing feature misalignment noise caused by camera shake.

[0014] As a further preferred embodiment of the above scheme, the motion smoothing loss in step 7... The following formula is used for calculation: ; in, For hyperparameters, A stable optical flow field generated analytically from a network-predicted deformation field; The global motion estimation function is obtained by fitting the homography matrix using the RANSAC algorithm to extract the background dominant motion. The first term of the loss function uses the L2 norm to penalize the amplitude of the global camera motion to eliminate high-frequency jitter. The second term uses the L1 norm to penalize the residual motion, and uses the sparsity-induced properties of the L1 norm to protect the independent motion of local dynamic objects from being smoothed.

[0015] As a further preferred embodiment of the above scheme, the geometric consistency loss in step 7... The following formula is used for calculation: ; in, For the predicted deformation field, and These represent the spatial gradients of the deformation field in the horizontal and vertical directions, respectively. This represents the mean operation within a local neighborhood. The Charbonnier loss is more robust to outliers than the L2 loss; the geometric consistency loss is used to constrain the local rate of change of the deformation field, forcing the deformation field to maintain affine transformation properties in the local region and preventing non-physical shear or torsional deformation.

[0016] As a further preferred embodiment of the above scheme, step 7 also includes photometric reconstruction loss. Sum of total loss functions The construction of the photometric reconstruction loss is defined as follows: ; in, For the generated stable frames, this loss constrains the temporal continuity of the stable video sequence in terms of visual content, i.e., stable frames. Stable optical flow that can be predicted by the network from its preceding and following stable frames Reconstruction; the total loss function is defined as: ; in, and Hyperparameters are used to balance the weights of various losses.

[0017] The features of this invention are as follows: 1. Multimodal Input Construction and Early Fusion. To enable the network to be sensitive to motion boundaries from the early stages of feature extraction, this invention does not rely solely on RGB image input, but instead constructs a multimodal input fused with RGB and bidirectional optical flow. Specifically, for time steps... The center frame is pre-computed with a lightweight, high-precision optical flow network (such as SEA-RAFT) to calculate its bidirectional optical flow field with the preceding and following frames. The RGB image data and the horizontal and vertical components of the optical flow field are concatenated along the channel dimension to form a high-dimensional input tensor containing texture information and motion priors. This early fusion strategy allows the network's first convolutional kernel to learn both texture and motion gradients simultaneously, significantly improving its ability to capture motion features in weakly textured regions.

[0018] 2. Fully Differentiable ST-UNet Inference. A deep neural network called ST-UNet is constructed as the core inference engine. This network adopts a U-Net-type encoder-decoder architecture to preserve multi-scale spatial details. Unlike traditional CNNs, ST-UNet embeds specially designed decoupled spatiotemporal blocks (DST-Blocks) at each level of the encoder, bottleneck layer, and decoder. The network directly outputs a dense deformation field consistent with the input resolution, which indicates where each pixel of the output image should be sampled from the input image.

[0019] 3. Spatiotemporal Decoupling Modeling Based on Lagrange Perspective. Addressing the spatiotemporal heterogeneity of video data (static spatial texture and dynamic temporal evolution), this invention designs a decoupled spatiotemporal block (DST-Block), decomposing spatiotemporal feature extraction into two orthogonal steps: Spatial Self-Attention (SSA) employs a Shifted Window Self-Attention mechanism based on Eulerian perspective, extracting local and global spatial geometric features within a single frame to capture the shape and edges of objects. Trajectory-Aware Temporal Interactive Attention (TATA) is the core innovation of this invention. Addressing the feature misalignment problem caused by camera shake, TATA introduces a Lagrange perspective from fluid dynamics. That is, the observer is no longer fixed at a grid point but follows the movement of particles (pixels). Specifically, using the input optical flow field as a priori, the feature map of the reference frame is "warped" along the motion trajectory to align to the current frame coordinate system. The Query, Key, and Value matrices and attention weights are calculated in the aligned feature space. Guided by optical flow, TATA ensures that the attention mechanism always interacts between the historical and future states of the same physical object, eliminating feature misalignment from a physical perspective. This enables the network to robustly aggregate temporal contextual information and maintain feature consistency even under severe jitter.

[0020] 4. A "Three-Way Balance" Unsupervised Training System. To train the network without ground truth labels, this invention constructs a dynamic game-theoretic system consisting of three mutually constraining loss functions: motion smoothing loss, photometric reconstruction loss, and geometric consistency loss. The motion smoothing loss differs from traditional smoothing terms that only penalize optical flow gradients; this invention proposes a loss based on global motion decomposition. It uses the RANSAC algorithm to fit global homography from the predicted stable optical flow and applies an L2 penalty to the global motion amplitude (eliminating high-frequency jitter) while applying an L1 penalty to the residual motion (protecting sparse dynamic object motion). This design forces the network to automatically learn to distinguish between background jitter and foreground motion. The photometric reconstruction loss uses the Charbonnier loss function to constrain the visual content consistency between the stabilized video frame and its neighboring frames, preventing the network from generating completely black or content-missing images. The geometric consistency loss constrains the spatial gradient of the predicted deformation field, penalizing severe non-affine deformations in local areas to prevent "jelly effect" or image tearing, ensuring the output video conforms to the geometry of the physical world.

[0021] Compared with the prior art, the present invention has the following beneficial effects: This invention solves the problem of cascaded error accumulation. Through an end-to-end ST-UNet architecture, feature extraction, motion estimation, and path planning are integrated into a unified differentiable function, allowing gradients to propagate back through the entire link. The network can adaptively adjust the front-end feature extraction strategy based on the final image stabilization effect, avoiding the downstream collapse caused by upstream errors in traditional methods.

[0022] This invention overcomes the bottleneck of feature alignment under severe jitter. By introducing the TATA mechanism with a Lagrange perspective, it achieves feature-level physical alignment using optical flow priors. Experiments demonstrate that in scenes with large parallax and severe rotation, this mechanism effectively avoids feature misalignment in traditional Transformers, significantly improving distortion rate and stability scores.

[0023] This invention achieves robust unsupervised implicit motion separation. It does not rely on expensive and difficult-to-obtain pairwise supervised data, nor does it require the introduction of additional object detection or mask generation networks. Through a dynamic game of a "three-way balance" loss function, the network automatically converges to the optimal solution: that is, while eliminating global jitter, it preserves the natural motion of dynamic objects (such as pedestrians and vehicles) in the scene to the greatest extent possible, avoiding the common problem of a stable background but distorted foreground.

[0024] It performs excellently in weak texture and high parallax scenes. Thanks to the constraints of early multimodal fusion and geometric consistency loss, this invention can maintain the integrity of the geometric structure in weak texture areas such as ice and grass, as well as in indoor and outdoor high parallax scenes, effectively suppressing the mesh distortion phenomenon common in traditional methods. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the overall architecture of the end-to-end video stabilization method based on decoupled spatiotemporal Transformer provided in this embodiment of the invention. The diagram shows the entire process from inputting an unstable video frame sequence and bidirectional optical flow, through processing by the ST-UNet deep network, to finally outputting a stable video frame. Figure 2 This is a detailed topology diagram of the ST-UNet neural network in an embodiment of the present invention. The diagram clearly shows the hierarchical relationship between the block embedding layer, encoder, bottleneck layer, decoder, and output layer, as well as the resolution changes, channel number changes, and skip connection methods of the feature maps at each stage.

[0026] Figure 3 This is a schematic diagram of the internal microstructure of the decoupled spatiotemporal block in an embodiment of the present invention. The diagram details the serial connection between the Spatial Self-Attention (SSA) module and the Trajectory Aware Temporal Interactive Attention (TATA) module, as well as the signal flow of feature distortion alignment, Q / K / V generation, and attention weighting in the TATA module.

[0027] Figure 4 This is a comparison of the image stabilization effect of the present invention and existing technologies (DIFRINT, DUT, VidStabFormer) in weak texture scenes (such as ice surfaces and grass). The red boxes and arrows in the figure indicate the geometric distortion and drift phenomena in the image stabilization results of existing technologies, as well as the superiority of the present invention in maintaining a straight-line structure. Figure 5 This is a comparison of the image stabilization effect of the present invention and the prior art in a large parallax scene, demonstrating the geometric fidelity capability of the present invention when dealing with the relative motion of objects at different depths (such as a sofa in the foreground and a wall in the distance); the red boxes and arrows in the figure indicate the geometric distortion and artifacts in the image stabilization results of the comparison methods; Figure 6 This is a comparison of the image stabilization effects of the present invention and existing technologies in dynamic local motion scenarios; it demonstrates the ability of the present invention to effectively remove camera shake while accurately preserving the natural movement of foreground figures; the red boxes and arrows in the figure indicate the geometric distortion and artifacts present in the image stabilization results of the comparison methods. Detailed Implementation

[0028] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, but the present invention is not limited to the following technical solutions.

[0029] Example 1

[0030] Part 1: Algorithm Principles and Mathematical Models 1. Problem Definition and End-to-End Paradigm The mathematical essence of video stabilization is an inverse problem. Given a video sequence affected by camera shake... Each frame All are potentially ideal stable images After a complex geometric transformation (Including rotation, translation, perspective projection, etc.) and degradation processes. Traditional methods attempt to explicitly solve for camera trajectory parameters. Filtering it yields a smooth trajectory. Then calculate the transformation matrix. .

[0031] This invention proposes an end-to-end structured prediction paradigm that, instead of explicitly solving for intermediate camera trajectory parameters, learns a nonlinear mapping function. Directly predict from the current unstable frame To the ideal stable frame pixel-level dense deformation field Stable frames By analyzing the original frame Application of Deformation Field Obtained by bilinear interpolation resampling: This paradigm integrates geometric solution, feature extraction, and image generation into a unified computational graph, making the entire process fully differentiable.

[0032] 2. Fusion of Lagrange's perspective and Euler's perspective In fluid mechanics and computer vision, there are two basic perspectives for describing motion fields: the Eulerian perspective and the Lagrange perspective. In the Eulerian perspective, the observer is stationary at a fixed point in space (a grid point) and observes the changes in fluid properties as the fluid flows past that point. In video processing, this corresponds to focusing on fixed pixel coordinates. Time Changing pixel values. Traditional 3D-CNNs and standard Video Transformers both use this perspective. Its limitation lies in its inability to handle sharp camera shakes with fixed coordinates. Different frames may correspond to completely different physical objects, leading to semantic discontinuity in temporal feature extraction. In a Lagrange perspective, an observer follows the movement of a fluid particle, observing the changes in its properties over time. In video processing, this corresponds to tracking the trajectory of the same physical point on the image plane.

[0033] The core innovation of this invention lies in organically integrating these two perspectives into a deep network: In the spatial dimension (within a single frame), a stable geometric structure and texture features are extracted using an Eulerian perspective (SSA module). In the temporal dimension (across frames), a Lagrange perspective (TATA module) is used to warp and align the feature maps with the help of an optical flow field, ensuring that the attention mechanism follows the particle's motion trajectory, thereby solving the feature misalignment problem. The flowchart is as follows. Figure 1 As shown.

[0034] Part Two: Detailed Network Architecture Design The DSTS-Net used in this embodiment consists of two main parts: a data preprocessing module and a core inference network ST-UNet.

[0035] 1. Data preprocessing and multimodal input The input terminal uses a "sliding window" strategy, with the window size set to [value missing]. (e.g., 5 frames). For the center frame of each window This invention not only inputs its RGB pixel values, but also introduces motion priors.

[0036] Optical flow estimator selection: The SEA-RAFT (Simple Efficient Accurate RAFT) network was chosen. Compared to traditional RAFT, SEA-RAFT introduces rigid motion pre-training. Background motion in video stabilization is mainly caused by rigid camera motion. SEA-RAFT has extremely high estimation accuracy and robustness for this type of motion, and its inference speed is fast, making it suitable as a preprocessing module.

[0037] Input tensor construction: for the center frame The present invention takes the frames before and after it. Calculate bidirectional optical flow: forward optical flow Backward optical flow .Will (3 channels) (2 channels) (2 channels) are spliced ​​together along the channel dimension to form The feature tensor. For a window containing 5 frames, the total input dimension is... In this implementation, B is set to 4.

[0038] 2. ST-UNet Network Architecture ST-UNet uses a variant of the U-Net architecture, which includes a block embedding layer, encoder, bottleneck layer, decoder, and output layer.

[0039] Patch Embedding: Using The convolution kernel, with a stride of 8, reduces the input resolution from... downsampling to The number of channels has been expanded to This step achieves preliminary compression and feature mapping of spatial information.

[0040] The encoder consists of four cascaded DST blocks, with a 2×2 convolution inserted after each of the first three blocks for downsampling. Stage 1: Feature map size. Number of channels Phase 2: Feature Map Size Number of channels Phase 3: Feature Map Size Number of channels Phase 4: Feature Map Size Number of channels .

[0041] Bottleneck layer: Located between the encoder and decoder, it is the core component for capturing global motion semantics. Essentially, it is an independent DST-Block that operates on the lowest resolution feature map after the fourth stage of the encoder. Here, the receptive field is the largest, responsible for capturing global camera motion patterns (such as overall translation and rotation trends).

[0042] The decoder gradually recovers the high-resolution, dense deformation field through upsampling and skip connections, and fuses multi-scale features to compensate for the loss of detail. The first three stages use 2×2 transposed convolutions for upsampling, doubling the feature map resolution and adjusting the number of channels; the last stage keeps the resolution unchanged. After each upsampling, the decoder concatenates the current feature with the cached features from the corresponding stage of the encoder, and then inputs it into a DST-Block for refinement.

[0043] Output Layer: The number of channels is mapped to 2 using 1×1 convolution (corresponding to the offsets in the x and y directions of the dense deformation field), and then an 8x bilinear interpolation upsampling is used to predict a dense deformation field with the same resolution as the input. This deformation field indicates from which position in the input image each pixel of the output image should be sampled. (Stable frame) By warping the original frame based on bilinear interpolation (Warp) get: .

[0044] 3. Detailed Explanation of Decoupled Spacetime Block (DST-Block) DST-Block is the basic building block of a network, designed to efficiently process spatiotemporal features. Its structure is as follows: Figure 3As shown, it includes cascaded spatial self-attention (SSA) and trajectory-aware temporal interactive attention (TATA) modules.

[0045] The Spatial Self-Attention (SSA) module is implemented based on the Swing Transformer architecture and employs a windowed multi-head self-attention mechanism. Specifically, SSA divides the input feature map into... (In this embodiment) The module uses local windows, and self-attention is computed independently within each window. To enhance cross-window information interaction, an optional window shifting mechanism is introduced, which enables communication between windows by cyclically shifting the feature map.

[0046] The Trajectory-Aware Temporal Interactive Attention (TATA) module is responsible for fusing temporal information. For the features of the current frame... and adjacent reference frame features Using pre-calculated optical flow field The first Feature map of a frame "Warp" to the In the frame coordinate system After this operation, In position The semantic content at that location is already consistent with In position The content at that location theoretically achieves physical alignment. Cross-attention is calculated: Query(Q) is derived from the features of the current frame. Key(K) comes from the aligned features of neighboring frames. Value(V) also comes from the aligned features of neighboring frames. .in This is a linear projection matrix. Finally, the trajectory-aware temporal interactive attention (TATA) is calculated using the following formula: .

[0047] TATA's design cleverly utilizes the geometric priors provided by optical flow, allowing the Transformer to focus on refining features and handling occlusion instead of learning complex motion correspondences from scratch. The Softmax operation acts as a learnable "confidence gating": if the optical flow is accurate and the alignment is good, and High similarity results in a weight close to 1; if occlusion or optical flow errors occur, and Content mismatch leads to a decrease in weight, and the network automatically suppresses the spread of misinformation.

[0048] Part Three: Loss Function and Training Strategy This invention does not require any manually labeled stable videos, but instead uses a carefully designed loss function for self-supervised learning.

[0049] 1. Analytical Derivation of Steady Optical Flow To calculate the loss, it is necessary to know "if the predicted deformation field were applied". New video streaming optical stream "What will it be?" Explicitly calling the optical flow network to calculate the optical flow of the stabilized video during training is extremely expensive and non-differentiable. This invention analytically calculates the stabilized optical flow based on the input optical flow and the predicted deformation field: tracking the stabilized frame. Each of them is located in The pixels, see how they are in a stable frame. Where has the China Mobile traversed to (denoted as...)? The difference between the two is the stable optical flow of that pixel. This process can be decomposed into a composite of three coordinate transformation steps: First, find stable frames. Central Pixels in the original unstable frame The corresponding position in .because This refers to the mapping from the stable frame coordinate system to the unstable frame coordinate system (i.e., backward warping). For a stable frame with position... The pixel, whose position in the unstable frame is: Then, using the original unstable optical flow... ,get In the next frame The position in the middle Substituting the result from the previous step: Unstable frames The position in the middle Mapping back to the new stable frame The position in the middle : .in, Defined from unstable frames coordinate system to stable frame The mapping of the coordinate system. Because and Because they are in different coordinate systems, therefore This invention utilizes forward sputtering to achieve... To approximate the solution This is a fully differentiable operation. The final stable optical flow vector... Defined as displacement Substituting the above steps, we get: ; in, , Stabilized frames are achieved through... The calculation is obtained. This process allows, in each iteration, the predictions based on the network's current forecasts to be updated. and and fixed input Dynamically generate stable optical flow and stable frames Then, the loss function is calculated.

[0050] 2. Detailed Explanation of Losses in the Tripartite Balance of Power Due to the lack of real labels, DSTS-Net training relies entirely on a carefully designed loss function system. This invention constructs a loss function constrained by three forces. Its core lies in this "three-way balance" loss function system, which not only prevents the network from learning meaningless trivial solutions but also implicitly separates camera motion from object motion. It should be noted that although SEA-RAFT is strong, it still contains noise. Before calculating the loss function, this invention uses the RANSAC algorithm and forward-backward consistency check to filter the optical flow, eliminating occluded points and mismatched points. Filtered pixels do not participate in loss calculation or backpropagation. Figure 4 As shown, this process illustrates the steps involved in calculating the loss function and the data sources. This visualization clarifies the dynamic interaction of the three-way losses, helping to understand the application of physical constraints in unsupervised training. The loss function of DSTS-Net consists of motion smoothing loss (… ), photometric reconstruction loss ( ) and geometric consistency loss ( It consists of three parts: ; in, and These are the weights of the corresponding loss terms.

[0051] Motion smoothing loss ( The spatial gradient of the stable optical flow is the core driving force for image stabilization. Many existing methods use the spatial gradient of the stable optical flow as the loss. However, this has serious theoretical flaws: This only implies smoothness within the flow field (i.e., rigid motion), but it does not eliminate camera shake. If the camera undergoes large overall translational shaking, its optical flow spatial gradient remains small, and the network cannot eliminate this global shake. To achieve a tripod-like image stabilization effect, this invention proposes penalizing the amplitude of the global motion: ; in, This is a hyperparameter (taken as 2 in this embodiment). The function is a global motion estimator. Its input is the current steady optical flow field. The output is the best globally parameterized fit to the flow field. This function first uses the RANSAC algorithm from... Using mid-sampling points, fit a homography matrix. This step robustly extracts the dominant flow representing camera motion while ignoring motion interference from local objects. Then, based on the fitted matrix... Regenerate a dense flow field that covers the entire graph: . This represents the portion of the optical flow in the video caused solely by camera movement (i.e., the dominant motion of the background). The first term in the above equation uses the L2 norm to penalize the magnitude of the global motion. This forces the camera trajectory to be as slow and smooth as possible, eliminating high-frequency jitter; the second term uses the L1 norm to penalize the residuals. The L1 norm allows for the existence of sparse outliers. This means that the network is allowed to retain pixels that do not conform to the global motion (i.e., dynamic objects) without forcibly smoothing them out.

[0052] Photometric reconstruction loss ( The goal is for the stabilized video to appear visually continuous. That is, the stabilized frames... Stable optical flow that can be predicted by the network from its preceding and following stable frames reconstruction: ; in (generally The Charbonnier loss is more robust to outliers than the L2 loss. This forces the network to preserve the texture details of the original video and the realistic motion of objects in the scene.

[0053] Geometric consistency loss ( This prevents the network from arbitrarily distorting images without physical means (such as tearing or crumpling them) in pursuit of smoothness. The deformation vectors of adjacent pixels should closely approximate this objective reality. Calculate using the following formula: ; in, These represent the spatial gradients in the horizontal and vertical directions, respectively. It penalizes the deformation field. The degree to which the local affine transformation deviates from the rigid transformation. Ensuring that the local deformation is conformal, i.e., does not change the local shape, avoids the "jelly effect".

[0054] 3. Training process Data source: The unstable video portion of the DeepStab dataset was used. Five frames, including the center frame and the two frames before and after it, were selected as training samples, resulting in a training set of approximately 48,000 training samples.

[0055] Preprocessing: All video frames were uniformly adjusted to... The resolution was adjusted to accommodate GPU memory limitations. Pre-computed optical flow inevitably contains noise. Before calculating the loss function, the optical flow was filtered using the RANSAC algorithm and bidirectional optical flow consistency detection to remove occlusion points and mismatched points. The filtered regions were not included in loss calculations or backpropagation.

[0056] Hyperparameters: Set extremely high geometric consistency loss weights. The reason is that it reflects the primary physical constraint in predicting dense deformation fields: preventing mesh topology collapse (such as folding or tearing). If geometric constraints are insufficient, the network is prone to compromise in pursuit of maximum motion smoothness (reducing...). This results in non-physical image distortion. This is determined by the characteristics of image stabilization tasks.

[0057] This embodiment is implemented on a high-performance workstation equipped with an NVIDIA RTX A6000 GPU, using the PyTorch deep learning framework. The network parameters are initialized using the Xavier method to maintain consistent signal variance during forward propagation and prevent gradient vanishing or exploding in deep Transformer networks. The optimizer is Adam, and the initial learning rate is set to 10. -5 Combined with a cosine annealing scheduling strategy, the maximum cycle time is 50 rounds, and the minimum learning rate is reduced to 1×10. -6 The batch size B=4. This learning rate scheduling strategy is crucial for optimizing a non-convex loss function that contains multiple mutually restraining terms. It allows the model to quickly traverse flat regions in the early stages of training and finely converge to the optimal solution of the "three-way restraint" loss function in the later stages.

[0058] Part Four: Experimental Results and Performance Analysis This invention has been extensively tested on the standard NUS dataset. The evaluation metrics adopted are the generally accepted standards in the field of video stabilization: Cropping Ratio (C), Distortion Value (D), and Stability Score (S). Cropping Ratio is defined as the ratio of the area of ​​the image retained after stabilization to the area of ​​the original image. This metric directly reflects the degree of information loss during the stabilization process. The closer the value is to 1, the less field of view (FOV) is lost, and the stronger the algorithm's ability to retain information. The distortion rate is calculated based on the affine components of the homography matrix before and after image stabilization. Specifically, it quantifies the degree of anisotropic scaling between the original frame and the stabilized frame. The closer the value is to 1, the better the image geometry is preserved, and the less severe the stretching or shearing deformation has occurred. The stability score is based on frequency domain analysis, calculated by Fourier transform to determine the energy proportion of low-frequency components in the motion trajectory. The closer the value is to 1, the more thoroughly the high-frequency jitter is suppressed, and the smoother the video appears.

[0059] The comparison covers three key technological generations in video stabilization development: Traditional geometry-driven methods—MeshFlow. As a widely adopted 2D stabilization standard in industry, MeshFlow utilizes sparse feature points to construct and smooth a mesh, representing the "ceiling" of traditional non-deep learning methods. Early deep-supervised methods—StabNet. As an early representative of deep learning in stabilization, StabNet attempted to transform parameters through CNN regression, but was limited by the scarcity of paired data and a simple motion model. Current state-of-the-art (SOTA) methods. These methods include DIFRINT, based on deep iterative frame interpolation; DUT, based on unsupervised keypoint trajectory optimization; the recently emerging TranStable, based on Transformer and Generative Adversarial Networks (GANs); and VidStabFormer, based on Transformer for image inpainting and stabilization.

[0060] Table 1 presents detailed comparative results for five challenging scenarios on the NUS dataset. The experimental results for the comparative methods in Table 1 are from the original papers, with "-" indicating items for which results were not provided in the original papers. The original papers for TranStable and VidStabFormer do not provide quantitative experimental results for specific scenarios, only average results.

[0061] Table 1 compares the quantitative evaluation results of DSTS-Net and mainstream methods on the NUS dataset. C, D, and S represent the clipping rate, distortion rate, and stability score, respectively; higher values ​​are better. Bold indicates the optimal value, and italic indicates the suboptimal value.

[0062]

[0063] *Note: DIFRINT and VidStabFormer use generative interpolation or inpainting techniques to fill the boundaries, so the theoretical cropping rate is 1.00. However, this does not mean that the original field of view is preserved. Instead, it fills the gaps by generating content at the expense of image realism.

[0064] Quantitative Analysis. As shown in Table 1, the present invention (DSTS-Net) achieved the best or second-best results in terms of clipping rate, distortion rate, and stability. The clipping rate of 0.94 is significantly higher than MeshFlow (0.55) and StabNet (0.54). This indicates that the present invention largely preserves the original field of view during image stabilization, without being forced to clip a large number of edges due to incorrect path planning. The distortion rate of 0.96 is close to the perfect value of 1.0. This is due to the feature alignment and strong geometric consistency constraints of the Lagrange perspective, which maintains the structure well in scenes with large parallax. The stability of 0.87 is comparable to state-of-the-art methods, demonstrating the effectiveness of the "three-way balance" loss in eliminating jitter. These quantitative experimental results not only confirm the comprehensive superiority of DSTS-Net over existing methods but also verify its theoretical innovation in solving the core challenges mentioned in the introduction: the end-to-end design eliminates the information bottleneck between modules, spatiotemporal decoupling improves the accuracy of feature alignment, and the unsupervised loss game achieves efficient motion decomposition. These advantages make DSTS-Net more versatile in practical applications.

[0065] Qualitative Analysis. To visually demonstrate the superiority of DSTS-Net in handling complex real-world scenarios and to supplement the assessment of visual perception quality, which is difficult to quantify quantitatively, the most representative challenging scenarios from the NUS dataset were selected for visual comparison. This invention is visually compared with methods such as DIFRINT, DUT, and VidStabFormer. To ensure fairness in the comparison, all comparison results were generated using the authors' open-source official pre-trained models or code on the same input sequence without any additional post-processing.

[0066] Weak texture scenes ( Figure 5 Scenes with weak or similar textures (such as ice surfaces or lawns) have always been a challenge for video stabilization. Figure 5The first line shows a scene of skating on a large ice surface, and the second line shows a grassy area enclosed by a fence. In this scene, the DUT method produces a very obvious unnatural phenomenon: the ice rink barriers and the grassy area's metal fence exhibit wavy, bending deformations. This "geometric drift" phenomenon is caused by the DUT's reliance on sparse feature points for motion estimation. With the lack of texture on the ice surface and the interference of similar textures on the grassy area, feature points are extremely difficult to extract and match, causing the inverse problem of motion estimation to become ill-posed. To meet smoothness constraints, the DUT's mesh deformation module produces incorrect interpolation in areas lacking data constraints, causing the originally rigid physical structure (fence) to be distorted into a fluid shape. Similarly, DIFRINT and VidStabFormer, due to their Euler perspective attention mechanism, are prone to temporal misalignment in areas with inconspicuous features, resulting in blurred or distorted fence edges. In contrast, in the DSTS-Net results, the ice rink barriers and the grassy area fence maintain a perfectly straight structure. This is thanks to the Lagrange perspective design of the TATA module. Even with sparse ice surface texture, the dense optical flow field generated by SEA-RAFT can still propagate through the motion of surrounding textured areas (such as people and fence edges), providing a robust motion prior. TATA utilizes this prior to align features, enabling the network to "understand" the overall rigid motion of the scene, thus maintaining the physical realism of the geometric structure in areas with weak or similar textures.

[0067] Scenes with large parallax (such as when the camera pans, the sofa and people in the foreground move at different speeds than the cabinet and walls in the background) are a touchstone for testing the geometric fidelity of image stabilization algorithms. Figure 6 The examples showcase indoor sofa scenes and supermarket scenes with significant depth variations. In such scenes, methods like DUTs often exhibit a "cut-and-paste" phenomenon. (Observation) Figure 6In videos stabilized by the DUT, faces, sofa edges, and the walls and cabinets behind them appear misaligned, stretched, and distorted. This is because these methods attempt to explain complex 3D parallax motion using a single 2D transformation model, failing to simultaneously maintain planar stability at different depths. In the DIFRINT stabilized results, the wall paintings and the cashier cabinet appear distorted, a common problem with frame interpolation methods. VibStabFormer, due to image rotation, shows noticeable blurring at the edges of the stabilized results. DSTS-Net's advantage lies in maintaining strict geometric rigidity. Straight lines in the image remain straight, and the relative motion between foreground and background is natural. DSTS-Net demonstrates the advantage of end-to-end dense prediction here. The network output is not a simple homography matrix, but a pixel-level deformation field. Guided by the "three-way balance" loss, especially constrained by the geometric consistency loss, the network learns to apply different degrees of smoothing correction to regions at different depths. The results show that DSTS-Net not only stabilizes background jitter but also perfectly maintains the relative geometric relationship between the foreground sofa and cabinets, eliminating visual layering. Furthermore, the Lagrange perspective introduced by the TATA module also plays a role here. TATA aggregates features along optical flow trajectories. This means that regardless of how objects move in the frame (foreground objects move faster, background objects move slower), the attention mechanism always focuses on the spatiotemporal evolution of the same physical object. This physical consistency ensures that the network does not confuse motion patterns at different depths, thus achieving a 3D-like image stabilization effect without explicit 3D reconstruction.

[0068] In scenarios with significant local motion, the scene typically presents two major challenges: dynamic semantic interference and information loss. The local optical flow generated by rapid foreground motion is easily confused with global camera jitter, causing existing methods to introduce significant errors during motion separation, thereby amplifying background distortion or foreground artifacts. Figure 6The study showcases outdoor motion scenes and indoor crowd sequences with significant local motion. In these highly dynamic environments, the comparative methods all exhibit limitations to varying degrees. DIFRINT, failing to effectively handle dense local motion, often leads to incorrect separation of the foreground and background, resulting in noticeable distortions or artifacts. For example, pillars in flowerbeds show non-rigid bending deformation, and unnatural blurry artifacts appear in the hair area of ​​pedestrians. DUT exhibits significant blurring and miscompensation in the area indicated by the arrow, such as abnormal distortion of facial contours or local magnification of backpack straps. This is because its cascading paradigm misinterprets foreground dynamics as camera shake, leading to overcompensation and amplifying semantic interference. VidStabFormer, in this scene, introduces a noticeable tilt to structures that should remain vertical (such as pillars), reflecting its inadequacy in capturing highly dynamic spatiotemporally dependent elements and its inability to accurately balance camera motion and object independence. In contrast, DSTS-Net achieves implicit and precise decomposition of motion signals through a "three-way balance" unsupervised loss system and the trajectory awareness mechanism of DST-Block. This drives the network to effectively separate high-frequency unintentional camera jitter and local foreground dynamics in a high-dimensional feature space, while avoiding information loss caused by over-cropping. Foreground motion in the region indicated by the arrow is naturally preserved, and the background structure remains rigid and intact, without visible distortion or artifacts. This result not only eliminates semantic confusion but also provides a smoother and more natural visual output.

[0069] As can be seen from the detailed description of the above embodiments, the present invention provides a theoretically complete, highly efficient and robust video stabilization solution, which has significant practical value and broad market prospects.

Claims

1. A decoupled spatiotemporal Transformer end-to-end video stabilization method, characterized in that, The method includes the following steps: Step 1: Obtain the unstable video sequence to be processed. The video sequence consists of multiple consecutive frames arranged in chronological order. Select the current frame to be processed as the center frame. , t The frame number is used as the reference, and a set time window of 2 is selected before and after it. k +1, k The window radius is defined, and adjacent frames within it are used as context input. Step 2: Calculate the bidirectional optical flow between adjacent frames in the unstable video sequence, generating the forward optical flow field from the previous frame to the next frame and the reverse optical flow field from the next frame to the previous frame. Then, calculate the forward optical flow of each frame in the RGB 3 channels. 2 represents 2 channels. H and W These are the image height and width, and the backward optical flow of the frame, respectively. By splicing along the channel dimension, a structure is formed. The feature tensor, for containing 2 k A window of +1 frame is used to construct a multimodal input tensor with dimension . B represents the batch size; Step 3: Input the multimodal input tensor into the pre-built spatiotemporal Transformer-UNet neural network model ST-UNet; the ST-UNet model adopts an encoder-decoder architecture, and a decoupled spatiotemporal block DST-Block is embedded in each stage of the encoder, bottleneck layer and decoder. Step 4: The input features are decoupled in time and space using the decoupled spatiotemporal block in the ST-UNet model. The decoupled spatiotemporal block alternately executes the spatial self-attention (SSA) mechanism and the trajectory-aware temporal interactive attention (TATA) mechanism. The TATA mechanism is based on the Lagrange perspective and uses the bidirectional optical flow field as motion guidance to distort and align the feature maps of adjacent frames along the motion trajectory to the coordinate system of the current frame. Attention weights and aggregated temporal features are then calculated in the aligned feature space. Step 5: Generate a dense deformation field corresponding to the resolution of the center frame through the output layer of the ST-UNet model; wherein the dense deformation field has two channels, which respectively represent the horizontal displacement component and the vertical displacement component required to eliminate jitter and parallax effects for each pixel in the image under the original jitter frame coordinate system. Step 6: Based on the dense deformation field, for the target stable frame Each pixel in the image is determined based on the dense deformation field. The corresponding displacement component is determined in the original jitter center frame. The corresponding sampling coordinates are used, and a differentiable resampling warp operation is performed on the center frame using bilinear interpolation to generate the stable video frame. ; Step 7: During the model training phase, construct a composite loss function that includes motion smoothing loss, photometric reconstruction loss, and geometric consistency loss, and optimize the parameters of the ST-UNet model through backpropagation under unsupervised conditions.

2. The method according to claim 1, characterized in that, The spatial self-attention (SSA) mechanism in the decoupled spatiotemporal block (DST-Block) is configured as follows: the input feature map is divided into rectangular windows of fixed size, and multi-head self-attention is independently calculated within each window; Between consecutively stacked DST-Blocks, a cyclic displacement operation is used to offset the window grid to establish a cross-window spatial information exchange channel. The SSA mechanism is based on the Euler perspective and focuses on extracting texture geometric features within a fixed pixel grid, thereby enhancing the structural representation capability within a single frame.

3. The method according to claim 1, characterized in that, The specific calculation steps of the trajectory-aware temporal interactive attention (TATA) mechanism in the decoupled spatiotemporal block are as follows: Let... For the current frame Feature map, adjacent frames Feature map, From Frame to The optical flow field of the frame, and The coordinate systems are different, respectively the first... Frame and the Frame coordinate system; first, for each position Utilizing optical flow field Feature maps of adjacent frames Perform a space-warp Warp transform to obtain the first... Feature maps in frame coordinates, i.e., physically aligned. ; Then, based on the feature map of the current frame and aligned feature maps Generate query vectors respectively Key vector Sum value vector : , , ； in The linear projection matrix is ​​used; finally, attention weights are calculated and features are aggregated. ； in The scaling factor is the feature dimension; the TATA mechanism, guided by optical flow, ensures that the attention mechanism focuses on the historical and future states of the same physical object, suppressing feature misalignment noise caused by camera shake.

4. The method according to claim 1, characterized in that, Motion smoothing loss in step 7 The following formula is used for calculation: ； in, For hyperparameters, A stable optical flow field generated analytically from a network-predicted deformation field; The global motion estimation function is obtained by fitting the homography matrix using the RANSAC algorithm to extract the background dominant motion. The first term of the loss function uses the L2 norm to penalize the amplitude of the global camera motion to eliminate high-frequency jitter. The second term uses the L1 norm to penalize the residual motion, and uses the sparsity-induced properties of the L1 norm to protect the independent motion of local dynamic objects from being smoothed.

5. The method according to claim 1, characterized in that, Geometric consistency loss in step 7 The following formula is used for calculation: ； in, For the predicted deformation field, and These represent the spatial gradients of the deformation field in the horizontal and vertical directions, respectively. This represents the mean operation within a local neighborhood. The Charbonnier loss is more robust to outliers than the L2 loss; the geometric consistency loss is used to constrain the local rate of change of the deformation field, forcing the deformation field to maintain affine transformation properties in the local region and preventing non-physical shear or torsional deformation.

6. The method according to claim 5, characterized in that, Step 7 also includes photometric reconstruction loss. Sum of total loss functions The construction of the photometric reconstruction loss is defined as follows: ； in, For the generated stable frames, this loss constrains the temporal continuity of the stable video sequence in terms of visual content, i.e., stable frames. Stable optical flow that can be predicted by the network from its preceding and following stable frames Reconstruction; the total loss function is defined as: ； in, and Hyperparameters are used to balance the weights of various losses.

Images