A video stabilization method and system based on inter-frame homography transformation

Abstract

The application discloses a video stabilization method and system based on video inter-frame homography transformation, belongs to the technical field of video supervision, and effectively distinguifies foreground and background in a video through a foreground mask module; after the foreground part is covered by using the mask, effective shallow semantic features of a background area are acquired through a credible shallow feature extraction module; for the effective shallow features of the background area, a motion estimation module associated with local features and global features is used to estimate the homography relationship between continuous adjacent frames; through a video stabilization smoothing module, the homography between the transformation frame and the reference frame obtained by using the motion relationship between adjacent frames is used to realize the stabilization of the whole video by using STN transformation. The effective combination of the foreground mask and the credible shallow feature avoids the artifact or distortion phenomenon caused by large disparity, the local and global feature relationship is established based on the credible shallow feature, accurate camera motion is obtained, and finally, the video stabilization is realized.

Description

A video stabilization method and system based on inter-frame homography transformation Technical Field

[0001] This invention relates to the field of video supervision technology, and in particular to an unsupervised method and system for stabilizing deep video based on inter-frame homography transformation. Background Technology

[0002] Due to the high cost and complex structure of professional video stabilization equipment, the development of video stabilization algorithms began. Traditional video stabilization methods are divided into 2D methods and 3D methods based on whether 3D reconstruction is required. The main difference between the two lies in the methods of feature trajectory smoothing and stable frame generation.

[0003] 2D methods typically use feature matching techniques to stabilize video frames using inter-frame transformation matrices, estimating only the linear transformation between consecutive frames. These methods are robust and faster to implement; however, because the linear motion model of 2D methods is too weak, they cannot fundamentally address the parallax problem caused by changes in scene depth.

[0004] 3D methods can address the problems caused by parallax. These methods typically utilize SFM (Sparse Focal Path Measuring) technology to record feature trajectories, reconstruct 3D positions, and ultimately produce strongly stable results. For example, Liu et al. used 3D camera trajectories and sparse 3D point clouds reconstructed by SFM to guide warping, achieving video stabilization by smoothing the camera trajectory. While the stabilization effect is superior to 2D methods, this approach is relatively fragile and lacks robustness. Furthermore, 3D video stabilization requires expensive 3D reconstruction costs and significant time investment; therefore, although 3D stabilization yields better results, its applicability is not as broad as that of 2D methods.

[0005] With the rise of deep neural networks, deep learning-based video stabilization methods have emerged. Most of these methods construct a supervised learning framework, learning from the transformations of historical or adjacent frames to predict unstable frames and thus stabilize the video. For example, Xu et al.'s deep learning-based video stabilization method utilizes historical frames of the video, performing supervised learning from the content captured simultaneously by two cameras (stable and unstable). Zhao et al. proposed the deep learning-based pixel-level video stabilization structure PWStableNet to address videos with significant parallax. This method takes the sequence of unstable frames and their adjacent frames as input, establishing a multi-level cascaded encoder-decoder structure, allowing subsequent stages to learn residuals from the feature maps of previous stages, thereby achieving stabilization. In addition, some researchers have introduced other techniques into the video stabilization process. For instance, Yu et al. introduced optical flow maps to represent inter-frame changes, using different optical flows to distinguish foreground and background within the video. Karpenko et al. also used gyroscopes for 3D rotation estimation to obtain more accurate camera data. However, these methods still present certain challenges in generating the final output frame, and inevitably produce visible distortion and severe artifacts when fusing information from multiple frames.

[0006] Therefore, designing a video stabilization method and system to effectively avoid artifacts or distortion is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a video stabilization method and system based on inter-frame homography transformation, which can effectively avoid the generation of artifacts or distortion while fusing information from adjacent frames. It uses continuous video frames and their foreground masks as input to ultimately achieve video stabilization.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] On one hand, this invention discloses a video stabilization method based on inter-frame homography transformation, which includes the following steps:

[0010] S1: Divide the video into background and foreground regions, cover the foreground region, and generate video frame I. k Foreground mask M k ;

[0011] S2: Extract the video frame I based on a shallow network. k low-level features F k Simultaneously, the video frame I is obtained through a convolutional network. k Confidence graph A k ; the foreground mask Mk The low-level feature F k and the confidence graph A k The video frame I is obtained by fusion. k credible low-level features G in the background region k ;

[0012] S3: Let k = k + 1, repeat S1-S2 to obtain video frame I. k+1 credible low-level features G in the background region k+1 ;

[0013] S4: Construct a motion estimation neural network and, based on the video frame I k credible low-level features G in the background region k and video frame I k+1 credible low-level features G in the background region k+1 Obtain the homography relationship matrix H between adjacent video frames. k+1,k ;

[0014] S5: Based on the homography relationship matrix H between adjacent video frames k+1,k The overall motion path of the video is smoothed to achieve video stabilization.

[0015] Preferably, S1 includes:

[0016] S11: Based on the semantic segmentation model, different objects in the video are distinguished to determine candidate objects. The area other than the candidate objects is determined as the background area; the candidate objects are movable objects.

[0017] S12: Filter according to the position information of the candidate object. For the same candidate object, if its position change in the previous frame and the next frame is less than the set lower threshold, the area where the candidate object is located is determined to be the background area. If its position change is greater than the set upper threshold, it is determined to be the foreground area.

[0018] S13: Pixels identified as background areas are represented by 1, and candidate objects identified as foreground objects are represented by 0. The foreground area is covered, and the foreground mask M is finally generated. k .

[0019] Preferably, the video frame I k credible low-level features G in the background region k The calculation formula is:

[0020] G k =M k ·F k ·A k , 1≤k≤T;

[0021] In the formula, T represents the total number of frames in the video.

[0022] Preferably, S4 includes:

[0023] S41: Construct feature map B = [G k G k+1 The feature map B undergoes a convolutional layer to extract high-level semantic features Z and reliable low-level features G for the background region. k and G k+1 The size is H×W×C;

[0024] S42: Reshape the high-level semantic feature Z into a sequence {z} i ∈R C |i=1,...,N},N=H×W;

[0025] S43: Feature z i The features are converted into different formats using different 1×1 convolution functions. The correlation between features in adjacent frames is calculated based on the different formats of features. The calculation method is as follows:

[0026] k i =W k z i ;q i =W q z i ;v i =W v z i ;

[0027]

[0028] Among them, W k W q W v These represent three different 1×1 convolution functions; k i q i v i Representing z respectively i Features of the three different formats converted; β i，j This represents the correlation between the i-th position and the j-th position; i and j represent points in the features of the k-th frame and the (k+1)-th frame, respectively.

[0029] S44: Based on the correlation β of the features between adjacent frames obtained. i，j This yields the high-level features y = (y1, y2, ..., y3). i , ..., y N )∈R C×N The calculation process is as follows:

[0030]

[0031] S45: Reshape the high-level feature y to obtain the reshaped feature Y, the format of which is H×W×C:

[0032] S46: The reconstructed feature Y is combined with the high-level semantic feature Z using residual links as input to the subsequent fully connected layer to obtain four motion displacement vectors from frame k to frame (k+1). Then, DLT is used to convert these four motion displacement vectors into a homography relation matrix H between adjacent video frames. k+1，k .

[0033] Preferably, the formula for extracting the high-level semantic feature Z is as follows:

[0034] Z = AvgPool(σ(f) 3×3 (B)));

[0035] Where σ is the ReLU function; f 3×3 This indicates a convolution operation with a kernel size of 3; AvgPool represents the max pooling operation.

[0036] Preferably, S4 further includes: constructing a loss function and optimizing the motion estimation neural network based on the loss function;

[0037] The loss function is:

[0038]

[0039] Where α and γ are the equilibrium hyperparameters, and E is a third-order identity matrix.

[0040]

[0041] L(I k+1 I k )=||F k+1 -F k ||1.

[0042] Preferably, S5 includes:

[0043] The overall video stabilization is achieved based on the following formula:

[0044] I' k+1 =I k+1 *P k *H k+1，k , 1≤k≤T;

[0045] Among them, I′ k+1 Represents the stabilized video frame I k+1 P kThe motion path of the camera from the first frame to the kth frame is represented by the following method:

[0046]

[0047] In the formula, H t，t-1 This represents the homography relation matrix from frame t to frame (t-1).

[0048] On the other hand, the present invention also discloses a video stabilization system based on video frame homography transformation, used to implement the above-mentioned video stabilization method based on video frame homography transformation, comprising:

[0049] The foreground mask module is used to divide the background and foreground regions of the video, cover the foreground region, and generate a foreground mask for the video frame;

[0050] A reliable shallow feature extraction module is used to obtain reliable low-level features of the background region of the video frame;

[0051] The motion estimation module is used to construct a motion estimation network and obtain the homography relationship matrix between adjacent video frames based on the reliable low-level features of the background region of adjacent video frames;

[0052] The path smoothing module is used to smooth the overall motion path of the video based on the homography relationship matrix between adjacent video frames, thereby achieving video stabilization.

[0053] Preferably, the reliable shallow feature extraction module includes:

[0054] The low-level feature extraction unit is used to extract low-level features of video frames based on shallow networks.

[0055] A confidence map acquisition unit is used to obtain the confidence map of the video frame through a convolutional network;

[0056] The fusion unit is used to fuse the foreground mask, the low-level features, and the confidence map of the video frame to obtain the reliable low-level features of the background region of the video frame.

[0057] As can be seen from the above technical solution, compared with the prior art, this invention discloses a video stabilization method and system based on inter-frame homography transformation. It effectively distinguishes the foreground and background in the video using a foreground mask module; after covering the foreground portion with a mask, a reliable shallow feature extraction module obtains effective shallow semantic features of the background region; for the effective shallow features of the background region, a motion estimation module that correlates local and global features estimates the homography relationship between consecutive adjacent frames; and a video stabilization and smoothing module uses STN transformation to stabilize the entire video by utilizing the homography between the transformed frame and the reference frame obtained from the motion relationship between adjacent frames. This invention, through the effective combination of the proposed foreground mask and reliable shallow features, avoids artifacts or distortion caused by large parallax. Based on reliable shallow features, it establishes local and global feature relationships to obtain accurate camera motion, ultimately achieving video stabilization. Attached Figure Description

[0058] Figure 1 is a flowchart of a video stabilization method based on inter-frame homography transformation;

[0059] Figure 2 is a schematic diagram of the mask obtained by the foreground mask module for video frames;

[0060] Figure 3 is a schematic diagram illustrating the expected video stabilization operation;

[0061] Figure 4 is a schematic diagram of the principle analysis based on the path smoothing module;

[0062] Figure 5 shows the structure of the loss function;

[0063] Figure 6 is a schematic diagram comparing the scene regions of existing full-frame video stabilization methods and the proposed method;

[0064] Figure 7 is a schematic diagram comparing the imaging of the dynamic foreground region of two optical flow-dependent methods and the method described in this paper;

[0065] Figure 8 is a schematic diagram of the visual comparison of ablation experiments. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Referring to Figure 1, this embodiment discloses a video stabilization method based on inter-frame homography transformation. The framework shown in Figure 1 is encapsulated into a loop scheme. Starting from the first frame of the video, the motion relationships between adjacent frames are estimated, and this process iteratively continues until the last frame. The obtained motion relationships are then used to perform stabilization operations on each frame, ultimately resulting in a stable video segment. The foreground mask M at input... k This is obtained using the foreground mask module. The reliable shallow feature extraction module extracts low-level features F. k At the same time, an attention mechanism is used to obtain a confidence map A. k The process involves removing regions with indistinct features from the image. Then, the foreground mask, confidence map, and low-level features are fused to obtain low-level features of reliable background regions, which are then fed into subsequent networks. In the motion estimation module, a homography matrix representing the motion relationship from frame (k+1) to frame (k) is estimated. Finally, based on the cumulative motion relationship from frame 1 to frame (k+1), video smoothing is performed on frame (k+1), thus achieving stable operation on a video of length k+1 frames.

[0068] The core idea of ​​this method is to smooth the motion trajectory of objects within the camera by utilizing the motion relationship of the background between adjacent video frames. This achieves a good stabilization effect even when dealing with videos containing "obvious moving objects." The specific steps are as follows:

[0069] S1: Divide the video into background and foreground regions, cover the foreground region, and generate video frame I. k Foreground mask M k .

[0070] As is well known, video content is complex, containing not only static background elements but also dynamic foreground elements, such as pedestrians, animals, and vehicles. The movement of these dynamic foreground elements is random and unpredictable. Because they are moving, the trajectories of foreground objects in the video do not match the camera's trajectory. Therefore, dynamic foreground regions in video frames can affect the accuracy of homography estimation. To avoid the influence of dynamic foreground elements in the video, this embodiment uses a foreground mask to occlude the influencing foreground elements, retaining only the background portion of the video, and then utilizes the homography changes of the background portion to stabilize the video content.

[0071] This embodiment uses the processing of a foreground mask as part of the data preprocessing. First, a pre-trained semantic segmentation model PSPNet is used to identify people, cars, ships, airplanes, animals, trucks, motorcycles, and bicycles as candidate moving objects. Regions other than candidate objects are directly identified as background regions. Then, the regions containing these candidate objects are further filtered.

[0072] In this embodiment, for the same candidate object in adjacent frames, if its position change between the previous and next frames is less than 5 pixels, the object can also be determined as background content. This embodiment takes into account that in addition to the object's own motion, large parallax can also cause the candidate object's position to change. Therefore, it does not directly determine the candidate object with a position change greater than 5 pixels as the foreground area. Instead, it sets 50 pixels as the threshold, and only candidate objects with a position change greater than this threshold are determined as the foreground area. The determination rule is as shown in formula (1):

[0073]

[0074] Where T represents the total number of frames in the video; p represents the number of pixels in the region where the candidate object is located; M k (p) represents the pixel value of the region where the candidate object is located; M represents the average coordinates of all pixels within the i-th candidate object region in the k-th frame; i represents the candidate object number; M k This represents the foreground mask of the video frame. In mask M... k In the middle, M k (p) = 1 indicates a pixel that is identified as a background region, M k (p) = 0 indicates a pixel that is identified as a foreground region. This embodiment shows a sample mask generated using the above-mentioned indicators in Figure 2.

[0075] S2: Extract the video frame I based on a shallow network. k low-level features F k Simultaneously, the video frame I is obtained through a convolutional network. k Confidence graph A k ; the foreground mask M k The low-level feature F k and the confidence graph A k The video frame I is obtained by fusion. k credible low-level features G in the background region k .

[0076] After obtaining the foreground mask, the main target of this module is the background region of the video frame. The background region is usually a non-planar scene with small parallax, and the result obtained by directly estimating motion from adjacent frames is not accurate. Therefore, this embodiment proposes a shallow network to learn the low-level features of the video frame, instead of using the RANSAC method to find the best matching feature points (such as SIFT feature points) as in traditional algorithms. This is because the shallow network tends to present details such as texture and contour in addition to feature points, and the low-level features extracted by the shallow network are more advantageous for motion estimation. The process of low-level feature extraction is shown in formula (2):

[0077] F k =f θ (I k )1≤k≤T (2)

[0078] Where θ represents the parameters of the low-level feature learning network; f θ This represents a shallow network that extracts low-level features; F k Indicates video frame I k The low-level characteristics.

[0079] In addition, this embodiment also considers low-texture and low-light regions in the background of the video frame. Low-texture and low-light regions in the image directly affect the stable results because neural networks are prone to extracting outlier feature points from these regions and generating incorrect motion estimates, which makes it impossible for DLT (Direct Linear Transform) to obtain an accurate homography transformation matrix. To avoid outlier features in motion estimation, this embodiment proposes a five-layer convolutional network with a kernel size of 3, which learns a confidence map. The learning process is shown in formula (3):

[0080] A k =g η (I k )1≤k≤T (3)

[0081] Where η represents the parameters of the confidence graph learning network; g η The network structure representing the learning confidence graph; A k Indicates video frame I k The confidence graph. The obtained confidence graph A... k For the extracted low-level features F k The background region is weighted, and the weighting process is shown in formula (4):

[0082] G k =M k ·F k ·A k 1≤k≤T (4)

[0083] Where T represents the total number of frames in the video; M k Indicates video frame I k Foreground mask; F k Indicates video frame I k The low-level features; A k Indicates video frame I k Confidence plot; G k Indicates video frame I k Believable low-level features in the background region. Due to M k The mask is 0-1, where the dynamic foreground region is all 0, therefore confidence map A k It only applies to low-level features F of video frames.k The background part. The significance of integrating these three elements lies in the fact that the low-level feature F... k Only the portion capable of accurately estimating inter-frame motion is retained, improving the reliability of obtaining the homography matrix in the next stage. This embodiment demonstrates through ablation experiments that adding a confidence map significantly outperforms the previous method, as shown in the experimental results.

[0084] S3: Let k = k + 1, repeat S1-S2 to obtain video frame I. k+1 credible low-level features G in the background region k+1 .

[0085] S4: Construct a motion estimation neural network and perform motion estimation based on video frame I. k credible low-level features G in the background region k and video frame I k+1 credible low-level features G in the background region k+1 Obtain the homography relationship matrix H between adjacent video frames. k+1，k .

[0086] In this embodiment, a motion estimation network is used for homography estimation in this step. Image homography estimation is a process of establishing correspondence by comparing the identical parts of two frames. In deep learning, due to the narrow receptive field of convolution kernels, such correspondence cannot be captured by convolution operations alone. Therefore, this embodiment employs a local-global hybrid feature extraction method to enhance the long-distance dependence of features, thereby better capturing the spatial correspondence between two frames.

[0087] In this embodiment, G with dimensions H×W×C is used. k and G k+1 Concatenate at the Channel level to construct feature map B = [G k G k+1 The input to the motion estimation network is shown below. The proposed motion estimation network structure in this embodiment is as follows:

[0088] Feature map B first undergoes a convolutional layer to extract high-level semantic features of size H×W×C. This process is described in equation (5):

[0089] Z = AvgPool(σ(f) 3×3 (B))) (5)

[0090] Where σ is the ReLU function; f 3×3 represents a convolution operation with a kernel size of 3; AvgPool represents a max pooling operation; Z represents the extracted high-level semantic features.

[0091] Then, to capture the connections between adjacent frames, the network reshapes the features Z into a sequence. This sequence is denoted as {z... i ∈R C |i=1,...,N}, where N=H×W. Feature z i Features can be converted into different formats using different 1×1 convolution functions, as described in Equation (6):

[0092] k i =W k z i q i =W q z i v i =W v z i (6)

[0093] Among them, W k W q W v These represent three different 1×1 convolution functions; k i q i v i Representing z respectively i Features of the three different formats converted.

[0094] The method for calculating the correspondence between adjacent frames is shown in formula (7):

[0095]

[0096] Where, β i，j This represents the correlation between the i-th and j-th positions; i and j represent points in the features of the k-th and (k+1)-th frames, respectively. The correlation β between the features of adjacent frames is calculated based on this correlation. i，j The resulting high-level features are y = (y1, y2, ..., y3). i , ..., y N )∈R C×N The calculation process is shown in formula (8):

[0097]

[0098] Secondly, the high-level feature y is reshaped to obtain the reshaped feature Y, which has the format H×W×C. The reshaped feature Y is then combined with the input feature Z using residual connections as input to the subsequent network structure, as shown in formula (9).

[0099] U=γY+Z (9)

[0100] Here, γ is a learnable, scalable factor with an initial value of 0. This step gradually trains Y to have more weights by leveraging the connections between adjacent frames. The motion estimation network is obtained by combining the two structures described above, as shown in Figure 1.

[0101] Finally, a fully connected layer is used to obtain four motion displacement vectors from frame k to frame (k+1), and then the four motion displacement vectors are transformed into a homography matrix H with 8 degrees of freedom through DLT (Direct Linear Transform). k+1，k .

[0102] This step also includes: constructing a loss function and optimizing the motion estimation neural network based on the loss function. The process of constructing the training loss function in this embodiment of the invention is shown in Figure 5.

[0103] The main method used in the embodiments of this invention is to utilize the estimated homography matrix H k+1 , to video frame I k+1 Twisted to I k This is to achieve video stabilization. Intuitively, if the homography matrix H... k+1 Precise enough, I k+1 Should be with I k If they are well aligned, then the L1 loss between them should be low. However, considering that in real-world scenarios, a single homography matrix is ​​often insufficient for the transformation between two views, this embodiment of the invention also uses F′ to represent the L1 loss. k+1 and F k Normalization. Here, I′ k+1 isI k+1 The result after stabilization. Therefore, I′ k+1 and I k The losses in between are as follows:

[0104]

[0105] However, directly minimizing Eq.5 can easily lead to worthless solutions because evaluating the estimated H becomes meaningless once the two input images are identical. To ensure the two input images are distinct when evaluating H, this embodiment of the invention adds F... k+1 and F k Another loss between them is:

[0106] L(I k+1 I k )=||F k+1 -F k ||1(14)

[0107] After adding this loss, the embodiments of the present invention evaluate H k+1While maintaining accuracy, it effectively avoids worthless zero solutions.

[0108] In practice, embodiments of the present invention exchange I k+1 and I k Based on the characteristics, another homography matrix H was obtained. k The embodiments of the present invention also add a constraint, forcing H k+1 and H k This is the inverse. Therefore, the loss function used to optimize the motion estimation neural network is defined as follows:

[0109]

[0110] Where α and γ are equilibrium hyperparameters, and E is a third-order identity matrix. α = 2.0 and γ = 0.01.

[0111] S5: Based on the homography relationship matrix H between adjacent video frames k+1，k The overall motion path of the video is smoothed to achieve video stabilization.

[0112] This embodiment uses the obtained adjacent frame homography relation matrix H k+1，k For the I k+1 The frame is warped to obtain I′ k+1 This achieves stable operation of the content of the (k+1)th video frame relative to the kth frame. During the stabilization of the video frames, to ensure spatial alignment of identical scenes in adjacent frames, an STN network is used to warp the image, as shown in equation (10):

[0113] I' k+1 =I k+1 *H k+1，k (10)

[0114] Thus, this embodiment achieves content stabilization between adjacent frames. However, a video is composed of several frames. When considering stabilizing the entire video, this embodiment of the invention not only needs to consider the continuity between adjacent frames, but also the continuity between non-adjacent frames. That is, the path of the entire video content needs to be smoothed, as shown in Figure 3.

[0115] To achieve this, this embodiment, in addition to aligning adjacent frames, also requires aligning subsequent video frames with their preceding video frames. That is, when this embodiment uses I... k+1 Multiply by the transformation matrix H k+1，k When stabilizing the (k+1)th frame, H k+1，kThe content should include not only the transformation relationship from k+1 to k (i.e., the homography relationship matrix), but also the transformation relationship from the (k+1)th frame to the first frame. This is to ensure that the camera path in the (k+1)th frame is not too different from that in the previous frames. Therefore, if the video composed of T frames is to be stabilized, it should satisfy formula (11):

[0116]

[0117] Where T is the total number of frames; P k Defined as the camera path after stabilization from frame 1 to frame k, or the cumulative transformation relationship from frame 1 to frame (k-1); H t，t-1 It is defined as the transformation relationship from frame t to frame (t-1).

[0118] To align subsequent video frames with their preceding frames, after the stabilization of the k-th frame is completed in this embodiment of the invention, video frame I needs to be... k The content has been updated to I′ k Then I′ k As the stable target for the next round of subsequent iterations, this embodiment of the invention can transform formula (10) into formula (12):

[0119] I k+1 =I k+1 *P k *H k+1，k 1≤k≤T (12)

[0120] Where T represents the total number of video frames; P k This represents the camera's motion path from the first frame to the kth frame.

[0121] The embodiments of the present invention can be viewed as correcting video frames from their deviated trajectories to the smooth trajectories they should theoretically follow, as shown in Figure 4.

[0122] This invention continuously stabilizes the background region of the video frame because the motion trajectory of the background region in the video can be used to represent the camera's motion trajectory. After the background region is stabilized, the camera's motion trajectory will also be smooth. Therefore, the motion trajectory of moving foreground objects in the video will also be smoother than before.

[0123] To evaluate the method of the present invention, the embodiments of the present invention also conducted extensive comparisons with representative stable methods of the past decade in terms of imaging effect, quantitative analysis and running time through experiments.

[0124] 1. Imaging Comparison

[0125] Since the continuity of the video cannot be directly compared here, this embodiment of the invention selects some stabilized scene frames for comparison. This allows this embodiment of the invention to intuitively compare the imaging quality of different methods from the stabilized results. Figure 6 shows the stabilized results obtained by representative methods over the past decade, including two traditional methods, L1-stabilizer and Bundle, the optical flow method Mesh-Flow, and several deep learning methods, StabNet, Yu and Ramamoorthi, and Hybrid.

[0126] In addition to comparing some complex scenes, this embodiment of the invention also selected videos from the Selfie dataset for imaging comparison of dynamic foreground regions. Here, this embodiment primarily selected two video stabilization methods relying on optical flow (YuandRamamoorthi and Hybrid) to demonstrate the superiority of the homography method used in this embodiment for imaging. The imaging results are shown in Figure 7.

[0127] Based on the comparison results of Figures 6 and 7, it can be seen that the embodiment of the present invention is superior to other methods in terms of distortion, and there are no obvious distortions or ghosting phenomena.

[0128] 2. Quantitative analysis

[0129] This invention uses metrics proposed by Liu et al. and representative methods from the past decade to evaluate the NUS, Deepstab, and Selfie datasets. Better stability results result in the three metrics proposed by Liu et al.—clipping degree, distortion degree, and stability—approaching 1.

[0130] The bold text in Table 1 indicates the best method for this metric, and * indicates an unsupervised deep learning method.

[0131] Table 1

[0132]

[0133] Table 1 shows the performance of the proposed method and other methods on the NUS dataset, Selfie dataset, and Deepstab dataset.

[0134] On the NUS dataset, the Hybrid full-frame method achieved the highest cropping rate (C). The cropping degree of this embodiment is the highest among methods other than the full-frame method. The method of this embodiment ranks first in distortion (D), outperforming other methods. In terms of stability (S), the two traditional methods, L1-stabilizer and Bundle, are the highest, while this embodiment is slightly inferior.

[0135] The videos in the Selfie dataset all contain large moving foregrounds, which aligns with the design expectations of this invention. Because this invention uses foreground masks to process the foreground, its overall performance on this dataset is good.

[0136] For the Deepstab dataset, this embodiment still achieved the highest level in terms of distortion.

[0137] Since this invention is a deep learning method based on homography, it is significantly faster than optimization-based and optical flow-based methods. The last column of Table 1 summarizes the frame-by-frame execution time of multiple methods on an RTX 30910 GPU. On average, this invention requires only 76ms per frame for stabilization. This stabilization speed is faster than both traditional methods and optimization-based deep learning methods. Compared to Deep3D methods, this invention achieves faster speeds by avoiding large data overhead while maintaining similar evaluation metrics.

[0138] This invention, while distorting video frames, strives to maintain consistency with previous video frame content. This prevents severe distortion of the video frame content and maintains a high level of distortion. Because this invention uses a mask to occlude moving foregrounds, it achieves better results with videos containing highly moving foregrounds. Since this method uses a confidence map to filter background features, the resulting homography matrix requires only a small distortion to achieve a stable effect, further ensuring the integrity of the video frame content.

[0139] 3. Ablation test

[0140] This embodiment uses the Selfie dataset for ablation experiments. Each video in the Selfie dataset contains a clear moving foreground, which aligns with the issues considered when designing this embodiment. The experiment performed four comparisons, and the results are shown in Table 2. The visual results are shown in Figure 8.

[0141] As shown in Figure 8, distortion occurs due to the dynamic foreground region without using a mask. Without a confidence map, the estimated homography matrix cannot effectively process video frames. However, using both methods together yields better results. This embodiment of the invention also selects the facial region of the dynamic foreground as a comparison object. The comparison in Figure 8 shows that significant distortion occurs in the facial region without a mask.

[0142] Table 2

[0143]

[0144] Table 2 shows that adding the confidence map significantly improves video stability. This is because the confidence map filters out unnecessary features, improving the accuracy of obtaining the homography matrix. It also shows that adding the foreground mask significantly improves distortion. This is because the foreground mask prevents the moving foreground from participating in the homography estimation process, maximizing the consistency between the stabilized foreground and the original content. Furthermore, it can be seen that when both the mask and the confidence map are added, all three metrics improve to some extent, fully demonstrating the rationality of the stabilization framework design.

[0145] This invention also discloses a video stabilization system based on video frame homography transformation, used to implement the above-mentioned video stabilization method based on video frame homography transformation, including:

[0146] The foreground mask module is used to divide the background and foreground regions of the video, cover the foreground region, and generate the foreground mask for the video frame.

[0147] The reliable shallow feature extraction module is used to obtain reliable low-level features of the background region of video frames, specifically including:

[0148] Low-level feature extraction unit, used to extract low-level features of video frames based on shallow networks;

[0149] The confidence map acquisition unit is used to obtain confidence maps of video frames through a convolutional network;

[0150] The fusion unit is used to fuse the foreground mask, low-level features, and confidence map of a video frame to obtain the reliable low-level features of the background region of the video frame.

[0151] The motion estimation module is used to construct a motion estimation network and obtain the homography relationship matrix between adjacent video frames based on the reliable low-level features of the background regions of adjacent video frames.

[0152] The path smoothing module is used to smooth the overall motion path of the video based on the homography relationship matrix between adjacent video frames, thereby achieving video stabilization.

[0153] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0154] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video stabilization method based on inter-frame homography transformation, characterized in that, Includes the following steps: S1: Divide the video into background and foreground regions, cover the foreground region, and generate video frame I. k Foreground mask M k The process includes: S11: Differentiating different objects in the video based on a semantic segmentation model to determine candidate objects; regions other than candidate objects are classified as background regions; the candidate objects are movable objects; S12: Filtering based on the position information of the candidate objects; for the same candidate object, if its position change between the previous and next frames is less than a set lower threshold, the region where the candidate object is located is classified as a background region; if its position change is greater than a set upper threshold, it is classified as a foreground region; S13: Pixels classified as background regions are represented by 1, and regions where candidate objects are classified as foreground are represented by 0, covering the foreground regions to finally generate a foreground mask. S2: Extract the video frame I based on a shallow network. k low-level features F k Simultaneously, the video frame I is obtained through a convolutional network. k Confidence graph A k ; the foreground mask M k The low-level feature F k and the confidence graph A k The video frame I is obtained by fusion. k credible low-level features G in the background region k The video frame I k credible low-level features of the background region The calculation formula is: In the formula, T represents the total number of video frames; S3: Let k = k + 1, repeat S1-S2 to obtain video frame I. k+1 credible low-level features G in the background region k+1 S4: Construct a motion estimation neural network and, based on the video frame I... k credible low-level features G in the background region k and video frame I k+1 credible low-level features G in the background region k+1 Obtain the homography relationship matrix H between adjacent video frames. k+1,k S5: Based on the homography relationship matrix H between adjacent video frames k+1,k The overall motion path of the video is smoothed to achieve video stabilization.

2. The video stabilization method based on inter-frame homography transformation according to claim 1, characterized in that, S4 includes: S41: Constructing feature map B=[ ， The feature map B undergoes a convolutional layer to extract high-level semantic features Z and reliable low-level features G for the background region. k and G k+1 The size is H×W×C; S42: Reshape the high-level semantic feature Z into a sequence { |i=1,...,N},N=H×W;S43:Features The features are converted into different formats using different 1×1 convolution functions. The correlation between features in adjacent frames is calculated based on the different formats of features. The calculation method is as follows: ； ;in, 、 、 These represent three different 1×1 convolution functions; 、 、 Represent Features of the three different formats converted; This represents the correlation between the i-th position and the j-th position; i and j represent points in the features of the k-th frame and the (k+1)-th frame, respectively; S44: Based on the obtained correlation between features of adjacent frames... , to obtain high-level features y= ( , ,..., ,..., ） The calculation process is as follows: S45: Reshape the high-level feature y to obtain the reshaped feature Y, which has the format H×W×C; S46: Combine the reshaped feature Y with the high-level semantic feature Z using residual connections as input to the subsequent fully connected layer to obtain four motion displacement vectors from frame k to frame (k+1). Then, use DLT to convert the four motion displacement vectors into a homography relation matrix between adjacent video frames. 。 3. The video stabilization method based on inter-frame homography transformation according to claim 2, characterized in that, The formula for extracting the high-level semantic feature Z is as follows: Where σ is the ReLU function; This indicates a convolution operation with a kernel size of 3; This is a max pooling operation.

4. A video stabilization method based on inter-frame homography transformation according to claim 2, characterized in that, S4 further includes: constructing a loss function, and optimizing the motion estimation neural network based on the loss function; the loss function is: L= Where α and γ are equilibrium hyperparameters, and E is a third-order identity matrix. ； 。 5. A video stabilization method based on inter-frame homography transformation according to claim 1, characterized in that, S5 includes: stabilizing the overall video based on the following formula: ;in, Represents the stabilized video frame The motion path of the camera from the first frame to the kth frame is represented by the following method: In the formula, This represents the homography relation matrix from frame t to frame (t-1).

6. A video stabilization system based on video frame homography transformation, used to implement the video stabilization method based on video frame homography transformation as described in any one of claims 1-5, characterized in that, include: The foreground mask module is used to divide the background and foreground regions of the video, cover the foreground region, and generate a foreground mask for the video frame; A reliable shallow feature extraction module is used to obtain reliable low-level features of the background region of the video frame; The motion estimation module is used to construct a motion estimation network and obtain the homography relation matrix between adjacent video frames based on the reliable low-level features of the background region of adjacent video frames; the path smoothing module is used to smooth the overall motion path of the video based on the homography relation matrix between adjacent video frames to achieve video stabilization.

7. A video stabilization system based on inter-frame homography transformation according to claim 6, characterized in that, The trusted shallow feature extraction module includes: a low-level feature extraction unit, used to extract low-level features of video frames based on shallow networks; a confidence map acquisition unit, used to obtain a confidence map of the video frames through a convolutional network; and a fusion unit, used to fuse the foreground mask, the low-level features, and the confidence map of the video frames to obtain trusted low-level features of the background region of the video frames.

Images