A real-time full-frame video stabilization method, system and device

Abstract

A real-time full-frame video stabilization method, system and device, the method of which, for real-time input unstable video images, first extracts the key points of the current input image through RFnet, inputs the current image and the previous frame image into the RAFT network model to obtain the optical flow map of the previous frame image; combines the optical flow map and the key points to generate the grid vertex motion of the frame, places the grid vertex motion into a sliding window for motion smoothing; and through the Realtime Smoother network, finally obtains the difference between the grid motion trajectories before and after smoothing, obtains the stable image and the corresponding filling mask through homography transformation, processes the stable image and the filling mask through E2FGVI to obtain the filled full-frame image, and outputs the full-frame image; the system and device based on the real-time full-frame video stabilization method stabilize the output of the full-frame video; the present application has the advantages of good real-time performance and high stability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a real-time full-frame video stabilization method, system, and device. Background Technology

[0002] Currently, a plethora of shooting devices, such as digital cameras, mobile phones, and various mobile smart devices, have emerged, enabling users to shoot in various scenarios at low cost. However, these handheld devices, lacking professional equipment support, are susceptible to various adverse factors, often resulting in shaky videos. This shakiness typically leads to blurring or distortion, significantly reducing video quality, and the instability is easily noticeable to the average person. Physical and optical stabilization methods require the shooter to prepare appropriate equipment or special hardware before shooting, which is costly and difficult to obtain. Furthermore, this method cannot stabilize already captured videos. Therefore, research on video stabilization algorithms has significant practical implications and broad application prospects. Most current video stabilization methods rely on global temporal motion information to calculate the warping of the current frame. These algorithms often require global video information to infer the camera's motion trend and then use image distortion to achieve stabilization.

[0003] Existing video stabilization methods often require global video information to infer camera motion trends, or cannot achieve real-time video stabilization. Furthermore, most video stabilization methods achieve stabilization by applying homography transformation to the image, which results in missing edges in the output video image, preventing the output of full-frame images.

[0004] Patent application CN108564554A discloses a video stabilization method based on motion trajectory optimization. First, it performs global camera motion trajectory stabilization on the input video by estimating the original camera motion trajectory using feature point matching and smoothing the trajectory by optimizing its energy function. Then, it performs local stabilization adjustments on the stabilized video by dividing the video image into grids, estimating the camera motion within each grid, and smoothing the motion. Finally, it performs global motion re-stabilization adjustments to further improve video stability. However, this application struggles with real-time image processing, and the image generated based on homography transformation is distorted from the original image, resulting in sharp edges and preventing output at the original image resolution. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, the present invention aims to provide a real-time full-frame video stabilization method, system, and device. Addressing the common problems of poor real-time performance and inability to output full frames in video stabilization methods, this invention achieves real-time full-frame video stabilization through grid motion estimation, real-time grid motion smoothing, and video inpainting algorithms. Furthermore, it utilizes a video filling method for post-processing, achieving full-frame image output through edge filling. This method offers the advantages of high image output stability and good real-time performance.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A real-time full-frame video stabilization method specifically includes the following steps:

[0008] Step 1: Collect continuous video with unstable factors such as jitter, divide it into image sequences, and preprocess the image sequences;

[0009] Step 2: Using the preprocessed image sequence as the input image, extract the key points of the input image sequence using the RFNet key point extraction algorithm;

[0010] Step 3: Extract the optical flow map of the input image sequence using the RAFT optical flow extraction algorithm;

[0011] Step 4: Based on the key points obtained in Step 2 and the optical flow map of the input image sequence extracted in Step 3, the image is meshed, and the mesh motion trajectory is generated by training the MotionPro network.

[0012] Step 5: Based on the grid motion trajectory obtained in Step 4, set a sliding window with a size of 2k+1 frames. Smooth the grid motion trajectory of the first 2k+1 frames. At the same time, the smoothing results of the first k sliding windows are used as the smoothed trajectory. Using the grid motion trajectory and the smoothed trajectory as input, the stable grid motion trajectory of the k+1th frame in the sliding window is generated through the Realtime Smoother network. After completing this process, the sliding window moves one frame to the right, discards the content of the first frame in the sliding window, reads the content of the new frame to the end of the window, obtains the smoothed motion trajectory corresponding to the new window, and updates the smoothed trajectory result. Repeat the above process until the operation is completed for the entire input image sequence.

[0013] Step 6: Based on the difference in motion trajectories before and after smoothing in Step 5, calculate the homography transformation matrix required to output a stable image using grids as deformation units. Perform homography transformation on the input image according to the homography transformation matrix of each grid, and output a stable image sequence, which is a stable video image. At the same time, perform the same homography transformation on a blank image to generate a corresponding mask as input for filling.

[0014] Step 7: Based on the stabilized image and mask obtained in Step 6, fill the missing edge parts using the E2FGVI video fill algorithm.

[0015] The image preprocessing in step 1 is as follows:

[0016] The input image is scaled to compress the input resolution to a uniform H*W.

[0017] The specific method in step 2 is as follows:

[0018] Step 2.1: Input the image into the RFD detector to obtain the score map S, the orientation map θ, and the scale map.

[0019] Step 2.2, analyze the score map S, orientation pattern θ, and scale map obtained in Step 2.1 respectively. Perform feature extraction;

[0020] Step 2.3: Input the features obtained in step 2.2 into the descriptor network, and extract fixed-length feature vectors for matching to obtain image key points.

[0021] The specific process of step 3 includes:

[0022] Step 3.1: Input the image into the Feature / Context network module for training to extract the texture content and local pixel features of the image;

[0023] Step 3.2: Input the texture content and local pixel features extracted in Step 2.1 into the association layer. Construct a 4-dimensional W×H×W×H association body by taking the inner product of all paired feature vectors. Then, perform pooling downsampling on the 4-dimensional association to generate a dimension of... The multidimensional correlation weights (where t is the number of pooling operations);

[0024] Step 3.3: Initialize the optical flow to be generated to 0. Based on the GRU update operator, use the multidimensional correlation weights obtained from the correlation layer in step 3.2 as the weight parameters of the GRU operator. Input the optical flow as input and iteratively pass it into the GRU operator. The output result is the updated optical flow. By iteratively updating the optical flow through this method, the final result is obtained.

[0025] The specific steps of step 4 are as follows:

[0026] Step 4.1: Combine the key points obtained in Step 2 with the optical flow map obtained in Step 3, and extract the optical flow of the corresponding points in the optical flow map based on the key point coordinates to obtain the key point coordinates and the optical flow of the corresponding key points.

[0027] Step 4.2: Input the key point positions and optical flow information obtained in Step 4.2 into the MotionPro network to obtain the mesh vertex motion;

[0028] 4.2.a. Mesh the image, calculate the Euclidean distance between each keypoint and each grid vertex to obtain the distance information between the keypoint and the grid vertex, and pass the distance information between the keypoint and the grid into the DistanceEncoder submodule to output the distance features and grid vertex weights.

[0029] 4.2.b, input the image key point locations and optical flow information into the Motion Encoder submodule, and output the optical flow features;

[0030] 4.2.c, Combine the distance features obtained in 4.2.a with the optical flow features in 4.2.b, and input them into the DM (Distance-Motion) submodule to obtain the mesh vertex motion features;

[0031] 4.2.d, multiply the mesh vertex motion features by the mesh vertex weights to obtain the mesh vertex motion;

[0032] Step 4.3: For the mesh vertex motion obtained in Step 4.2, calculate the loss value based on three aspects: the difference between the motion of each vertex and the motion of nearby keypoints, the consistency of the overall motion of the four vertices in a single mesh, and the consistency of the mesh shape after motion. Backpropagate to update the MotionPro network until the loss converges, so that the network can generate a correct representation of the motion direction and velocity of the mesh pixels in the frame, and obtain the mesh motion trajectory through multiple iterations; its loss function is shown in the following equation (1):

[0033] L MR =L vm +λ s Lsp, (1)

[0034] Among them, L sp The calculation formula for shape protection loss is shown in equation (2) below:

[0035]

[0036] L vm The loss includes the motion difference between each vertex and its neighboring key points, and the consistency of the overall motion of the four vertices in a single mesh, as shown in the following formula (3):

[0037]

[0038] The specific steps of step 5 are as follows:

[0039] Step 5.1: Based on the grid motion trajectory obtained in Step 4, set the sliding window to a size of 2k+1 frames, and smooth the grid motion trajectory of the first 2k+1 frames.

[0040] Step 5.2: Input the grid motion trajectory in the sliding window and the smoothed grid motion trajectory of the previous k frames into the Realtime Smoother network for the first inference, and output the stable grid motion trajectory of the (k+1)th frame in the sliding window.

[0041] Step 5.2.a: Input the grid motion trajectory of the first 2k+1 frames read in step 5.1 into the Origin_embedding submodule and input the smoothed grid motion trajectory of k frames into the Smooth_embedding submodule. Expand their coordinate dimensions through a fully connected layer. The results are the original trajectory encoding feature Ori and the smoothed trajectory encoding feature Smooth, respectively.

[0042] Step 5.2.b: Pass Ori and Smooth into a convolutional module. Each convolutional module has n convolutional units. Each convolutional unit consists of a 3*3 three-dimensional convolutional layer and a ReLU layer. The outputs are the original trajectory convolutional feature Ori_conv and the smooth trajectory convolutional feature Smooth_conv, respectively.

[0043] Step 5.2.c: Pass Ori_conv and Smooth_conv to the original trajectory decoding module Ori_decoder and the smooth trajectory decoding module Smoother_decoder respectively, decode the content, and output the new motion trajectory s after stabilization in the (k+1)th frame within the sliding window;

[0044] Step 5.3: Add the next frame of video content to the sliding window, remove the first frame of video content of the current sliding window, add the smoothing result s obtained in step 5.2 to the smoothing result of the previous k sliding windows as the smoothed trajectory result, and use the new sliding window and smoothed trajectory as input. Repeat step 5.2 until the entire video is processed.

[0045] Step 5.4: After all video processing is complete, the motion trajectory of the entire video after smoothing by the Realtime Smoother network is obtained;

[0046] By using the loss function, Equation (4), the adjacent frames of the output trajectory are made to have temporal consistency;

[0047]

[0048] Where Ω i A local region was defined. Indicates v ik A smooth trajectory at the location;

[0049] By using the loss function as shown in Equation (5), the smoothness of the trajectory is subject to certain restrictions, so that its motion trajectory conforms to the global motion trend of the original trajectory:

[0050]

[0051] By adding content-protected loss L to the loss function cp To avoid partial image distortion that affects video quality, the loss function is shown in formula (6):

[0052]

[0053] A shape preservation loss is added to avoid image discontinuities between image grids or severe grid shape distortion. The loss is shown in the following formula (7):

[0054]

[0055] in, G represents im At the four vertices in a clockwise direction, the motion vectors corresponding to the four vertices are respectively Rot90 means rotating the motion vector by 90°;

[0056] Motion trend consistency loss: The first and last frames of the sliding window indicate the trend of grid motion changes within the sliding window, but direct use will lead to new jitter. By calculating the difference between the motion and ave of each frame, assigning weights and summing them, the target tar get of the smoothing function is obtained, avoiding the direct use of the average ave of the first and last frames which will lead to new jitter. Its loss formula is shown in equations (8) to (11) below:

[0057] ave=(f0+f 14 ) / 2 (8)

[0058]

[0059]

[0060]

[0061] Among them, f i ω represents the motion trajectory of the i-th frame within the sliding window, where ω is a constant.

[0062] The loss value is calculated based on the loss function described above, and the Realtime Smoother network is updated through backpropagation until the loss result converges and can correctly generate a smoother and more continuous grid motion trajectory.

[0063] The specific steps of step 6 are as follows:

[0064] Step 6.1: Based on the Realtime Smoother smoothed network motion trajectory obtained after training in Step 5, subtract the motion trajectory before smoothing from the motion trajectory after smoothing to obtain the difference between the mesh motion trajectory before and after smoothing.

[0065] Step 6.2: Calculate the homography transformation matrix of each grid based on the grid motion trajectory difference. Perform homography transformation on the image in units of grids according to the homography transformation matrix of each grid to obtain a stable image sequence. At the same time, perform the same homography transformation on a blank image to generate a corresponding mask as input for the filling process.

[0066] The specific steps of step 7 are as follows:

[0067] Step 7.1: Input the stable image sequence and mask obtained in step 6 into the Flow_completion module to complete the optical flow of the missing parts, and obtain the filled optical flow Flow.

[0068] Step 7.2: Input the stabilized image sequence output from Step 6 into the local neighbor feature extraction module and the non-local reference module in the E2FGVI video filling algorithm respectively; for the missing part of each frame of video content in the image sequence, extract the neighboring frame feature C0 and the non-neighboring frame feature C1 of the current processing frame respectively for filling.

[0069] Step 7.3: Input the adjacent frame features C0 and the filling optical flow Flow into the Feature Propagation module, i.e. the feature propagation module, to propagate the adjacent frame features to the missing region through the filling optical flow, and obtain feature C2;

[0070] Step 7.4: Take C1 and C2 as inputs and pass them to the content hallucination module. Combine the non-adjacent frame features C1 for the parts that cannot be filled by local features, and output the final filled image.

[0071] A system based on a real-time full-frame video stabilization method includes:

[0072] The data processing module is used to divide the input real-time video into image sequences and preprocess the input images, corresponding to the preprocessing operation in step 1.

[0073] The key point extraction module extracts key points from the current image, corresponding to the key point extraction operation in step 2.

[0074] The optical flow extraction module is used for the optical flow extraction operation in step 3; it uses the RAFT optical flow extraction algorithm to extract the optical flow map of the input image sequence from the current image and the input image.

[0075] The mesh motion estimation module is used for the mesh vertex trajectory establishment operation in step 4. It uses the key point information extracted in step 2 and the optical flow map extracted in step 3 to generate mesh motion trajectories by training the MotionPro network. It includes a Distance Encoder submodule, which outputs distance features and mesh vertex weights; a Motion Encoder submodule, which outputs optical flow features; and a DM (Distance-Motion) submodule, which is used to obtain mesh vertex motion features.

[0076] The real-time motion trajectory smoothing module smooths and stabilizes the existing grid motion trajectory in real time, outputting a smoother and more continuous motion trajectory, corresponding to the grid motion trajectory smoothing operation in step 5.

[0077] The video rendering and filling module, based on the smoothed grid motion trajectory obtained in step 5, obtains a stable video image through homography transformation in step 6, and fills the missing parts in the homography-transformed image through step 7, thereby achieving real-time full-frame stable video output.

[0078] A device based on a real-time full-frame video stabilization method includes:

[0079] Video image capture device, used to capture video images in real time;

[0080] A program processor is used to store computer programs and execute the computer programs to implement the real-time full-frame video stabilization method described in steps 1 to 7.

[0081] A display used to show the results of video stabilization.

[0082] Compared with existing technologies, the present invention exhibits significant advantages in the following aspects:

[0083] Traditional video stabilization methods do not support real-time processing and require processing as a whole output after the entire video is captured. This invention takes into account the need for stable video output in real-time transmission of videos such as live broadcasts and surveillance. By smoothing the motion trajectory of the grid in real time, it achieves real-time requirements in terms of both video input format and inference speed, and its video stabilization quality is comparable to existing stabilization algorithms.

[0084] By utilizing keypoints and optical flow maps of an image, a CNN network is used to propagate the motion of keypoints to grid vertices, generating grid vertex trajectories. The motion trajectories of grid vertices are read in real-time through a sliding window. The read sliding window data is then smoothed using a Realtime Smoother network. Finally, based on the difference between the smoothed and unsmoothed trajectories, a grid homography transformation is used to achieve stable video output. Through the structural and loss design of the Realtime Smoother network, video stabilization is achieved in an unsupervised environment, i.e., without the need for paired stable-unstable video datasets, and under the condition of a limited sliding window field of view.

[0085] Compared to traditional methods that stabilize video by cropping the image size, this invention uses the E2FGVI video filling algorithm to fill in the edge parts, achieving real-time stable video output across the entire frame. It solves the video cropping problem that exists in stabilizing video images using mesh transformation methods. This allows methods that stabilize images using rendering techniques such as homography transformation, which result in missing image information, to fill in the missing parts after rendering, enabling the image to be output with the same resolution and content as the original video image. Attached Figure Description

[0086] Figure 1 This is a structural diagram of the real-time full-frame video stabilization method of the present invention.

[0087] Figure 2 This is a diagram of the MotionPro network structure of the present invention.

[0088] Figure 3 This is a diagram of the Realtime Smoother network structure of the present invention.

[0089] Figure 4 This is a flowchart of the MotionPro network training process of the present invention.

[0090] Figure 5 This is a flowchart of the Realtime Smoother network training process of the present invention.

[0091] Figure 6 This is a flowchart of the testing process for this invention.

[0092] Figure 7 This is a schematic diagram of the input and partial output images of the present invention, wherein, Figure 7 (a) is the image sequence that needs to be stable. Figure 7 (b) is the optical flow map corresponding to this sequence. Figure 7 (c) is the stable video image obtained through homography transformation. Figure 7 (d) is the video image after filling. Detailed Implementation

[0093] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0094] Example:

[0095] This embodiment provides a specific training process, including the following steps:

[0096] See Figure 1 Step 1: Collect continuous video with stable factors such as jitter, divide it into image sequences, compress the input images, and obtain images with a size of 640*360.

[0097] Step 2: Using the preprocessed image sequence as input, extract the key points of the input image using a keypoint extraction algorithm (RFNet algorithm). The specific steps are as follows:

[0098] Step 2.1: Input the image into the RF-Det detector to obtain the score map S, the orientation map θ, and the scale map.

[0099] Step 2.2, analyze the score map S, orientation map θ, and scale map obtained in Step 2.1 respectively. Extract their respective features;

[0100] Step 2.3: Input the features obtained in Step 2.2 into the descriptor network and extract fixed-length feature vectors for matching to obtain image key points;

[0101] Step 3: Extract the optical flow map of the input image sequence using the optical flow extraction algorithm proposed in the article "RAFT: Recurrent All-Pairs Field Transforms for Optical Flow" (RAFT algorithm for short). The specific steps are as follows:

[0102] Step 3.1: Input the image into the Feature / Context network module to extract the image content and features;

[0103] Step 3.2: Input the result obtained in Step 2.1 into the association layer, and construct a 4D W×H×W×H association body by taking the inner product of all paired feature vectors; then perform pooling downsampling on the 4-dimensional association to generate a dimension of... The multidimensional correlation weights (where t is the number of pooling operations);

[0104] Step 3.3: Initialize the optical flow to be generated to 0. Based on the GRU update operator, use the multidimensional correlation weights obtained from the correlation layer in step 3.2 as the weight parameters of the GRU operator. Input the optical flow as input and iteratively pass it into the GRU operator. The output result is the updated optical flow. By iteratively updating the optical flow through this method, the final optical flow result is obtained.

[0105] Step 4: During the training phase, based on the optical flow map and key points of the input image sequence, the image is meshed to generate mesh motion trajectories. The MotionPro network training process is as follows: Figure 4 As shown, the specific steps are as follows:

[0106] Step 4.1: Combine the key points obtained in Steps 2 and 3 with the optical flow map to obtain the key point coordinates and the corresponding optical flow K of the key points. MD ∈R G×4×N Where G is the number of grid vertices, N is the number of keypoints, and of the four dimensions in the middle, the first two represent the optical flow of the keypoints, and the last two represent the distance between the keypoints and the grid vertices.

[0107] Step 4.2: Combine the key point locations obtained in Step 4.2 with the optical flow information K. MD Input the data into the MotionPro network to obtain the motion of the mesh vertices.

[0108] The specific process of step 4.2 is as follows:

[0109] 4.2.a. The image is meshed, the distance between keypoints and mesh vertices is calculated, and the keypoint and distance information K is stored. MD The code is passed to the Distance Encoder submodule, which first passes K through the embedding layer. MD The second dimension increases from 4 to 64, and then two 1*1 convolutional layers are used to obtain the distance feature E between grid vertices and keypoints. d ∈R G×64×N E d After passing through another 1x1 convolutional layer, we get W. d ∈R G×1×N W d This represents the distance weight of the key point to each grid vertex;

[0110] 4.2.b, link the image key point locations with optical flow information K MD The input is passed to the Motion Encoder submodule, which outputs optical flow features. This module first passes the K-axis through the embedding_motion layer. MD The second dimension increases from 4 to 64, and then two 1*1 convolutional layers are used to obtain the distance feature E between grid vertices and keypoints.m ∈R G×64×N ;

[0111] 4.2.c combines the distance features obtained in 4.2.a with the optical flow features obtained in 4.2.b, and feeds them into the DM (Distance-Motion) module. This module consists of two 1*1 convolutional layers to obtain E. dm ∈R G×1×N This represents the motion relationship between mesh vertices and their corresponding keypoints;

[0112] 4.2.d, multiply the mesh vertex motion features by the mesh vertex weights to obtain the mesh vertex motion Ori. m ∈R G×2 .

[0113] Step 4.3, for the mesh vertex motion Ori obtained in step 4.2 m The loss value is calculated based on three aspects: the difference between the motion of each vertex and the motion of nearby key points, the consistency of the overall motion of the four vertices in a single mesh, and the consistency of the mesh shape after motion. The MotionPro network is updated by backpropagation until the loss converges so that the network can correctly generate the mesh vertex motion. The loss design of each module is shown in the following formula (1):

[0114]

[0115] In formula (1), B represents the number of images, M represents the number of vertices in a single row of the grid, N represents the number of vertices in a single column of the grid, and n ik Ori m The motion estimation vector of the k-th keypoint in the i-th frame, Ω ik Let m represent the grid vertex in the i-th image, centered at the k-th grid vertex and with a radius of a specific value r. ij This indicates the movement of key points within the circle. L n The motion of the mesh vertices should be similar to the motion of the adjacent key points.

[0116]

[0117] In formula (2), p ij H represents the coordinates of the j-th keypoint in the i-th frame. ij (p ij ) represents p ij The grid belongs to Ori m The obtained mesh vertex motion is then subjected to homography transformation. L vm This means that the result of the homography transformation of the mesh should be similar to the result of the motion of the key points within the mesh.

[0118]

[0119] In formula (3), This represents the four vertices of the m-th grid in the i-th frame in a clockwise direction, and the motion vectors corresponding to the four vertices are respectively... Rot90 indicates that the motion vector is rotated by 90°. This loss is used to protect the mesh shape and prevent severe deformation of the mesh. See the MotionPro network structure diagram. Figure 2 .

[0120] Step 5: During the training phase, a sliding window of length 15k is set. Based on the sliding window, the grid motion trajectory within the corresponding range is read, and the smoothed motion trajectory of the grid corresponding to the sliding window is generated. After completing this process, the sliding window moves one frame to the right and continues processing until the entire image sequence has been processed. This sliding window enables the real-time inference process. The Realtime Smoother network training process is as follows: Figure 5 As shown, the specific process is as follows:

[0121] Step 5.1: Based on the grid motion trajectory obtained in Step 4, set the sliding window to a size of 2k+1 frames, and smooth the grid motion trajectory of the first 2k+1 frames.

[0122] Step 5.2: Input the grid motion trajectory in the sliding window and the smoothed grid motion trajectory of the previous k frames into the Realtime Smoother network for the first inference, and output the stable grid motion trajectory of the (k+1)th frame in the sliding window.

[0123] Step 5.2.a, extract the original trajectory Ori from 15k frames. wm and k-frame smoothed trajectory wm Input the Origin_embedding submodule and the Smooth_embedding submodule respectively, and expand their coordinate dimensions. The results are Ori em ∈R B×64×H×W×T Smooth em ∈R B×64×H×W×S , where B represents the batch size, H is the number of vertices in a single row of the grid, W is the number of vertices in a single column of the grid, T represents the original trajectory window size of 15k, and S represents the smooth trajectory window size of 7;

[0124] Step 5.2.b, respectively Ori em With Smooth em The input is a convolutional module, each containing three convolutional units. Each convolutional unit consists of a 3x3 three-dimensional convolutional layer and a ReLU layer. The outputs are Ori... conv With Smoothconv ;

[0125] Step 5.2.c, respectively Ori conv With Smooth conv The code is passed to the Ori_decoder and Smoother_decoder modules respectively to decode the content and output the stable motion trajectory of the 8th frame within the sliding window: Smooth8∈R. B×2×H×W×1 .

[0126] Step 5.3: Move the sliding window one frame backward, remove the first frame of the smoothed trajectory, add the smoothing result Smooth8 obtained in Step 5.2 to the smoothed trajectory, and set the new sliding window Ori... wm 'Smooth trajectory' wn As input, repeat step 5.2 until the entire video has been processed.

[0127] Step 5.4: After all video processing is complete, the motion trajectory of the entire video smoothed by the Realtime Smoother network will be obtained. Based on loss functions such as temporal consistency, motion trend estimation, mesh transformation consistency, and shape preservation, the loss value is calculated, and the Realtime Smoother network is updated via backpropagation until the loss converges, enabling the network to correctly smooth the motion trajectory in real time. The loss design is as follows:

[0128] Considering temporal consistency, adjacent frames of the output trajectory should be similar. Small trajectory changes indicate good trajectory smoothness. The loss function is shown in formula (4) below:

[0129]

[0130] Among them, Ω i A local region was defined. Indicates v ik A smooth trajectory at the location;

[0131] Considering only trajectory smoothing would cause the video to ignore all motion elements, including reasonable and overall motion. Therefore, we need to impose certain restrictions on trajectory smoothing to ensure that the motion trajectory conforms to the global motion trend of the original trajectory. The loss function is shown in the following formula (5):

[0132]

[0133] Image smoothing may cause distortion in parts of the image, which will severely affect video quality. Therefore, a content-preserving loss L should be added to the loss function. cp The loss function is shown in equation (6):

[0134]

[0135] Since this algorithm is based on grid-based image transformation, a shape preservation loss is added to avoid image discontinuities between grids or severe grid shape distortion. The loss is shown in the following formula (7):

[0136]

[0137] in, G represents im At the four vertices in a clockwise direction, the motion vectors corresponding to the four vertices are respectively Rot90 means rotating the motion vector by 90°.

[0138] Motion trend consistency loss: The first and last frames of the sliding window indicate the trend of grid motion changes within the sliding window. However, directly using the average value ave of the first and last frames will lead to new jitter. By calculating the difference between the motion and ave in each frame, assigning weights to them, and summing them, the target of the smoothing function is obtained. The loss design is shown in the following series of formulas:

[0139] ave=(f0+f 14 ) / 2

[0140]

[0141]

[0142]

[0143] Where, f i This represents the motion trajectory of the i-th frame within the sliding window, where ω is a constant. See the Realtime Smoother network structure diagram. Figure 3 .

[0144] Step 6: Based on the difference in motion trajectories before and after smoothing, distort the image in grid units, output the stabilized image, and generate the corresponding mask as input for the filling process.

[0145] Furthermore, the specific steps of step 6 are as follows:

[0146] Step 6.1: After obtaining the smoothed network motion trajectory using Realtime Smoother after training, calculate the difference between the mesh motion trajectory before and after smoothing.

[0147] Step 6.2: Calculate the homography transformation matrix of each grid based on the difference in grid motion trajectories, perform homography transformation on the image in units of grid, distort the image to obtain a stable image, and perform the same homography transformation on a blank image as a mask for subsequent video filling.

[0148] Step 7: Based on the stabilized image and the mask, fill in the missing edge parts using the E2FGVI algorithm.

[0149] Furthermore, the specific steps of step 7 are as follows:

[0150] Step 7.1: After obtaining the stable image sequence and corresponding filling mask obtained in Step 6, input the image and mask into the Flow_completion module to complete the optical flow of the missing part and obtain the filling optical flow Flow.

[0151] Step 7.2: Input the stabilized image sequence output from Step 6 into the local neighbor feature extraction module and the non-local reference module in the E2FGVI video filling algorithm respectively; for the missing part of each frame of video content in the image sequence, extract the neighboring frame feature C0 and the non-neighboring frame feature C1 of the current processing frame respectively for filling.

[0152] Step 7.3: Input the adjacent frame features C0 and the filling optical flow Flow into the Feature Propagation module, i.e. the feature propagation module, to propagate the adjacent frame features to the missing region through the filling optical flow, and obtain feature C2;

[0153] Step 7.4: Pass C1 and C2 as input to the content hallucination module. For parts that cannot be filled by local features, combine the non-adjacent frame features C1 to output the final filled image. See [link to module]. Figure 7 .

[0154] Systems based on the above real-time full-frame video stabilization method include:

[0155] The data processing module is used to divide the input real-time video into image sequences and preprocess the input images, corresponding to the preprocessing operation in step 1.

[0156] The key point extraction module extracts key points from the current image, corresponding to the key point extraction operation in step 2.

[0157] The optical flow extraction module is used for the optical flow extraction operation in step 3; it uses the RAFT optical flow extraction algorithm to extract the optical flow map of the input image sequence from the current image and the input image.

[0158] The mesh motion estimation module is used for the mesh vertex trajectory establishment operation in step 4. It uses the key point information extracted in step 2 and the optical flow map extracted in step 3 to generate mesh motion trajectories by training the MotionPro network. It includes a Distance Encoder submodule, which outputs distance features and mesh vertex weights; a Motion Encoder submodule, which outputs optical flow features; and a DM (Distance-Motion) submodule, which is used to obtain mesh vertex motion features.

[0159] The real-time motion trajectory smoothing module smooths and stabilizes the existing grid motion trajectory in real time, outputting a smoother and more continuous motion trajectory, corresponding to the grid motion trajectory smoothing operation in step 5.

[0160] The video rendering and filling module, based on the smoothed grid motion trajectory obtained in step 5, obtains a stable video image through homography transformation in step 6, and fills the missing parts in the homography-transformed image through step 7, thereby achieving real-time full-frame stable video output.

[0161] Devices based on the above real-time full-frame video stabilization method include:

[0162] Video image capture device, used to capture video images in real time;

[0163] A program processor is used to store computer programs and execute the computer programs to implement the real-time full-frame video stabilization method described in steps 1 to 7.

[0164] A display used to show the results of video stabilization.

[0165] Experimental results

[0166] according to Figure 6 To verify the performance of the improved model, we compared it with five advanced video stabilization algorithms from recent years on the NUS public dataset in terms of metrics and results. These algorithms include DIFRINT, FuSta, LVS, DeepStab, and DUT. The metrics for this invention compared with those of the five advanced video stabilization algorithms from recent years on the public dataset are shown in Table 1.

[0167] Table 1. Stability, distortion, and clipping rates of different models

[0168]

[0169]

[0170] Table 1 Stabilization rate, distortion rate and clipping rate of different models

[0171] As shown in Table 1, compared with existing technologies, this invention achieves real-time video stabilization while leading other existing technologies in terms of stabilization rate and being comparable to other methods in terms of distortion rate. Furthermore, by combining a video filling algorithm, it solves the edge missing problem, allowing the stabilized video to be output in full frame at its original resolution without cropping.

Claims

1. A real-time full-frame video stabilization method, characterized in that, Specifically, the following steps are included: Step 1: Collect continuous video with jitter and instability factors, divide it into image sequences, and preprocess the image sequences using bilinear interpolation. Step 2: Using the preprocessed image sequence as the input image, extract the key points of the input image sequence using the RFNet key point extraction algorithm; Step 3: Extract the optical flow map of the input image sequence using the RAFT optical flow extraction algorithm; Step 4: Based on the key points obtained in Step 2 and the optical flow map of the input image sequence extracted in Step 3, the image is meshed, and the mesh motion trajectory is generated by training the MotionPro network. Step 5: Based on the mesh motion trajectory obtained in Step 4, set a sliding window. The sliding window is... Frame size, for the previous The frame's grid motion trajectory is smoothed, while the preceding... The smoothed result of the second sliding window is used as the smoothed trajectory. The mesh motion trajectory and the smoothed trajectory are taken as input and processed through a Realtime Smoother network to generate the first sliding window... After stabilizing the frame, the grid motion trajectory is obtained. Following this process, the sliding window moves one frame to the right, discarding the content of the first frame within the sliding window. The content of the new frame is read to the end of the window to obtain the smoothed motion trajectory corresponding to the new window. Simultaneously, the smoothed trajectory result is updated. This process is repeated until the entire input image sequence is processed. The specific processing flow of the RealtimeSmoother network is as follows: Step 5.1: Based on the mesh motion trajectory obtained in Step 4, set the sliding window to... Frame size, and for the previous The frame's grid motion trajectory is smoothed; Step 5.2, set the grid motion trajectory in the sliding window and the previous... The smoothed mesh motion trajectory of each frame is input into the Realtime Smoother network for the first inference, and the output in the sliding window is the [frame name missing]. The stable grid motion trajectory after the frame is completed; Step 5.2.a, read the previous data from step 5.1 The frame's grid motion trajectory is input into the Origin_embedding submodule. The smoothed grid motion trajectory of the frame is input into the Smooth_embedding submodule, and their coordinate dimensions are expanded through a fully connected layer. The results are the original trajectory encoding feature Ori and the smoothed trajectory encoding feature Smooth, respectively. Step 5.2.b: Pass Ori and Smooth into a convolutional module. Each convolutional module has n convolutional units. Each convolutional unit consists of a 3*3 three-dimensional convolutional layer and a ReLU layer. The outputs are the original trajectory convolutional feature Ori_conv and the smooth trajectory convolutional feature Smooth_conv, respectively. Step 5.2.c: Pass Ori_conv and Smooth_conv to the original trajectory decoding module Ori_decoder and the smooth trajectory decoding module Smoother_decoder respectively, decode the content, and output the first line within the sliding window. New motion trajectory after frame stabilization ; Step 5.3: Add the next frame of video content from the sliding window to the sliding window, remove the first frame of video content from the current sliding window, and obtain the smoothing result from Step 5.

2. Before joining The smoothed result of the next sliding window is used as the smoothed trajectory result. The new sliding window and the smoothed trajectory are used as input. Step 5.2 is repeated until the entire video is processed. Step 5.4: After all video processing is complete, the motion trajectory of the entire video after smoothing by the Realtime Smoother network is obtained; By using the loss function, Equation (4), the adjacent frames of the output trajectory are made to have temporal consistency; in A local region was defined. express A smooth trajectory at the location; By using the loss function as shown in equation (5), certain restrictions are placed on the smoothness of the trajectory, so that its motion trajectory conforms to the global motion trend of the original trajectory: Protect the loss by adding content to the loss function To avoid partial image distortion that affects video quality, the loss function is shown in formula (6): A shape preservation loss is added to avoid image discontinuities between image grids or severe grid shape distortion. The loss is shown in the following formula (7): in, , , express At the four vertices in a clockwise direction, the motion vectors corresponding to the four vertices are respectively Rot90 means rotating the motion vector by 90°; Motion trend consistency loss: The first and last frames of the sliding window indicate the trend of grid motion changes within the sliding window, but direct use will lead to new jitter. This is addressed by calculating the motion in each frame... The differences between them are weighted and summed to obtain the objective function of the smoothing function. Avoid directly using the average of the first and last frames. This leads to the generation of new jitter, and the loss formula is shown in equations (8) to (11) below: （8） （9） （10） （11） in, Indicates the first [number]th ... The motion trajectory of the frame, It is a constant; The loss value is calculated based on the above loss function, and the Realtime Smoother network is updated by backpropagation until the loss result converges and can correctly generate a smoother and more continuous grid motion trajectory. Step 6: Based on the difference in motion trajectories before and after smoothing in Step 5, calculate the homography transformation matrix required to output a stable image using grids as deformation units. Perform homography transformation on the input image according to the homography transformation matrix of each grid, and output a stable image sequence, which is a stable video image. At the same time, perform the same homography transformation on a blank image to generate a corresponding mask as input for filling. Step 7: Based on the stabilized image and mask obtained in Step 6, fill the missing edge parts using the E2FGVI video fill algorithm.

2. The real-time full-frame video stabilization method according to claim 1, characterized in that, The image preprocessing in step 1 is as follows: The input image is scaled to compress the input resolution to a uniform resolution. .

3. The real-time full-frame video stabilization method according to claim 1, characterized in that, The specific method in step 2 is as follows: Step 2.1: Input the image into the RFD detector to obtain the score map. , directional map Scale diagram ; Step 2.2, analyze the score graphs obtained in Step 2.

1. , Direction chart and scale diagram Perform feature extraction; Step 2.3: Input the features obtained in step 2.2 into the descriptor network, and extract fixed-length feature vectors for matching to obtain image key points.

4. The real-time full-frame video stabilization method according to claim 1, characterized in that, The specific process of step 3 includes: Step 3.1: Input the image into the Feature / Context network module for training to extract the texture content and local pixel features of the image; Step 3.2: Input the texture content and local pixel features extracted in Step 2.1 into the association layer, and construct a 4-dimensional layer by taking the inner product of all paired feature vectors. Related entities; pooling and downsampling the relationships in a 4-dimensional dimension to generate a dimension of... The multidimensional correlation weights, where t is the number of pooling operations; Step 3.3: Initialize the optical flow to be generated to 0. Based on the GRU update operator, use the multidimensional correlation weights obtained from the correlation layer in step 3.2 as the weight parameters of the GRU operator. Input the optical flow as input and iteratively pass it into the GRU operator. The output result is the updated optical flow. By iteratively updating the optical flow through this method, the final result is obtained.

5. The real-time full-frame video stabilization method according to claim 1, characterized in that, The specific steps of step 4 are as follows: Step 4.1: Combine the key points obtained in Step 2 with the optical flow map obtained in Step 3, and extract the optical flow of the corresponding points in the optical flow map based on the key point coordinates to obtain the key point coordinates and the optical flow of the corresponding key points. Step 4.2: Input the key point positions and optical flow information obtained in Step 4.2 into the MotionPro network to obtain the mesh vertex motion; 4.2.a. Mesh the image, calculate the Euclidean distance between each keypoint and each grid vertex to obtain the distance information between the keypoint and the grid vertex, and pass the distance information between the keypoint and the grid into the Distance Encoder submodule to output the distance features and grid vertex weights. 4.2.b, input the image key point locations and optical flow information into the Motion Encoder submodule, and output the optical flow features; 4.2.c: Combine the distance features obtained in 4.2.a with the optical flow features obtained in 4.2.b, and input them into the DM submodule to obtain the mesh vertex motion features; 4.2.d, multiply the mesh vertex motion features by the mesh vertex weights to obtain the mesh vertex motion; Step 4.3: For the mesh vertex motion obtained in Step 4.2, calculate the loss value based on three aspects: the difference between the motion of each vertex and the motion of nearby keypoints, the consistency of the overall motion of the four vertices in a single mesh, and the consistency of the mesh shape after motion. Backpropagate to update the MotionPro network until the loss converges, so that the network can generate the correct expression of the motion direction and velocity of the mesh pixels in the frame, and obtain the mesh motion trajectory through multiple iterations; its loss function is shown in the following formula (1): （1） in, The calculation formula for shape protection loss is shown in equation (2) below: （2） The loss includes the motion difference between each vertex and its neighboring keypoints, and the consistency of the overall motion of the four vertices in a single mesh, as shown in the following formula (3): （3）。 6. The real-time full-frame video stabilization method according to claim 1, characterized in that, The specific steps of step 6 are as follows: Step 6.1: Based on the Realtime Smoother smoothed network motion trajectory obtained after training in Step 5, subtract the motion trajectory before smoothing from the motion trajectory after smoothing to obtain the difference between the mesh motion trajectory before and after smoothing. Step 6.2: Calculate the homography transformation matrix of each grid based on the grid motion trajectory difference. Perform homography transformation on the image in units of grids according to the homography transformation matrix of each grid to obtain a stable image sequence. At the same time, perform the same homography transformation on a blank image to generate a corresponding mask as input for the filling process.

7. The real-time full-frame video stabilization method according to claim 1, characterized in that, The specific steps of step 7 are as follows: Step 7.1: Input the stable image sequence and mask obtained in step 6 into the Flow_completion module to complete the optical flow of the missing parts, and obtain the filled optical flow Flow. Step 7.2: Input the stabilized image sequence output from Step 6 into the local neighbor feature extraction module and the non-local reference module in the E2FGVI video filling algorithm respectively; for the missing part of each frame of video content in the image sequence, extract the neighboring frame feature C0 and the non-neighboring frame feature C1 of the current processing frame respectively for filling. Step 7.3: Input the adjacent frame features C0 and the filling optical flow Flow into the Feature Propagation module, i.e. the feature propagation module, to propagate the adjacent frame features to the missing region through the filling optical flow, and obtain feature C2; Step 7.4: Take C1 and C2 as inputs and pass them to the content hallucination module. Combine the non-adjacent frame features C1 for the parts that cannot be filled by local features, and output the final filled image.

8. The system of the real-time full-frame video stabilization method according to any one of claims 1 to 7, characterized in that, include: The data processing module is used to divide the input real-time video into image sequences and preprocess the input images, corresponding to the preprocessing operation in step 1. The key point extraction module extracts key points from the current image, corresponding to the key point extraction operation in step 2. The optical flow extraction module is used for the optical flow extraction operation in step 3; it uses the RAFT optical flow extraction algorithm to extract the optical flow map of the input image sequence from the current image and the input image. The mesh motion estimation module is used for the mesh vertex trajectory establishment operation in step 4. It uses the key point information extracted in step 2 and the optical flow map extracted in step 3 to generate mesh motion trajectories by training the MotionPro network. It includes the DistanceEncoder submodule, which outputs distance features and mesh vertex weights; and the Motion Encoder submodule, which outputs optical flow features. Pass the data to the DM submodule to obtain the motion characteristics of the mesh vertices; The real-time motion trajectory smoothing module smooths and stabilizes the existing grid motion trajectory in real time, outputting a smoother and more continuous motion trajectory, corresponding to the grid motion trajectory smoothing operation in step 5. The video rendering and filling module, based on the smoothed grid motion trajectory obtained in step 5, obtains a stable video image through homography transformation in step 6, and fills the missing parts in the homography-transformed image through step 7, thereby achieving real-time full-frame stable video output.

9. The apparatus for the real-time full-frame video stabilization method according to any one of claims 1 to 7, characterized in that, include: Video image capture device, used to capture video images in real time; A program processor is used to store computer programs and execute the computer programs to implement the real-time full-frame video stabilization method described in steps 1 to 7. A display used to show the results of video stabilization.

Images