Video stabilization method and device, processing equipment and storage medium

Abstract

The invention relates to a video stabilization method and device, processing equipment and a storage medium, and belongs to the technical field of image processing, and the video stabilization method comprises the following steps: employing an IMU sensor to collect an IMU data sequence of a camera, and generating an IMU inter-frame transformation field; for the image sequence, generating an image inter-frame transformation field by using a track-based unsupervised network structure; fusing the IMU inter-frame transformation field and the image inter-frame transformation field by using the trained fusion network to form a stable inter-frame transformation field; and transforming the unstable video by using the stable inter-frame transformation field to generate stable video output. The IMU sensor is used for collecting the global motion of the camera, the influence of local motion of a scene and image quality is avoided, meanwhile, the motion of each pixel is estimated based on image data, the problem that motion is inconsistent under different depths in the scene can be solved, and the success rate and stability of image stabilization are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video stabilization method, device, processing equipment and storage medium. Background Art

[0002] Video stabilization technology plays a vital role in the process of video shooting and processing. Traditional video stabilization methods mainly rely on image processing technologies such as optical flow and feature point tracking, but these methods are prone to failure in complex scenes, such as scenes with more local motion, single background, dim light and other situations where there are insufficient effective features. In recent years, stabilization methods based on convolutional neural networks have made significant progress. Convolutional networks can extract some deep features, but they still face challenges, especially in fast and complex motion environments. On the other hand, the angular velocity and acceleration data provided by the IMU sensor can provide stable camera motion information, but relying solely on IMU data for stabilization also has limitations. The IMU can only calculate the overall motion of the camera and cannot consider the local motion caused by different depths of the scene, which will cause distortion in the stabilized video. Therefore, there is an urgent need for a video stabilization method that fuses image and IMU data to improve the stabilization effect. Summary of the invention

[0003] In order to solve the problem that traditional video stabilization methods are prone to failure in scenes with insufficient effective features and large depth changes, and the video is still distorted after stabilization, resulting in unsatisfactory stabilization effects, the present invention proposes a video stabilization method, device, processing equipment and storage medium based on IMU and image fusion.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A video stabilization method, the method comprising the following steps:

[0006] Step 1: Generate the IMU inter-frame transformation field based on the IMU data sequence. The specific process is as follows:

[0007] Step 1.1: Obtain an image sequence of an unstable video captured by a camera and an IMU data sequence synchronously collected by an IMU sensor, wherein the IMU data sequence includes acceleration data and angular velocity data of the camera at each moment;

[0008] Step 1.2: Calculate the corresponding Euler angle according to the angular velocity data, convert the Euler angle into a rotation matrix, and solve the corresponding translation vector according to the acceleration data;

[0009] Step 1.3: Correct the rotation matrix and translation vector obtained in step 1.2 according to the calibration matrix between the IMU sensor and the camera to obtain the corrected rotation matrix and translation vector;

[0010] Step 1.4: Convert each corrected rotation matrix to the Riemann manifold space for smoothing to obtain a smoothed rotation matrix, and use the Kalman filter method to smooth each corrected translation vector to generate a smoothed translation vector;

[0011] Step 1.5: transform the coordinates of each pixel point of two adjacent frames in the image sequence according to formula (3), and obtain the IMU inter-frame transformation field after traversing all the frame images in the image sequence;

[0012] P 2 =KRK -1 P 1 +Kt (3)

[0013] Among them, P 1 is the coordinate of a pixel in a frame image, P 2 is the coordinate of the pixel after transformation, K is the memory matrix of the camera, R is the smoothed rotation matrix corresponding to the frame image, and t is the smoothed translation vector corresponding to the frame image;

[0014] Step 2: Generate an image frame transformation field based on the image sequence. The specific process is as follows:

[0015] The image sequence is input into a trained trajectory-based unsupervised network structure for processing, and the network outputs a smooth motion trajectory to obtain an image frame transformation field. The processing process of the trajectory-based unsupervised network structure includes the following steps:

[0016] Step 2.1: Use RFNet detector to identify key points in each frame, and use PWCNet to calculate the optical flow between frames to obtain the motion vector of the key points;

[0017] Step 2.2: Estimate the homography transformation matrix from the key point trajectory, calculate the residual motion vector of each key point after applying the homography transformation, and the distance between each key point and the grid point according to the estimated homography transformation matrix, and then use the 1D convolution branch to encode the residual motion information and distance information. Use the attention mechanism to assign higher weights to key points with similar motion and smaller distance, adaptively select important key points and aggregate their motion in a weighted sum manner to accurately obtain the motion vector of each grid point;

[0018] Step 2.3: The motion trajectory is obtained by temporally associating the motion vectors of the corresponding mesh vertices of multiple frames. The 1D convolutional trajectory encoder is used to slide along the time window to encode the temporal features of the trajectory, which is further input into several convolutional layers stacked with 3D convolutions. The kernel decoder and scale decoder are then used for parallel embedding, and the smoothing kernel and scale are predicted to form the final smoothing kernel. The network backpropagation is used to minimize the accumulated corresponding grid point position differences in the time dimension, and iterative smoothing is performed to output a smooth motion trajectory.

[0019] Step 3: Use the trained fusion network to fuse the IMU inter-frame transformation field and the image inter-frame transformation field, and the network outputs the fused stable inter-frame transformation field;

[0020] Step 4: Use the stable inter-frame transformation field to transform the unstable video to generate a stable video output.

[0021] A video stabilization device, comprising:

[0022] A first generation module for generating an IMU inter-frame transformation field based on an IMU data sequence, wherein the first generation module comprises:

[0023] A data acquisition submodule, used to acquire an image sequence of unstable video collected by a camera and an IMU data sequence synchronously collected by an IMU sensor, wherein the IMU data sequence includes acceleration data and angular velocity data of the camera at each moment;

[0024] A calculation submodule, used for calculating the corresponding Euler angle according to the angular velocity data, converting the Euler angle into a rotation matrix, and solving the corresponding translation vector according to the acceleration data;

[0025] A correction submodule, used to correct the rotation matrix and translation vector obtained by the calculation submodule according to the calibration matrix between the IMU sensor and the camera, so as to obtain a corrected rotation matrix and translation vector;

[0026] A smoothing submodule is used to convert each corrected rotation matrix into a Riemann manifold space for smoothing to obtain a smoothed rotation matrix, and to smooth each corrected translation vector using a Kalman filter method to generate a smoothed translation vector;

[0027] A coordinate transformation submodule is used to transform the coordinates of each pixel point of two adjacent frames in the image sequence according to formula (3), and obtain an IMU inter-frame transformation field after traversing all frame images in the image sequence;

[0028] P 2 =KRK- 1 P 1+Kt (3)

[0029] Among them, P 1 is the coordinate of a pixel in a frame image, P 2 is the coordinate of the pixel after transformation, K is the memory matrix of the camera, R is the smoothed rotation matrix corresponding to the frame image, and t is the smoothed translation vector corresponding to the frame image;

[0030] The second generation module for generating the image frame transformation field based on the image sequence is used to input the image sequence into the trained trajectory-based unsupervised network structure for processing. The network outputs a smooth motion trajectory to obtain the image frame transformation field. The trajectory-based unsupervised network structure includes:

[0031] The first submodule is used to identify the key points in each frame of the image using the RFNet detector and calculate the optical flow between frames using PWCNet to obtain the motion vector of the key points;

[0032] The second submodule is used to estimate the homography transformation matrix from the key point trajectory, calculate the residual motion vector of each key point after applying the homography transformation, and the distance between each key point and the grid point according to the estimated homography transformation matrix, and then use the 1D convolution branch to encode the residual motion information and distance information. The attention mechanism assigns higher weights to key points with similar motion and smaller distance, adaptively selects important key points and aggregates their motion in a weighted sum manner to accurately obtain the motion vector of each grid point;

[0033] The third submodule is used to obtain the motion trajectory by temporally associating the motion vectors of the corresponding mesh vertices of multiple frames, and uses a 1D convolutional trajectory encoder to slide along the time window to encode the temporal features of the trajectory, which is further input into several convolutional layers stacked with 3D convolutions, and then embedded in parallel through a kernel decoder and a scale decoder, predicting the smoothing kernel and scale to form the final smoothing kernel, and minimizing the accumulated corresponding grid point position differences in the time dimension through back propagation of the network, and iteratively smoothing is performed to output a smooth motion trajectory;

[0034] A fusion module, used to fuse the IMU inter-frame transformation field and the image inter-frame transformation field using a trained fusion network, and the network outputs a fused stable inter-frame transformation field;

[0035] The transformation module is used to transform the unstable video using the stable inter-frame transformation field to generate a stable video output.

[0036] A processing device includes a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor implements the above-mentioned video stabilization method when executing the computer program.

[0037] A storage medium stores a computer program, and when the computer program is read and executed, the above-mentioned video stabilization method is implemented.

[0038] In view of the fact that the existing image stabilization methods are insufficient for complex scenes and scenes with large depth changes, the present invention proposes a video stabilization method, device, processing equipment and storage medium based on IMU and image data fusion. The present invention uses an IMU sensor to obtain the global motion of the camera, including the three-dimensional rotation and three-dimensional translation of the camera, and converts the three-dimensional rotation and translation into an IMU inter-frame transformation field based on the IMU data sequence; generates an image inter-frame transformation field for the image sequence using a trajectory-based unsupervised network structure; and then fuses the IMU inter-frame transformation field and the image inter-frame transformation field to form a final stable inter-frame transformation field, which acts on unstable video to generate a stable video. The beneficial effects of the present invention are:

[0039] Compared with the existing methods, the present invention combines IMU data and image data. The IMU data is used to collect the global motion of the camera, including three-dimensional rotation and three-dimensional translation. The motion estimation is not affected by the local motion of the scene and the image quality. The image data is used to estimate the motion of each pixel, which can solve the problem of inconsistent motion at different depths in the scene. The fusion of IMU data and image data can not only integrate the global motion of the camera, but also combine the local motion caused by different depths and moving objects in the scene, thereby improving the success rate and stability of image stabilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of a video stabilization method according to an embodiment of the present invention;

[0041] Figure 2 Flowchart for obtaining the calibration matrix. DETAILED DESCRIPTION

[0042] The technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0043] In one embodiment, if Figure 1 As shown, a video stabilization method is provided, the method comprising the following steps:

[0044] Step 1: Generate IMU inter-frame transformation field based on IMU data sequence;

[0045] Step 2: Generate an image frame transformation field based on the image sequence;

[0046] Step 3: Use the trained fusion network to fuse the IMU inter-frame transformation field and the image inter-frame transformation field, and the network outputs the fused stable inter-frame transformation field;

[0047] Step 4: Use the stable inter-frame transformation field to transform the unstable video to generate a stable video output.

[0048] Specifically, the specific process of generating the IMU inter-frame transformation field based on the IMU data sequence in step 1 is as follows:

[0049] Step 1.1: Collect image sequence and IMU data sequence: directly obtain the image sequence of the unstable video captured by the camera, and use the IMU sensor to synchronously collect the IMU data sequence. The IMU data sequence includes the acceleration data and angular velocity data of the camera at each moment. By integrating the angular velocity and quadratically integrating the acceleration, the three-dimensional rotational motion and translational motion of the camera are obtained, which can be used to describe the camera motion trajectory.

[0050] Step 1.2: Solve the transformation matrix and translation vector: Calculate the Euler angle based on the angular velocity data obtained by the IMU sensor, and then convert the Euler angle into a rotation matrix R′; use the acceleration data as translation information, and solve the corresponding translation vector t′ based on the acceleration data.

[0051] Step 1.3: Calibrate the IMU sensor and the camera to obtain the calibration matrix, and then correct the rotation matrix and translation vector obtained in step 1.2 according to the calibration matrix between the IMU sensor and the camera to obtain the corrected rotation matrix and translation vector.

[0052] Since the center of the IMU sensor does not coincide with the center of the camera, the motion information measured by the IMU sensor cannot directly reflect the motion information of the camera. There is a rotation and translation deviation between the two, so they need to be calibrated. Figure 2 As shown, the calibration matrix between the IMU sensor and the camera is obtained by the following steps:

[0053] Step 1.3.1: Data preprocessing: Time axis alignment of IMU data sequence and image sequence;

[0054] Step 1.3.2: Calculate the camera's rotation matrix and translation vector from the IMU data sequence in the same way as step 1.2;

[0055] Step 1.3.3: Use the ORB algorithm to match the feature points of the image sequence, calculate the basic matrix F by the 8-point method, and then calculate the essential matrix E from the basic matrix F through the camera intrinsic parameters. Then perform SVD decomposition on the essential matrix E to obtain the rotation matrix and translation vector of the camera;

[0056] Step 1.3.4: Taking the first frame image as a reference, fit the rotation matrix and translation vector calculated in steps 1.3.2 and 1.3.3 respectively to obtain the optimal calibration matrix, which contains a three-dimensional rotation matrix and a three-dimensional translation vector.

[0057] Step 1.4: Motion smoothing: For each rotation matrix sequence of a video, each rotation matrix in the rotation matrix sequence is converted to the Riemann manifold space for smoothing to obtain a smoothed rotation matrix.

[0058] Assuming that the video includes N frames of images, the three-dimensional rotation matrix sequence of the N-frame image sequence can be expressed as:

[0059] x=[R′ 1 , R′ 2 , …, R N ] (1)

[0060] Step 1.4.1: Construct the objective function:

[0061]

[0062] Among them, {R n} represents the smoothed rotation matrix sequence, {R′ n} represents the original rotation matrix sequence, α is the parameter that controls the smooth path, d g (R′ n , R n )=||logm(R′ n T , R n )|| F , d g (R n , R n+1 )=||logm(R n T , R n+1 )|| F , logm represents the matrix logarithm, and N is the total number of frames in the image sequence.

[0063] Step 1.4.2: Use the steepest gradient descent method to solve formula (2) and obtain the smoothed rotation matrix R.

[0064] The Kalman filter method is used to smooth each translation vector to generate a smoothed translation vector t.

[0065] Step 1.5: Generate transformation field: Use the smoothed rotation matrix and translation vector to transform the unstable image, where the transformation refers to the transformation of pixel coordinates. Let the coordinates of a pixel in a frame of the unstable video be P 1, after transforming the coordinates of the pixel point, the coordinates of the new pixel point can be calculated according to formula (3):

[0066] P 2 =KRK -1 P 1 +Kt (3)

[0067] Among them, P 2 is the coordinate of the pixel after the stabilization transformation, K is the memory matrix of the camera, R is the smoothed rotation matrix corresponding to the frame image, and t is the smoothed translation vector corresponding to the frame image.

[0068] Formula (3) is used to calculate each pixel point of two adjacent frames in the image sequence. That is, after the coordinates of each pixel point are transformed, the motion vector of the pixel point between frames can be obtained, and then the adjacent image transformation field can be generated. After calculating the entire unstable image sequence, the IMU inter-frame transformation field based on the IMU data sequence can be obtained. This transformation field can describe the motion vector of each pixel point between frames in the process of converting the unstable video to the stable video calculated based on the IMU data (including angular velocity and acceleration data).

[0069] The specific process of generating the image frame transformation field based on the image sequence in step 2 is as follows:

[0070] The inter-frame transformation field in this step is processed using an unsupervised network structure based on trajectories. The continuous video frame images, i.e., the image sequence obtained in step 1.1, are input into the above network structure, and the network is trained. The trained network structure can generate an inter-frame transformation field based on the image information. The inter-frame transformation field can describe the motion vector of each pixel between frames in the process of image processing from unstable video to stable video. The specific steps of processing the unsupervised network structure based on trajectories are as follows:

[0071] Step 2.1, key point detection:

[0072] Motion estimation is performed using RFNet detector and PWCNet optical flow. RFNet detector is used to identify key points in video frames, and PWCNet is used to calculate the optical flow between frames to obtain the motion vectors of key points.

[0073] Step 2.2, motion propagation: propagate the motion of key points to dense mesh vertices.

[0074] The homography transformation matrix H is estimated from the key point trajectory, and the residual motion vector of each key point after applying the homography transformation and the distance between each key point and the grid point are calculated according to the estimated homography transformation matrix H. The residual motion information and distance information are then encoded using the 1D convolution branch. The key points with similar motion and smaller distance are assigned higher weights through the attention mechanism. Important key points are adaptively selected and their motions are aggregated in a weighted sum manner to accurately obtain the motion vector of each grid point.

[0075] Step 2.3, trajectory smoothing:

[0076] The motion trajectory can be obtained by temporally associating the motion vectors of the corresponding mesh vertices of multiple frames. The trajectory encoder of 1D convolution is used to slide along the time window to encode the temporal features of the trajectory, which is further input into the convolutional layer of several 3D convolution stacks, and then embedded in parallel through the kernel decoder and scale decoder to predict the smoothing kernel and scale to form the final smoothing kernel. The network's back propagation is used to minimize the accumulated corresponding grid point position differences in the time dimension, and iterative smoothing is performed to output a smooth motion trajectory, thereby obtaining a sequence of image frame transformation fields.

[0077] After processing through the above steps 2.1 to 2.3, an image frame transformation field based on the image sequence can be obtained.

[0078] The specific process of fusing the inter-frame transformation field based on the IMU data sequence and the image sequence in step 3 is as follows:

[0079] The IMU inter-frame transformation field and the image inter-frame transformation field are fused using the attention mechanism, and the two are used as the input of the trained fusion network. The output of the fusion network is the fused stable inter-frame transformation field. At this time, this new inter-frame transformation field can be directly used to distort the unstable video.

[0080] The specific structure of the fusion network is as follows:

[0081] 1) Fusion network input and output: The input of the network is two inter-frame transformation fields, one is the inter-frame transformation field generated by IMU, and the other is the inter-frame transformation field generated by image sequence; the output of the network is a stable inter-frame transformation field that can be used to directly transform unstable video.

[0082] 2) Fusion module based on spatiotemporal attention mechanism:

[0083] The fusion network uses the spatiotemporal attention mechanism to fuse the two inter-frame transformation fields. The specific processing process is as follows:

[0084] Extracting spatiotemporal feature information: Use convolution to extract the spatial and temporal information of the input frame-to-frame transformation field. The convolution operation can capture local features and enhance the model's understanding of spatial structure. The input is processed through convolution to extract the temporal transformation in order to capture dynamic information.

[0085] Calculate attention weights: After adding spatial features and temporal features, the attention weights are generated through the activation function. The weights here reflect the importance of different features.

[0086] Feature fusion: Use the calculated attention weights to perform weighted calculations on the input inter-frame transformation field, which can highlight important features while suppressing unimportant features, and finally output a fused stable inter-frame transformation field.

[0087] The fusion network is trained using supervised learning. The training process includes the following steps:

[0088] The first camera is fixedly mounted on the stabilizer to acquire a stable video;

[0089] An IMU sensor is rigidly fixed on the second camera, and the second camera is rigidly fixedly connected to the base of the stabilizer. The second camera collects unstable video, and the IMU sensor synchronously collects corresponding IMU data.

[0090] The stabilized video captured by the first camera is used as a label to perform supervised training on the fusion network, and a trained fusion network is obtained after training.

[0091] The following loss functions are set during the network training process, and weighted summation is used to obtain the final loss function in the supervised learning process:

[0092] Optical flow loss function: ensures smooth transition of video frames by calculating the difference between predicted optical flow and true optical flow;

[0093] Smooth loss function: By imposing smooth constraints on motion vectors and reducing local drastic changes, the smoothness of the video is ensured and abrupt changes are reduced;

[0094] Content loss function: Ensure image quality and consistency by comparing the content of video frames before and after stabilization to ensure the consistency of video content and reduce content distortion caused by stabilization.

[0095] The trained fusion network can fuse two inter-frame transformation fields, and the stable inter-frame transformation field output by the network can be directly used to warp unstable videos.

[0096] In step 4, when applying the stable inter-frame transformation field, the following steps are specifically included:

[0097] Step 4.1: Apply the fused stable inter-frame transformation field to each frame image in the unstable video and adjust the position of each pixel;

[0098] Step 4.2: The adjusted pixel position may not be aligned with the integer pixel coordinates, resulting in sub-pixel offset and aliasing. Therefore, bilinear interpolation is used to eliminate aliasing and restore image quality. At the same time, the black border area with missing content in the transformed image is cropped to retain the maximum effective area in the center to ensure the video viewing experience.

[0099] Aiming at the situation that the existing stabilization methods are insufficient in stabilizing complex scenes and scenes with large depth changes, the present invention proposes a video stabilization method based on IMU and image data fusion. First, the IMU sensor is used to obtain the global motion of the camera, including the three-dimensional rotation and three-dimensional translation of the camera, and the three-dimensional rotation and translation are converted into an inter-frame transformation field; secondly, a neural network is used to generate an inter-frame transformation field for the image sequence; thirdly, the calibration matrix between the camera and the IMU is used to convert the inter-frame transformation field obtained by the IMU sensor into a transformation field centered on the optical center of the camera, and the IMU inter-frame transformation field is obtained; finally, the IMU inter-frame transformation field and the image sequence transformation field are fused to form the final stable transformation field, which is applied to the unstable video to generate a stable video.

[0100] Compared with the existing methods, the present invention proposes a video stabilization method that integrates IMU and image data. The method combines IMU data and image data. The IMU data is used to collect the global motion of the camera, including three-dimensional rotation and translation. The motion estimation is not affected by the local motion of the scene and the image quality; the image data is used to estimate the motion of each pixel, which can solve the problem of inconsistent motion at different depths in the scene; the fusion of IMU data and image data can not only integrate the global motion of the camera, but also combine the local motion caused by different depths and moving objects in the scene, thereby improving the success rate and stability of image stabilization.

[0101] It should be noted that the step numbers in the above method embodiment do not necessarily represent the order of implementation of the steps. For those skilled in the art, the order of the steps can be changed without departing from the protection scope of the present invention. For example, the order of step 1 and step 2 can be changed, and the changed technical solution still falls within the protection scope of the present invention.

[0102] The video stabilization device, processing equipment, storage medium, etc. used to execute the video stabilization method provided by the present invention are described below. The specific implementation process and technical effects thereof refer to the relevant contents of the above-mentioned video stabilization method, which will not be repeated below.

[0103] In another embodiment, the present invention provides a video stabilization device, the device comprising:

[0104] A first generation module for generating an IMU inter-frame transformation field based on an IMU data sequence, the first generation module comprising:

[0105] The data acquisition submodule is used to acquire the image sequence of the unstable video collected by the camera and the IMU data sequence synchronously collected by the IMU sensor. The IMU data sequence includes the acceleration data and angular velocity data of the camera at each moment;

[0106] The calculation submodule is used to calculate the corresponding Euler angle according to the angular velocity data, convert the Euler angle into a rotation matrix, and solve the corresponding translation vector according to the acceleration data;

[0107] A correction submodule is used to correct the rotation matrix and translation vector obtained by the calculation submodule according to the calibration matrix between the IMU sensor and the camera, and obtain the corrected rotation matrix and translation vector;

[0108] A smoothing submodule is used to convert each corrected rotation matrix into a Riemann manifold space for smoothing to obtain a smoothed rotation matrix, and to smooth each corrected translation vector using a Kalman filter method to generate a smoothed translation vector;

[0109] The coordinate transformation submodule is used to transform the coordinates of each pixel point of two adjacent frames in the image sequence according to formula (3), generate an adjacent image transformation field, and obtain the IMU inter-frame transformation field after traversing all frame images in the image sequence;

[0110] P 2 =KRK- 1 P 1 +Kt (3)

[0111] Among them, P 1 is the coordinate of a pixel in a frame image, P 2 is the coordinate of the pixel after transformation, K is the memory matrix of the camera, R is the smoothed rotation matrix corresponding to the frame image, and t is the smoothed translation vector corresponding to the frame image;

[0112] The second generation module for generating the image frame transformation field based on the image sequence is used to input the image sequence into the trained trajectory-based unsupervised network structure for processing. The network outputs a smooth motion trajectory to obtain the image frame transformation field. The trajectory-based unsupervised network structure includes:

[0113] The first submodule is used to identify the key points in each frame of the image using the RFNet detector and calculate the optical flow between frames using PWCNet to obtain the motion vector of the key points;

[0114] The second submodule is used to estimate the homography transformation matrix from the key point trajectory, calculate the residual motion vector of each key point after applying the homography transformation, and the distance between each key point and the grid point according to the estimated homography transformation matrix, and then use the 1D convolution branch to encode the residual motion information and distance information. The attention mechanism assigns higher weights to key points with similar motion and smaller distance, adaptively selects important key points and aggregates their motion in a weighted sum manner to accurately obtain the motion vector of each grid point;

[0115] The third submodule is used to obtain the motion trajectory by temporally associating the motion vectors of the corresponding mesh vertices of multiple frames, and uses a 1D convolutional trajectory encoder to slide along the time window to encode the temporal features of the trajectory, which is further input into several convolutional layers stacked with 3D convolutions, and then embedded in parallel through a kernel decoder and a scale decoder, predicting the smoothing kernel and scale to form the final smoothing kernel, and minimizing the accumulated corresponding grid point position differences in the time dimension through back propagation of the network, and iteratively smoothing is performed to output a smooth motion trajectory;

[0116] The fusion module is used to fuse the IMU inter-frame transformation field and the image inter-frame transformation field using the trained fusion network, and the network outputs the fused stable inter-frame transformation field;

[0117] The transformation module is used to transform the unstable video using the stable inter-frame transformation field to generate a stable video output.

[0118] Optionally, the smoothing submodule is also used to construct the objective function:

[0119]

[0120] Among them, {R n} represents the smoothed rotation matrix sequence, {R' n} represents the original rotation matrix sequence, α is the parameter that controls the smooth path, d g (R' n ,R n )=||logm(R' n T ,R n )|| F , d g (R n ,R n+1 )=||logm(R n T ,R n+1 )|| F , logm represents the matrix logarithm, N is the total number of frames in the image sequence;

[0121] Then, the steepest gradient descent method is used to solve formula (2) to obtain the smoothed rotation matrix R.

[0122] Optionally, the correction submodule is also used to:

[0123] Data preprocessing: align the time axis of the IMU data sequence and the image sequence;

[0124] Calculate the camera's rotation matrix and translation vector based on the IMU data sequence;

[0125] The ORB algorithm is used to match the feature points of the image sequence, and the basic matrix F is calculated by the 8-point method. Then, the essential matrix E is calculated from the basic matrix F through the camera intrinsic parameters. The essential matrix E is then decomposed by SVD to obtain the rotation matrix and translation vector of the camera.

[0126] Taking the first frame image as a reference, the calculated rotation matrix and translation vector are fitted respectively to obtain an optimal calibration matrix, which includes a three-dimensional rotation matrix and a three-dimensional translation vector.

[0127] Optionally, the fusion network uses a spatiotemporal attention mechanism to fuse the transformation fields. The processing of the fusion network includes:

[0128] Extracting spatiotemporal feature information: Use convolution to extract the spatial and temporal information of the input IMU inter-frame transformation field and image inter-frame transformation field respectively;

[0129] Calculate attention weight: After adding spatial features and temporal features, generate attention weight through activation function;

[0130] Feature fusion: Use the calculated attention weights to perform weighted calculations on the input transformation field, and finally output the fused stable inter-frame transformation field.

[0131] Optionally, the training process of the fusion network includes the following steps:

[0132] The first camera is fixedly mounted on the stabilizer to acquire a stable video;

[0133] A second camera rigidly fixed with an IMU sensor is rigidly connected to the base of the stabilizer to collect unstable video and corresponding IMU data;

[0134] The stabilized video captured by the first camera is used as a label to perform supervised training on the fusion network, and a trained fusion network is obtained after training.

[0135] Furthermore, the loss function used when training the fusion network is a weighted sum of an optical flow loss function, a smoothing loss function, and a content loss function.

[0136] Optionally, the transformation module is also used to apply a stable inter-frame transformation field to each frame image in the unstable video to adjust the position of each pixel; a bilinear interpolation method is used to eliminate jagged edges generated after the pixel position adjustment, and to crop black edge areas where content is missing in the image.

[0137] The above-mentioned device is used to execute the video stabilization method provided by the above-mentioned embodiment. Its implementation principle and technical effect are similar and will not be described in detail here.

[0138] The above modules may be one or more integrated circuits configured to implement the above video stabilization method, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0139] In another embodiment, the present invention provides a processing device including a memory and a processor.

[0140] The memory is used to store computer programs, and the processor calls the computer programs stored in the memory to execute the various steps of the video stabilization method provided in the above embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0141] In another embodiment, the present invention provides a program product, such as a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the computer program implements the various steps of the video stabilization method provided in the above embodiment.

[0142] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes. The above-mentioned computer-readable storage medium may be included in the computing device, or may exist independently without being assembled into the computing device.

[0143] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system.

[0144] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A video stabilization method, characterized in that: The following steps are involved: Step 1: Generate the IMU inter-frame transformation field based on the IMU data sequence. The specific process is as follows: Step 1.1: Obtain an image sequence of an unstable video captured by a camera and an IMU data sequence synchronously collected by an IMU sensor, wherein the IMU data sequence includes acceleration data and angular velocity data of the camera at each moment; Step 1.2: Calculate the corresponding Euler angle according to the angular velocity data, convert the Euler angle into a rotation matrix, and solve the corresponding translation vector according to the acceleration data; Step 1.3: Correct the rotation matrix and translation vector obtained in step 1.2 according to the calibration matrix between the IMU sensor and the camera to obtain the corrected rotation matrix and translation vector; Step 1.4: Convert each corrected rotation matrix to the Riemann manifold space for smoothing to obtain a smoothed rotation matrix, and use the Kalman filter method to smooth each corrected translation vector to generate a smoothed translation vector; Step 1.5: transform the coordinates of each pixel point of two adjacent frames in the image sequence according to formula (3), and obtain the IMU inter-frame transformation field after traversing all the frame images in the image sequence; P2=KRK -1 P1+Kt (3) Among them, P1 is the coordinate of a pixel point in a frame image, P2 is the coordinate of the pixel point after transformation, K is the memory matrix of the camera, R is the smoothed rotation matrix corresponding to the frame image, and t is the smoothed translation vector corresponding to the frame image; Step 2: Generate an image frame transformation field based on the image sequence. The specific process is as follows: The image sequence is input into a trained trajectory-based unsupervised network structure for processing, and the network outputs a smooth motion trajectory to obtain an image frame transformation field. The processing process of the trajectory-based unsupervised network structure includes the following steps: Step 2.1: Use RFNet detector to identify key points in each frame, and use PWCNet to calculate the optical flow between frames to obtain the motion vector of the key points; Step 2.2: Estimate the homography transformation matrix from the key point trajectory, calculate the residual motion vector of each key point after applying the homography transformation, and the distance between each key point and the grid point according to the estimated homography transformation matrix, and then use the 1D convolution branch to encode the residual motion information and distance information. Use the attention mechanism to assign higher weights to key points with similar motion and smaller distance, adaptively select important key points and aggregate their motion in a weighted sum manner to accurately obtain the motion vector of each grid point; Step 2.3: The motion trajectory is obtained by temporally associating the motion vectors of the corresponding mesh vertices of multiple frames. The 1D convolutional trajectory encoder is used to slide along the time window to encode the temporal features of the trajectory, which is further input into several convolutional layers stacked with 3D convolutions. The kernel decoder and scale decoder are then used for parallel embedding, and the smoothing kernel and scale are predicted to form the final smoothing kernel. The network backpropagation is used to minimize the accumulated corresponding grid point position differences in the time dimension, and iterative smoothing is performed to output a smooth motion trajectory. Step 3: Use the trained fusion network to fuse the IMU inter-frame transformation field and the image inter-frame transformation field, and the network outputs the fused stable inter-frame transformation field; Step 4: Use the stable inter-frame transformation field to transform the unstable video to generate a stable video output.

2. The video stabilization method according to claim 1, characterized in that: The process of smoothing the rotation matrix comprises the following steps: Step 1.4.1: Construct the objective function: Among them, {R n } represents the smoothed rotation matrix sequence, {R' n } represents the original rotation matrix sequence, α is the parameter that controls the smooth path, d g (R n ,R n+1 )=||logm(R n T ,R n+1 )|| F , logm represents the matrix logarithm, N is the total number of frames in the image sequence; Step 1.4.2: Use the steepest gradient descent method to solve formula (2) and obtain the smoothed rotation matrix R.

3. The video stabilization method according to claim 1 or 2, characterized in that: Obtain the calibration matrix between the IMU sensor and the camera by following these steps: Step 1.3.1: performing time axis alignment on the IMU data sequence and the image sequence; Step 1.3.2: Calculate the rotation matrix and translation vector of the camera according to the IMU data sequence; Step 1.3.3: Use the ORB algorithm to match feature points of the image sequence, calculate the basic matrix F by the 8-point method, and then calculate the essential matrix E from the basic matrix F through the camera intrinsic parameters, and then perform SVD decomposition on the essential matrix E to obtain the rotation matrix and translation vector of the camera; Step 1.3.4: Taking the first frame image as a reference, the rotation matrix and translation vector calculated in steps 1.3.2 and 1.3.3 are fitted respectively to obtain a calibration matrix, wherein the calibration matrix includes a three-dimensional rotation matrix and a three-dimensional translation vector.

4. The video stabilization method according to claim 1 or 2, characterized in that: The fusion network adopts the spatiotemporal attention mechanism to fuse the transformation fields. The processing process of the fusion network includes: Extracting spatiotemporal feature information: using convolution to respectively extract the spatial and temporal information of the input IMU inter-frame transformation field and the image inter-frame transformation field; Calculate attention weight: After adding spatial features and temporal features, generate attention weight through activation function; Feature fusion: Use the calculated attention weights to perform weighted calculations on the input inter-frame transformation field, and finally output the fused stable inter-frame transformation field.

5. The video stabilization method according to claim 1 or 2, characterized in that: The training process of the fusion network includes the following steps: The first camera is fixedly mounted on the stabilizer to acquire a stable video; A second camera rigidly fixed with an IMU sensor is rigidly connected to the base of the stabilizer to collect unstable video and corresponding IMU data; The fusion network is supervisedly trained using the stable video captured by the first camera as a label, and a trained fusion network is obtained after the training.

6. The video stabilization method according to claim 5, characterized in that: The loss function used when training the fusion network is a weighted sum of an optical flow loss function, a smoothing loss function, and a content loss function.

7. The video stabilization method according to claim 1 or 2, characterized in that: Step 4 includes the following steps: Step 4.1: Apply the stable inter-frame transformation field to each frame image in the unstable video to adjust the position of each pixel; Step 4.2: Use the bilinear interpolation method to eliminate the jagged edges caused by adjusting the pixel position, and crop the black border areas where content is missing in the image.

8. A video stabilization device, characterized in that: include: A first generation module for generating an IMU inter-frame transformation field based on an IMU data sequence, wherein the first generation module comprises: A data acquisition submodule, used to acquire an image sequence of unstable video collected by a camera and an IMU data sequence synchronously collected by an IMU sensor, wherein the IMU data sequence includes acceleration data and angular velocity data of the camera at each moment; A calculation submodule, used for calculating the corresponding Euler angle according to the angular velocity data, converting the Euler angle into a rotation matrix, and solving the corresponding translation vector according to the acceleration data; A correction submodule, used to correct the rotation matrix and translation vector obtained by the calculation submodule according to the calibration matrix between the IMU sensor and the camera, so as to obtain a corrected rotation matrix and translation vector; A smoothing submodule is used to convert each corrected rotation matrix into a Riemann manifold space for smoothing to obtain a smoothed rotation matrix, and to smooth each corrected translation vector using a Kalman filter method to generate a smoothed translation vector; A coordinate transformation submodule is used to transform the coordinates of each pixel point of two adjacent frames in the image sequence according to formula (3), and obtain an IMU inter-frame transformation field after traversing all frame images in the image sequence; P2=KRK -1 P1+Kt (3) Among them, P1 is the coordinate of a pixel point in a frame image, P2 is the coordinate of the pixel point after transformation, K is the memory matrix of the camera, R is the smoothed rotation matrix corresponding to the frame image, and t is the smoothed translation vector corresponding to the frame image; The second generation module for generating the image frame transformation field based on the image sequence is used to input the image sequence into the trained trajectory-based unsupervised network structure for processing. The network outputs a smooth motion trajectory to obtain the image frame transformation field. The trajectory-based unsupervised network structure includes: The first submodule is used to identify the key points in each frame of the image using the RFNet detector and calculate the optical flow between frames using PWCNet to obtain the motion vector of the key points; The second submodule is used to estimate the homography transformation matrix from the key point trajectory, calculate the residual motion vector of each key point after applying the homography transformation, and the distance between each key point and the grid point according to the estimated homography transformation matrix, and then use the 1D convolution branch to encode the residual motion information and distance information. The attention mechanism assigns higher weights to key points with similar motion and smaller distance, adaptively selects important key points and aggregates their motion in a weighted sum manner to accurately obtain the motion vector of each grid point; The third submodule is used to obtain the motion trajectory by temporally associating the motion vectors of the corresponding mesh vertices of multiple frames, and uses a 1D convolutional trajectory encoder to slide along the time window to encode the temporal features of the trajectory, which is further input into several convolutional layers stacked with 3D convolutions, and then embedded in parallel through a kernel decoder and a scale decoder, predicting the smoothing kernel and scale to form the final smoothing kernel, and minimizing the accumulated corresponding grid point position differences in the time dimension through back propagation of the network, and iteratively smoothing is performed to output a smooth motion trajectory; A fusion module, used to fuse the IMU inter-frame transformation field and the image inter-frame transformation field using a trained fusion network, and the network outputs a fused stable inter-frame transformation field; The transformation module is used to transform the unstable video using the stable inter-frame transformation field to generate a stable video output.

9. A processing device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the video stabilization method according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is read and executed, the video stabilization method according to any one of claims 1 to 7 is implemented.

Images