Video jitter elimination method and system and electronic equipment

Abstract

Embodiments of the invention provide a video jitter elimination method and system, and an electronic device, and the method can obtain video data comprising continuous video frames, extract feature points of the video frames, and track point pairs corresponding to the feature points in a cross-frame manner. Constructing a feature point track matrix according to the point pairs, and decomposing the feature point track matrix into a feature matrix and a coefficient matrix; wherein the feature point track matrix comprises motion tracks of the feature points in different video frames, the feature matrix is used for representing directions of the motion tracks corresponding to the feature points, and the coefficient matrix is used for representing projection coefficients of the feature points in the directions. Performing filtering processing on the feature matrix, constructing a mask matrix used for describing feature point loss information, and generating a reconstruction track matrix through the feature matrix, the coefficient matrix and the mask matrix; according to the method, the geometric transformation parameters of the video data are set by reconstructing the track matrix, the jitter of the picture frame in the video data can be eliminated, and the video display effect is further improved.

Description

Technical Field

[0001] The present application relates to the field of image communication, and in particular to a video de-jittering method, system and electronic device. Background Art

[0002] Video de-shaking technology is an image processing technology that aims to reduce or eliminate unstable video images caused by camera shake. In scenes such as handheld shooting, drone aerial photography, and sports camera recording, unstable camera movement often causes obvious jitter in the video, affecting the viewing experience. Video de-shaking technology can compensate or correct camera movement by analyzing and processing motion information in the video, thereby outputting a more stable and smooth video image. For example, video de-shaking technology may involve multiple key technical links such as motion estimation, motion model establishment, motion smoothing, and video reconstruction.

[0003] Video de-shaking technology can be implemented based on camera motion models, which can be divided into two categories: 2D motion models and 3D motion models. 2D motion models include affine transformation and homography transformation, which can describe the motion of a spatial plane and are suitable for video de-shaking in flat scenes. 3D motion models mainly include SFM (Structure from Motion), which obtains 3D coordinates through 3D reconstruction to describe spatial motion and is more suitable for complex dynamic scenes than 2D motion models.

[0004] However, among the above-mentioned camera motion models, the 2D motion model can only describe a limited number of scenarios, such as lens zoom, parallax effect, etc., which cannot be directly described by the 2D camera model. Although the 3D motion model can describe the complete three-dimensional motion scene, the 3D motion model has a high computational complexity in the three-dimensional reconstruction process, making it difficult to run the 3D motion model on resource-constrained devices, thereby increasing the limitations of video de-shaking and affecting the user experience. Summary of the invention

[0005] The present application provides a video de-shaking method, system and electronic device to solve the problem of high limitations of video de-shaking.

[0006] In a first aspect, some embodiments of the present application provide a video de-shaking method, comprising the following steps:

[0007] Acquire video data, wherein the video data includes continuous video frames;

[0008] Extracting feature points of the video frame, and tracking point pairs corresponding to the feature points across frames;

[0009] Constructing a feature point trajectory matrix according to the point pairs, wherein the feature point trajectory matrix includes motion trajectories of the feature points in different video frames;

[0010] Decomposing the feature point trajectory matrix into a feature matrix and a coefficient matrix, wherein the feature matrix is ​​used to represent the direction of the motion trajectory corresponding to the feature point, and the coefficient matrix is ​​used to represent the projection coefficients of the feature point in the direction;

[0011] Performing filtering processing on the feature matrix and constructing a mask matrix, wherein the mask matrix is ​​generated based on loss information of the feature points in different video frames;

[0012] Generate a reconstruction trajectory matrix through the feature matrix, the coefficient matrix and the mask matrix;

[0013] The geometric transformation parameters of the video data are set according to the reconstructed trajectory matrix.

[0014] In some embodiments of the present application, extracting the feature points of the video frames includes: reading a preset number of video frames to obtain a subset of video frames; performing feature extraction on the video frames in the subset of video frames to obtain the feature points; the feature points are ORB feature points or feature points based on an accelerated segmentation test.

[0015] In some embodiments of the present application, decomposing the feature point trajectory matrix into a feature matrix and a coefficient matrix includes: projecting the feature point trajectory matrix to a target subspace, the target subspace including directions of a preset number of dimensions; decomposing the feature trajectory matrix into the feature matrix and the coefficient matrix according to the projection directions and projection coefficients of the feature point trajectories in the target subspace; the feature matrix is ​​a matrix of a preset number of dimensions.

[0016] In some embodiments of the present application, constructing a mask matrix includes: creating an initial mask matrix, which is the same type as the feature matrix; initializing the elements in the initial mask matrix to default values; if the tracked feature point is lost in the video frame, setting the elements at the corresponding positions of the initial mask matrix to target values; and outputting the initial mask matrix to generate the mask matrix.

[0017] In some embodiments of the present application, the reconstructed trajectory matrix is ​​generated according to the following formula:

[0018] M′ 2n×k ≈W⊙C 2n×r (E r×k K)

[0019] Among them, M′ 2n×k is the reconstruction trajectory matrix, W is the mask matrix, C 2n×r is the coefficient matrix, E r×k is the feature matrix after filtering, and K is the number of video frames.

[0020] In some embodiments of the present application, after the step of constructing a feature point trajectory matrix based on the point pairs, it also includes: reading sliding window parameters of a preset time domain sliding window, the sliding window parameters including the size of the time domain sliding window and the moving step of the time domain sliding window; sliding the time domain sliding window in the video data according to the sliding window parameters; if the current time domain sliding window is the first sliding window of the video data, executing the step of decomposing the feature point trajectory matrix into a feature matrix and a coefficient matrix; if the current time domain sliding window is not the first sliding window of the video data, updating the feature matrix and the coefficient matrix according to the overlapping elements of the current time domain sliding window and the reference time domain sliding window, the reference time domain sliding window being the previous time domain sliding window of the current time domain sliding window.

[0021] In some embodiments of the present application, updating the feature matrix and the coefficient matrix according to the overlapping elements of the current time domain sliding window and the reference time domain sliding window includes: detecting the overlapping elements of the first trajectory matrix and the second trajectory matrix, the first trajectory matrix is ​​the feature point trajectory matrix corresponding to the reference time domain sliding window, and the second trajectory matrix is ​​the feature point trajectory matrix corresponding to the current time domain sliding window; reading the first feature matrix and the first coefficient matrix corresponding to the first trajectory matrix; calculating the second feature matrix and the second coefficient matrix corresponding to the current time domain sliding window according to the overlapping elements, the first trajectory matrix and the second trajectory matrix; updating the feature matrix and the coefficient matrix through the second feature matrix and the second coefficient matrix, so as to generate the reconstructed trajectory matrix according to the updated feature matrix and the coefficient matrix.

[0022] In some embodiments of the present application, the calculation of the second characteristic matrix corresponding to the current time domain sliding window includes: constructing an objective function based on the LM algorithm, the overlapping elements, the first trajectory matrix and the second trajectory matrix, the objective function including a residual vector; solving the second characteristic matrix according to the objective function.

[0023] In a second aspect, some embodiments of the present application further provide a video de-shaking system, including a matrix construction module, a motion subspace restriction module, and a filtering and smoothing module, wherein:

[0024] A matrix construction module is configured to obtain video data, the video data including continuous video frames; extract feature points of the video frames, and track point pairs corresponding to the feature points across frames; construct a feature point trajectory matrix based on the point pairs, the feature point trajectory matrix including motion trajectories of the feature points in different video frames;

[0025] A motion subspace restriction module is configured to decompose the feature point trajectory matrix into a feature matrix and a coefficient matrix, wherein the feature matrix is ​​used to characterize the direction of the motion trajectory corresponding to the feature point, and the coefficient matrix is ​​used to characterize the projection coefficients of the feature point in the direction;

[0026] The filtering and smoothing module is configured to perform filtering processing on the feature matrix and construct a mask matrix, wherein the mask matrix is ​​generated based on the loss information of the feature points in different video frames; generate a reconstructed trajectory matrix through the feature matrix, the coefficient matrix and the mask matrix; and set the geometric transformation parameters of the video data according to the reconstructed trajectory matrix.

[0027] In a third aspect, some embodiments of the present application further provide an electronic device, comprising a processor, a memory and a bus; wherein the processor and the memory are communicatively connected via the bus, the memory stores computer program instructions executable by the processor, and the processor is configured to perform the following program steps:

[0028] Acquire video data, wherein the video data includes continuous video frames;

[0029] Extracting feature points of the video frame, and tracking point pairs corresponding to the feature points across frames;

[0030] Constructing a feature point trajectory matrix according to the point pairs, wherein the feature point trajectory matrix includes motion trajectories of the feature points in different video frames;

[0031] Decomposing the feature point trajectory matrix into a feature matrix and a coefficient matrix, wherein the feature matrix is ​​used to represent the direction of the motion trajectory corresponding to the feature point, and the coefficient matrix is ​​used to represent the projection coefficients of the feature point in the direction;

[0032] Performing filtering processing on the feature matrix and constructing a mask matrix, wherein the mask matrix is ​​generated based on loss information of the feature points in different video frames;

[0033] Generate a reconstruction trajectory matrix through the feature matrix, the coefficient matrix and the mask matrix;

[0034] The geometric transformation parameters of the video data are set according to the reconstructed trajectory matrix.

[0035] It can be seen from the above technical solutions that some embodiments of the present application provide a video de-jittering method, system, and electronic device, and the method can obtain video data including continuous video frames, and then extract feature points of the video frames, and track point pairs corresponding to the feature points across frames. Construct a feature point trajectory matrix based on the point pairs, and decompose the feature point trajectory matrix into a feature matrix and a coefficient matrix. Among them, the feature point trajectory matrix includes the motion trajectories of the feature points in different video frames, the feature matrix is ​​used to characterize the direction of the motion trajectory corresponding to the feature points, and the coefficient matrix is ​​used to characterize the projection coefficients of the feature points in the direction. Then perform filtering processing on the feature matrix, and construct a mask matrix for describing the lost information of the feature points, and generate a reconstructed trajectory matrix through the feature matrix, the coefficient matrix, and the mask matrix. The method sets the geometric transformation parameters of the video data by reconstructing the trajectory matrix, which can eliminate the jitter of the picture frames in the video data, thereby improving the display effect of the video data and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A flowchart of a video de-shaking method provided in some embodiments of the present application;

[0038] Figure 2 A schematic diagram of a process for extracting feature points provided in some embodiments of the present application;

[0039] Figure 3 A schematic diagram of a sliding window process provided in some embodiments of the present application;

[0040] Figure 4 A schematic diagram of a process flow of matrix block processing provided in some embodiments of the present application;

[0041] Figure 5 A schematic diagram of the relationship between feature point trajectory matrices provided in some embodiments of the present application;

[0042] Figure 6 A schematic diagram of the structure of a video de-shaking system provided in some embodiments of the present application;

[0043] Figure 7 An example diagram of a video de-shaking process provided in some embodiments of the present application;

[0044] Figure 8 A schematic diagram of the structure of an electronic device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the exemplary embodiments of the present application clearer, the technical solutions in the exemplary embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0046] Based on the exemplary embodiments shown in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete technical solution separately.

[0047] It should be understood that the terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented according to an order other than those given in the diagrams or descriptions of the embodiments of this application.

[0048] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0049] Video de-shaking technology can compensate or correct camera motion by analyzing and processing motion information in the video, thereby outputting a more stable and smooth video image. For example, video de-shaking technology may involve multiple key technical links such as motion estimation, motion model establishment, motion smoothing and video reconstruction. Among them, the motion model may include a 2D motion model and a 3D motion model.

[0050] In some embodiments, the 2D motion model can eliminate or reduce jitter in the video by directly transforming pixels or feature points on the image plane, such as affine transformation and homography transformation. Among them, affine transformation is a relatively simple linear transformation that can handle basic transformations such as translation, rotation, and scaling of images; homography transformation can describe the projection relationship of points in the same plane under different viewing angles, and is suitable for video jitter elimination in planar scenes.

[0051] However, 2D motion models are suitable for scenes that are flat or nearly flat, and cannot describe scenes with designed depth changes or complex three-dimensional motion, such as the different scales of objects at different distances when the lens is zoomed, that is, the parallax effect; or the complex motion changes caused by the rotation and movement of the camera in three-dimensional space. Therefore, for scenes with depth differences or complex geometric structures, 2D motion models cannot provide sufficient accuracy to accurately estimate and compensate for the camera's motion, resulting in jitter in the video output by the camera.

[0052] In some embodiments, the 3D motion model can more accurately describe the motion trajectory of the camera by constructing the three-dimensional geometric structure of the scene, such as the SFM method. SFM analyzes multiple frames in a video sequence, extracts and matches feature points, and then estimates the camera's pose (position and posture) and the three-dimensional structure of the scene. The 3D motion model can handle complex dynamic scenes and achieve a more accurate video shake removal effect.

[0053] However, 3D motion models require complex 3D reconstruction and camera pose estimation. These processes involve a lot of calculations and require high hardware computing power. They are difficult to run in real time on resource-constrained devices, which increases the limitations of video de-shaking.

[0054] To this end, some embodiments of the present application provide a video de-shaking method, which can describe the motion between video frames based on the tracking and transformation of 2D feature points. The method can describe complex spatial motion information without requiring excessive computation, thereby improving the video de-shaking effect and the applicability of the video de-shaking.

[0055] like Figure 1 As shown, in some embodiments, the video de-shaking method includes the following steps:

[0056] S1: Acquire video data, where the video data includes continuous video frames.

[0057] When performing video de-shaking, first obtain continuous video frames taken by the camera, that is, video data, where each video frame is a separate image frame. For ease of processing, when obtaining video data, a preset number of image frames at a certain moment can be read. For example, K frames of continuous video data at time t can be read from the video data for subsequent processing.

[0058] S2: Extract feature points of the video frame, and track point pairs corresponding to the feature points across frames.

[0059] After acquiring the video data, the video frames of the video data are read, and the feature points of each video frame are extracted, and the position of each feature point in different video frames is tracked across frames to form a pair of feature points. Among them, the feature point is a pixel point extracted according to the image features in the video frame, and a single feature point may include one pixel point or multiple pixel points.

[0060] like Figure 2 As shown, in some embodiments, when extracting feature points of video frames, a preset number of video frames may be read to obtain a subset of video frames. Feature extraction is then performed on the video frames in the subset of video frames to obtain feature points. The feature points are ORB (Oriented FAST and Rotated BRIEF) feature points or feature points based on accelerated segmentation test (Features from Accelerated Segment Test, FAST).

[0061] For ORB feature points, in some embodiments, the feature points of the video frame may be detected and described by the ORB algorithm, wherein the ORB algorithm first uses the FAST method to find the feature points, then calculates the direction of the feature points, and uses the rotated BRIEF descriptor to encode the feature points.

[0062] For FAST feature points, in some embodiments, the FAST algorithm can be used to examine a circle of pixels around each pixel, and to determine whether the central pixel is a corner point by comparing the brightness values ​​of the pixels. When extracting feature points, the FAST algorithm can determine the feature points by a preset.

[0063] In some embodiments, when tracking the point pairs corresponding to the feature points across frames, the feature points extracted from the video frames are tracked and processed by the LK (Lucas-Kanade) optical flow method. The LK optical flow method is an optical flow estimation method based on constant brightness and small area motion consistency. It assumes that the brightness of the object does not change between adjacent frames, and the movement of the object is smooth, that is, adjacent pixels have similar movements. Through these assumptions, the LK optical flow method can estimate the movement direction and speed of each pixel or feature point in the image.

[0064] S3: constructing a feature point trajectory matrix according to the point pairs, wherein the feature point trajectory matrix includes movement trajectories of the feature points in different video frames.

[0065] By extracting and tracking the point pairs of feature points, a corresponding feature point trajectory matrix can be constructed. The feature point trajectory matrix includes the motion trajectories of the feature points in different video frames. For example, K frames of video data are read, N feature points are extracted from the video frames of the K frames of video data, and the N feature points are tracked.

[0066] For example, since some feature points will appear and disappear continuously when tracking feature points, the constructed trajectory matrix M is a sparse matrix, as shown in the following formula:

[0067]

[0068] At this time, if the feature point trajectory matrix is ​​directly low-pass filtered M ′ =MK, which is equivalent to directly low-pass filtering each feature point trajectory. However, since there are no constraints between the trajectory points in M, the motion behavior of the feature points will be inconsistent after filtering, resulting in local distortion after video de-shaking. And M is a sparse matrix, the feature point pairs are constantly changing, and the convolution kernel of the filter is also required to change constantly. For this reason, the embodiment of the present application is provided with a motion subspace constraint to improve the above problems.

[0069] Among them, the motion subspace constraint is set based on the subspace constraint and the actual motion model of the camera. The subspace constraint assumes that when a moving affine camera shoots a rigid body, the feature trajectory observed by it can be projected into a low-dimensional subspace, so the feature point trajectory matrix can be decomposed into two low-dimensional subvectors. The camera is a perspective imaging model, and the object is non-rigid, so the feature motion trajectory is a nonlinear manifold, which can be approximated by a linear model with a specific low-dimensional subspace constraint.

[0070] S4: Decomposing the feature point trajectory matrix into a feature matrix and a coefficient matrix.

[0071] Therefore, after constructing the feature point trajectory matrix, the feature point trajectory matrix can be decomposed into a feature matrix and a coefficient matrix through the motion subspace constraint. The feature matrix is ​​used to characterize the direction of the motion trajectory corresponding to the feature point, and the coefficient matrix is ​​used to characterize the projection coefficient of the feature point in the direction.

[0072] In some embodiments, in a short period of time, the degrees of freedom of motion can be up to 9 dimensions at most, so the nonlinear manifold can be approximated by a linear model constrained by a 9-dimensional subspace.

[0073] Accordingly, in some embodiments, when decomposing the feature point trajectory matrix, the feature point trajectory matrix can be decomposed into a 9-dimensional feature matrix and a corresponding coefficient matrix. For example, the short-term feature trajectory matrix M is decomposed into an r=9-dimensional feature matrix E and its corresponding coefficient matrix C.

[0074] M 2n×k ≈C 2n×r E r×k

[0075] S5: performing filtering processing on the feature matrix and constructing a mask matrix, wherein the mask matrix is ​​generated based on loss information of the feature points in different video frames.

[0076] After decomposing the feature point trajectory matrix into a feature matrix and a coefficient matrix, filtering can be performed on the feature matrix, and a corresponding Mask matrix can be constructed to describe the loss information of feature points between different video frames through the Mask matrix. r×k Low-pass filtering is performed to preserve the motion dimensionality property.

[0077] In some embodiments, when constructing a mask matrix, an initial mask matrix may be created first. The initial mask matrix is ​​of the same type as the feature matrix, that is, the dimensions of the mask matrix and the feature matrix are the same. The elements of the initial mask matrix are then initialized to default values. If the tracked feature point is lost in the video frame, the element at the corresponding position of the initial mask matrix is ​​set to the target value, and the initial mask matrix is ​​output to generate the final mask matrix.

[0078] That is to say, first create a Mask matrix of the same type as the feature matrix, then initialize the Mask matrix, and then set the corresponding element values ​​according to the loss information of the feature points in the video frame to generate a binary Mask matrix. The Mask can be used to describe the inter-frame loss information of the feature points.

[0079] S6: Generate a reconstruction trajectory matrix through the feature matrix, the coefficient matrix and the mask matrix.

[0080] After filtering the feature matrix and constructing the mask matrix, a reconstructed trajectory matrix is ​​generated according to the filtered feature matrix, coefficient matrix and mask matrix, that is, the feature point trajectory matrix is ​​reconstructed by the above matrix to generate a reconstructed trajectory matrix.

[0081] In some embodiments, the reconstructed trajectory matrix is ​​generated according to the following formula:

[0082] M′ 2n×k ≈W⊙C 2n×r (E r×k K)

[0083] Among them, M′ 2n×k is the reconstruction trajectory matrix, W is the mask matrix, C 2n×r is the coefficient matrix, E r×k is the feature matrix after filtering, and K is the number of video frames.

[0084] S7: Setting geometric transformation parameters of the video data according to the reconstructed trajectory matrix.

[0085] After the reconstruction trajectory matrix is ​​generated, the geometric transformation parameters used for video jitter elimination can be calculated according to the reconstruction trajectory matrix, such as rotation, translation, etc. The calculated geometric transformation parameters are then applied to the video frames of the original video data to achieve stable display of the video data, thereby eliminating the jitter problem of the video.

[0086] The above steps S1-S7 are for de-shaking the video at a certain moment. Based on the above principle, in some embodiments, for the processing of long videos, all video frames of the video data can be analyzed and processed through a time domain sliding window to ensure the stable display of the long video.

[0087] For this reason, Figure 3 As shown, in some embodiments, after the step of constructing a feature point trajectory matrix according to the point pairs (i.e., step S3), the sliding window parameters of the preset time domain sliding window are read. Among them, the sliding window parameters include the size of the time domain sliding window and the moving step of the time domain sliding window. Then the time domain sliding window is slid in the video data according to the sliding window parameters. If the current time domain sliding window is the first sliding window of the video data, the step of decomposing the feature point trajectory matrix into a feature matrix and a coefficient matrix is ​​performed, i.e., step S4; if the current time domain sliding window is not the first sliding window of the video data, the feature matrix and the coefficient matrix are updated according to the overlapping elements of the current time domain sliding window and the reference time domain sliding window, and the reference time domain sliding window is the previous time domain sliding window of the current time domain sliding window.

[0088] That is, when processing long videos, the sliding window technology can be used to divide the video data into multiple shorter segments (i.e., the time domain sliding window) through the time domain sliding window, and process the data in each sliding window. In addition, since the main matrix of the feature point trajectory matrix remains unchanged during the sliding process, the factor decomposition of the feature point trajectory matrix in the new window can be accelerated by overlapping the new sliding window with the old sliding window.

[0089] For example, the sliding window parameters may include a size of K and a moving step of δ; if k=50 and δ=5, the number of frames of the time domain sliding window is 50 video frames, and the time domain sliding window moves 5 frames each time. For the first 50 frames of feature point trajectory matrices, the above S4 method is used to perform factor decomposition and corresponding spatial filtering and smoothing processing.

[0090] like Figure 4As shown, in some embodiments, when the feature matrix and the coefficient matrix are updated according to the overlapping elements of the current time domain sliding window and the reference time domain sliding window, the overlapping elements of the first trajectory matrix and the second trajectory matrix are detected. Among them, the first trajectory matrix is ​​the feature point trajectory matrix (old feature point trajectory matrix) corresponding to the reference time domain sliding window, and the second trajectory matrix is ​​the feature point trajectory matrix (new feature point trajectory matrix) corresponding to the current time domain sliding window. Then read the first feature matrix and the first coefficient matrix corresponding to the first trajectory matrix, and calculate the second feature matrix and the second coefficient matrix corresponding to the current time domain sliding window according to the overlapping elements, the first trajectory matrix and the second trajectory matrix, and update the feature matrix and the coefficient matrix through the second feature matrix and the second coefficient matrix, so as to generate a reconstructed trajectory matrix according to the updated feature matrix and the coefficient matrix.

[0091] For example, the sliding window parameter is K=50, and the moving step is δ=5. The first trajectory matrix is ​​M 0 , after moving 5 frames, the second trajectory matrix is ​​defined as M 1 The relationship between the two is as follows Figure 5 As shown, that is:

[0092]

[0093] Combined with the factor decomposition formula, we get M 1 The formula is:

[0094]

[0095] Among them, C 1 and E 1 M 0 The known vector after factorization, so C 2 and E 2 The solution expression is as follows:

[0096] C 2 =A 21 E 1T (E 1 E 1T ) -1

[0097]

[0098] Among them, due to E 1 The dimension is small, and the inverse calculation is small, so C 2 can be solved analytically; and E 2 The solution can be obtained analytically or by optimization.

[0099] Therefore, for E 2In some embodiments, when calculating the second characteristic matrix corresponding to the current time domain sliding window, an objective function can be constructed based on the LM algorithm, overlapping elements, the first trajectory matrix and the second trajectory matrix, and then the second characteristic matrix is ​​solved according to the objective function. The objective function includes the residual vector. That is, the second characteristic matrix E is optimized by the LM algorithm. 2 to improve the video de-jittering efficiency.

[0100] For example, Figure 5 The feature point trajectory matrix based on the time domain sliding window is solved using the LM optimization algorithm, and the optimization objective function is:

[0101]

[0102] Where F is the residual vector.

[0103] pass Figure 3-4 The factor decomposition shown, as well as steps S1-S7, can continuously transform the video data to achieve the de-shaking effect. The description of the 3D motion model can be improved and complex problems such as parallax can be solved by using subspace constraints and time domain sliding windows; at the same time, based on the matrix block processing method, the calculation amount of the matrix decomposition motion can be simplified, so that the de-shaking method can be run in real time on a variety of devices to reduce the limitations of video de-shaking.

[0104] Based on the above video de-shaking method, some embodiments of the present application also provide a video de-shaking system, which can be used to execute the video de-shaking method provided in the above embodiment. Figure 6 As shown, the video de-shaking system includes a matrix construction module 601, a motion subspace restriction module 602 and a filtering and smoothing module 603, wherein:

[0105] The matrix construction module 601 is configured to obtain video data, the video data including continuous video frames; extract feature points of the video frames, and track point pairs corresponding to the feature points across frames; construct a feature point trajectory matrix based on the point pairs, the feature point trajectory matrix including motion trajectories of the feature points in different video frames;

[0106] The motion subspace restriction module 602 is configured to decompose the feature point trajectory matrix into a feature matrix and a coefficient matrix, wherein the feature matrix is ​​used to represent the direction of the motion trajectory corresponding to the feature point, and the coefficient matrix is ​​used to represent the projection coefficient of the feature point in the direction;

[0107] The filtering and smoothing module 603 is configured to perform filtering processing on the feature matrix and construct a mask matrix, wherein the mask matrix is ​​generated based on the loss information of the feature points in different video frames; generate a reconstructed trajectory matrix through the feature matrix, the coefficient matrix and the mask matrix; and set the geometric transformation parameters of the video data according to the reconstructed trajectory matrix.

[0108] like Figure 7 As shown, the above-mentioned video de-shaking method and video de-shaking system are explained below through an example.

[0109] Figure 7 In the paper, the video de-shaking method is represented as four parts: A is the construction of the feature point trajectory matrix, B is the motion subspace constraint, C is the motion subspace filter smoothing, and D is the time domain sliding window factor decomposition, where:

[0110] A-Construction of feature point trajectory matrix, including:

[0111] a1: Read k frames of continuous video data at time t.

[0112] a2: Extract ORB or FAST feature points of k frames of video data respectively.

[0113] a3: Use the LK optical flow method to track the N feature points extracted in A2.

[0114] a4: Use the N feature points of the above k frames to construct the feature point trajectory matrix M.

[0115] B-Motion subspace constraints, including:

[0116] b1: Decompose the short-term feature point trajectory matrix M into an r=9-dimensional feature matrix E and its corresponding coefficient matrix C.

[0117] M 2n×k ≈C 2n×r E r×k

[0118] C-Motion subspace filtering and smoothing, including:

[0119] c1: feature matrix E r×k Perform filtering.

[0120] c2: Construct a Mask matrix to describe the missing information of feature points.

[0121] c3: Reconstruct the feature point trajectory matrix after filtering to generate the reconstructed trajectory matrix M ′ .

[0122] M ′ 2n×k ≈W⊙C 2n×r(E r×k K)

[0123] D-time domain sliding window factorization, including:

[0124] d1: Sliding window size K and moving step size δ settings.

[0125] d2: Factor decomposition of the first trajectory matrix: For the first k-frame trajectory matrix, the above b3 method is used to perform factor decomposition and corresponding spatial filtering and smoothing. The number of feature points that completely cover the k-frame image is m, which is required to be greater than r / 2.

[0126]

[0127] d3: Decomposition and solution of the second feature point trajectory: The first trajectory matrix is ​​M 0 , after moving δ frames, the second trajectory matrix is ​​defined as M 1 ,in:

[0128]

[0129] Combined with the factor decomposition formula, we can get M 1 :

[0130]

[0131] It turns out that:

[0132] C 2 =A 21 E 1T (E 1 E 1T ) -1

[0133]

[0134] E 2 Using the LM optimization algorithm to solve, the optimization objective function is:

[0135]

[0136] d4: Get the reconstructed trajectory matrix, filter and reconstruct it through the method of C, and finally output the reconstructed trajectory matrix. The reconstructed trajectory matrix can be used to perform geometric transformation processing for video de-shaking.

[0137] Based on the above-mentioned video de-shake method and video de-shake system, some embodiments of the present application also provide an electronic device 500, including the video de-shake system described in the above-mentioned embodiments.

[0138] In some embodiments, Figure 8As shown, the electronic device 500 includes: at least one processor 501, at least one communication interface 502, at least one memory 503 and at least one bus 504. Among them, the bus 504 is used to realize the direct connection and communication of these components, the communication interface 502 is used to communicate signaling or data with other node devices, and the memory 503 stores computer program instructions that can be executed by the processor 501. When the electronic device 500 is running, the processor 501 communicates with the memory 503 through the bus 504, the processor 501 can call the computer program stored in the memory 503, and the execution of the computer program can realize the video de-jittering method provided in the embodiment of the present application.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0140] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A video de-shaking method, characterized in that: include: Acquire video data, wherein the video data includes continuous video frames; Extracting feature points of the video frame, and tracking point pairs corresponding to the feature points across frames; Constructing a feature point trajectory matrix according to the point pairs, wherein the feature point trajectory matrix includes motion trajectories of the feature points in different video frames; Decomposing the feature point trajectory matrix into a feature matrix and a coefficient matrix, wherein the feature matrix is ​​used to represent the direction of the motion trajectory corresponding to the feature point, and the coefficient matrix is ​​used to represent the projection coefficients of the feature point in the direction; Performing filtering processing on the feature matrix and constructing a mask matrix, wherein the mask matrix is ​​generated based on loss information of the feature points in different video frames; Generate a reconstruction trajectory matrix through the feature matrix, the coefficient matrix and the mask matrix; The geometric transformation parameters of the video data are set according to the reconstructed trajectory matrix.

2. The video de-shaking method according to claim 1, characterized in that: The extracting feature points of the video frame comprises: Reading a preset number of video frames to obtain a subset of video frames; Feature extraction is performed on the video frames in the video frame subset to obtain the feature points; the feature points are ORB feature points or feature points based on an accelerated segmentation test.

3. The video de-shaking method according to claim 1, characterized in that: The step of decomposing the feature point trajectory matrix into a feature matrix and a coefficient matrix comprises: Projecting the feature point trajectory matrix to a target subspace, the target subspace including directions of a preset number of dimensions; According to the projection direction and projection coefficient of the feature point trajectory in the target subspace, the feature trajectory matrix is ​​decomposed into the feature matrix and the coefficient matrix; the feature matrix is ​​a matrix of a preset number of dimensions.

4. The video de-shaking method according to claim 1, characterized in that: The constructing of the mask matrix comprises: Creating an initial mask matrix, wherein the initial mask matrix is ​​of the same type as the feature matrix; Initializing the elements in the initial mask matrix to default values; If the tracked feature point is lost in the video frame, the element at the corresponding position of the initial mask matrix is ​​set to a target value; The initial mask matrix is ​​output to generate the mask matrix.

5. The video de-shaking method according to claim 1, characterized in that: The reconstructed trajectory matrix is ​​generated according to the following formula: M′ 2n×k ≈W⊙C 2n×r (E r×k K) Among them, M′ 2n×k is the reconstruction trajectory matrix, W is the mask matrix, C 2n×r is the coefficient matrix, E r×k is the feature matrix after filtering, and K is the number of video frames.

6. The video de-shaking method according to claim 1, characterized in that: After the step of constructing a feature point trajectory matrix according to the point pairs, the method further includes: Reading a preset sliding window parameter of a time domain sliding window, wherein the sliding window parameter includes a size of the time domain sliding window and a moving step of the time domain sliding window; Sliding the time domain sliding window in the video data according to the sliding window parameter; If the current time domain sliding window is the first sliding window of the video data, executing the step of decomposing the feature point trajectory matrix into a feature matrix and a coefficient matrix; If the current time domain sliding window is not the first sliding window of the video data, the feature matrix and the coefficient matrix are updated according to overlapping elements of the current time domain sliding window and a reference time domain sliding window, and the reference time domain sliding window is a time domain sliding window before the current time domain sliding window.

7. The video de-shaking method according to claim 6, characterized in that: The updating of the feature matrix and the coefficient matrix according to the overlapping elements of the current time domain sliding window and the reference time domain sliding window includes: Detecting overlapping elements of a first trajectory matrix and a second trajectory matrix, wherein the first trajectory matrix is ​​a feature point trajectory matrix corresponding to the reference time domain sliding window, and the second trajectory matrix is ​​a feature point trajectory matrix corresponding to the current time domain sliding window; Reading a first characteristic matrix and a first coefficient matrix corresponding to the first trajectory matrix; Calculate a second feature matrix and a second coefficient matrix corresponding to the current time domain sliding window according to the overlapping elements, the first trajectory matrix and the second trajectory matrix; The feature matrix and the coefficient matrix are updated by using the second feature matrix and the second coefficient matrix, so as to generate the reconstructed trajectory matrix according to the updated feature matrix and the coefficient matrix.

8. The video de-shaking method according to claim 7, characterized in that: The calculating the second characteristic matrix corresponding to the current time domain sliding window includes: constructing an objective function based on the LM algorithm, the overlapping elements, the first trajectory matrix and the second trajectory matrix, wherein the objective function includes a residual vector; The second characteristic matrix is ​​solved according to the objective function.

9. A video de-shaking system, characterized in that: include: A matrix construction module is configured to acquire video data, the video data comprising continuous video frames; extract feature points of the video frames, and track point pairs corresponding to the feature points across frames; Constructing a feature point trajectory matrix according to the point pairs, wherein the feature point trajectory matrix includes motion trajectories of the feature points in different video frames; A motion subspace restriction module is configured to decompose the feature point trajectory matrix into a feature matrix and a coefficient matrix, wherein the feature matrix is ​​used to characterize the direction of the motion trajectory corresponding to the feature point, and the coefficient matrix is ​​used to characterize the projection coefficients of the feature point in the direction; The filtering and smoothing module is configured to perform filtering processing on the feature matrix and construct a mask matrix, wherein the mask matrix is ​​generated based on the loss information of the feature points in different video frames; generate a reconstructed trajectory matrix through the feature matrix, the coefficient matrix and the mask matrix; and set the geometric transformation parameters of the video data according to the reconstructed trajectory matrix.

10. An electronic device, characterized in that: Includes processor, memory and bus; The processor and the memory are communicatively connected via the bus; The memory stores computer program instructions executable by the processor, the processor being configured to: Acquire video data, wherein the video data includes continuous video frames; Extracting feature points of the video frame, and tracking point pairs corresponding to the feature points across frames; Constructing a feature point trajectory matrix according to the point pairs, wherein the feature point trajectory matrix includes motion trajectories of the feature points in different video frames; Decomposing the feature point trajectory matrix into a feature matrix and a coefficient matrix, wherein the feature matrix is ​​used to represent the direction of the motion trajectory corresponding to the feature point, and the coefficient matrix is ​​used to represent the projection coefficients of the feature point in the direction; Performing filtering processing on the feature matrix and constructing a mask matrix, wherein the mask matrix is ​​generated based on loss information of the feature points in different video frames; Generate a reconstruction trajectory matrix through the feature matrix, the coefficient matrix and the mask matrix; The geometric transformation parameters of the video data are set according to the reconstructed trajectory matrix.

Images