Video real-time anti-shake method and related equipment

Abstract

The invention provides a video real-time anti-shake method and related equipment. The method comprises the following steps: acquiring a motion affine matrix of each target frame in a video sequence relative to a previous frame; performing parameter decomposition on the motion affine matrix to obtain each motion parameter; for each motion parameter, acquiring an original motion track corresponding to the motion parameter and a smoothed motion track relative to the original motion track; and carrying out real-time anti-shake processing on the target frame based on the smoothed motion trail corresponding to each motion parameter and the corresponding original motion trail. In the scheme, real-time anti-shake can be carried out on the current target frame image by utilizing the smoothed motion trail and the corresponding original motion trail, so that anti-shake delay is eliminated, and real-time anti-shake is realized.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, specifically to a real-time video stabilization method and related equipment. Background Technology

[0002] During video recording, camera shake is unavoidable, resulting in shaky footage. Video stabilization algorithms can eliminate this shake. These algorithms typically consist of three parts: motion estimation, motion smoothing, and motion compensation. Motion estimation obtains the affine matrix between two adjacent frames, which is then decomposed using parametric decomposition to obtain motion parameters for subsequent motion smoothing and compensation.

[0003] Most current image stabilization algorithms construct motion affine matrices that apply to the previous frame. This results in a one-frame delay in the stabilized image, leading to insufficient real-time performance of video stabilization.

[0004] In summary, how to achieve real-time video stabilization is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a real-time video stabilization method and related equipment to achieve the purpose of real-time stabilization.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of this invention discloses a real-time video stabilization method, the method comprising:

[0008] Obtain the motion affine matrix of each target frame in the video sequence relative to the previous frame;

[0009] The motion affine matrix is ​​decomposed parametrically to obtain the various motion parameters;

[0010] For each of the aforementioned motion parameters, obtain its corresponding original motion trajectory and a smoothed motion trajectory relative to the original motion trajectory;

[0011] Based on the smoothed motion trajectory corresponding to each motion parameter and the corresponding original motion trajectory, the target frame is subjected to real-time anti-shake processing.

[0012] Preferably, the real-time image stabilization process for the target frame based on the smoothed motion trajectory corresponding to each motion parameter and the corresponding original motion trajectory includes:

[0013] Based on the smoothed motion trajectory and the original motion trajectory corresponding to each motion parameter, a correction formula for each motion parameter acting on the target frame is calculated.

[0014] The target frame is subjected to real-time stabilization based on the correction formula.

[0015] Preferably, the real-time image stabilization processing of the target frame based on the correction formula includes:

[0016] The target frame is subjected to real-time image stabilization using the following formula:

[0017]

[0018]

[0019] ;

[0020] in,( ) refers to the pixel points in the target frame ( , Pixels after real-time image stabilization and This represents the original trajectory of the translation. and This represents the smoothed trajectory of the translation. and This represents the original trajectory of the sheared motion. and This represents the smoothed motion trajectory after shearing. and This represents the original motion trajectory after scaling. and This represents the smoothed motion trajectory after scaling. Represents the original trajectory of rotation. This represents the smoothed trajectory of the rotation.

[0021] Preferably, the step of parametrically decomposing the motion affine matrix to obtain various motion parameters includes:

[0022] The motion affine matrix is ​​decomposed parametrically using the quasi-Newton method to obtain the various motion parameters.

[0023] Preferably, the step of performing parametric decomposition on the affine matrix of motion based on the quasi-Newton method to obtain various motion parameters includes:

[0024] The motion affine matrix is ​​transformed to obtain the motion affine matrix to be decomposed; the motion affine matrix to be decomposed represents the affine transformation relationship from the previous frame to the target frame when the center of the frame is at the origin.

[0025] In the DFP algorithm to be initialized, an initial transformation vector and an initial approximation matrix are set; the approximation matrix is ​​a positive definite symmetric matrix that approximates the Hessian inverse matrix, and the initial transformation vector is composed of multiple initial first motion parameters;

[0026] Using the affine matrix to be decomposed and the initialized transformation vector as algorithm parameters, the initialized DFP algorithm is run to output multiple reference values ​​corresponding to the first motion parameters;

[0027] The second motion parameter is calculated using reference values ​​corresponding to multiple first motion parameters; wherein each motion parameter includes multiple first motion parameters and the second motion parameter, the second motion parameter is one of scaling parameter, shearing parameter, translation parameter and rotation parameter, and the multiple first motion parameters are parameters other than the second motion parameter among scaling parameter, shearing parameter, translation parameter and rotation parameter.

[0028] Preferably, the step of running the initialized DFP algorithm using the affine matrix to be decomposed and the initialized transformation vector as algorithm parameters to output multiple reference values ​​corresponding to the first motion parameters includes:

[0029] Based on the affine matrix to be decomposed and the initialized transformation vector, the gradient vector is calculated, and based on the gradient vector and the latest approximation matrix, the search direction is calculated.

[0030] Based on the search direction, the step size vector is calculated, and the transformation vector is updated using the step size vector;

[0031] The gradient vector is updated based on the updated transformation vector;

[0032] Determine whether the size of the updated gradient vector is less than the precision threshold;

[0033] If so, then multiple reference values ​​corresponding to the first motion parameters are obtained from the updated transformation vector;

[0034] If not, then correct the approximation matrix, and calculate the gradient vector for the next iteration based on the affine matrix to be decomposed and the updated transformation vector, and return to execute the step of calculating the search direction based on the gradient vector and the latest approximation matrix.

[0035] Preferably, correcting the approximate matrix includes:

[0036] The difference between the updated gradient vector and the original gradient vector is calculated to obtain the gradient vector difference value;

[0037] The corrected approximation matrix is ​​calculated based on the difference between the approximation matrix, the step size vector, and the gradient vector.

[0038] A second aspect of this invention discloses a real-time video stabilization device, the device comprising:

[0039] The acquisition unit is used to acquire the motion affine matrix of each target frame in the video sequence relative to the previous frame.

[0040] The decomposition unit is used to perform parametric decomposition on the motion affine matrix to obtain various motion parameters;

[0041] The trajectory calculation unit is used to obtain the original motion trajectory and the smoothed motion trajectory relative to the original motion trajectory for each of the motion parameters.

[0042] The image stabilization unit is used to perform real-time image stabilization processing on the target frame based on the smoothed motion trajectory corresponding to each of the motion parameters and the corresponding original motion trajectory.

[0043] A third aspect of this invention discloses an electronic device, comprising:

[0044] Memory, used to store computer programs;

[0045] A processor is configured to execute the computer program to implement the method as described in any of the first aspects of the embodiments of the present invention.

[0046] A fourth aspect of the present invention discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the method described in any of the first aspects of the present invention.

[0047] The fifth aspect of this application discloses a computer program product, wherein when the computer program product is executed by a processor, it implements the method described in any of the first aspects of this invention.

[0048] Based on the above embodiments of the present invention, a real-time video stabilization method and related equipment are provided. This method involves obtaining the motion affine matrix of each target frame in a video sequence relative to the previous frame; performing parametric decomposition on the motion affine matrix to obtain various motion parameters; for each motion parameter, obtaining its corresponding original motion trajectory and a smoothed motion trajectory relative to the original motion trajectory; and performing real-time stabilization processing on the target frame based on the smoothed motion trajectory and the corresponding original motion trajectory for each motion parameter. In this solution, the motion parameters of the current target frame relative to the previous frame, along with its corresponding original motion trajectory and the smoothed motion trajectory relative to the original motion trajectory, are applied to the current target frame for stabilization. Therefore, the target frame image after stabilization is real-time, reducing the delay by one frame, thereby achieving the purpose of real-time stabilization and solving the technical problem of insufficient real-time performance in existing video stabilization technologies. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0050] Figure 1 This is an optional flowchart of a real-time video stabilization method disclosed in an embodiment of the present invention;

[0051] Figure 2 This is an optional structural diagram of a real-time video stabilization device disclosed in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of an optional structure of an electronic device disclosed in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0055] First, the technical terms appearing in this application will be explained as follows:

[0056] Image stabilization algorithm: An algorithm that can effectively reduce image shake caused by camera shake, thereby improving video stability and quality.

[0057] Affine transformation: A linear transformation that maps two-dimensional coordinates (x, y) to another set of two-dimensional coordinates (u, v), resulting in a motion affine matrix. In this transformation, the relative positions and properties of the coordinate points remain unchanged. Affine transformations mainly include operations such as rotation, scaling, translation, and shearing.

[0058] Parametric decomposition: The motion affine matrix is ​​obtained through a series of affine transformations such as rotation, scaling, translation, and shearing. To obtain the actual value of each motion parameter, it is necessary to perform parametric decomposition on the motion affine matrix.

[0059] Smoothing trajectory: Removes jitter from the original motion trajectory, making the motion trajectory smooth.

[0060] As the background technology indicates, the speed and accuracy of parameter decomposition directly affect the effectiveness of image stabilization. Currently, the main parameter decomposition methods are gradient descent and Newton's method. Among them, gradient descent is the simplest and most commonly used optimization algorithm. The core idea of ​​gradient descent is to adjust the parameters along the gradient direction to gradually reduce the objective function until a minimum value is found.

[0061] Gradient descent updates parameters through the following steps:

[0062] (1)

[0063] in, It is the value of x in the k-th iteration. It is the step size of the iteration. Represents the gradient operator, This represents the objective function; although the approach is simple, the convergence speed is slow when approaching the minimum value, which will seriously affect the real-time performance and effectiveness of the stabilization.

[0064] Newton's method is a numerical method for solving nonlinear equations. The basic idea of ​​Newton's method is to perform a second-order Taylor expansion of the objective function near the existing minimum estimate, thereby finding the next estimate of the minimum. It combines first-order and second-order information, enabling rapid convergence. The update formula is as follows:

[0065] (2)

[0066] in, The Hessian matrix of a function is a matrix composed of the second-order partial derivatives of the function, defined as follows:

[0067] (3)

[0068] Although Newton's method converges quickly, each step requires solving for the inverse of the Hessian matrix of the objective function, which is computationally complex. Furthermore, the Hessian matrix must be positive definite; otherwise, the algorithm may fail to converge.

[0069] In summary, there is a need for a method that has fast convergence speed and is computationally simple to perform parameter decomposition, which can quickly and accurately obtain motion parameters.

[0070] Furthermore, most current image stabilization algorithms apply the parameter decomposition to the frame preceding the two adjacent frames. This results in a one-frame delay in the stabilized image, leading to insufficient real-time performance of the stabilization.

[0071] To address at least one of the aforementioned technical problems, this invention discloses a real-time video stabilization method and related equipment. In this solution, parameter decomposition is performed based on the quasi-Newton method and the motion affine matrix. This method not only converges faster than the gradient descent method but also eliminates the need to calculate the second-order partial derivatives of Newton's method. Furthermore, a positive definite symmetric matrix that approximates the Hessian inverse matrix is ​​constructed, enabling faster and more accurate decomposition of motion parameters. In addition, this method applies the parameter decomposition to the current target frame for stabilization, eliminating a one-frame delay, thereby achieving the goals of reducing computational complexity and real-time stabilization.

[0072] like Figure 1 The diagram shows a flowchart of a real-time video stabilization method disclosed in an embodiment of the present invention. The method includes the following steps:

[0073] Step S101: Obtain the motion affine matrix of each target frame in the video sequence relative to the previous frame.

[0074] In step S101, the video sequence is a video sequence captured in real time by the shooting device. The video sequence contains multiple consecutive frames of images, including a target frame. The target frame is the currently captured video / image frame, which is to be subjected to image stabilization processing. This solution aims to perform real-time image stabilization processing on the currently captured target frame.

[0075] In the specific implementation of step S101, for each target frame (image) of the video sequence, the corner points of the previous frame (image) can be extracted using a preset corner point extraction algorithm, and the corner points of the target frame image can be obtained based on the optical flow method and the corner points of the previous frame image; the motion affine matrix is ​​estimated based on the corner points of the previous frame image and the corner points of the target frame image.

[0076] For example, the affine transformation from the previous frame image to the target frame image is represented as:

[0077] (4)

[0078] The affine matrix can be obtained by multiplying the motion vectors such as translation, rotation, scaling, and shearing in sequence, i.e., formula (5):

[0079] (5)

[0080] Then, by combining the motion vectors in (5), we have:

[0081] (6)

[0082] in, , This is the scaling parameter in the motion parameters. , The shear parameter is a component of the motion parameters. , The translation parameter is one of the motion parameters. For the rotational parameters in the motion parameters, ( , () represents the pixels in the target frame.

[0083] Step S102: Perform parametric decomposition on the motion affine matrix to obtain each motion parameter.

[0084] In step S102, the motion affine matrix can be decomposed parametrically based on the quasi-Newton method to obtain the various motion parameters.

[0085] It should be noted that the quasi-Newton method not only converges faster than the gradient descent method, but also does not require the calculation of the second-order partial derivatives of Newton's method. Furthermore, it constructs a positive definite symmetric matrix (called the approximation matrix) that can approximate the Hessian inverse matrix, which can decompose the motion parameters more quickly and accurately. Thus, it provides a parameter decomposition scheme that balances accuracy and processing speed.

[0086] Of course, in other examples, other methods, such as Newton's method or gradient descent, can be used to parametrically decompose the affine matrix of motion.

[0087] The iterative process of the quasi-Newton method involves the correction of the approximate matrix. The following will explain in detail the derivation of the quasi-Newton conditions (i.e., the quasi-Newton equations) and the construction process of the correction matrix:

[0088] Suppose that after k+1 iterations At this point, the objective function exist Performing a Taylor expansion nearby and taking a second-order approximation, we obtain:

[0089] (7)

[0090] A gradient operator is applied simultaneously in formula (7). We can obtain:

[0091] (8)

[0092] Where H represents the Hessian matrix, which is taken in formula (8) After organizing, we can obtain:

[0093] (9)

[0094] If we introduce a notation, that is, let:

[0095] (10)

[0096] Then formula (10) can be written compactly as:

[0097] (11)

[0098] Formula (11) is the quasi-Newton condition, which applies to the Hessian matrix during the iteration process. To impose constraints, therefore, for Make approximations Formula (12) can be used as a guide:

[0099] (12)

[0100] From the above, we can conclude that in order to iterate... Only calculation is needed. That's it. Commonly used quasi-Newton methods include the DFP (Davidon-Fletcher-Powell) algorithm and the BFGS (Broyden–Fletcher–Goldfarb–Shanno) algorithm. The DFP algorithm will be used below to... Approximate calculation.

[0101] The core of the DFP algorithm (quasi-Newton method) is: to iteratively calculate... Make an approximation. The iteration format is:

[0102] (13)

[0103] Among them, the initial approximation matrix It is usually taken as the identity matrix. Therefore, the key is the correction matrix at each step. How to construct it.

[0104] Assumption Possibly with An association occurs. Here, the "method of undetermined" is used, meaning that first... The form to be determined is to be in a certain form, and then the derivation is carried out in combination with the quasi-Newtonian condition (11).

[0105] Will Pending:

[0106] (14)

[0107] in, For undetermined coefficients, Let be the vector to be determined. Formally, this undetermined formula at least guarantees the matrix... Symmetry.

[0108] Substituting formula (14) into formula (13) and combining it with the guiding condition (12), we can obtain:

[0109] (15)

[0110] Formula (15) can be rewritten as:

[0111] (16)

[0112] Among them, the parentheses and Since there are two numbers, we can assign them the following simple values:

[0113] (17)

[0114] Right now:

[0115] (18)

[0116] Where vector It remains to be determined. Substituting formula (17) into formula (16), we get:

[0117] (19)

[0118] For formula (19) to hold true, we can directly take:

[0119] (20)

[0120] Substituting formula (20) into formula (18), we get:

[0121] (twenty one)

[0122] In formula (21), the second equation uses... Symmetry.

[0123] Substituting equations (20) and (21) into equation (14), we obtain the correction matrix. :

[0124] (twenty two)

[0125] In summary, the complete description of the DFT algorithm is as follows:

[0126] (1) Given an initial value (i.e., the initial transformation parameters) and precision threshold And let the initial approximate matrix k is the number of iterations.

[0127] (2) Confirm the search direction . This is the gradient vector.

[0128] (3) Obtain the step size using a one-dimensional search. ,make , .

[0129] (4) If The algorithm ends when the condition is met.

[0130] (5) Calculation .

[0131] (6) Calculation .

[0132] make Proceed to step (2).

[0133] Correspondingly, the specific implementation of step S102 includes the following steps:

[0134] Step S201: Transform the motion affine matrix to obtain the motion affine matrix to be decomposed.

[0135] Among them, the motion affine matrix to be decomposed represents the affine transformation relationship from the previous frame to the target frame when the center of the frame is at the origin of the coordinate system.

[0136] For example, formula (6) moves the origin of the image to the center:

[0137] (twenty three)

[0138] Where cw and ch are the x and y coordinates of the image center, respectively. Let , , , ;

[0139] At this point, the affine matrix of the motion to be decomposed is known. The subsequent calculations can be performed using the quasi-Newton method combined with the affine matrix of the motion to be decomposed. , , , , then calculate and Solve ,calculate and Solve .

[0140] Step S202: Set the initial transformation vector and the initial approximation matrix in the DFP algorithm to be initialized.

[0141] The approximation matrix is ​​a positive definite symmetric matrix that approximates the Hessian inverse matrix, and the initialized transformation vector consists of multiple initial first motion parameters.

[0142] In step S202, as described in the complete DFT algorithm above, given initial values... (i.e., the initial transformation parameters) and precision threshold And let the initial approximate matrix k is the number of iterations.

[0143] For example, .

[0144] Step S203: Using the affine matrix to be decomposed and the initialized transformation vector as algorithm parameters, run the initialized DFP algorithm to output multiple reference values ​​corresponding to the first motion parameters.

[0145] Step S204: Calculate the second motion parameters using the reference values ​​corresponding to the multiple first motion parameters.

[0146] The motion parameters include multiple first motion parameters and second motion parameters. The second motion parameter is one of the scaling parameter, shearing parameter, translation parameter, and rotation parameter. The multiple first motion parameters are parameters other than the second motion parameter among the scaling parameter, shearing parameter, translation parameter, and rotation parameter.

[0147] In the specific implementation of steps S203 and S204 above, each motion parameter can be used as a first motion parameter and a second motion parameter. When a certain motion parameter is used as the second motion parameter to be solved, all other motion parameters besides the second motion parameter can be used as first motion parameters, and the initial values ​​of these first motion parameters are iteratively updated to obtain the actual reference values ​​of the first motion parameters that finally meet the accuracy requirements. Then, the final second motion parameter is obtained through these actual reference values.

[0148] In some examples, step S203 above may include the following steps:

[0149] Step S301: Calculate the gradient vector based on the affine matrix of the motion to be decomposed and the initialized transformation vector.

[0150] It should be noted that the loss function in the DFP algorithm iteration process is as follows:

[0151]

[0152] make , , , .

[0153] So:

[0154]

[0155]

[0156]

[0157]

[0158]

[0159]

[0160]

[0161]

[0162]

[0163]

[0164] In step S301, the gradient vector exist The value below.

[0165] Step S302: Calculate the search direction based on the gradient vector and the latest approximation matrix.

[0166] In step S302, the search direction can be confirmed. .

[0167] in, For the search direction, This is the current approximate matrix.

[0168] Step S303: Based on the search direction, calculate the step size vector and update the transformation vector using the step size vector;

[0169] The step size can be obtained using a one-dimensional search: ,make .in, This is the step size vector. The transformation vector is further updated using the step size vector and the step size itself.

[0170] .

[0171] Step S304: Update the gradient vector based on the updated transformation vector.

[0172] In step S304, the updated gradient vector exist The value below.

[0173] Step S305: Determine whether the size of the updated gradient vector is less than the precision threshold. If yes, proceed to step S306; otherwise, proceed to step S307.

[0174] In step S305, the magnitude of the updated gradient vector can be obtained by taking its absolute value or by mapping, or other methods. If the magnitude of the updated gradient vector... If so, the algorithm ends after executing step S306.

[0175] Step S306: Obtain reference values ​​corresponding to multiple first motion parameters from the updated transformation vector.

[0176] Step S307: Correct the approximate matrix, and calculate the gradient vector for the next iteration based on the affine matrix to be decomposed and the updated transformation vector, then return to step S302.

[0177] This allows us to calculate the difference between the updated gradient vector and the original gradient vector, obtaining the gradient vector difference. This can be expressed by the following formula:

[0178] Gradient vector difference: .

[0179] Furthermore, the corrected approximate matrix can be calculated based on the difference between the approximate matrix, the step size vector, and the gradient vector.

[0180] Among them, the corrected approximate matrix .

[0181] These examples present an optional implementation of parameter decomposition using the quasi-Newton method, which can efficiently and accurately decompose the motion affine matrix and accelerate the processing efficiency of real-time stabilization algorithms.

[0182] Continue reading Figure 1 Step S103: For each of the motion parameters, obtain its corresponding original motion trajectory and the smoothed motion trajectory relative to the original motion trajectory.

[0183] In step S103, the original motion trajectory can be smoothed using a Kalman filter to obtain a smoothed motion trajectory.

[0184] Step S104: Based on the smoothed motion trajectory corresponding to each motion parameter and the corresponding original motion trajectory, perform real-time image stabilization on the target frame.

[0185] It should be noted that in this solution, the motion parameters of the current target frame relative to the previous frame, as well as its corresponding original motion trajectory and the smoothed motion trajectory relative to the original motion trajectory, are applied to the current target frame for image stabilization. Therefore, the target frame image after image stabilization is real-time, reducing the delay of one frame, thereby achieving the purpose of real-time image stabilization. This solves the technical problem of insufficient real-time performance of video image stabilization in the prior art.

[0186] In the specific implementation of step S104, the correction formula for each motion parameter acting on the target frame can be calculated based on the smoothed motion trajectory and the corresponding original motion trajectory for each motion parameter; and the target frame is subjected to real-time anti-shake processing based on the correction formula.

[0187] In some examples, the correction formula for the effect of these partial motion parameters on the target frame can be calculated based on the difference between the smoothed motion trajectory corresponding to some motion parameters and the corresponding original motion trajectory.

[0188] For example, the correction formulas for these motion parameters acting on the target frame can be calculated based on the differences between the smoothed motion trajectories corresponding to the translation parameters, shear parameters, and rotation parameters and the corresponding original motion trajectories.

[0189] In some examples, the correction formula for the effect of these partial motion parameters on the target frame can be calculated based on the quotient between the smoothed motion trajectory corresponding to some motion parameters and the corresponding original motion trajectory.

[0190] For example, the correction formula for these partial motion parameters acting on the target frame can be calculated based on the quotient between the smoothed motion trajectory corresponding to the scaling parameter and the corresponding original motion trajectory.

[0191] After obtaining the correction formulas for each motion parameter acting on the target frame, these formulas can be superimposed to form the overall correction formula for all motion parameters acting on the target frame.

[0192] Taking translation parameters as an example, suppose the translation matrix from the previous frame image to the target frame image is constructed as follows:

[0193] (twenty four)

[0194] The correction formula applied to the previous frame is:

[0195] (25)

[0196] in, and The translation parameters are obtained from parametric decomposition. and This indicates the translation from the previous frame to the current frame. and Represents the original trajectory of the translation. and This represents the smoothed trajectory after translation.

[0197] Formula (25) can be rewritten as:

[0198] (26)

[0199] Combining (24) and (26), the correction formula applied to the target frame image is:

[0200] (27)

[0201] It should be noted that the derivation process of the correction formula for other motion parameters acting on the target frame is the same / similar to that of the translation parameters mentioned above. After calculating the correction formula for each motion parameter acting on the target frame, a total correction formula can be formed. The total correction formula for all motion parameters acting on the target frame can be referred to the following formula (28). Real-time image stabilization can be performed on the target frame to obtain the pixel points of each pixel in the target frame after real-time image stabilization, which finally constitutes the target frame image after real-time image stabilization.

[0202]

[0203]

[0204] (28)

[0205] in,( ) refers to the pixel points in the target frame. , Pixels after real-time image stabilization and This represents the original trajectory of the translation. and This represents the smoothed trajectory of the translation. and This represents the original trajectory of the sheared motion. and This represents the smoothed motion trajectory after shearing. and This represents the original motion trajectory after scaling. and This represents the smoothed motion trajectory after scaling. Represents the original trajectory of rotation. This represents the smoothed trajectory of the rotation.

[0206] In this embodiment of the invention, a correction formula applied to the target frame is obtained by using the smooth motion trajectory and the original motion trajectory. Then, the target frame image is adjusted by using the correction formula applied to the current frame. This avoids the problem of a one-frame delay caused by the anti-shake effect applied to the previous frame image in the prior art, and can achieve the effect of real-time anti-shake.

[0207] Based on the video real-time stabilization method disclosed in the above embodiments of the present invention, this solution uses the quasi-Newton method and motion affine matrix for parameter decomposition, which not only converges faster than the gradient descent method, but also eliminates the need to calculate the second-order partial derivatives of Newton's method. Furthermore, it constructs a positive definite symmetric matrix that can approximate the Hessian inverse matrix, enabling faster and more accurate decomposition of motion parameters. In addition, the stabilization applied to the current frame eliminates the delay of one frame, thereby achieving the goal of reducing computational complexity and real-time stabilization.

[0208] like Figure 2 The diagram shown is a structural diagram of a real-time video stabilization device disclosed in an embodiment of the present invention. The device includes: an acquisition unit 201, a decomposition unit 202, a trajectory calculation unit 203, and a stabilization unit 204.

[0209] The acquisition unit 201 is used to acquire the motion affine matrix of each target frame in the video sequence relative to the previous frame.

[0210] Decomposition unit 202 is used to perform parametric decomposition on the motion affine matrix to obtain various motion parameters;

[0211] The trajectory calculation unit 203 is used to obtain the original motion trajectory and the smoothed motion trajectory relative to the original motion trajectory for each motion parameter.

[0212] The image stabilization unit 204 is used to perform real-time image stabilization processing on the target frame based on the smoothed motion trajectory corresponding to each motion parameter and the corresponding original motion trajectory.

[0213] Optional, the image stabilization unit 204 includes:

[0214] The correction formula calculation subunit is used to calculate the correction formula for each motion parameter acting on the target frame based on the smoothed motion trajectory and the corresponding original motion trajectory for each motion parameter.

[0215] The anti-shake subunit is used to perform real-time anti-shake processing on the target frame based on the correction formula.

[0216] Optional, anti-shake subunit, specifically used for:

[0217] The target frame is stabilized in real time using the following formula:

[0218]

[0219]

[0220] ;

[0221] in,( ) refers to the pixel points in the target frame. , Pixels after real-time image stabilization and This represents the original trajectory of the translation. and This represents the smoothed trajectory of the translation. and This represents the original trajectory of the sheared motion. and This represents the smoothed motion trajectory after shearing. and This represents the original motion trajectory after scaling. and This represents the smoothed motion trajectory after scaling. Represents the original trajectory of rotation. This represents the smoothed trajectory of the rotation.

[0222] Optionally, decomposition unit 202 is specifically used for:

[0223] The affine matrix of motion is decomposed parametrically using the quasi-Newton method to obtain the various motion parameters.

[0224] Specifically, the motion affine matrix is ​​transformed to obtain the motion affine matrix to be decomposed; the motion affine matrix to be decomposed represents the affine transformation relationship from the previous frame to the target frame when the center of the frame is at the origin.

[0225] In the DFP algorithm to be initialized, the initial transformation vector and the initial approximation matrix are set; the approximation matrix is ​​a positive definite symmetric matrix that approximates the Hessian inverse matrix, and the initial transformation vector consists of multiple initial first motion parameters.

[0226] Using the affine matrix to be decomposed and the initialized transformation vector as algorithm parameters, the initialized DFP algorithm is run to output reference values ​​corresponding to multiple first motion parameters;

[0227] The second motion parameter is calculated using reference values ​​corresponding to multiple first motion parameters. Each motion parameter includes multiple first motion parameters and second motion parameters. The second motion parameter is one of scaling parameters, shearing parameters, translation parameters, and rotation parameters. The multiple first motion parameters are parameters other than the second motion parameter among scaling parameters, shearing parameters, translation parameters, and rotation parameters.

[0228] Specifically, using the affine matrix to be decomposed and the initialized transformation vector as algorithm parameters, the initialized DFP algorithm is run to output reference values ​​corresponding to multiple first motion parameters, including:

[0229] Based on the affine matrix of motion to be decomposed and the initialized transformation vector, the gradient vector is calculated, and based on the gradient vector and the latest approximation matrix, the search direction is calculated.

[0230] Based on the search direction, the step size vector is calculated, and the transformation vector is updated using the step size vector;

[0231] Update the gradient vector based on the updated transformation vector;

[0232] Determine whether the size of the updated gradient vector is less than the precision threshold;

[0233] If so, then the reference values ​​corresponding to the first motion parameters are obtained from the updated transformation vector;

[0234] If not, then correct the approximation matrix, and calculate the gradient vector for the next iteration based on the affine matrix to be decomposed and the updated transformation vector. Then return to perform the step of calculating the search direction based on the gradient vector and the latest approximation matrix.

[0235] Specifically, the correction approximation matrix includes:

[0236] Calculate the difference between the updated gradient vector and the original gradient vector to obtain the gradient vector difference value;

[0237] The corrected approximate matrix is ​​calculated based on the difference between the approximate matrix, the step size vector, and the gradient vector.

[0238] Based on the video real-time image stabilization device disclosed in the above embodiments of the present invention, in this solution, parameter decomposition is performed based on the quasi-Newton method and the motion affine matrix. This not only has a faster convergence speed than the gradient descent method, but also eliminates the need to calculate the second-order partial derivatives of Newton's method. Furthermore, a positive definite symmetric matrix that can approximate the Hessian inverse matrix is ​​constructed, which can decompose motion parameters more quickly and accurately. In addition, the image stabilization applied to the current frame eliminates the delay of one frame, thereby achieving the goal of reducing computational complexity and real-time image stabilization.

[0239] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions stored in the memory 330 to execute the aforementioned real-time video stabilization method.

[0240] Furthermore, the logic instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0241] Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0242] The aforementioned storage media include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media that can store program code.

[0243] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the real-time video stabilization methods provided by the above methods.

[0244] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the real-time video stabilization methods provided by the methods described above.

[0245] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0246] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0247] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for video real-time image stabilization, characterized in that, The method comprises: obtaining a motion affine matrix of each target frame in a video sequence relative to a previous frame; performing parameter decomposition on the motion affine matrix to obtain various motion parameters; for each motion parameter, obtaining a corresponding original motion trajectory and a smoothed motion trajectory relative to the original motion trajectory; based on the smoothed motion trajectory and the original motion trajectory corresponding to each motion parameter, performing real-time anti-shake processing on the target frame.

2. The method of claim 1, wherein, The real-time anti-shake processing on the target frame based on the smoothed motion trajectory and the original motion trajectory corresponding to each motion parameter comprises: based on the smoothed motion trajectory and the original motion trajectory corresponding to each motion parameter, calculating a correction formula of each motion parameter acting on the target frame; based on the correction formula, performing real-time anti-shake processing on the target frame.

3. The method of claim 2, wherein, The real-time anti-shake processing on the target frame based on the correction formula comprises: performing real-time anti-shake processing on the target frame by the following formula: ； wherein (x, y) is a pixel point in the target frame, (x, y) is a pixel point after real-time anti-shake processing, represents a translation original motion trajectory, represents a translation smoothed motion trajectory, represents a skew original motion trajectory, represents a skew smoothed motion trajectory, represents a scaling original motion trajectory, represents a scaling smoothed motion trajectory, represents a rotation original motion trajectory, represents a rotation smoothed motion trajectory.​​​​​​​​ 4. The method according to any one of claims 1 to 3, characterized in that, The parameter decomposition on the motion affine matrix to obtain various motion parameters comprises: performing parameter decomposition on the motion affine matrix based on a quasi-Newton method to obtain various motion parameters.

5. The method of claim 4, wherein, The parameter decomposition on the motion affine matrix based on a quasi-Newton method to obtain various motion parameters comprises: performing transformation processing on the motion affine matrix to obtain a to-be-decomposed motion affine matrix; the to-be-decomposed motion affine matrix represents an affine transformation relationship from the previous frame to the target frame when the center of the frame is at the coordinate origin; setting an initialized transformation vector and an initialized approximate matrix in a to-be-initialized DFP algorithm; the approximate matrix is a positive definite symmetric matrix of an approximate Hessian inverse matrix, and the initialized transformation vector is composed of a plurality of initial first motion parameters; taking the to-be-decomposed motion affine matrix and the initialized transformation vector as algorithm parameters, running the initialized DFP algorithm to output reference values corresponding to a plurality of first motion parameters; calculating second motion parameters by using the reference values corresponding to a plurality of first motion parameters; wherein each motion parameter comprises a plurality of first motion parameters and the second motion parameters, the second motion parameter is one of a scaling parameter, a skew parameter, a translation parameter and a rotation parameter, and the plurality of first motion parameters are parameters other than the second motion parameter among the scaling parameter, the skew parameter, the translation parameter and the rotation parameter.

6. The method of claim 5, wherein, The taking the to-be-decomposed motion affine matrix and the initialized transformation vector as algorithm parameters, running the initialized DFP algorithm to output reference values corresponding to a plurality of first motion parameters comprises: based on the to-be-decomposed motion affine matrix and the initialized transformation vector, calculating a gradient vector, and based on the gradient vector and the current latest approximate matrix, calculating a search direction; based on the search direction, calculating a step vector, and updating the transformation vector by using the step vector; updating the gradient vector based on the updated transformation vector; determining whether a size of the updated gradient vector is less than a precision threshold value; if yes, obtaining reference values corresponding to the first motion parameters from the updated transformation vector; if no, correcting the approximation matrix, and calculating a gradient vector for a next round of iteration based on the motion affine matrix to be decomposed and the updated transformation vector, and returning to performing the step of calculating the search direction based on the gradient vector and the most recently updated approximation matrix.

7. The method of claim 6, wherein, The correcting the approximation matrix comprises: calculating a difference between the updated gradient vector and the gradient vector before updating to obtain a gradient vector difference value; calculating the corrected approximation matrix based on the approximation matrix, the step vector and the gradient vector difference value.

8. A video real-time image stabilization apparatus, characterized by comprising: The apparatus comprises: an acquisition unit configured to acquire a motion affine matrix of each target frame in a video sequence relative to a previous frame; a decomposition unit configured to perform parameter decomposition on the motion affine matrix to obtain each motion parameter; a trajectory calculation unit configured to, for each motion parameter, acquire a corresponding original motion trajectory and a smoothed motion trajectory relative to the original motion trajectory; a stabilization unit configured to perform real-time stabilization processing on the target frame based on the smoothed motion trajectory corresponding to each motion parameter and the corresponding original motion trajectory.

9. An electronic device, comprising: comprise: a memory configured to save a computer program; a processor configured to execute the computer program to implement the video real-time stabilization method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, a memory configured to save a computer program; wherein the computer program is executed by a processor to implement the video real-time stabilization method of any one of claims 1 to 7. a memory configured to save a computer program; wherein the computer program is executed by a processor to implement the video real-time stabilization method of any one of claims 1 to 7.

Images