Video stabilization method, device, terminal equipment and computer readable storage medium

Abstract

This application relates to the field of video processing technology, and provides a video stabilization method, apparatus, terminal device, and computer-readable storage medium. The method includes: determining a first distortion restoration mapping diagram and a second distortion restoration mapping diagram; determining a motion matrix based on the second distortion restoration mapping diagram and gyroscope motion information; determining a stabilization motion matrix and a first projective transformation matrix based on the motion matrix; and determining the video stabilization result based on the original video frame, the first distortion restoration mapping diagram, and the first projective transformation matrix. By using the original video frame, the distortion restoration mapping diagram, and the first projective transformation matrix, the video stabilization result is obtained in real time. This simultaneously solves three problems: stabilization of distortion-free real-time video, real-time lens distortion correction, and real-time video rolling shutter distortion correction, improving video / image quality, and reducing computational load and cost.

Description

Technical Field

[0001] This application belongs to the field of video processing technology, and in particular relates to a video stabilization method, apparatus, terminal equipment, and computer-readable storage medium. Background Technology

[0002] With the development of technology, different types of terminal devices are equipped with shooting functions.

[0003] However, some handheld devices, due to the inability to control device stability, are more prone to various image distortion problems compared to other devices, thus reducing the quality of videos or images. Summary of the Invention

[0004] This application provides a video stabilization method, apparatus, terminal device, and computer-readable storage medium, which can solve the problem of low quality of videos or images captured by unstable terminal devices.

[0005] In a first aspect, embodiments of this application provide a video stabilization method, including:

[0006] Determine the first distortion restoration mapping diagram and the second distortion restoration mapping diagram;

[0007] The motion matrix is ​​determined based on the second distortion restoration mapping diagram and the gyroscope motion information.

[0008] The stabilized motion matrix is ​​determined based on the motion matrix, and the first projective transformation matrix is ​​determined based on the motion matrix and the stabilized motion matrix.

[0009] The video stabilization result is determined based on the original video frames, the first distortion restoration mapping diagram, and the first projective transformation matrix.

[0010] Secondly, embodiments of this application provide a video stabilization device, including:

[0011] The first calculation module is used to determine the first distortion restoration mapping diagram and the second distortion restoration mapping diagram;

[0012] The second calculation module is used to determine the motion matrix based on the second distortion restoration mapping diagram and the gyroscope motion information;

[0013] The third calculation module is used to determine the stabilized motion matrix based on the motion matrix, and to determine the first projective transformation matrix based on the motion matrix and the stabilized motion matrix.

[0014] The fourth calculation module is used to determine the video stabilization result based on the original video frame, the first distortion restoration mapping diagram, and the first projective transformation matrix.

[0015] Thirdly, embodiments of this application provide a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the video stabilization method as described in any of the first aspects above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the video stabilization method as described in any of the first aspects above.

[0017] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the video stabilization method of any one of the first aspects described above.

[0018] By drawing video stabilization results in real time based on the original video frames, the distortion restoration mapping diagram, and the first projective transformation matrix, the system simultaneously solves three problems: stabilization of distortion-free real-time video, real-time lens distortion correction, and real-time video roll-up distortion correction. This improves the quality of video / images while requiring less computation and incurring lower costs. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the video stabilization method provided in an embodiment of this application;

[0021] Figure 2 This is a schematic diagram illustrating an application scenario of video frame transformation provided in an embodiment of this application;

[0022] Figure 3 This is a three-dimensional schematic diagram of the ideal source point of the lens, the checkerboard plane and its perpendicular line provided in the embodiments of this application;

[0023] Figure 4a This is a schematic diagram of the calibration plate image provided in the embodiments of this application;

[0024] Figure 4b This is a schematic diagram of the stabilized calibration image provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the first original video frame and the corresponding initial stabilized video frame provided in the embodiments of this application;

[0026] Figure 6 This is the video stabilization result obtained by projective transformation matrix transformation provided in the embodiments of this application. A schematic diagram;

[0027] Figure 7a , 7b 7c, 7d, 7e, and 7f are schematic diagrams illustrating the application scenarios of the vertex set and corresponding drawing effect diagrams provided in the embodiments of this application;

[0028] Figure 8 This is a schematic diagram of an application scenario for vertex-based image rendering provided in an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of the video stabilization device provided in the embodiments of this application;

[0030] Figure 10 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0031] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0032] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0033] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0034] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0035] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0036] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0037] The video stabilization method provided in this application can be applied to terminal devices such as mobile phones, tablets, laptops, and smart cameras. This application does not impose any restrictions on the specific type of terminal device.

[0038] Figure 1 A schematic flowchart of the video stabilization method provided in this application is shown. It is an example and not a limitation. The method can be applied to the smart camera described above.

[0039] S101. Determine the first distortion restoration mapping diagram and the second distortion restoration mapping diagram.

[0040] In practical applications, a single checkerboard image is pre-set for distortion correction of a wide-angle camera. By capturing this checkerboard image, a calibration image is obtained. The camera is calibrated based on the calibration image, and the camera's intrinsic parameter matrix L and rotation displacement matrix R are calculated. A projective transformation is performed based on the rotation displacement matrix R, and the lens distortion model function is determined through a lens distortion correction algorithm. The first distortion parameter vector and the second distortion parameter vector are calculated, and then the first distortion restoration mapping diagram and the second distortion restoration mapping diagram are determined.

[0041] In practical applications, the first original video frame refers to the original image frame of the video acquired at time i (i=1) (that is, the first video frame of the video that needs to be stabilized).

[0042] like Figure 2 The diagram illustrates an application scenario for video frame transformation.

[0043] Among them, the first original video frame After the motion matrix Transformation yields the original video frames. The first original video frame via projective matrix Changes can be used to obtain stabilized video frames. Original video frames After the motion matrix Transformation yields the original video frames. Original video frames via projective matrix Changes can be used to obtain stabilized video frames. And so on.

[0044] In practical applications, upon detecting the first raw video frame... At that time, due to the second projective transformation matrix corresponding to the first original video frame The parameters of the first original video frame and the stabilization cropping ratio can be directly determined; therefore, the second projective transformation matrix can be directly used. For the first original video frame The process is performed to obtain the initial stabilized video frames. The parameters of the first original video frame include its height and width.

[0045] S102. Determine the motion matrix based on the second distortion restoration mapping diagram and the gyroscope motion information.

[0046] In practical applications, the second original video frame This refers to the original image frame of the video acquired at time i (i>1) (i.e., the non-first video frame of the video that needs stabilization processing, such as...). , ...); Based on the second original video frame, determine the third original video frame, obtain the first distortion-restored thumbnail corresponding to the second original video frame, and the second distortion-restored thumbnail corresponding to the third original video frame, and calculate the first feature point set of the first distortion-restored thumbnail and the first feature point set of the second distortion-restored thumbnail; determine the motion estimation matrix based on the first feature point set and the second feature point set, and determine the motion matrix based on the motion estimation matrix. In this embodiment, the third original video frame refers to the second original video frame. Previous original video frame .

[0047] S103. Determine the stabilized motion matrix based on the motion matrix, and determine the first projective transformation matrix based on the motion matrix and the stabilized motion matrix.

[0048] In practical applications, the motion matrix is ​​smoothed to obtain the smoothed, stabilized motion matrix. Smoothing algorithms include, but are not limited to, the Kalman filter algorithm.

[0049] In practical applications, the first projective transformation matrix is ​​determined based on the relationship between the motion matrix, the stabilized motion matrix, and the projective transformation matrix.

[0050] S104. Determine the video stabilization result based on the original video frame, the first distortion restoration mapping diagram, and the first projective transformation matrix.

[0051] In practical applications, the first vertex set of the original video frame is identified, and the vertex set is processed according to the first distortion restoration mapping relationship to obtain the second vertex set after lens distortion removal. The second vertex set is then processed according to the first projective transformation matrix to obtain the third vertex set after roll-up distortion removal. The original video frame is used as the texture, and the third vertex set is used to draw the video using the GPU to determine the video stabilization result.

[0052] In one embodiment, step S101 includes:

[0053] Acquire calibration board image;

[0054] The camera is calibrated based on the calibration plate image to obtain the camera's rotational displacement matrix;

[0055] The first distortion parameter vector and the second distortion parameter vector are determined based on the rotation displacement matrix;

[0056] The first distortion restoration mapping diagram is determined based on the first distortion parameter vector, and the second distortion restoration mapping diagram is determined based on the second distortion parameter vector.

[0057] In practical applications, an image of the calibration board (in this embodiment, the calibration board is specifically a checkerboard used for camera calibration) is obtained by photographing the calibration board. The camera is then calibrated based on the calibration board image. The ideal source point of the pinhole camera lens is defined as the center point. The size and orientation of the checkerboard are determined, and the camera's intrinsic parameter matrix L and rotation / displacement matrix are calculated. .

[0058] in, , ... The resulting 3x3 matrix is ​​an orthogonal matrix. The r component is multiplied and accumulated with the original point position to form the rotational motion, and the t component is multiplied with 1 and then added to represent the displacement. This transformation enables the reference point of the checkerboard object (i.e., the corner point of the checkerboard) to coincide with the camera source point, and makes the checkerboard plane perpendicular to the lens orientation.

[0059] like Figure 3 As shown, a three-dimensional schematic diagram of the ideal source point of the lens, the checkerboard plane and its perpendicular line is provided.

[0060] In the diagram, a 3D coordinate system is constructed using the ideal source point of the lens, the checkerboard plane, and its perpendicular line. Any point in this 3D coordinate system is: By performing a projective transformation on the camera using the rotation-translation matrix, the normalized imaging points on the camera plane are obtained. As shown in Formula 1.

[0061] (Formula 1);

[0062] Among them, the horizontal focal length of the camera lens Vertical focal length The optical centers are respectively When lens distortion exists, the coordinates u and v of the image point in the camera frame differ from the lens-normalized image distortion point. It has the relationship shown in Formula 2:

[0063] (Formula 2);

[0064] Since the optical axis of a lens cannot be perfectly perpendicular to the imaging plane, optical axis deviation distortion is introduced. This distortion is corrected through lens distortion correction. The normalized point coordinates become the coordinates without optical axis deflection distortion, as shown in Formula 3.

[0065] (Formula 3);

[0066] in, This indicates the angle of deviation of the optical axis from the reference x-axis. This indicates the angle of deviation of the optical axis from the reference y-axis. , , respectively with Formula 1 , and correspond.

[0067] Equation 3 is transformed to obtain Equation 4:

[0068]

[0069] (Formula 4);

[0070] Make For normalized imaging points without lens distortion, it can be determined The lens distortion model function is as follows:

[0071] (Formula 5);

[0072] (Formula 6);

[0073] .

[0074] in, This is the distortion parameter vector.

[0075] In practical applications, the corner coordinates of the calibration board (i.e., the checkerboard image) in the calibration image are detected using a line-scanning statistical method. The checkerboard corner positions in the world coordinate system are then generated as follows: Wherein, since the length of the chessboard grid is a relative reference value (generally set to 1); i and j represent the row and column numbers of the corner points in the chessboard grid, respectively; , Let represent the width and height of a single chessboard cell, respectively. Then, using formulas 5-6 above, the first distortion parameter vector problem can be solved, thereby determining the first distortion restoration mapping diagram. .

[0076] By cropping and scaling the chessboard grid coordinates by a factor of W, and substituting this into formula 5-6, the second distortion parameter vector corresponding to the thumbnail can be calculated, thereby determining the second distortion restoration mapping relationship diagram corresponding to the thumbnail. .

[0077] Figure 4 shows a schematic diagram of a calibration plate image and a stabilized calibration plate image.

[0078] like Figure 4a As shown, the captured calibration board image suffers from lens distortion and roller blind distortion, requiring distortion correction processing to obtain the image shown. Figure 4b The image shown is a stabilized calibration plate image that is close to what the human eye can see.

[0079] In one embodiment, step S104 includes:

[0080] When the original video frame is detected to be the first original video frame, the parameters of the first original video frame are obtained;

[0081] Determine the second projective transformation matrix corresponding to the first original video frame based on the parameters;

[0082] The first original video frame is projectively transformed based on the first distortion restoration mapping diagram and the second projective transformation matrix to obtain the initial stabilized video frame.

[0083] In practical applications, when the original video frame is detected as the first original video frame... At that time, obtain the parameters of the first original video frame and the second projective transformation matrix corresponding to the first original video frame. It can be represented as:

[0084] (Formula 7);

[0085] Therefore, the height of the first original video frame can be used as a basis. ,Width And the stabilization cutting ratio is Determine the second projective transformation matrix. Then, for the first original video frame... according to The initial stabilized video frame can be obtained by performing a projective transformation. .

[0086] like Figure 5 As shown, a first raw video frame is provided. and the corresponding initial stabilized video frames A schematic diagram;

[0087] Among them, the initial stabilized video frames The content of the image comes from the first original video frame. A sub-image region whose height and width are both the same as the first original video frame. of The horizontal offset of the top left corner origin is times, and the horizontal offset is... The vertical offset is To stabilize the initial video frames Screen size and the first original video frame Consistent, the final initial stabilized video frames can be obtained through interpolation using the GPU. .

[0088] In one embodiment, step S102 includes:

[0089] S1021. When the original video frame is detected to be the second original video frame, the first scaled original video frame corresponding to the second original video frame is obtained.

[0090] S1022. Obtain the previous original video frame of the second original video frame. The previous original video frame of the second original video frame is the third original video frame.

[0091] S1023. Obtain the second scaled original video frame corresponding to the third original video frame according to the second distortion restoration mapping diagram;

[0092] S1024. Determine the motion estimation matrix based on the first scaled original video frame and the second scaled original video frame;

[0093] S1025. Determine the motion matrix based on the motion estimation matrix and the motion information from the gyroscope.

[0094] In practical applications, existing original video frames and motion estimation matrix The relationship satisfies: , representing the original video frame The image content can be obtained from the previous video frame. (The image content is estimated using the motion estimation matrix) The 2D planar projective transformation is approximated by this. Wherein, the original video frame... Any homogeneous coordinate point is The previous video frame of the original video frame The coordinates of any second point are .

[0095] Motion estimation matrix It can be represented as:

[0096] (Formula 8);

[0097] Then we can get:

[0098] (Formula 9);

[0099] In practical applications, when the original video frame is detected as the second original video frame... At that time, acquire the second original video frame. The corresponding first scaled original video frame The preceding original video frame of the second original video frame is determined to be the third original video frame. The corresponding second-scaled original video frame is determined through scaling and cropping processing and the second distortion restoration mapping relationship diagram. According to Formula 9, the motion estimation matrix can be determined based on the corresponding pairs of feature points in the two feature point sets of the first scaled original video frame and the second scaled original video frame. The motion matrix is ​​then determined based on the motion estimation matrix and the gyroscope motion information.

[0100] In one embodiment, step S1024 includes:

[0101] The second scaled original video frame is processed to obtain the first set of feature points;

[0102] The first scaled original video frame, the second scaled original video frame, and the first feature point set are processed to obtain the second feature point set.

[0103] The motion estimation matrix is ​​determined based on the first set of feature points and the second set of feature points.

[0104] The first feature point set is obtained by processing the second scaled original video frame through the first algorithm.

[0105] Specifically, the second feature point set is obtained by processing the first scaled original video frame, the second scaled original video frame, and the first feature point set through the second algorithm.

[0106] In practical applications, the first algorithm is used to process the second scaled original video frame to obtain the first set of feature points. The second algorithm processes the first scaled original video frame, the second scaled original video frame, and the first feature point set to obtain the second feature point set. .

[0107] The first algorithm includes, but is not limited to, a feature point detection algorithm; the second algorithm includes, but is not limited to, a coefficient optical flow algorithm.

[0108] In specific applications, the detection of the first feature point set includes: dividing the second scaled original video frame into several regions, determining the corner points in each region, and removing locally redundant corner points and corner points with low response values ​​through a grid filtering algorithm to obtain a certain number of first feature point sets; the second feature point set is the set of positions of the first feature point set in the first scaled original video frame.

[0109] Through multiple samplings, four random numbers are selected from the first and second feature point sets each time. The points are used to establish a system of equations as shown in Equation 10:

[0110] (Formula 10)

[0111] in, Represents the points in the second feature point set; Indicates and The points in the corresponding first feature point set; The coefficients of the motion estimation matrix can be obtained by solving the system of equations. This allows for the determination of a single motion estimation matrix. Multiple motion estimation matrices can be determined by selecting four random pairs of points from the first and second feature point sets multiple times.

[0112] In practical applications, after determining multiple motion estimation matrices, the accuracy of the motion estimation matrices can be verified using formulas 11-12, based on each pair of points selected from the first and second feature point sets.

[0113] (Formula 11)

[0114] (Formula 12)

[0115] in, Let represent the motion estimation matrix for the t-th verification. The inverse of the motion estimation matrix validated for the t-th time; T represents a small distance threshold (generally set to be less than 1 pixel). The motion estimation matrix with the most points satisfying formulas 11-12 above is determined through calculation. And through the following equation 13 (for the motion estimation matrix) Components in Multiply by W and divide by the component Divide by W to determine the final motion estimation matrix. .

[0116] (Formula 13).

[0117] In one embodiment, step S1025 includes:

[0118] Acquire timestamp alignment offset, video single-frame imaging time, and gyroscope motion information;

[0119] The motion matrix is ​​determined based on the timestamp alignment offset, the imaging time of a single video frame, the motion estimation matrix, and the motion information from the gyroscope.

[0120] In practical applications, due to differences in imaging time between different rows of images, automatic calibration can be initiated by combining gyroscope motion information. Dynamic calibration based on gyroscope motion information includes:

[0121] The offset between the image imaging time and the gyroscope motion information timestamp was obtained. And estimate the single-frame imaging time of the original video. At that time, (among which, This is an unknown constant that remains constant after the current device is started. Let be the time required for imaging the i-th frame. The gyroscope information is converted using the following formula to determine the motion matrix:

[0122] Any point in the plane of the original video frame The new coordinates can be obtained by transforming the coordinates using the rotation and displacement matrix R. As shown in the following formula:

[0123] (Formula 14);

[0124] In practical applications, gyroscope motion information This represents the instantaneous rotational speed of the device at a specific moment, with a sampling time of [time value missing]. (Approximate duration of motion). By extracting the modulus... Among them, the modulus This indicates the rotational speed, measured in radians per second. Indicates the direction of rotation.

[0125] To simplify the calculation, the gyroscope motion information needs to be converted into a 4-tuple format, defined as follows: Gyroscope motion information can be converted into 4-tuples. Furthermore, the cumulative rotational motion quartet between the two quartets can be calculated using the following formula, as shown below:

[0126] (Formula 15);

[0127] The 4-tuple rotation vector is processed by a 3x3 rotation matrix of the following form (i.e., the transformed gyroscope motion information).

[0128] (Formula 16)

[0129] In this embodiment, the difference between the gyroscope motion information acquisition time and the video imaging time is determined by multiple different interpolation algorithms. The interpolation algorithms include, but are not limited to, nearest neighbor interpolation, linear interpolation, and radial interpolation algorithms.

[0130] Alignment offset between image imaging time and gyroscope motion information timestamp The single-frame imaging time of the original video The converted gyroscope motion information is used to determine the motion matrix using formula 17-19:

[0131] (Formula 17);

[0132] (Formula 18);

[0133] (Formula 19);

[0134] in, This represents half the time required to image one frame. This indicates that cumulative motion calculations are performed based on the image imaging center time. Represents rows in an image.

[0135] Understandably, when non-convergence of the calibration is detected, the motion matrix can be set. .

[0136] In one embodiment, when the alignment offset between the image imaging time and the gyroscope motion information timestamp cannot be determined... And the single-frame imaging time of the original video At this time, multi-frame iterative estimation is required:

[0137] 1. Randomly select from the range of values. And, calculate the gyroscope motion information, and then based on the gyroscope motion information and Determine the motion estimation matrix .

[0138] Define the range of values ​​for the parameters Where d represents the maximum time delay of the video data relative to the gyroscope motion information, and fps represents the frame rate. A value greater than 0 indicates that the imaging time of the last row of the image is greater than the imaging time of the first row. When When the value is less than 0, it means that the imaging time of the last row of the image is less than the imaging time of the first row.

[0139] 2. The motion estimation matrix will be calculated based on the steps described above. The motion estimation matrix determined according to formulas 11-13 above The comparison was performed, and the adaptation error was calculated by adjusting the image center points according to... and Perform a transformation and calculate the distance difference between the two transformed points. Specifically, when... and When similar to the identity matrix, parameters are not calculated. and The adaptation error is minimal. Because the image content or the current device experiences only very slight jitter at this point, the adaptation error for any parameter sub-range will be very small.

[0140] To ensure accuracy and speed, parameters will be used during the implementation process. The range of values ​​is divided into K parts. A circular queue L of structure length K is defined to record these values. The mean loss of the adaptation error for parameter sub-intervals. upper and lower bounds of parameter search, and the direction of parameter search. , Current optimized parameter values and Number of hits When performing motion estimation for the i-th original video frame, Newton's method is used only a finite number of times within a single parameter sub-interval L[k]. and The specific steps are as follows: Based on the current optimization parameter values... and Calculate the adaptation error L1 by performing multiple iterative searches based on the current device's computing performance, including: randomly selecting updates. or If updated and ,make If updated ,make The updated or Substitute the fitting error L2 into the calculation. If L2 > L1, let... or ;when or If the value is less than the specified threshold, perform random assignment on the alignment and return to execute multiple loop searches; if L1>=L2, let or If L1 = L2, return and perform multiple iterations of the loop search. Exit the loop search; if L1 is less than the threshold, adjust the number of hits for that parameter sub-interval. Update the mean error in L[k] using L1.

[0141] Finally, iterate through the circular queue L and select the current optimized parameter value for the parameter sub-interval where the number of fits is greater than the threshold and the mean loss is minimized. and The gyroscope motion estimation result for the current frame is calculated.

[0142] By ensuring that the number of fit hits exceeds a threshold, algorithm convergence can be guaranteed, and minimizing the mean loss ensures local optima for the solved parameters. When the number of frames is large and there are parameter sub-intervals with a high number of hits in the circular queue L, parameter sub-intervals with a low number of hits can be skipped from the search.

[0143] In practical applications, during the initial stabilization phase, the gyroscope motion information has not yet converged. At this time, the motion matrix between the original video frames Fi-1 and Fi can only be estimated from the image content. Once the gyroscope motion information converges, the motion estimation matrix determined based on the gyroscope motion information is used preferentially.

[0144] Additionally, gyroscope motion estimation can be used to filter out erroneous optical flow point pairs in the first and second feature point sets. Specifically, this involves using the current optimized parameter values ​​from the previous frame's state. and The gyroscope motion estimation result of the current frame is calculated, and the first feature point set is... Input to calculate the new position of the current frame ,Will and Compare and discard point pairs with large distance differences. Obtain an optimized set of optical flow point pairs. and And then according to and The process involves subsequent steps such as verification, filtering, and calculation of the determined motion matrix. This approach avoids errors in the motion estimation matrix caused by multiple moving objects obscuring the image content.

[0145] In one embodiment, determining the stabilized motion matrix based on the motion matrix includes:

[0146] The motion matrix is ​​decomposed to obtain the variables;

[0147] Smooth the variables to obtain smoothed variables;

[0148] The stabilized motion matrix is ​​determined based on the smoothed variables.

[0149] In practical applications, the stabilization motion matrix needs to be determined. To ensure uniform motion and preserve the original video frames To stabilize the frame The projective transformation matrix between them To ensure distortion-free motion, the variables are obtained by decomposing the motion matrix, smoothing the variables, and then determining the stable motion matrix based on the smoothed variables. .

[0150] The relationship between the stabilizing motion matrix and the moving matrix can be expressed as:

[0151] ;(Formula 20)

[0152] When i=0 .

[0153] The motion matrix needs to be adjusted using formulas 21-22. Decompose:

[0154]

[0155] (Formula 21)

[0156] (Formula 22)

[0157] The perspective transformation is achieved through the above formula. Convert the camera intrinsic parameter matrix L into a normalized coordinate system motion matrix. Then the normalized perspective transformation matrix is ​​applied. The transformation is decomposed into continuous transformations of similarity, affine, and perspective, and then the Kalman filter algorithm is used to process the transformed motion matrix. variables in Perform smoothing to obtain the smoothed variables. Then, the normalized smooth perspective transformation matrix is ​​determined. Then, the stabilization motion matrix is ​​obtained through formula 23. . (Formula 23).

[0158] The projective matrix is ​​calculated as follows: by formula It can be seen that when i=0, At this point, it's impossible to limit the degree of distortion when converting the current frame to a stabilized frame. Therefore, the setting is modified through post-processing. This limits the degree of distortion:

[0159] (Formula 24);

[0160] Using the above formula 24 Decomposed into scaling (s) and displacement An approximate representation of three-dimensional rotation (a, b, c) is obtained by substituting the four corner points of the image into... and In the middle, calculate the displacement difference of the four corner points under these two transformations. Define a maximum permissible displacement difference based on the image size. ,make The following formula uses a linear weighting method to ensure the fused product... The maximum distortion.

[0161] (Formula 25)

[0162] Finally, to ensure The transformation will not exceed the limits, and the video stabilization result is satisfactory. The image content at the four corners of the image is located in the original video frame. Internally, it is processed using post-processing methods:

[0163] like Figure 6 As shown, a method is provided using the projective transformation matrix. Video stabilization results after conversion A schematic diagram.

[0164] like Figure 6 , by the first projective transformation matrix The maximum planar rotation angle of the maximum video stabilization result can be calculated. The video stabilization result was calculated. If the rotation amplitude of the four corners of the image is greater than 1, the rotation amplitude of the four corners of the image is greater than 1. Let the excess part of the rotation be Then let Perform a reverse rotation.

[0165] Then calculate and determine the video stabilization result. The four corner points correspond to the positions in the original image, and the width of the corresponding original video frame can then be calculated. ,high Let the original video frame width be W and the height be H. Calculate the scaling factor. If S>1, then let Perform scaling by 1 / s.

[0166] Determine the video stabilization result Are the positions of the four corner points in the original image correct? Inside. if not Then perform a translation operation to ensure the original The corner point that crossed the boundary is located at Boundary location.

[0167] In one embodiment, step S05 further includes:

[0168] When the original video frame is detected to be the second original video frame, the first set of vertices of the second original video frame is identified;

[0169] The second vertex set is obtained by calculating the first vertex set based on the first distortion restoration mapping diagram;

[0170] The third vertex set is obtained by calculating the second vertex set based on the first projective transformation matrix;

[0171] The video stabilization result is determined based on the third vertex set, the second original video frame, and the first original video frame.

[0172] In practical applications, the first original video is identified by recognition. First vertex set If directly using the first original video frame (like Figure 7a As shown), drawing according to the first vertex set results in image frames with issues of roll-up distortion and lens distortion (e.g., Figure 7b (As shown).

[0173] In practical applications, the first vertex set can be substituted into the first distortion-restored projective relation graph to identify the second vertex set. (like Figure 7c As shown), the image drawn at this time exhibits roller blind distortion (such as...). Figure 7d (As shown); the mapping matrix should be determined using the motion matrix and the smoothing matrix, and then the second vertex set should be processed using the mapping matrix to obtain the third vertex set after removing the curtain distortion. (like Figure 7e As shown), by combining the third vertex set and the original video frames, the video stabilization results without lens distortion and without rolling shutter distortion are plotted (e.g.) Figure 7f (As shown).

[0174] In practical applications, the first vertex set The second vertex set is obtained by performing one transformation through CPU computation. Second vertex set The data is passed to the GPU vertex buffer. During rendering each frame, the GPU vertex renderer and segment renderer perform roll-up distortion removal to obtain the third vertex set. .

[0175] Among them, the first vertex set To the set of the second vertex The conversion process is as follows:

[0176] The custom value N represents the number of rows / blocks to eliminate the roller blind effect (this can be set according to actual needs, for example, N = 20). Define a total of N rows and M columns. There are 2D vertex coordinates, where the coordinates of the i-th row and j-th column are... At this point, a first vertex set is generated. The first vertex set is then processed using the first distortion restoration mapping graph to obtain a second vertex set. The texture coordinates corresponding to the vertices in the second vertex set are defined as follows: The vertex coordinates in the second vertex set are normalized to the GPU-adapted coordinate range and passed into the vertex buffer for further processing.

[0177] Set the second vertex Set to the third vertex The conversion process is as follows:

[0178] Before each conversion, the CPU needs to upload N de-rolling stabilization matrices to the GPU. ;

[0179] ;

[0180] Using vertex coordinates as The texture coordinates corresponding to the vertex coordinates are... Locate the row number of the vertex using the vertex indexing method. Motion matrix of corresponding row block Perform vertex transformation to obtain The transformed vertex coordinates are obtained from the output. and the corresponding texture coordinates ; Define the texture sampler through the segment renderer, using vertex coordinates With texture coordinates Draw and render the image; the drawing process is further divided into drawing... A triangle.

[0181] like Figure 8 As shown, this diagram illustrates an application scenario for rendering images based on vertex drawing.

[0182] like Figure 8 As shown, if there are 4 rows and 5 columns of vertices, a total of 2*(4-1)*(5-1)=24 triangles can be drawn.

[0183] This embodiment obtains the video stabilization result in real time by drawing based on the original video frame, the distortion restoration mapping relationship diagram and the first projective transformation matrix. It solves three problems at the same time: stabilization of distortion-free real-time video, real-time lens distortion correction and real-time video roll-up distortion correction, thereby improving the quality of video / images. It also has low computational load and low cost.

[0184] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0185] Corresponding to the video stabilization method in the above embodiments, Figure 9 A structural block diagram of the video stabilization device provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0186] Reference Figure 9 The video stabilization device 100 includes:

[0187] The first calculation module 101 is used to determine the first distortion restoration mapping diagram and the second distortion restoration mapping diagram;

[0188] The second calculation module 102 is used to determine the motion matrix based on the second distortion restoration mapping diagram and the gyroscope motion information;

[0189] The third calculation module 103 is used to determine the stabilized motion matrix based on the motion matrix, and to determine the first projective transformation matrix based on the motion matrix and the stabilized motion matrix.

[0190] The fourth calculation module 104 is used to determine the video stabilization result based on the original video frame, the first distortion restoration mapping diagram, and the first projective transformation matrix.

[0191] This embodiment obtains the video stabilization result in real time by drawing based on the original video frame, the distortion restoration mapping relationship diagram and the first projective transformation matrix. It solves three problems at the same time: stabilization of distortion-free real-time video, real-time lens distortion correction and real-time video roll-up distortion correction, thereby improving the quality of video / images. It also has low computational load and low cost.

[0192] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0193] Figure 10 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Figure 10 As shown, the terminal device 11 of this embodiment includes: at least one processor 110 ( Figure 10 (Only one is shown in the image) a processor, a memory 111, and a computer program 112 stored in the memory 111 and executable on at least one processor 110, wherein the processor 110 executes the computer program 112 to implement the steps in any of the above-described video stabilization method embodiments.

[0194] Terminal device 11 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This terminal device may include, but is not limited to, a processor 110 and a memory 111. Those skilled in the art will understand that... Figure 10 This is merely an example of terminal device 11 and does not constitute a limitation on terminal device 11. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0195] The processor 110 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0196] In some embodiments, memory 111 may be an internal storage unit of terminal device 11, such as a hard disk or memory of terminal device 11. In other embodiments, memory 111 may be an external storage device of terminal device 11, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on terminal device 11. Furthermore, memory 111 may include both internal and external storage units of terminal device 11. Memory 111 is used to store operating system, applications, bootloader, data, and other programs, such as program code of computer programs. Memory 111 can also be used to temporarily store data that has been output or will be output.

[0197] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments.

[0198] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0199] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0200] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0201] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0202] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0203] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0204] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0205] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A video stabilization method, characterized in that, include: Determining the first distortion restoration mapping diagram and the second distortion restoration mapping diagram includes: acquiring a calibration board image; calibrating the camera based on the calibration board image to obtain the camera's rotation displacement matrix; determining the first distortion parameter vector and the second distortion parameter vector based on the rotation displacement matrix; determining the first distortion restoration mapping diagram based on the first distortion parameter vector; and determining the second distortion restoration mapping diagram corresponding to the thumbnail based on the second distortion parameter vector. The motion matrix is ​​determined based on the second distortion restoration mapping diagram and the gyroscope motion information; the motion matrix is ​​used to transform the previous original video frame to the next original video frame. A stabilized motion matrix is ​​determined based on the motion matrix, and a first projective transformation matrix is ​​determined based on the motion matrix and the stabilized motion matrix. Based on the original video frame, the first distortion restoration mapping diagram, and the first projective transformation matrix, the video stabilization result is determined; the first projective transformation matrix is ​​used to transform the original video frame into a stabilized video frame.

2. The method as described in claim 1, characterized in that, The step of determining the motion matrix based on the second distortion restoration mapping diagram and the gyroscope motion information includes: When the original video frame is detected to be the second original video frame, the first scaled original video frame corresponding to the second original video frame is obtained; Obtain the previous original video frame of the second original video frame, and the previous original video frame of the second original video frame is the third original video frame. The second scaled original video frame corresponding to the third original video frame is obtained according to the second distortion restoration mapping diagram. Determine the motion estimation matrix based on the first scaled original video frame and the second scaled original video frame; The motion matrix is ​​determined based on the motion estimation matrix and the gyroscope motion information.

3. The method as described in claim 2, characterized in that, The step of determining the motion estimation matrix based on the first scaled original video frame and the second scaled original video frame includes: The second scaled original video frame is processed to obtain the first feature point set; The first scaled original video frame, the second scaled original video frame, and the first feature point set are processed to obtain the second feature point set. The motion estimation matrix is ​​determined based on the first set of feature points and the second set of feature points.

4. The method as described in claim 2, characterized in that, The step of determining the motion matrix based on the motion estimation matrix and the gyroscope motion information includes: Acquire timestamp alignment offset, video single-frame imaging time, and gyroscope motion information; The motion matrix is ​​determined based on the timestamp alignment offset, the single-frame imaging time of the video, the motion estimation matrix, and the gyroscope motion information.

5. The method according to any one of claims 1-4, characterized in that, The step of determining the stabilized motion matrix based on the motion matrix includes: The motion matrix is ​​decomposed to obtain variables; The variables are smoothed to obtain smoothed variables; The stabilized motion matrix is ​​determined based on the smoothed variables.

6. The method as described in claim 5, characterized in that, The step of determining the video stabilization result based on the original video frame, the first distortion restoration mapping diagram, and the first projective transformation matrix includes: When the original video frame is detected to be the first original video frame, the parameters of the first original video frame are obtained; Determine the second projective transformation matrix corresponding to the first original video frame based on the parameters; Based on the first distortion restoration mapping diagram and the second projective transformation matrix, the first original video frame is projectively transformed to obtain the initial stabilized video frame.

7. The method as described in claim 5, characterized in that, The step of determining the video stabilization result based on the original video frame, the first distortion restoration mapping diagram, and the first projective transformation matrix further includes: When the original video frame is detected to be a second original video frame, the first set of vertices of the second original video frame is identified; The second vertex set is obtained by calculating the first vertex set based on the first distortion restoration mapping diagram; The third vertex set is obtained by calculating the second vertex set based on the first projective transformation matrix; The video stabilization result is determined based on the third vertex set, the second original video frame, and the first original video frame.

8. A video stabilization device, characterized in that, include: A first calculation module is used to determine a first distortion restoration mapping diagram and a second distortion restoration mapping diagram, including: acquiring a calibration board image; calibrating the camera according to the calibration board image to obtain a camera rotation displacement matrix; determining a first distortion parameter vector and a second distortion parameter vector according to the rotation displacement matrix; determining a first distortion restoration mapping diagram according to the first distortion parameter vector; and determining a second distortion restoration mapping diagram corresponding to the thumbnail according to the second distortion parameter vector. The second calculation module is used to determine a motion matrix based on the second distortion restoration mapping diagram and the gyroscope motion information; the motion matrix is ​​used to transform the previous original video frame to the next original video frame. The third calculation module is used to determine the stabilized motion matrix based on the motion matrix, and to determine the first projective transformation matrix based on the motion matrix and the stabilized motion matrix. The fourth calculation module is used to determine the video stabilization result based on the original video frame, the first distortion restoration mapping diagram, and the first projective transformation matrix; the first projective transformation matrix is ​​a projective transformation matrix used to transform the original video frame into a stabilized video frame.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images