Video stabilization method based on inertial measurement unit data and feature point matching
By combining the data and feature points of the inertial measurement unit, the second fine -tuning is performed to obtain the final transformation parameter, which solves the problem of poor image stability effects in the existing technology, and achieves better image stability effects.
Patent Information
- Application Number
- CN202411459935.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-01-21
AI Technical Summary
In the prior art, when compensating for random jitter of the camera, it is difficult to accurately estimate the angular velocity and acceleration data, resulting in poor image stability effect; while the feature point matching method is difficult to find sufficient feature points in some scenarios or there are problems of mismatch, which affects the compensation effect.
A video stabilization method based on inertial measurement unit data and feature point matching is adopted. The compensation parameters are initially estimated through IMU data, feature point matching is performed within the feature point deviation range, and secondary fine-tuning is performed to obtain the final transformation parameters.
By narrowing the search range for feature point matching, it saves computing power and reduces the probability of mismatching, significantly improving the image stabilization effect.
Smart Images
Figure CN119342344A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a video stabilization method based on inertial measurement unit data and feature point matching. Background Art
[0002] In recent years, vehicle-mounted cameras and portable video recorders have been increasingly used in public services, administrative traffic police, and other fields. In these scenarios, since the cameras are not fixed, they will encounter the problem of random camera jitter. This will then cause random jitter in the captured images. Watching such videos will cause obvious dizziness, and the user experience is very poor.
[0003] There are two main types of motion in video images. One is the motion caused by the camera's carrier turning; the other is the random jitter of the carrier (such as the vibration caused by the vehicle engine, the slight tremor of the arm, etc.). The former is a normal motion and does not need to be offset. The latter is the motion that really needs to be compensated. As for how to distinguish the two types of motion in the video, there have been many studies. The present invention does not involve this aspect of the problem, but only solves the problem of how to better compensate for the second type of random jitter.
[0004] There are currently two main ways to compensate for random jitter: 1) Install an inertial measurement unit (IMU) on the camera to collect the camera's angular velocity and acceleration data in real time, and then estimate the offset of the stationary target in the image between the previous and next frames based on the camera's internal parameter matrix and other parameters. This offset is generally the number of pixels displaced in the horizontal (x direction) and vertical (y direction) directions, as well as the rotation angle in the vertical screen direction (z direction). Then, based on these estimated parameters, the compensated image is transformed. 2) Use image processing methods to find feature points in the image, and then match these feature points between the previous and next frames to obtain a certain number of feature point pairs. Then, based on the feature point pairs, estimate the transformation parameters between the two frames (generally using affine transformation or perspective transformation). Then, based on these estimated parameters, the compensated image is transformed.
[0005] Both methods can achieve image stabilization to a certain extent, but they also have their own disadvantages. For the first method, due to factors such as the error of the IMU itself and the error of the production and installation process, it is difficult to accurately estimate the accurate angular velocity and acceleration data, so it is not possible to achieve very accurate image compensation, which affects the final image stabilization effect. For the second method, in theory, if the feature point pair is matched very accurately, the compensation parameters can be very accurately estimated to achieve the optimal compensation effect, but there are various limitations in practical applications, and it is impossible to achieve very accurate feature point matching. For example: when the picture itself is relatively flat (such as facing a white wall), it is difficult to find a sufficient number of feature points; when the picture shakes a large amplitude, the distance between the correct feature point pairs is too large, and there will be great problems in matching feature points using image processing methods. The search radius must be set very large, which will lead to a square increase in the amount of calculation. It is difficult to provide sufficient computing power on some cheap edge devices. In addition, the expansion of the search radius will significantly increase the probability of mismatching, thereby affecting the estimation of the correct transformation parameters. Summary of the invention
[0006] Based on the technical problems existing in the background technology, the present invention proposes a video stabilization method based on inertial measurement unit data and feature point matching, which has a better image stabilization effect.
[0007] The video stabilization method based on inertial measurement unit data and feature point matching proposed by the present invention comprises the following steps:
[0008] Step 1, obtaining IMU data and estimating compensation parameters based on the IMU data, calculating the compensated image relative to the previous frame image based on the compensation parameters, setting the feature point deviation range of the current frame image relative to the compensated image, and reselecting N feature points in the compensated image as reference feature points, wherein the IMU data is the angular velocity and acceleration data of the camera collected by the inertial measurement unit;
[0009] Step 2: Match feature points within the feature point deviation range estimated by IMU according to the current frame image and the previous frame compensated image, and determine whether the matched feature point pair is greater than N. If so, proceed to step 3, otherwise proceed to step 4;
[0010] Step 3: Fine-tune the compensation parameters estimated by the IMU data. After each fine-tuning, calculate the reference feature points and the compensation parameters after the current fine-tuning to obtain the tuning parameter GSAD of the current fine-tuning amount. Select the fine-tuning parameter corresponding to the optimal tuning parameter GSAD to calculate the final transformation parameter.
[0011] Step 4: Use the IMU data estimated compensation parameters as the final transformation parameters;
[0012] Step 5: transform the current frame image according to the final transformation parameters to obtain the current frame compensated image;
[0013] Step 6: Buffer the compensated image of the current frame and loop into step 2 to achieve stable video transmission.
[0014] Furthermore, it is characterized in that, in step 1, the current frame image is first buffered for backup.
[0015] Furthermore, in step 2, a method similar to estimateAffinePartial2D of opencv is used to perform feature point matching.
[0016] Furthermore, the video stabilization method based on inertial measurement unit data and feature point matching is characterized in that, in step three, the final transformation parameters are obtained specifically as follows: step 201 to step 210:
[0017] Step 201, setting the optimal rotation, horizontal and vertical fine-tuning amount Δθ opt , Δx opt , Δy opt Both are 0, GSAD opt Set to a maximum value;
[0018] Step 202, performing a traversal cycle of the first layer rotation angle on the nth image block of the kth frame within a set range according to a set step length, and the current value is recorded as Δθ;
[0019] Step 203, performing a traversal cycle of the second layer horizontal x offset on the n-th image block of the k-th frame within a set range according to the set step length, and the current value is recorded as Δx;
[0020] Step 204, performing a traversal cycle of the third layer vertical y offset on the n-th image block of the k-th frame within a set range according to the set step length, and the current value is recorded as Δy;
[0021] Step 205, calculate the current fine-tuning amount adjustment parameter GSAD, adjustment parameter GSAD and GSAD Dpt Compare, if the parameter GSAD is less than GSAD opt , indicating that the current fine-tuning amount is better, and Δθ opt , Δx opt , Δy opt Change the value of Δθ, Δx, Δy to the current one, and change the parameter GSAD to the current one opt ; If the parameter GSAD is greater than GSAD opt Then continue the cycle directly;
[0022] Step 206, after the triple loop is completed, the current Δθ opt , Δx opt , Δy optThe corresponding optimal fine-tuning parameters are the final transformation parameters.
[0023] Further, in step 205, in calculating the adjustment parameter GSAD of the current fine-tuning amount, the calculation formula is as follows:
[0024]
[0025] Among them, σ θ , σ x , σ y is empirical data, G(·) is a Gaussian function, SAD(·) is the sum of the absolute values of the pixel-by-pixel differences between the feature region block and the averaged feature region block, and is a metric. MX(·) and MY(·) represent rotation transformations with offsets in the X and Y directions, respectively.
[0026] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the video stabilization method as described above when executing the computer program.
[0027] A computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the video stabilization method as described above.
[0028] The advantage of the video stabilization method based on inertial measurement unit data and feature point matching provided by the present invention is that the video stabilization method based on inertial measurement unit data and feature point matching provided in the structure of the present invention obtains accurate image transformation parameters by means of secondary estimation. The first estimation process calculates the approximate offset parameters through IMU data; the second estimation process is performed by means of feature point matching in a smaller area near this approximate offset. Since the first estimation greatly reduces the search range of feature point matching, it not only saves search computing power, but also reduces the probability of feature point mismatching, and the image stabilization effect is better. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a schematic diagram of the process of the present invention;
[0030] Figure 2 This is a schematic diagram of the image after the previous frame is compensated by the video stabilization method, in which there is a target A;
[0031] Figure 3 This is a schematic diagram of the current frame image. The target A in the previous frame changes to A1 due to jitter;
[0032] Figure 4 This is a schematic diagram of the results after IMU data compensation. There is still a certain deviation between target A2 and target A. DETAILED DESCRIPTION
[0033] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.
[0034] like Figures 1 to 4 As shown, the video stabilization method based on inertial measurement unit data and feature point matching proposed by the present invention includes the following steps:
[0035] Step 1 (corresponding to Figure 1 102 to 104), obtaining IMU data and estimating compensation parameters according to the IMU data, calculating an image after compensation relative to a previous frame based on the compensation parameters, setting a feature point deviation range of a current frame image relative to the compensated image, and reselecting N feature points in the previous frame image as reference feature points, wherein the IMU data is angular velocity and acceleration data collected by an inertial measurement unit from a camera;
[0036] First, the camera must be equipped with an IMU device, and the installation posture and the x, y, and z axes of the camera's photosensitive device must be kept parallel as much as possible. Then buffer the current frame image for backup, such as Figure 3 As shown, the target A1 in the current frame image should have been captured as follows if there was no jitter: Figure 2 The image is at position A. Due to the jitter of the camera, the position of target A has accumulated offsets. The actual offset can be expressed by the horizontal deviation p_x, the vertical deviation p_y, and the z-axis deviation angle p_a.
[0037] Among them, obtaining IMU data and estimating compensation parameters based on the IMU data, and calculating the compensated image relative to the previous frame based on the compensation parameters can be implemented using existing technologies, which will not be described in detail in this embodiment. For example, the document "A Non-Linear Filter for Gyroscope-Based Video Stabilization" (A Nonlinear Filter for Gyroscope Video Stabilization).
[0038] In addition, if Figure 4As shown in the figure, after compensating target A1 with the compensation parameters estimated by IMU data, the compensated image at A2 generally has a certain deviation relative to target A. Target A2 has a horizontal deviation d_x, a vertical deviation d_y, and a z-axis deviation angle d_a relative to target A. However, this deviation is generally not too large, and a rough upper limit can be obtained based on experience, which is the approximate range of feature point deviation.
[0039] Step 2 (corresponding to Figure 1 101, 105 and 106), according to the current frame image and the compensated image of the previous frame, feature point matching is performed within the feature point deviation range estimated by the IMU, and it is determined whether the matched feature point pair is greater than N, if so, proceed to step three, if not, proceed to step four;
[0040] The method similar to OpenCV's estimateAffinePartial2D is used for feature point matching. In addition, N is an empirical value, generally not less than 10.
[0041] Step 3 (corresponding to Figure 1 107), the compensation parameters estimated by the IMU data are fine-tuned, and after each fine-tuning, the reference feature points and the compensation parameters after the current fine-tuning are calculated to obtain the adjustment parameter GSAD of the current fine-tuning amount, and the fine-tuning parameters corresponding to the optimal adjustment parameter GSAD are selected to calculate the final transformation parameters;
[0042] Step 4 (corresponding to Figure 1 108), using the IMU data estimated compensation parameters as the final transformation parameters.
[0043] Step 5 (corresponding to Figure 1 109), transforming the current frame image according to the final transformation parameters to obtain the compensated image of the current frame.
[0044] Step 6 (corresponding to Figure 1 110), buffering the compensated image of the current frame, looping into step 2, and realizing stable video transmission.
[0045] According to steps 1 to 6, this embodiment obtains accurate image transformation parameters by means of secondary estimation. The first estimation process calculates the approximate offset parameters through IMU data; the second estimation process is performed by feature point matching in a smaller area near the approximate offset. Since the first estimation greatly reduces the search range of feature point matching, it not only saves search computing power, but also reduces the probability of feature point mismatching.
[0046] In step three, we first clarify a few definitions:
[0047] Definition 1: Gaussian function As shown in formula (1), the reason for deriving σ is that the empirical values of σ are different when the Gaussian function is used to weight the horizontal direction, vertical direction, and rotation direction;
[0048]
[0049] Definition 2: Feature area block FB(k,n,x,y): It means the nth image block of the kth frame, the center point is x,y, and the side length is a fixed value (usually 8-16 pixels).
[0050] Definition 3: Mean-centered feature region block AFB(k, n, x, y): It means to subtract the mean of the region block from each pixel value of FB(k, n, x, y) (this operation can offset the change of light brightness to a certain extent) to obtain a new pixel value. When AFB(k, n, x, y) is used as a reference block with a certain position, the last two parameters (x, y) are fixed values, so it is simplified to two parameters AFB(k, n). When AFB(k, n, x, y) is used to match a candidate block when searching for the best match, x and y are variable, so the form of 4 parameters is retained.
[0051] Definition 4. Metric SAD(k,n,x,y) is the sum of the absolute value of the pixel-by-pixel difference between AFB(k-1,n) and AFB(k,n,x,y). It is essentially a metric to measure the difference between two block regions in different regions of two frames.
[0052] Definition 5: GSAD(k, Δθ, Δx, Δy) is as shown in formula (2):
[0053]
[0054] where σ θ , σ x , σ y It is empirical data, which is related to the accuracy of the IMU device. MX(·) and MY(·) represent the rotation transformation with offset in the X direction and Y direction respectively. The calculation method of MX(△θ, Δx, Δy) and MY(Δθ, Δx, Δy) is shown in the formula:
[0055]
[0056] where θ b 、x b ,y bIt is the rotation angle and displacement of the corresponding feature point in the k-1 frame calculated by the IMU data of the n-th feature point. It can be regarded as the most likely position estimated by the IMU. It is used as a reference position for optimal search. Among them, Δθ, Δx, Δy are the rotation, horizontal and vertical fine-tuning amounts relative to the corresponding reference position during the fine-tuning process.
[0057] Based on the above definition, the final transformation parameters in step 3 are obtained in the following steps 201 to 210:
[0058] Step 201, setting the optimal fine-tuning amount Δθ opt , Δx opt , Δy opt Both are 0, GSAD opt Set to a maximum value; for example, 1000000;
[0059] Step 202, perform a traversal cycle of the first layer rotation angle within a certain range (the empirical value is not greater than [-10, 10]) according to a certain step size (the empirical value is 1-2 degrees), and the current value is recorded as Δθ;
[0060] Step 203, perform a traversal loop of the second layer x offset in a certain range (experienced value is 2 to 4 pixels) according to a certain step size (empirical value is not greater than [-20, 20]), and the current value is recorded as Δx;
[0061] Step 204, perform a traversal loop of the y offset of the third layer in a certain range (the empirical value is not greater than [-16, 16]) according to a certain step size (the empirical value is 2 to 4 pixels), and the current value is recorded as Δy;
[0062] Step 205, calculate the current fine-tuning parameter GSAD according to formula (2), and GSAD opt Compare, if less than GSAD opt , indicating that the current fine-tuning amount is better, and Δθ opt , Δx opt , Δy opt Change the value of △θ, Δx, Δy to the current value, and change GSAD to the current value. opt ; If GSAD is greater than GSAD opt Then continue the cycle directly;
[0063] Step 206, after the triple loop is completed, the current Δθ opt , Δx opt , Δy opt The corresponding optimal fine-tuning parameters are the final transformation parameters required in step three.
[0064] Practice shows that the method of this embodiment has more accurate parameter estimation and better image stabilization effect than the IMU estimation method. Compared with the feature point matching method, in addition to having comparable effects in some scenes, it can also work robustly and normally in scenes where other feature point matching methods fail.
[0065] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A video stabilization method based on inertial measurement unit data and feature point matching, characterized in that: The steps include: Step 1, obtaining IMU data and estimating compensation parameters based on the IMU data, calculating the compensated image relative to the previous frame image based on the compensation parameters, setting the feature point deviation range of the current frame image relative to the compensated image, and reselecting N feature points in the compensated image as reference feature points, wherein the IMU data is the angular velocity and acceleration data of the camera collected by the inertial measurement unit; Step 2: Match feature points within the feature point deviation range estimated by IMU according to the current frame image and the previous frame compensated image, and determine whether the matched feature point pair is greater than N. If so, proceed to step 3, otherwise proceed to step 4; Step 3: Fine-tune the compensation parameters estimated by the IMU data. After each fine-tuning, calculate the reference feature points and the compensation parameters after the current fine-tuning to obtain the tuning parameter GSAD of the current fine-tuning amount. Select the fine-tuning parameter corresponding to the optimal tuning parameter GSAD to calculate the final transformation parameter. Step 4: Use the IMU data estimated compensation parameters as the final transformation parameters; Step 5: transform the current frame image according to the final transformation parameters to obtain the current frame compensated image; Step 6: Buffer the compensated image of the current frame and loop into step 2 to achieve stable video transmission.
2. The video stabilization method based on inertial measurement unit data and feature point matching according to claim 1, characterized in that: In step 1, the current frame image is first buffered for backup.
3. The video stabilization method based on inertial measurement unit data and feature point matching according to claim 1, characterized in that: In step 2, the estimateAffinePartial2D method similar to OpenCV is used to match feature points.
4. The video stabilization method based on inertial measurement unit data and feature point matching according to claim 1, characterized in that: In step three, the final transformation parameters are obtained in the following steps 201 to 210: Step 201, setting the optimal rotation, horizontal and vertical fine-tuning amount Δθ opt ,Δx opt ,Δy opt Both are 0, GSAD opt Set to a maximum value; Step 202, performing a traversal cycle of the first layer rotation angle on the nth image block of the kth frame within a set range according to a set step length, and the current value is recorded as Δθ; Step 203, performing a traversal cycle of the second layer horizontal x offset on the n-th image block of the k-th frame within a set range according to the set step length, and the current value is recorded as Δx; Step 204, performing a traversal cycle of the third layer vertical y offset on the n-th image block of the k-th frame within a set range according to the set step length, and the current value is recorded as Δy; Step 205, calculate the current fine-tuning amount adjustment parameter GSAD, adjustment parameter GSAD and GSAD opt Compare, if the parameter GSAD is less than GSAD opt , indicating that the current fine-tuning amount is better, and Δθ opt , Δx opt , Δy opt Change the value of Δθ, Δx, Δy to the current one, and change the parameter GSAD to the current one opt ; If the parameter GSAD is greater than GSAD opt Then continue the cycle directly; Step 206, after the triple loop is completed, the current Δθ opt , Δx opt , Δy opt The corresponding optimal fine-tuning parameters are the final transformation parameters.
5. The video stabilization method based on inertial measurement unit data and feature point matching according to claim 4, characterized in that: In step 205, the current fine-tuning amount adjustment parameter GSAD is calculated using the following calculation formula: Among them, σ θ , σ x , σ y is empirical data, G(·) is a Gaussian function, SAD(·) is the sum of the absolute values of the pixel-by-pixel differences between the feature region block and the averaged feature region block, and is a metric. MX(·) and MY(·) represent rotation transformations with offsets in the X and Y directions, respectively.
6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the video stabilization method according to any one of claims 1 to 5 is implemented.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the video stabilization method according to any one of claims 1 to 5.