A video de-shaking method suitable for large zoom cameras
By using a combination method of feature point matching and Kalman filters in a large zoom camera, the jitter problem caused by focal length changes is solved, stable image repair and user experience improvement are achieved, and hardware costs are reduced.
Patent Information
- Application Number
- CN202111424140.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-26
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-11-26
AI Technical Summary
The existing camera deshaking algorithm fails to effectively handle the jitter caused by large zoom cameras when the focal length changes, resulting in blurred image and reduced user experience.
By discrete video images into pictures and perform grayscale processing, feature points are marked, feature points are matched using Mahayana distance, transform relationship equations between adjacent images are constructed, and motion compensation is combined with Kalman filter to smooth the camera's Movement trajectory, realizes motion repair of images.
It effectively eliminates the jitter of the large zoom camera under focal length changes, improves image stability and user experience, reduces hardware costs, and avoids the black edge problem when image zooming.
Smart Images

Figure CN114143459B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of camera video image processing, and is particularly suitable for eliminating the shaking phenomenon caused by zooming in a large zoom camera to view local details. Background Art
[0002] Camera shake removal has always been a research focus in the field of image processing. Cameras will shake due to interference from various factors in the natural environment, as well as involuntary shaking by the operator. These shakes will cause image blur and reduce the viewer's experience. The presence of jittery images will have a serious impact on subsequent target monitoring and tracking functions. Therefore, research on camera shake removal is of great significance.
[0003] The essence of camera image jitter is that there is a tiny, randomly shifted displacement between two adjacent frames of images. Existing jitter elimination methods can be divided into hardware jitter elimination and software algorithm jitter elimination.
[0004] The existing methods of camera shake reduction processing in hardware are from two directions: the camera bracket and the lens. For the fixed bracket of the camera, the existing methods mostly use materials with high rigidity to make the bracket, for example, using professional photography stands, etc. This method is suitable for fixed cameras. For movable cameras, such as mobile phones, camcorders, etc., lenses with physical anti-shake function are usually used. Using hardware camera shake reduction processing will increase the overall cost of the camera system. The camera usage environment is relatively complex, and the camera shake in the natural environment is mostly complex and irregular.
[0005] The camera is de-shaked by software, and the de-shake algorithm can be divided into 2D de-shake algorithm and 3D de-shake algorithm.
[0006] The 2D de-jitter algorithm obtains the feature points on two adjacent video images through video, calculates the changes of the camera between the two frames of images by matching the feature points, smoothes multiple groups of postures, predicts and estimates the camera posture, and performs motion repair on the image.
[0007] The 3D de-shaking algorithm performs de-shaking processing on the depth camera, matches the point cloud data obtained by the depth camera through the point cloud, obtains the camera posture through the matching result, and performs image restoration based on the camera posture.
[0008] Most of the existing camera shake reduction algorithms do not take into account changes in the camera's focal length, especially for cameras that can achieve large zoom. When the user zooms the lens in, the camera's focal length changes significantly, and the original video shake phenomenon will become more serious. The existing technology does not take image scaling into consideration, so there will be large errors in feature point extraction and matching. The existing shake reduction algorithms are not suitable for the shake reduction processing of cameras with large zoom. Summary of the invention
[0009] In view of the deficiencies in the above technical background, the present invention proposes a video de-shaking method suitable for a large zoom camera. The large zoom camera changes the focal length of the camera, zooms in the picture, and views local details. As the picture is zoomed in, the camera shake will also be magnified. The present invention establishes the relationship between the focal length and the shake amplitude. The motion trajectory of the camera is smoothed by Kalman filtering, and motion compensation is performed on the image, so that the camera has a smoother effect at different focal lengths.
[0010] In order to achieve the above-mentioned invention purpose and solve the problems existing in the existing construction technology, the technical solution adopted by the present invention is:
[0011] A video de-shaking method applicable to a large-zoom camera comprises the following steps:
[0012] Step 1: Discretize the continuous video into a continuous sequence of pictures, and grayscale the pictures. Compare the grayscale values of each pixel with those of the surrounding pixels. If the grayscale value is greater than the grayscale values of all surrounding pixels, it is marked as a feature point.
[0013] Step 2: Take two adjacent frames of images, match the feature points on the two adjacent frames of images according to the principle of minimum Mahalanobis distance, and construct a set of corresponding feature points on the adjacent images;
[0014] Step 3: Calculate the threshold of feature point matching according to the focal length of the camera, and eliminate the feature point combination whose Mahalanobis distance between corresponding feature points on adjacent images in the feature point set is greater than the threshold;
[0015] Step 4: Construct the transformation relationship equation between two adjacent frames of image data, bring the corresponding feature point set obtained in step 3 into the least squares algorithm, and obtain the relative transformation equation between the two frames of image data, including the scaling coefficient, rotation transformation matrix and translation vector;
[0016] Step 5: Based on the relative transformation equation between the images obtained in step 4, the prediction equation of the Kalman filter is constructed, and the coordinate points of one image in two adjacent frames of images are used as state variables and the coordinate points of the other image are used as observation variables to construct the observation equation of the Kalman filter, and the Kalman gain matrix is solved to form a Kalman filter;
[0017] Step 6: Perform image filtering based on the Kalman filter to obtain an image after jitter elimination.
[0018] Wherein, step 3 specifically includes the following process:
[0019] (1) Record the maximum focal length of the camera and the minimum focal length of the camera, and normalize the focal length of the camera;
[0020] (2) Adjust the camera to the maximum focal length and traverse the feature point combinations in the feature point set to find the minimum Mahalanobis distance;
[0021] (3) Calculate the post-fusion matching threshold, expressed as:
[0022] h=f m h min
[0023] Among them, f m Normalized value for the focal length of the current camera, h min is the minimum Mahalanobis distance at maximum focal length.
[0024] Among them, in step 4, the transformation relationship equation between two adjacent frames of image data is expressed as:
[0025]
[0026] Where k is the scaling coefficient, (x, y) and (x′, y′) are the pixel coordinates of the two frames of images, and θ is the rotation angle;
[0027] is the rotation transformation matrix;
[0028] is the translation vector.
[0029] Compared with the background technology, the present invention has the following advantages:
[0030] (1) The present invention uses a software filtering algorithm to eliminate jitter in a large zoom camera, which has the advantage of low hardware cost;
[0031] (2) In the process of feature point matching, the present invention corrects the threshold of feature point matching in real time through the focal length value, thereby improving the accuracy and real-time performance of feature point matching;
[0032] (3) In addition to considering traditional translation and rotation, the present invention also takes the scaling of the image into consideration, improves the image relative motion model, and prevents the black edge problem from appearing in the repaired image when the image is corrected at different scaling ratios.
[0033] (4) The present invention takes into account the impact of focal length change on jitter. Compared with the traditional camera jitter removal algorithm, the repaired image is more stable and the user experience is better. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 It is a schematic diagram of system connection of an embodiment of the present invention.
[0035] Figure 2It is a workflow diagram of the anti-shake processing method for a large-zoom camera according to an embodiment of the present invention.
[0036] Figure 3 Schematic diagram of the image restoration workflow according to an embodiment of the present invention. DETAILED DESCRIPTION
[0037] The present invention is further described below in conjunction with specific examples, which are only used to illustrate the present invention and are not used to limit the scope of the present invention.
[0038] like Figure 1 As shown in the figure, the system connection diagram commonly used in this example is divided into a data acquisition unit, a data processing unit and a human-computer interaction unit. The data acquisition unit uses a large zoom camera, which is fixed by a 2-DOF motion pan-tilt. The video image data is sent to the data processing unit via UDP. The data processing unit runs the algorithm designed in this paper to process the collected video data. The human-computer interaction unit is used to display the video image processed by the camera. At the same time, the user can control the rotation of the camera, the zoom of the screen, etc. through the human-computer interaction unit.
[0039] like Figure 2 As shown, it is a flow chart of this embodiment. Referring to the flow chart, the implementation process of this embodiment is as follows:
[0040] Step 1: Obtain video stream data from a large zoom camera, divide it into a continuous image sequence, grayscale the acquired images, and take two adjacent frames of video images as the i-th frame image and the j-th frame image respectively;
[0041] Perform traversal calculations on the i-th frame image and the j-th frame image, and denote the pixel point on the i-th frame image as X i (x i ,y i ), the point on the j-th frame image is X j (x j ,y j ), compare the gray value of the point with the gray value of the surrounding points. If it is greater than the gray value of all the surrounding points, the point is recorded as a feature point.
[0042] Step 2: Match the feature points on the two frames of images. In order to reduce the number of feature point pairs and improve the accuracy and real-time performance of image matching, an agreement is made on the minimum Mahalanobis distance between feature points. When the Mahalanobis distance between feature points is less than the threshold, the point can be considered to match the feature point, otherwise it is considered to be an invalid feature point.
[0043] The calculation formula of the Mahalanobis distance between feature points of two adjacent frames is as follows:
[0044]
[0045] X i , X j are the feature points on the i-th frame and the j-th frame;
[0046] S is X i , X j The covariance matrix between ;
[0047] When the Mahalanobis distance is less than the set threshold, it is determined to be a feature point pair. When multiple pairs of data are less than the threshold, the one with the smallest Mahalanobis distance is taken as the feature point pair. When all the traversed points are greater than the threshold, no feature point pair is found for this point.
[0048] Step 3: When the camera changes its focal length to zoom in or out, the focal length of the camera at this time can be obtained, the maximum focal length and the minimum focal length of the camera can be recorded, the focal length of the camera can be normalized, the camera can be modulated to the maximum focal length, the feature points of two adjacent frames can be extracted, the feature points can be traversed to obtain the minimum Mahalanobis distance, and the minimum Mahalanobis distance of the feature point multiplied by the normalized focal length can be used as the threshold for feature point matching.
[0049] The threshold calculation expression for feature point matching is: h = f m h min
[0050] Among them, f m is the normalized focal length; h min is the minimum Mahalanobis distance at maximum focal length.
[0051] Step 4: Based on the transformation equation between the two frames of images, construct the least squares optimization target equation. The least squares target optimization equation is as follows:
[0052] min∑||X j -X i ||
[0053] The constraint equation is the coordinate transformation equation between images:
[0054]
[0055] Where k is the scaling coefficient, (x, y) and (x′, y′) are the pixel coordinates of the two frames of images, and θ is the rotation angle;
[0056] is the rotation transformation matrix;
[0057] is the translation vector;
[0058] The scaling transformation coefficient k, rotation matrix and translation vector are solved through optimization iteration.
[0059] Step 5: Based on the relative transformation equation between the images obtained in step 4, the prediction equation of the Kalman filter is constructed, and the coordinate points of one image in two adjacent frames of images are used as state variables and the coordinate points of the other image are used as observation variables to construct the observation equation of the Kalman filter, and the Kalman gain matrix is solved to form a Kalman filter; Figure 3 As shown;
[0060] The present invention adopts the construction of a Kalman filter to eliminate the jitter error, and constructs a prediction equation based on the above-obtained scaling transformation coefficients, rotation matrix and translation vector, as shown below:
[0061]
[0062] in, is the predicted coordinate, X i is the pixel coordinate of the i-th frame image.
[0063] The Kalman gain matrix solution equation is as follows:
[0064] KPH T (HPH T +R) -1
[0065] Among them, P is the covariance matrix, H is the observation matrix, and R is the observation error.
[0066] Step 6: Filter the image using the Kalman filter to obtain the image after the jitter is eliminated. As shown below:
[0067]
[0068] in, is the corrected pixel coordinate, X j is the pixel coordinate of the j-th frame image.
[0069] The Kalman filter is used to achieve image restoration compensation. While completing the restoration of the moving image, the purpose of eliminating the jitter of the large zoom camera is achieved.
Claims
1. A video de-shaking method suitable for a large zoom camera, characterized in that: The following steps are involved: Step 1: Discretize the continuous video into a continuous sequence of pictures, and grayscale the pictures. Compare the grayscale values of each pixel with those of the surrounding pixels. If the grayscale value is greater than the grayscale values of all surrounding pixels, it is marked as a feature point. Step 2: Take two adjacent frames of images, match the feature points on the two adjacent frames of images according to the principle of minimum Mahalanobis distance, and construct a set of corresponding feature points on the adjacent images; Step 3: Calculate the threshold of feature point matching according to the focal length of the camera, and eliminate the feature point combination whose Mahalanobis distance between corresponding feature points on adjacent images in the feature point set is greater than the threshold; Step 4: Construct the transformation relationship equation between two adjacent frames of image data, bring the corresponding feature point set obtained in step 3 into the least squares algorithm, and obtain the relative transformation equation between the two frames of image data, including the scaling coefficient, rotation transformation matrix and translation vector; Step 5: Based on the relative transformation equation between the images obtained in step 4, the prediction equation of the Kalman filter is constructed, and the coordinate points of one image in two adjacent frames of images are used as state variables and the coordinate points of the other image are used as observation variables to construct the observation equation of the Kalman filter, and the Kalman gain matrix is solved to form a Kalman filter; Step 6: Perform image filtering based on the Kalman filter to obtain an image after jitter elimination; Wherein, step 3 specifically includes the following process: (1) Record the maximum focal length of the camera and the minimum focal length of the camera, and normalize the focal length of the camera; (2) Adjust the camera to the maximum focal length and traverse the feature point combinations in the feature point set to find the minimum Mahalanobis distance; (3) The minimum Mahalanobis distance of the feature point multiplied by the normalized focal length is used as the threshold of feature point matching. The calculation expression of the threshold of feature point matching is: h=f m h min Among them, f m Normalized value for the focal length of the current camera, h min is the minimum Mahalanobis distance at maximum focal length.
2. The video de-shaking method applicable to a large zoom camera according to claim 1, characterized in that: In step 4, the transformation relationship equation between two adjacent frames of image data is expressed as: Where k is the scaling coefficient, (x, y) and (x′, y′) are the pixel coordinates of the two frames of images, and θ is the rotation angle; is the rotation transformation matrix; is the translation vector.
