Image anti-shake method and anti-shake system

By optimizing feature point matching and global motion estimation, separating and compensating camera shake displacement, the unstable problem caused by camera shake in image processing is solved, and efficient and accurate image anti-shake effect is achieved, suitable for high-definition video shooting.

CN118828207BActive Publication Date: 2024-11-29GUOCHUANG RUISHI ARTIFICIAL INTELLIGENCE TECHNOLOGY (SICHUAN) CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411289273.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-14
Publication Date
2024-11-29
Estimated Expiration
2044-09-14

AI Technical Summary

Technical Problem

The prior art is difficult to effectively reduce or eliminate image instability caused by camera shake in image processing, especially in high-definition video shooting and motion shooting scenes. The traditional hardware anti-shake method is costly and complex in structure, making it difficult to implement on small devices.

Method used

By optimizing the feature point matching algorithm, the robustness of global motion estimation is enhanced, and an accurate jitter displacement calculation method is introduced to achieve image stabilization. The specific steps include matching feature points, displacement calculation, global transformation matrix estimation and displacement compensation for the frame image, separating the camera's physical displacement and jitter displacement, and performing jitter displacement compensation.

Benefits of technology

It achieves a more accurate and efficient image anti-shake effect, improves the accuracy and stability of anti-shake, is suitable for complex scenarios, and reduces equipment costs and structural complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118828207B_ABST
    Figure CN118828207B_ABST
Patent Text Reader

Abstract

The present invention provides an image anti-shake method and an anti-shake system, which relate to the field of image processing technology. The method comprises the following steps: matching feature points of a first frame image and a second frame image to obtain at least one feature point pair, wherein the first frame image and the second frame image have an overlapping area; calculating displacement of all matched feature point pairs to obtain displacement between the feature point pairs, wherein the displacement includes physical displacement of camera movement and jitter displacement caused by jitter; estimating a global transformation matrix between the first frame image and the second frame image based on all matched feature point pairs; stripping the physical displacement from the displacement based on the global transformation between the first frame image and the second frame image, and performing displacement compensation on the second frame image according to the jitter displacement after stripping. The present invention achieves a more accurate and efficient image anti-shake effect by optimizing a feature point matching algorithm, enhancing the robustness of global motion estimation, and introducing an accurate jitter displacement calculation method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image anti-shake method and an anti-shake system. Background Art

[0002] In the field of digital image processing, especially in the process of video capture and real-time transmission, image stabilization technology is a crucial technology. With the popularization of high-definition and even ultra-high-definition video content, users' requirements for video quality are increasing day by day. Especially in handheld shooting or sports shooting scenes, camera shake often causes video blur and shaking, seriously affecting the viewing experience. Therefore, how to effectively reduce or eliminate image instability caused by camera shake has become a technical problem that needs to be solved in the current image processing field.

[0003] Traditional image stabilization methods mainly rely on hardware means, such as using mechanical or optical stabilization devices to reduce the physical shake of the camera during shooting. However, such methods have limitations such as high cost, complex structure, and difficulty in implementation on small devices.

[0004] Therefore, it is necessary to provide an image stabilization method and an stabilization system to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an image stabilization method and an stabilization system, which achieve a more accurate and efficient image stabilization effect by optimizing the feature point matching algorithm, enhancing the robustness of the global motion estimation, and introducing an accurate jitter displacement calculation method.

[0006] The present invention provides an image anti-shake method, which comprises the following steps:

[0007] S1: performing feature point matching on a first frame image and a second frame image to obtain at least one feature point pair, wherein the first frame image and the second frame image have an overlapping area;

[0008] S2: Calculate the displacement of all matched feature point pairs to obtain the displacement between the feature point pairs, wherein the displacement includes the physical displacement caused by the camera movement and the jitter displacement caused by the jitter;

[0009] S3: estimating a global transformation matrix between the first frame image and the second frame image based on all matched feature point pairs, wherein the global transformation matrix represents a physical displacement between the first frame image and the second frame image caused by camera movement;

[0010] S4: stripping the physical displacement from the displacement based on a global transformation between the first frame image and the second frame image, and performing displacement compensation on the second frame image according to the jitter displacement after stripping.

[0011] Preferably, step S1 comprises the following steps:

[0012] S11: preprocessing the first frame image and the second frame image, wherein the preprocessing includes grayscale conversion, noise reduction, and image enhancement;

[0013] S12: using a feature point detection algorithm to detect feature points in the first frame image and the second frame image respectively;

[0014] S13: Describing the detected feature points using feature descriptors, and establishing at least one feature point pair between the first frame image and the second frame image by similarity matching between the feature descriptors.

[0015] Preferably, step S2 comprises the following steps:

[0016] S21: for each pair of matched feature points, calculating the coordinate positions of the feature point pairs in the first frame image and the second frame image;

[0017] S22: Based on the coordinate positions, the displacement between the feature point pairs is calculated using geometric transformation.

[0018] Preferably, step S3 comprises the following steps:

[0019] S31: Use a robust estimation method to select the inliers that meet the global transformation from all matching feature point pairs;

[0020] S32: Based on the screened inliers, a global transformation matrix is ​​estimated using a least square method, wherein the global transformation matrix represents a physical displacement between the first frame image and the second frame image caused by camera movement.

[0021] Preferably, step S4 comprises the following steps:

[0022] S41: for each matched feature point pair, using the global transformation matrix to remove the physical displacement in the calculated displacement, and retain the jitter displacement;

[0023] S42: calculating an offset according to the jitter displacement after stripping of all feature points;

[0024] S43: performing a displacement transformation in the opposite direction on the second frame image according to the offset to compensate for the displacement caused by the shaking, so that the second frame image is visually aligned with the first frame image.

[0025] Preferably, the offset calculation method is:

[0026] For all matched feature point pairs, the jitter displacement of each point pair in the X direction and the Y direction is calculated, and the average value of all jitter displacements in the X direction and the Y direction is taken as the offset.

[0027] Preferably, it also includes:

[0028] The second frame image after anti-shake processing is synthesized with the first frame image to generate a stable video stream.

[0029] The present invention also provides an image stabilization system, which is applied to an image stabilization method. The stabilization system comprises:

[0030] A feature point matching module, used for matching feature points of a first frame image and a second frame image to obtain at least one feature point pair, wherein the first frame image and the second frame image have an overlapping area;

[0031] A displacement calculation module, used to calculate the displacement of all matched feature point pairs to obtain the displacement between the feature point pairs, wherein the displacement includes the physical displacement of the camera movement and the jitter displacement caused by the jitter;

[0032] A matrix estimation module, configured to estimate a global transformation matrix between the first frame image and the second frame image based on all matched feature point pairs, wherein the global transformation matrix represents a physical displacement between the first frame image and the second frame image caused by camera movement;

[0033] The displacement compensation module is used to separate the physical displacement from the displacement based on the global transformation between the first frame image and the second frame image, and to perform displacement compensation on the second frame image according to the jitter displacement after the separation.

[0034] Compared with the related art, the image stabilization method and stabilization system provided by the present invention have the following beneficial effects:

[0035] The present invention establishes feature point pairs between consecutive frame images through feature point matching technology. These feature point pairs can accurately reflect the correspondence between images. Subsequently, the physical displacement caused by camera movement and the displacement caused by jitter are separated by calculating the displacement between the feature point pairs and combining the estimation of the global transformation matrix. Finally, the image is displacement compensated based on the stripped jitter displacement, thereby achieving image stabilization.

[0036] The image anti-shake method of the present invention not only improves the accuracy and effect of anti-shake, but also can maintain high stability and robustness in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of an image anti-shake method provided by the present invention;

[0038] Figure 2 A module structure diagram of an image anti-shake method provided by the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0040] It should also be noted that, for ease of description, only the parts related to the present invention, but not all of the contents, are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the various operations (or steps) as being processed sequentially, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the various operations can be rearranged. The processing can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0041] Embodiment 1

[0042] The present invention provides an image anti-shake method, referring to Figure 1 As shown, the anti-shake method comprises the following steps:

[0043] S1: Perform feature point matching on a first frame image and a second frame image to obtain at least one feature point pair, wherein the first frame image and the second frame image have an overlapping area.

[0044] In this embodiment, step S1 includes the following steps:

[0045] S11: preprocessing the first frame image and the second frame image, wherein the preprocessing includes grayscale conversion, noise reduction, and image enhancement.

[0046] In this embodiment, step S11 is to pre-process the first frame image and the second frame image to ensure the accuracy of subsequent feature point detection and matching. The pre-processing includes the following main steps:

[0047] Grayscale: Convert a color image into a grayscale image. This step can reduce the amount of computational work in image processing because a grayscale image only contains brightness information but not color information. At the same time, grayscale can also remove the interference of color on feature point detection and improve detection accuracy.

[0048] Noise reduction: Use Gaussian filter to perform noise reduction on grayscale images. The purpose of noise reduction is to remove noise points in the image, which may interfere with the detection of feature points. Noise reduction can improve the signal-to-noise ratio of the image and make feature point detection more stable and reliable.

[0049] Image enhancement: Enhance the denoised image to improve the contrast and clarity of the image, which usually includes histogram equalization, sharpening and other operations. Image enhancement can highlight the edge and texture information in the image, making it easier for the feature point detection algorithm to find stable feature points.

[0050] After preprocessing, the image quality is significantly improved, the noise is effectively suppressed, and the feature information is enhanced, which provides good input data for subsequent feature point detection and matching, and helps to improve the accuracy and robustness of anti-shake processing.

[0051] S12: using a feature point detection algorithm to detect feature points in the first frame image and the second frame image respectively.

[0052] In this embodiment, a feature point detection algorithm (including but not limited to SIFT, SURF, and ORB) is used to detect feature points in the first frame image and the second frame image respectively. The feature point detection algorithm can automatically identify key points in the image. These points are usually areas in the image where local information changes greatly, such as corner points, edge points, etc.

[0053] Through feature point detection, a series of stable feature points can be found in the first frame image and the second frame image respectively. These feature points will become the basis for subsequent feature matching and displacement calculation, which is crucial for anti-shake processing.

[0054] S13: Describing the detected feature points using feature descriptors, and establishing at least one feature point pair between the first frame image and the second frame image by similarity matching between the feature descriptors.

[0055] In this embodiment, the detected feature points are described using feature descriptors, which are high-dimensional vectors used to represent local image information around the feature points. Then, by calculating the similarity between the feature descriptors (which can be represented by including but not limited to Euclidean distance and Hamming distance), feature point pairs are established between the first frame image and the second frame image.

[0056] Feature point matching can find feature point pairs with similar local information between two frames of images. These feature point pairs not only contain the physical displacement information of the camera, but also the tiny displacement information caused by jitter. Feature point matching provides a reliable data basis for subsequent calculation of displacement and estimation of the global transformation matrix.

[0057] S2: Calculate the displacement of all matched feature point pairs to obtain the displacement between the feature point pairs, wherein the displacement includes the physical displacement caused by camera movement and the jitter displacement caused by jitter.

[0058] In this embodiment, step S2 includes the following steps:

[0059] S21: For each pair of matched feature points, calculate the coordinate positions of the feature point pair in the first frame image and the second frame image.

[0060] In this embodiment, for each pair of feature points obtained by matching feature descriptors, it is first necessary to determine the specific coordinate positions of these feature points in their respective frames (i.e., the first frame image and the second frame image). This is done by traversing the list of detected feature points and using the position information provided by the feature point detection algorithm, where the position information is pixel coordinates.

[0061] Specifically, the following steps can be used:

[0062] Read coordinates: Read the coordinate values ​​of each feature point from the output of the feature point detection algorithm ( x , y ).

[0063] Pairing association: According to the result of feature point pair matching, the coordinates of the feature points in the first frame image are associated with the coordinates of the corresponding matched feature points in the second frame image.

[0064] Store coordinates: Store the coordinates of the paired feature point pairs for subsequent displacement calculation.

[0065] Through step S21, the specific coordinate positions of each pair of matching feature points in the two frames of images are accurately obtained, which is the basis for the subsequent calculation of the displacement between feature point pairs and is crucial to ensuring the precision and accuracy of the displacement calculation. Accurate coordinate information can reflect the real image changes caused by camera movement and jitter, and provides reliable data support for subsequent global transformation matrix estimation and jitter displacement stripping.

[0066] S22: Based on the coordinate positions, the displacement between the feature point pairs is calculated using geometric transformation.

[0067] In this embodiment, based on the coordinate position of the feature point pair obtained in step S21, the displacement between the feature point pairs is calculated using the principle of geometric transformation, specifically, the Euclidean distance or vector difference between the two points in two-dimensional space is calculated to obtain the relative movement distance of the feature points in the X direction and the Y direction.

[0068] Specifically, the following steps can be used:

[0069] Calculate vector difference: For each pair of feature points, calculate the vector difference between their coordinates in the second frame image and their coordinates in the first frame image.

[0070] Determine the displacement: Use the vector difference as the displacement between the feature point pairs. This displacement includes both the physical displacement caused by camera movement and the jitter displacement caused by jitter.

[0071] Store displacement: Store the calculated displacement of each pair of feature points for use in subsequent steps.

[0072] The displacement between each pair of matching feature points is calculated through step S22, which includes both the physical displacement caused by camera movement and the small displacement caused by jitter. This provides the necessary data for the subsequent global transformation matrix estimation, and also lays the foundation for stripping off the physical displacement and retaining the jitter displacement for displacement compensation.

[0073] S3: Estimate a global transformation matrix between the first frame image and the second frame image based on all matched feature point pairs, wherein the global transformation matrix represents a physical displacement between the first frame image and the second frame image caused by camera movement.

[0074] In this embodiment, step S3 includes the following steps:

[0075] S31: A robust estimation method is used to select the inliers that conform to the global transformation from all matching feature point pairs.

[0076] In this embodiment, in order to screen out the inliers that truly conform to the global transformation (i.e., the physical displacement mainly caused by camera movement) from all matching feature point pairs, the present invention adopts robust estimation methods, including but not limited to the random sampling consensus algorithm (RANSAC) and maximum likelihood estimation (MLESAC). These methods can effectively process noise and outliers in the data and improve the accuracy and robustness of the global transformation matrix estimation.

[0077] Exemplarily, the RANSAC algorithm randomly selects a small number of feature point pairs as initial samples, assuming that these point pairs all conform to a certain global transformation model, and calculates the initial global transformation matrix based on these point pairs. Then, it calculates the errors of the remaining feature point pairs under the transformation matrix (i.e., the difference between the actual displacement and the predicted displacement), and regards the point pairs with errors less than a certain threshold as inliers, assuming that they truly conform to the global transformation. This process is repeated many times, each time selecting different initial samples, and finally selecting the transformation matrix with the largest number of inliers or the smallest total error as the final result.

[0078] By screening out the inliers that conform to the global transformation, outliers due to mismatch, noise or jitter can be excluded, thereby improving the accuracy of the global transformation matrix estimation.

[0079] S32: Based on the screened inliers, a global transformation matrix is ​​estimated using a least square method, wherein the global transformation matrix represents a physical displacement between the first frame image and the second frame image caused by camera movement.

[0080] In this embodiment, based on the inliers screened out in step S31, the least squares method is used to estimate the global transformation matrix. Specifically, the least squares method is a mathematical optimization technique that finds the best function match for the data by minimizing the sum of squares of errors. Here, the error refers to the displacement difference between the actual transformation of the inliers and the assumed global transformation.

[0081] In the specific implementation, we first construct an overdetermined system of equations (i.e., a system of equations with more equations than unknowns) based on the coordinate positions of the interior points, where the unknowns are the parameters of the global transformation matrix. Then, we use the least squares method to solve the system of equations to obtain the optimal estimate of the global transformation matrix.

[0082] By estimating the global transformation matrix using the least squares method, the physical displacement between the first frame image and the second frame image caused by camera movement can be accurately quantified; at the same time, the accurate global transformation matrix provides a solid foundation for the subsequent separation of the physical displacement from the displacement, making the calculation of the jitter displacement more accurate, thereby optimizing the anti-shake effect.

[0083] S4: stripping the physical displacement from the displacement based on a global transformation between the first frame image and the second frame image, and performing displacement compensation on the second frame image according to the jitter displacement after stripping.

[0084] In this embodiment, step S4 includes the following steps:

[0085] S41: For each matched feature point pair, the physical displacement in the calculated displacement is removed using the global transformation matrix, and the jitter displacement is retained.

[0086] In this embodiment, for each pair of feature point pairs that have been matched and calculated for displacement, the global transformation matrix estimated in the previous step S3 is used to decompose these displacements. The global transformation matrix accurately describes the physical displacement caused by camera movement (such as translation, rotation, etc.). By operating the displacement vector of each feature point pair with the global transformation matrix, the displacement caused by the physical movement of the camera can be stripped out, and the displacement caused only by camera shake, that is, the shake displacement, is retained.

[0087] This step enables subsequent steps to focus on compensating for the jitter displacement without being affected by the physical movement of the camera. By accurately stripping off the physical displacement, the accuracy of jitter displacement estimation and the accuracy of subsequent compensation are improved, thereby enhancing the effect of image stabilization.

[0088] S42: Calculate the offset according to the jitter displacement after stripping of all feature point pairs.

[0089] In this embodiment, based on the jitter displacement after stripping of all feature point pairs, an overall offset is calculated, and this offset represents the average displacement of the second frame image relative to the first frame image due to jitter. The specific calculation method is to calculate the jitter displacement in the X direction and the Y direction for all matching feature point pairs, and then take the average value of these jitter displacements in their respective directions to obtain the final offset.

[0090] By calculating the offset, the complex jitter displacement is simplified into a unified and easy-to-process vector, providing a direct basis for subsequent image displacement compensation. This offset accurately reflects the distance and direction that the second frame image needs to move in order to maintain visual alignment with the first frame image.

[0091] S43: performing a displacement transformation in the opposite direction on the second frame image according to the offset to compensate for the displacement caused by the shaking, so that the second frame image is visually aligned with the first frame image.

[0092] In this embodiment, the second frame image is subjected to a displacement transformation in the opposite direction according to the offset calculated in step S42. Specifically, the second frame image is moved the same distance in the opposite direction of the offset to compensate for the displacement caused by the shaking.

[0093] Through displacement compensation, the pixels in the second frame are repositioned to where they would be if there was no shaking, so that they are visually aligned with the first frame. This not only eliminates image blur and ghosting caused by camera shake, but also improves the continuity and viewing experience of the video.

[0094] Additionally, it includes:

[0095] The second frame image after anti-shake processing is synthesized with the first frame image to generate a stable video stream.

[0096] The working principle of an image anti-shake method provided by the present invention is as follows:

[0097] First, we start with feature point matching of consecutive frame images (i.e., the first frame and the second frame). In this step, we first preprocess the image, including grayscale conversion to reduce the amount of calculation, denoising to remove image noise interference, and image enhancement to highlight feature points for subsequent detection. Then, we use a feature point detection algorithm to detect key points in the two frames of images respectively, and describe these key points through feature descriptors. We use the similarity matching between descriptors to establish feature point pairs between the first frame and the second frame. These feature point pairs are the basis for subsequent analysis of camera motion and jitter.

[0098] Next, for each pair of matched feature points, the difference in their coordinate positions in the two frames of images is calculated. This difference is the displacement between the feature point pairs, which is the superposition of the physical displacement caused by camera movement and the displacement caused by jitter.

[0099] In order to separate the physical displacement caused only by camera movement from the mixed displacement, the method adopts the estimation of the global transformation matrix. This method screens out the internal points that conform to the global motion model (i.e., the feature point pairs of the displacement mainly caused by camera movement) through a robust estimation method, and uses these internal points to calculate the global transformation matrix through the least squares method. This matrix accurately describes the physical displacement caused by camera movement between the first frame image and the second frame image.

[0100] Once the global transformation matrix is ​​obtained, it can be used to separate the physical displacement from the displacement of each pair of feature points, thereby retaining only the displacement caused by jitter. This step is the core of the anti-shake processing and provides accurate data for subsequent compensation of jitter displacement.

[0101] Subsequently, an overall offset is calculated based on the jitter displacement after stripping of all feature points. This offset represents the average displacement of the second frame image relative to the first frame image due to jitter. Finally, this offset is used to perform a displacement transformation on the second frame image in the opposite direction to compensate for the displacement caused by the jitter, so that the second frame image is visually aligned with the first frame image, thereby achieving an anti-shake effect.

[0102] Finally, the second frame image after anti-shake processing is synthesized with the first frame image to generate a more stable video stream. This process is cyclically applied to consecutive video frames, which can significantly improve the stability and viewing experience of the video.

[0103] Embodiment 2

[0104] The present invention also provides an image anti-shake system, which is applied to an image anti-shake method, referring to Figure 2 As shown, the anti-shake system includes:

[0105] The feature point matching module 100 is used to perform feature point matching on a first frame image and a second frame image to obtain at least one feature point pair, wherein the first frame image and the second frame image have an overlapping area.

[0106] The displacement calculation module 200 is used to perform displacement calculation on all matched feature point pairs to obtain the displacement between the feature point pairs, wherein the displacement includes the physical displacement caused by camera movement and the jitter displacement caused by jitter.

[0107] The matrix estimation module 300 is used to estimate the global transformation matrix between the first frame image and the second frame image based on all matched feature point pairs, wherein the global transformation matrix represents the physical displacement between the first frame image and the second frame image caused by camera movement.

[0108] The displacement compensation module 400 is used to separate the physical displacement from the displacement based on the global transformation between the first frame image and the second frame image, and perform displacement compensation on the second frame image according to the jitter displacement after the separation.

[0109] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically-erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0111] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or includes elements that are inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

Claims

1. An image stabilization method, characterized in that: The anti-shake method comprises the following steps: S1: performing feature point matching on a first frame image and a second frame image to obtain at least one feature point pair, wherein the first frame image and the second frame image have an overlapping area; S2: Calculate the displacement of all matched feature point pairs to obtain the displacement between the feature point pairs, wherein the displacement includes the physical displacement caused by the camera movement and the jitter displacement caused by the jitter; S3: estimating a global transformation matrix between the first frame image and the second frame image based on all matched feature point pairs, wherein the global transformation matrix represents a physical displacement between the first frame image and the second frame image caused by camera movement; Wherein, step S3 comprises the following steps: S31: Use a robust estimation method to select the inliers that meet the global transformation from all matching feature point pairs; S32: estimating a global transformation matrix based on the screened inliers using a least squares method, wherein the global transformation matrix represents a physical displacement between the first frame image and the second frame image caused by camera movement; S4: stripping the physical displacement from the displacement based on a global transformation between the first frame image and the second frame image, and performing displacement compensation on the second frame image according to the jitter displacement after stripping; Step S4 includes the following steps: S41: for each matched feature point pair, using the global transformation matrix to remove the physical displacement in the calculated displacement, and retain the jitter displacement; S42: calculating an offset according to the jitter displacement after stripping of all feature points; S43: performing a displacement transformation in the opposite direction on the second frame image according to the offset to compensate for the displacement caused by the shaking, so that the second frame image is visually aligned with the first frame image.

2. The image stabilization method according to claim 1, characterized in that: Step S1 includes the following steps: S11: preprocessing the first frame image and the second frame image, wherein the preprocessing includes grayscale conversion, noise reduction, and image enhancement; S12: using a feature point detection algorithm to detect feature points in the first frame image and the second frame image respectively; S13: Describing the detected feature points using feature descriptors, and establishing at least one feature point pair between the first frame image and the second frame image by similarity matching between the feature descriptors.

3. The image stabilization method according to claim 2, characterized in that: Step S2 includes the following steps: S21: for each pair of matched feature points, calculating the coordinate positions of the feature point pairs in the first frame image and the second frame image; S22: Based on the coordinate positions, the displacement between the feature point pairs is calculated using geometric transformation.

4. The image stabilization method according to claim 3, characterized in that: In step S42, the offset calculation method is: For all matched feature point pairs, the jitter displacement of each point pair in the X direction and the Y direction is calculated, and the average value of all jitter displacements in the X direction and the Y direction is taken as the offset.

5. The image stabilization method according to claim 4, characterized in that: Also includes: The second frame image after anti-shake processing is synthesized with the first frame image to generate a stable video stream.

6. An image stabilization system, applied to an image stabilization method according to any one of claims 1 to 5, characterized in that: The anti-shake system comprises: A feature point matching module, used for matching feature points of a first frame image and a second frame image to obtain at least one feature point pair, wherein the first frame image and the second frame image have an overlapping area; A displacement calculation module, used to calculate the displacement of all matched feature point pairs to obtain the displacement between the feature point pairs, wherein the displacement includes the physical displacement of the camera movement and the jitter displacement caused by the jitter; A matrix estimation module, configured to estimate a global transformation matrix between the first frame image and the second frame image based on all matched feature point pairs, wherein the global transformation matrix represents a physical displacement between the first frame image and the second frame image caused by camera movement; The displacement compensation module is used to separate the physical displacement from the displacement based on the global transformation between the first frame image and the second frame image, and to perform displacement compensation on the second frame image according to the jitter displacement after the separation.

Citation Information

Patent Citations

  • Electronic image stabilization method based on inertial navigation information

    CN102780846A

  • Fast digital image stabilization method applicable to space robot visual system

    CN104144282A