Anti-shake control method and device, and electronic device

Abstract

This application proposes a method, apparatus, and electronic device for image stabilization control. The method includes: acquiring a target image to be stabilized and a reference image acquired before the target image; performing feature point matching on the target image and the reference image to determine at least one set of matching feature point pairs in the target image and the reference image; determining the positional difference of the at least one set of feature point pairs based on the position of at least one first feature point in the first image and the position of at least one second feature point in the second image; determining a reference error based on the positional difference of the at least one set of feature point pairs; and performing image stabilization based on the reference error and the number of feature point pairs in the at least one set of feature point pairs to obtain the stabilized target image. This method effectively improves the image stabilization capability of the camera, avoids the use of additional special components, reduces costs, and improves the image stabilization effect.

Description

Technical Field

[0001] This application relates to the field of computer vision processing technology, and in particular to a stabilization control method, device and electronic device. Background Technology

[0002] With the development of technology, cameras have enriched people's lives. However, during the shooting process, factors such as the user's hand shaking, the camera's photosensitive components, and sensor compensation errors can lead to blurry images. Therefore, image stabilization is necessary.

[0003] In related technologies, optical image stabilization compensates for light shakiness caused by hand shake using movable components, thereby reducing image blur. However, optical image stabilization requires additional components, making it relatively expensive. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in the related art.

[0005] Therefore, the following technical solution is proposed:

[0006] The first aspect of this application provides a stabilization control method, including:

[0007] Acquire the target image to be stabilized, and the reference image acquired before the target image;

[0008] Feature point matching is performed on the target image and the reference image to determine at least one pair of feature points that match each other in the target image and the reference image, wherein the pair of feature points includes a first feature point located in the target image and a second feature point located in the reference image;

[0009] The positional difference of the at least one set of feature point pairs is determined based on the first image position of at least one first feature point in the at least one set of feature point pairs and the second image position of at least one second feature point.

[0010] The reference error is determined based on the positional differences of the at least one set of feature point pairs;

[0011] Based on the reference error and the number of feature point pairs in the at least one set of feature point pairs, anti-shake processing is performed to obtain the target image after anti-shake processing.

[0012] Optionally, the step of performing image stabilization based on the reference error and the number of feature point pairs in the at least one set of feature point pairs to obtain the image stabilized target image includes at least one of the following:

[0013] If the number of feature point pairs is greater than or equal to the threshold number and the reference error is less than the first error value, a feature point matching method is used for image stabilization to obtain the image stabilization target image.

[0014] When the number of feature point pairs is greater than or equal to the threshold number, the reference error is greater than or equal to the first error value, and the reference error is less than or equal to the second error value, a combined anti-shake method of feature point matching and gyroscope anti-shake is used to perform anti-shake processing to obtain the target image after anti-shake processing.

[0015] When the number of feature point pairs is greater than or equal to the threshold number and the reference error is greater than the second error value, a gyroscope anti-shake method is used to perform anti-shake processing to obtain the target image after anti-shake processing.

[0016] Optionally, the step of performing image stabilization based on the reference error and the number of feature point pairs in the at least one set of feature point pairs to obtain the image stabilized further includes:

[0017] If the number of feature point pairs in at least one set of feature point pairs is less than the threshold number, a gyroscope anti-shake method is used to perform anti-shake processing to obtain the target image after anti-shake processing.

[0018] Optionally, the image stabilization process employing a combination of feature point matching and gyroscope stabilization to obtain the stabilized target image includes:

[0019] The target image is subjected to anti-shake processing using a feature point matching method to obtain the first offset of each pixel in the target image;

[0020] The target image is subjected to gyroscope stabilization processing to obtain the second offset of each pixel in the target image;

[0021] Based on the reference error, the first offset and the second offset are weighted and summed to obtain the weighted offset.

[0022] Based on the weighted offset, the position of each pixel in the target image is transformed to obtain the target image after image stabilization.

[0023] Optionally, the step of performing anti-shake processing on the target image using a feature point matching method to obtain the first offset of each pixel in the target image includes:

[0024] A first image position of at least one first feature point and a second image position of at least one second feature point are obtained, and a mapping matrix is ​​obtained by matching using optical flow; wherein, the mapping matrix is ​​used to indicate the mapping relationship from the second image position to the first image position;

[0025] Based on the mapping matrix, an inverse affine transformation is performed on the target image to obtain the first offset of each pixel in the target image.

[0026] Optionally, the step of applying the gyroscope stabilization method to the target image to obtain a second offset for each pixel in the target image includes:

[0027] Acquire multiple jitter angular velocities acquired by the gyroscope between acquiring the target image and the reference image;

[0028] Kalman filtering is used to filter the multiple jitter angular velocities;

[0029] The second offset is determined based on the filtered multiple jitter angular velocities.

[0030] A second aspect of this application provides a shake stabilization control device, comprising:

[0031] The acquisition module is used to acquire the target image to be stabilized, as well as the reference image acquired before the target image;

[0032] A matching module is used to perform feature point matching on the target image and the reference image to determine at least one pair of feature points that match each other in the target image and the reference image, wherein the pair of feature points includes a first feature point located in the target image and a second feature point located in the reference image;

[0033] An error module is configured to determine the positional difference of at least one set of feature point pairs based on the first image position of at least one first feature point and the second image position of at least one second feature point; and to determine a reference error based on the positional difference of at least one set of feature point pairs.

[0034] The processing module is used to perform anti-shake processing based on the reference error and the number of feature point pairs in the at least one set of feature point pairs, to obtain the target image after anti-shake processing.

[0035] Optionally, the processing module is specifically used for:

[0036] If the number of feature point pairs is greater than or equal to the threshold number and the reference error is less than the first error value, a feature point matching method is used for image stabilization to obtain the image stabilization target image.

[0037] When the number of feature point pairs is greater than or equal to the threshold number, the reference error is greater than or equal to the first error value, and the reference error is less than or equal to the second error value, a combined anti-shake method of feature point matching and gyroscope anti-shake is used to perform anti-shake processing to obtain the target image after anti-shake processing.

[0038] When the number of feature point pairs is greater than or equal to the threshold number and the reference error is greater than the second error value, a gyroscope anti-shake method is used to perform anti-shake processing to obtain the target image after anti-shake processing.

[0039] Optionally, the processing module is further configured to:

[0040] If the number of feature point pairs is less than the threshold number, a gyroscope-based anti-shake method is used to perform anti-shake processing to obtain the target image after anti-shake processing.

[0041] Optionally, the processing module is further configured to:

[0042] The target image is subjected to anti-shake processing using a feature point matching method to obtain the first offset of each pixel in the target image;

[0043] The target image is subjected to gyroscope stabilization processing to obtain the second offset of each pixel in the target image;

[0044] Based on the reference error, the first offset and the second offset are weighted and summed to obtain the weighted offset.

[0045] Based on the weighted offset, the position of each pixel in the target image is transformed to obtain the target image after image stabilization.

[0046] Optionally, the processing module is specifically used for:

[0047] A first image position of at least one first feature point and a second image position of at least one second feature point are obtained, and a mapping matrix is ​​obtained by matching using optical flow; wherein, the mapping matrix is ​​used to indicate the mapping relationship from the second image position to the first image position;

[0048] Based on the mapping matrix, an inverse affine transformation is performed on the target image to obtain the first offset of each pixel in the target image.

[0049] Optionally, the processing module is specifically used for:

[0050] Acquire multiple jitter angular velocities acquired by the gyroscope between acquiring the target image and the reference image;

[0051] Kalman filtering is used to filter the multiple jitter angular velocities;

[0052] The second offset is determined based on the filtered multiple jitter angular velocities.

[0053] A third aspect of this application provides an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to invoke and execute the executable instructions stored in the memory to implement the anti-shake control method proposed in the first aspect of this application.

[0054] The fourth aspect of this application provides a non-transitory computer-readable storage medium that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the anti-shake control method proposed in the first aspect of this application.

[0055] A fifth aspect of this application provides a computer program product that, when executed by a processor of an electronic device, enables the electronic device to perform the anti-shake control method proposed in the first aspect of this application.

[0056] The technical solution of this application acquires a target image to be stabilized and a reference image acquired before the target image. Feature point matching is performed on the target image and the reference image to determine at least one set of matching feature point pairs in the target image and the reference image. Each feature point pair includes a first feature point located in the target image and a second feature point located in the reference image. Based on the first image position of at least one first feature point and the second image position of at least one second feature point in the at least one set of feature point pairs, the positional difference of the at least one set of feature point pairs is determined. A reference error is determined based on the positional difference of the at least one set of feature point pairs. Stabilization processing is then performed based on the reference error and the number of feature point pairs in the at least one set of feature point pairs to obtain a stabilized target image. This effectively improves the camera's stabilization capability, avoids the use of additional special components, reduces costs, and improves the stabilization effect.

[0057] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0058] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0059] Figure 1A schematic flowchart illustrating a stabilization control method provided in an embodiment of this application;

[0060] Figure 2 A flowchart illustrating another anti-shake control method provided in an embodiment of this application;

[0061] Figure 3 This is a schematic diagram of the structure of a shake-resistant control device provided in an embodiment of this application;

[0062] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0063] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0064] The following description, with reference to the accompanying drawings, describes a stabilization control method, apparatus, and electronic device according to embodiments of this application.

[0065] Figure 1 This is a flowchart illustrating a stabilization control method provided in an embodiment of this application.

[0066] like Figure 1 As shown, the image stabilization control method may include the following steps:

[0067] Step 101: Acquire the target image to be stabilized, and the reference image acquired before the target image.

[0068] In this context, the reference image refers to the image used as a reference for image stabilization, while the target image to be stabilized refers to the image in the frame that has shifted from the reference image due to shaking or rapid movement of a person or object in the shot. By using the reference image preceding the target image as a reference, image stabilization is applied to the target image, making the image presented after stabilization more consistent with the reference image. This makes it less perceptible to the user through changes in the image, thus achieving the effect of image stabilization.

[0069] Optionally, by acquiring images through a camera, the first frame captured after the camera is started can be used as the reference image, and subsequent frames that show offset can be used as the target images to be stabilized.

[0070] Optionally, by using preset rules, a certain image is selected as the target image to be stabilized, and images acquired adjacent to it or at intervals of several frames are used as reference images for stabilization.

[0071] In addition, the image with the highest edge sharpness of the reference object can be selected from several frames of images previously acquired for the target image as the reference image.

[0072] Step 102: Perform feature point matching on the target image and the reference image to determine at least one set of matching feature point pairs in the target image and the reference image. The feature point pair includes a first feature point located in the target image and a second feature point located in the reference image.

[0073] In the target image and the reference image, there are some points where the gray values ​​of the image change drastically or points with large curvature on the image edge (i.e., the intersection of two edges). In this step, these points are identified from the target image and the reference image respectively as feature points, and the information of these feature points, such as color features and texture features, is described.

[0074] Since feature points in the target image and the reference image can reflect the essential features of the corresponding imaging object, feature points corresponding to the same imaging object, i.e., feature point pairs, can be determined from different images by matching feature points with the same or similar feature point information.

[0075] Optionally, feature point information of the target image and the reference image can be obtained, and feature point matching can be performed.

[0076] Optionally, texture feature information of the target image and the reference image is acquired, and feature point matching is performed. The texture feature information includes at least one of edge feature information and corner feature information. Edge feature information refers to feature points of the image edge contour, which are pixels with abrupt gray-level changes. Corner points are the intersections of two edges, meaning the local neighborhood of a corner point has boundaries of two different regions in different directions.

[0077] Optionally, pyramid image segmentation is performed on the target image and the reference image to obtain feature point information of the segmented image.

[0078] Optionally, edge detection methods can be used to obtain edge feature information of the image, and corner detection algorithms can be used to obtain corner feature information of the image.

[0079] Optionally, an optical flow algorithm can be used to match feature points between the target image and the reference image.

[0080] Understandably, after performing feature point matching on the target image and the reference image and determining multiple matching feature points, the mapping relationship between the feature points in the target image and the reference image can be obtained based on the image positions of the matched feature points in the target image and the reference image. This mapping relationship can then be used to adjust the pixel positions of the target image so that the processed target image presents a picture that is more consistent with the reference image.

[0081] Step 103: Determine the positional difference of at least one set of feature point pairs based on the first image position of at least one first feature point and the second image position of at least one second feature point in at least one set of feature point pairs; and determine the reference error based on the positional difference of at least one set of feature point pairs.

[0082] The reference error is a parameter that measures the difference in the positions of matched feature points between the target image and the reference image; in other words, it measures the magnitude of the difference between the two images. The greater the difference between the two images, the larger the reference error.

[0083] Optionally, the reference error can be the average absolute error of the positional differences of at least one set of feature point pairs.

[0084] Step 104: Perform image stabilization based on the reference error and the number of feature point pairs in at least one set of feature point pairs to obtain the image stabilized target image.

[0085] The number of feature point pairs refers to the number of feature point pairs in the target image that match the reference image.

[0086] Optionally, the image stabilization method includes at least one of the following: image stabilization using a feature point matching method, image stabilization using a combination of feature point matching and gyroscope image stabilization, and image stabilization using a gyroscope image stabilization method.

[0087] Based on the reference error and the number of feature point pairs that match at least one set of feature point pairs in the target image, various possible image stabilization scenarios are defined. Since each image stabilization method has certain limitations, selecting the appropriate stabilization method for the target image based on the suitable scenario for each method is beneficial for improving the stabilization effect.

[0088] This application embodiment acquires a target image to be stabilized and a reference image acquired before the target image. Feature point matching is performed on the target image and the reference image to determine at least one set of matching feature point pairs in the target image and the reference image. Each feature point pair includes a first feature point located in the target image and a second feature point located in the reference image. Based on the first image position of at least one first feature point and the second image position of at least one second feature point in the at least one set of feature point pairs, the positional difference of the at least one set of feature point pairs is determined. Based on the positional difference of the at least one set of feature point pairs, a reference error is determined. Stabilization processing is performed based on the reference error and the number of feature point pairs in the at least one set of feature point pairs to obtain a stabilized target image. This effectively improves the camera's stabilization capability, avoids the use of additional special components, reduces costs, and improves the stabilization effect.

[0089] Figure 2 This is a flowchart illustrating another anti-shake control method provided in an embodiment of this application.

[0090] like Figure 2 As shown, the image stabilization control method may include the following steps:

[0091] Step 201: Acquire the target image to be stabilized, and the reference image acquired before the target image.

[0092] Optionally, by acquiring images through a camera, the first frame captured after the camera is started can be used as the reference image, and the subsequent frames that exhibit jitter and shift can be used as the target images to be stabilized.

[0093] Step 202: Perform feature point matching on the target image and the reference image to determine at least one set of matching feature point pairs in the target image and the reference image. The feature point pair includes a first feature point located in the target image and a second feature point located in the reference image.

[0094] In some implementations, pyramid segmentation is performed on both the target image and the reference image, dividing the image into eight levels from 0 to 7. Pyramid segmentation refers to scaling the image and describing it through several different scaling levels.

[0095] Optionally, for each pyramid level image, edge detection and corner detection algorithms are used to obtain the image edges and corners as feature points, and edge features and corner features, i.e., feature point information, are extracted. For the same pyramid level images of the target image and the reference image, feature point matching is performed based on the feature point information to determine at least one pair of mutually matching feature points.

[0096] It is understandable that, for feature points within each pyramid level of the target image, matching feature points with the same or similar feature point information can be determined within the corresponding pyramid level of the reference image based on specific point information matching. Since feature points within each pyramid level of the target image have corresponding feature points in the target image, similarly, feature points within each pyramid level of the reference image also have corresponding feature points in the reference image. Therefore, based on the matching feature points within the same pyramid level in the target image and the reference image, mutually matching feature points can be determined in both the target image and the reference image.

[0097] After identifying the matching feature points in the target image and the reference image, it is necessary to further determine the mapping matrix from the reference image to the target image based on the image positions of the matching feature points in the target image and the reference image, respectively.

[0098] In some implementations, an optical flow algorithm is used to obtain the mapping matrix from the reference image to the target image. The optical flow algorithm can calculate the movement of each feature point between two frames over time t to t+δt. The specific steps are as follows:

[0099] Optionally, the feature information of the feature point at the second image position x, y, z in the reference image can be represented as I(x, y, z, t), and the feature information of the matched feature point at the first image position in the target image can be represented as I(x+δx, y+δy, z+δz, t+δt). Then the constraint equation for the image feature point is:

[0100] I(x,y,z,t)=I(x+δx,y+δy,z+δz,t+δt)

[0101] Assuming the movements of the two positions are sufficiently small, expanding the above constraint equations using the Taylor formula yields:

[0102]

[0103] Here, HOT refers to higher-order terms, which can be ignored when the shift is small enough.

[0104] From the above equation, we can obtain:

[0105]

[0106] Dividing both sides by δt, we get:

[0107]

[0108] Further simplification yields:

[0109]

[0110] in, These are the components of the optical flow vector I(x,y,z,t), namely x, y, and z. and It is the difference of the image at this point in the corresponding direction.

[0111] make Then: I x V x + y V y + z V z =-I t It can be written as:

[0112] Substituting the n feature points into the above formula yields a system of equations:

[0113]

[0114] The mapping matrix between feature points in the reference image and each matching feature point in the target image can be obtained through transformation:

[0115]

[0116] Step 203: Determine the positional difference of at least one set of feature point pairs based on the first image position of at least one first feature point and the second image position of at least one second feature point in at least one set of feature point pairs, and determine the reference error based on the positional difference of at least one set of feature point pairs.

[0117] Optionally, the reference error can be the average absolute error of the positional differences between the at least one set of feature point pairs.

[0118] Optionally, based on the mapping matrix obtained in step 202, the second image position of at least one second feature point in the reference image is transformed using the mapping matrix to obtain the mapped position. The positional difference between the mapped position and the second image position of the matched feature point in the reference image is calculated to determine the positional difference of at least one set of feature point pairs, and then the mean absolute error is calculated based on the positional difference of the at least one set of feature point pairs.

[0119] Step 204: Perform image stabilization based on the reference error and the number of feature point pairs in at least one set of feature point pairs to obtain the image stabilized target image.

[0120] The image stabilization process for the target image can be performed using one of the following methods: gyroscope stabilization, feature point matching, or a combination of feature point matching and gyroscope stabilization.

[0121] If a combination of feature point matching and gyroscope stabilization is used for stabilization, proceed to steps 205-208; if gyroscope stabilization is used, proceed to step 209; if feature point matching is used, proceed to step 210.

[0122] Based on the reference error and the number of feature point pairs that match at least one set of feature point pairs in the target image, several possible image stabilization scenarios are defined. Since each image stabilization method has certain limitations, selecting the appropriate stabilization method for the target image based on the suitable scenario for each method is beneficial to improving the stabilization effect. One possible implementation is as follows: when the number of feature point pairs is less than a threshold, gyroscope-based image stabilization is used; when the number of feature point pairs is greater than or equal to the threshold and the reference error is less than a first error value, feature point matching is used; when the number of feature point pairs is greater than or equal to the threshold, the reference error is greater than or equal to the first error value, and the reference error is less than or equal to a second error value, a combination of feature point matching and gyroscope-based image stabilization is used; and when the number of feature point pairs is greater than or equal to the threshold and the reference error is greater than the second error value, gyroscope-based image stabilization is used.

[0123] The number of thresholds, the first error value, and the second error value can be preset based on experience, or they can be flexibly set and adjusted according to different scenario requirements and processing capabilities. As one possible implementation, the number of thresholds can range from 16 to 24; for example, the number of thresholds could be 20. The first error value is 20, and the second error value is 40.

[0124] Step 205: If a combination of feature point matching and gyroscope stabilization is used for stabilization, then the feature point matching method is used to stabilize the target image to obtain the first offset of each pixel in the target image.

[0125] Optionally, based on the mapping matrix between the feature points of the target image and the reference image obtained in step 202, an inverse affine transformation is performed on the target image to obtain the first offset of each pixel in the target image.

[0126] Step 206: Apply gyroscope-based image stabilization to the target image to obtain the second offset of each pixel in the target image.

[0127] Optionally, the target image is stabilized using a gyroscope-based image stabilization method to obtain a second offset for each pixel in the target image, including:

[0128] Multiple jitter angular velocities acquired by the gyroscope between the acquisition of the target image and the reference image are obtained; Kalman filtering is applied to filter these multiple jitter angular velocities; and a second offset is determined based on the filtered multiple jitter angular velocities. Specifically, time integration is performed on the filtered multiple jitter angular velocities to obtain the gyroscope angle change of the target image relative to the reference image. Based on this angle change, a reverse angle change is applied to the target image to counteract the gyroscope angle change of the target image relative to the reference image, thus achieving image stabilization. The reverse angle change is applied to each pixel in the target image, changing the position of each pixel in the target image. A second offset for each pixel position is determined based on the change in the position of each pixel.

[0129] Among them, multiple jitter angular velocities include multiple angular velocities rotating around the three axes of X, Y, and Z.

[0130] Step 207: Based on the reference error, the first offset and the second offset are weighted and summed to obtain the weighted offset.

[0131] Optionally, the weighted offset = W1 * first offset + W2 * second offset. Where W1 is the weight value of the first offset and W2 is the weight value of the second offset.

[0132] When it is determined that a combination of feature point matching and gyroscope stabilization is used for image stabilization, the weight value of the first offset W1 = (40 - reference error) / 20, and the weight value of the second offset W2 = 1 - W1.

[0133] It should be noted that when the gyroscope-based image stabilization method is used, the weight value W1 of the first offset can be considered as 0 and the weight value W2 of the second offset as 1; when the feature point matching method is used, the weight value W1 of the first offset can be considered as 1 and the weight value W2 of the second offset as 0.

[0134] Step 208: Based on the weighted offset, perform position transformation on each pixel in the target image.

[0135] Optionally, based on the weighted offset obtained in step 207, the position of each pixel in the target image is transformed. That is, based on the weighted offset, the pixels in the target image are offset in the opposite direction to obtain the target image after image stabilization.

[0136] Step 209: If a gyroscope-based anti-shake method is used for anti-shake processing, the change in gyroscope angle of the target image relative to the reference image is obtained based on the multiple jitter angular velocities collected by the gyroscope between the target image and the reference image, and the target image is anti-shake processed based on the change in angular velocity.

[0137] Multiple jitter angular velocities are acquired by the gyroscope between acquiring the target image and the reference image. A Kalman filter is used to filter these jitter angular velocities to enhance their accuracy. The filtered jitter angular velocities are then integrated over time to obtain the gyroscope angle change of the target image relative to the reference image. Based on this angle change, the target image undergoes a reverse angle change to counteract the gyroscope angle change relative to the reference image, thus achieving image stabilization.

[0138] Step 210: If the feature point matching method is used for image stabilization, then according to the mapping matrix between the feature points of the target image and the reference image, the target image is subjected to an inverse affine transformation so that the transformed target image is used as the target image after image stabilization.

[0139] The image stabilization control method of this application embodiment acquires a target image to be stabilized and a reference image acquired before the target image. Feature point matching is performed on the target image and the reference image to determine at least one pair of matching feature points in the target image and the reference image. Each feature point pair includes a first feature point located in the target image and a second feature point located in the reference image. Based on the position of at least one first feature point in the first image and the position of at least one second feature point in the second image, the positional difference of the at least one pair of feature points is determined. A reference error is determined based on the positional difference of the at least one pair of feature points. Image stabilization is then performed based on the reference error and the number of feature point pairs in the at least one pair of feature points to obtain the stabilized target image. If a combined image stabilization method of feature point matching and gyroscope stabilization is used, then feature point matching is applied to the target image. The method employs image stabilization to obtain a first offset for each pixel in the target image. Then, gyroscope-based image stabilization is used to obtain a second offset for each pixel. Based on a reference error, the first and second offsets are weighted and summed to obtain a weighted offset. The position of each pixel in the target image is then repositioned based on this weighted offset. If gyroscope-based image stabilization is used, the gyroscope angle change of the target image relative to the reference image is obtained based on multiple jitter angular velocities acquired by the gyroscope between the target and reference images. Image stabilization is then applied to the target image based on this angular velocity change. If feature point matching is used, an inverse affine transformation is performed on the target image based on the mapping matrix between the feature points of the target and reference images, and the transformed target image is used as the stabilized target image. This method effectively improves the camera's image stabilization capability, avoids the use of additional special components, reduces costs, and enhances the stabilization effect.

[0140] To achieve the above embodiments, this application also proposes a shake control device.

[0141] Figure 3 This is a schematic diagram of the structure of a shake control device provided in an embodiment of this application.

[0142] like Figure 3 As shown, the image stabilization control device includes: a data acquisition module 310, a matching module 320, an error module 330, and a processing module 340.

[0143] The acquisition module 310 is used to acquire the target image to be stabilized, as well as the reference image acquired before the target image.

[0144] Matching module 320 is used to perform feature point matching on the target image and the reference image to determine at least one set of feature point pairs that match each other in the target image and the reference image, wherein the feature point pair includes a first feature point located in the target image and a second feature point located in the reference image;

[0145] Error module 330 is used to determine the positional difference of the at least one set of feature point pairs based on the first image position of at least one first feature point and the second image position of at least one second feature point in the at least one set of feature point pairs; and to determine a reference error based on the positional difference of the at least one set of feature point pairs.

[0146] The processing module 340 is used to perform anti-shake processing based on the reference error and the number of feature point pairs of the at least one set of feature point pairs to obtain the target image after anti-shake processing.

[0147] As one possible implementation, processing module 340 is specifically used for:

[0148] If the number of feature point pairs is greater than or equal to the threshold number and the reference error is less than the first error value, a feature point matching method is used for image stabilization to obtain the image stabilization target image.

[0149] When the number of feature point pairs is greater than or equal to the threshold number, the reference error is greater than or equal to the first error value, and the reference error is less than or equal to the second error value, a combined anti-shake method of feature point matching and gyroscope anti-shake is used to perform anti-shake processing to obtain the target image after anti-shake processing.

[0150] When the number of feature point pairs is greater than or equal to the threshold number and the reference error is greater than the second error value, a gyroscope anti-shake method is used to perform anti-shake processing to obtain the target image after anti-shake processing.

[0151] As one possible implementation, the processing module 340 is further configured to: when the number of feature point pairs is less than the threshold number, use a gyroscope anti-shake method to perform anti-shake processing to obtain the anti-shake target image.

[0152] As one possible implementation, the processing module 340 is further configured to: perform image stabilization processing on the target image using a feature point matching method to obtain a first offset of each pixel in the target image; perform image stabilization processing on the target image using the gyroscope image stabilization method to obtain a second offset of each pixel in the target image; perform a weighted summation of the first offset and the second offset according to the reference error to obtain a weighted offset; and perform position transformation on each pixel in the target image according to the weighted offset to obtain the image stabilized.

[0153] As one possible implementation, the processing module 340 is specifically used to: obtain the first image position of each feature point in the reference image and the second image position of each feature point in the target image, and perform matching using optical flow to obtain a mapping matrix; wherein, the mapping matrix is ​​used to indicate the mapping relationship from the second image position to the first image position; and perform an inverse affine transformation on the target image according to the mapping matrix to obtain the first offset of each pixel in the target image.

[0154] As one possible implementation, the processing module 340 is specifically used to: acquire multiple jitter angular velocities acquired by the gyroscope between acquiring the target image and the reference image; filter the multiple jitter angular velocities using Kalman filtering; and determine the second offset based on the filtered multiple jitter angular velocities.

[0155] The image stabilization control device of this application acquires a target image to be stabilized and a reference image acquired before the target image. It then performs feature point matching on the target image and the reference image to determine at least one set of matching feature point pairs in the target image and the reference image. Each feature point pair includes a first feature point located in the target image and a second feature point located in the reference image. Based on the first image position of at least one first feature point and the second image position of at least one second feature point in the at least one set of feature point pairs, a positional difference between the at least one set of feature point pairs is determined. A reference error is determined based on the positional difference between the at least one set of feature point pairs. Image stabilization processing is then performed based on the reference error and the number of feature point pairs in the at least one set of feature point pairs to obtain a stabilized target image. This effectively improves the camera's image stabilization capability, avoids the use of additional special components, reduces costs, and enhances the image stabilization effect.

[0156] It should be noted that the foregoing explanation of the embodiment of the image stabilization control method also applies to the image stabilization control device of this embodiment, and will not be repeated here.

[0157] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0158] like Figure 4 As shown, the electronic device 400 includes: a memory 410 and a processor 420, and a bus 430 connecting different components (including the memory 410 and the processor 420).

[0159] The memory 410 is used to store executable instructions of the processor 420; the processor 420 is configured to call and execute the executable instructions stored in the memory 410 to implement the anti-shake control method proposed in the above embodiments of this application.

[0160] Bus 430 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0161] Electronic device 400 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 400, including volatile and non-volatile media, removable and non-removable media.

[0162] Memory 410 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 440 and / or cache memory 450. Electronic device 400 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 460 may be used to read and write non-removable, non-volatile magnetic media (… Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 430 via one or more data media interfaces. Memory 410 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0163] A program / utility 480 having a set (at least one) of program modules 470 may be stored, for example, in memory 410. Such program modules 470 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 470 typically perform the functions and / or methods described in the embodiments of this application.

[0164] Electronic device 400 can also communicate with one or more external devices 490 (e.g., keyboard, pointing device, display 491, etc.), and with one or more devices that enable a user to interact with electronic device 400, and / or with any device that enables electronic device 400 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 492. Furthermore, electronic device 400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 493. As shown, network adapter 493 communicates with other modules of electronic device 400 via bus 430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0165] The processor 420 performs various functional applications and data processing by running programs stored in the memory 410.

[0166] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium, wherein the instructions in the storage medium, when executed by the processor of an electronic device, enable the electronic device to perform the anti-shake control method proposed in any of the foregoing embodiments.

[0167] To implement the above embodiments, this application also provides a computer program product, which, when executed by the processor of an electronic device, enables the electronic device to perform the anti-shake control method proposed in any of the foregoing embodiments.

[0168] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0169] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0170] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0171] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0172] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0173] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0174] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0175] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method of anti-shake control, characterized in that, Includes the following steps: Acquire the target image to be stabilized, and the reference image acquired before the target image; Feature point matching is performed on the target image and the reference image to determine at least one pair of feature points that match each other in the target image and the reference image, wherein the pair of feature points includes a first feature point located in the target image and a second feature point located in the reference image; The positional difference of the at least one set of feature point pairs is determined based on the first image position of at least one first feature point in the at least one set of feature point pairs and the second image position of at least one second feature point. The reference error is determined based on the positional differences of the at least one set of feature point pairs; Based on the reference error and the number of feature point pairs in the at least one set of feature point pairs, anti-shake processing is performed to obtain the anti-shake target image; wherein, when the number of feature point pairs is greater than or equal to the threshold number and the reference error is greater than the second error value, a gyroscope anti-shake method is used to perform anti-shake processing to obtain the anti-shake target image.

2. The method according to claim 1, characterized in that, The step of performing image stabilization based on the reference error and the number of feature point pairs in the at least one set of feature point pairs to obtain the image stabilization target image further includes at least one of the following: If the number of feature point pairs is greater than or equal to the threshold number and the reference error is less than the first error value, a feature point matching method is used for image stabilization to obtain the image stabilization target image. When the number of feature point pairs is greater than or equal to the threshold number, the reference error is greater than or equal to the first error value, and the reference error is less than or equal to the second error value, a combined anti-shake method of feature point matching and gyroscope anti-shake is used to perform anti-shake processing to obtain the target image after anti-shake processing.

3. The method of claim 2, wherein, The step of performing anti-shake processing based on the reference error and the number of feature point pairs in the at least one set of feature point pairs to obtain the anti-shake target image further includes: If the number of feature point pairs is less than the threshold number, a gyroscope-based image stabilization method is used to perform image stabilization, resulting in the image stabilized target image.

4. The method of claim 2, wherein, The image stabilization process employs a combination of feature point matching and gyroscope stabilization to obtain the stabilized target image, including: The target image is subjected to anti-shake processing using a feature point matching method to obtain the first offset of each pixel in the target image; The target image is subjected to gyroscope stabilization processing to obtain the second offset of each pixel in the target image; Based on the reference error, the first offset and the second offset are weighted and summed to obtain the weighted offset. Based on the weighted offset, the position of each pixel in the target image is transformed to obtain the target image after image stabilization.

5. The method of claim 4, wherein, The step of performing image stabilization using a feature point matching method on the target image to obtain the first offset of each pixel in the target image includes: A first image position of at least one first feature point and a second image position of at least one second feature point are obtained, and a mapping matrix is ​​obtained by matching using optical flow; wherein, the mapping matrix is ​​used to indicate the mapping relationship from the second image position to the first image position; Based on the mapping matrix, an inverse affine transformation is performed on the target image to obtain the first offset of each pixel in the target image.

6. The method of claim 4, wherein, The step of applying the gyroscope-based image stabilization method to the target image to obtain the second offset of each pixel in the target image includes: Acquire multiple jitter angular velocities acquired by the gyroscope between acquiring the target image and the reference image; Kalman filtering is used to filter the multiple jitter angular velocities; The second offset is determined based on the filtered multiple jitter angular velocities.

7. An anti-shake control device, characterized by comprising: include: The acquisition module is used to acquire the target image to be stabilized, as well as the reference image acquired before the target image; A matching module is used to perform feature point matching on the target image and the reference image to determine at least one pair of feature points that match each other in the target image and the reference image, wherein the pair of feature points includes a first feature point located in the target image and a second feature point located in the reference image; An error module is configured to determine the positional difference of at least one set of feature point pairs based on the first image position of at least one first feature point and the second image position of at least one second feature point; and to determine a reference error based on the positional difference of at least one set of feature point pairs. The processing module is used to perform anti-shake processing based on the reference error and the number of feature point pairs of the at least one set of feature point pairs to obtain the anti-shake target image; wherein, when the number of feature point pairs is greater than or equal to a threshold number and the reference error is greater than a second error value, a gyroscope anti-shake method is used to perform anti-shake processing to obtain the anti-shake target image.

8. The apparatus of claim 7, wherein, The processing module is also used for: If the number of feature point pairs is greater than or equal to the threshold number and the reference error is less than the first error value, a feature point matching method is used for image stabilization to obtain the target image after image stabilization. When the number of feature point pairs is greater than or equal to the threshold number, the reference error is greater than or equal to the first error value, and the reference error is less than or equal to the second error value, a combined anti-shake method of feature point matching and gyroscope anti-shake is used to perform anti-shake processing to obtain the target image after anti-shake processing.

9. The apparatus of claim 8, wherein, The processing module is also used for: If the number of feature point pairs is less than the threshold number, a gyroscope-based image stabilization method is used to perform image stabilization, resulting in the image stabilized target image.

10. The apparatus of claim 8, wherein, The processing module is also used for: The target image is subjected to anti-shake processing using a feature point matching method to obtain the first offset of each pixel in the target image; The target image is subjected to gyroscope stabilization processing to obtain the second offset of each pixel in the target image; Based on the reference error, the first offset and the second offset are weighted and summed to obtain the weighted offset. Based on the weighted offset, the position of each pixel in the target image is transformed to obtain the target image after image stabilization.

11. The apparatus of claim 10, wherein, The processing module is specifically used for: A first image position of at least one first feature point and a second image position of at least one second feature point are obtained, and a mapping matrix is ​​obtained by matching using optical flow; wherein, the mapping matrix is ​​used to indicate the mapping relationship from the second image position to the first image position; Based on the mapping matrix, an inverse affine transformation is performed on the target image to obtain the first offset of each pixel in the target image.

12. The apparatus of claim 10, wherein, The processing module is specifically used for: Acquire multiple jitter angular velocities acquired by the gyroscope between acquiring the target image and the reference image; Kalman filtering is used to filter the multiple jitter angular velocities; The second offset is determined based on the filtered multiple jitter angular velocities.

13. An electronic device, comprising: include: processor; Memory for storing the executable instructions of the processor; The processor is configured to invoke and execute executable instructions stored in the memory to implement the method as described in any one of claims 1-6.

14. A non-transitory computer-readable storage medium, comprising: When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-6.

Images