Anti-shake processing method and device, and electronic device

Abstract

The application discloses a method and device for anti-shake processing and electronic equipment, and belongs to the field of image processing. The method is executed by an electronic device, and the electronic device comprises a lens assembly. The method comprises the following steps: obtaining anti-shake compensation information according to eye movement data of a first user; and controlling the lens assembly to move according to the anti-shake compensation information. The eye movement data comprises an eye rotation amount and an eye translation amount. The anti-shake compensation amount comprises N movement amounts. N is an integer greater than 1.

Description

Technical Field

[0001] This application belongs to the field of image processing, and specifically relates to a stabilization processing method, device and electronic device. Background Technology

[0002] With the development of communication technology, mobile terminal devices have become increasingly popular shooting devices due to their improved shooting functions and portability. However, when shooting with mobile terminal devices handheld or with the help of third-party devices, sudden shaking often occurs, affecting the shooting quality.

[0003] Currently, the industry predominantly employs optical image stabilization (OIS) to mitigate the impact of camera shake on shooting. OIS works by using an onboard gyroscope to detect minute movements and transmitting the signal to a microprocessor. The microprocessor then calculates the amount of displacement to compensate for the shake, which is subsequently applied by a compensation lens group based on the direction and amount of lens movement. This effectively overcomes image blur caused by camera vibration. However, OIS requires a built-in gyroscope to detect camera shake, increasing hardware costs. Summary of the Invention

[0004] The purpose of this application is to provide a stabilization method, device, and electronic device that can obtain stabilization compensation information using the eye movement data of a first user, and then control the movement of the lens assembly according to the stabilization compensation information to complete shake compensation and achieve eye-motion stabilization.

[0005] In a first aspect, embodiments of this application provide a method for image stabilization, executed by an electronic device, the electronic device including a lens assembly, the method comprising: Based on the first user's eye movement data, image stabilization compensation information is obtained; Based on the image stabilization compensation information, control the movement of the lens assembly; The eye movement data includes eye rotation and eye translation; the anti-shake compensation includes N movement amounts; N is an integer greater than or equal to 1.

[0006] Optionally, obtaining image stabilization compensation information based on the first user's eye movement data includes: Based on the reference coordinate system and the mapping matrix, the eye movement data of the first user is converted into first motion data; Based on the first motion data and the proportional gain coefficients corresponding to the N movement amounts, the image stabilization compensation information is obtained; The first motion data includes a first angular velocity, a second angular velocity, and a third angular velocity; the first angular velocity is the angular velocity along the X-axis of the reference coordinate system; the second angular velocity is the angular velocity along the Y-axis of the reference coordinate system; and the third angular velocity is the angular velocity along the Z-axis of the reference coordinate system.

[0007] Optionally, obtaining image stabilization compensation information based on the first user's eye movement data includes: The eye movement data of the first user is input into the image stabilization model to obtain the image stabilization compensation information output by the image stabilization model.

[0008] Optionally, the method further includes: Acquire the second motion data and eye movement data of the electronic device in the reference coordinate system; A mapping matrix is ​​determined based on the second motion data and the eye movement data.

[0009] Optionally, determining the mapping matrix based on the second motion data and the eye movement data includes: Based on a reference coordinate system and a first matrix, the eye movement data is converted into third motion data; wherein, the first matrix is ​​determined by the second motion data and the eye movement data; Based on the third motion data and the proportional gain coefficients corresponding to the N movement amounts, the first compensation information is obtained; Based on the first compensation information, control the movement of the lens assembly; Acquire the first image captured by the lens assembly; Obtain the sharpness difference between the first image and the second image; wherein, the second image is an image captured by the lens assembly after it has moved based on optical image stabilization compensation information; If the sharpness difference is greater than or equal to a first threshold, the first matrix is ​​adjusted and the first image is reacquired; if the sharpness difference is less than the first threshold, the first matrix is ​​determined as a mapping matrix.

[0010] Optionally, the method further includes: If eye movement data of the second user is obtained, or if the working duration of the lens assembly exceeds a second threshold, a third image and a fourth image captured by the lens assembly are obtained; the shooting interval between the third image and the fourth image is a first duration. The mapping matrix is ​​corrected based on the third and fourth images.

[0011] Optionally, the step of correcting the mapping matrix based on the third image and the fourth image includes: Obtain the position information of M first feature points in the third image and the position information of M second feature points in the fourth image; M is an integer greater than or equal to 1; Based on the mapping matrix and the position information of the M first feature points, determine the M position information of the M second feature points in the fourth image; The mapping matrix is ​​corrected based on the location information of the M second feature points and the M location information.

[0012] Secondly, embodiments of this application provide a stabilization processing device, the device including a processing module for: Based on the eye movement data of the first user, image stabilization compensation information is obtained; based on the image stabilization compensation information, the movement of the lens assembly is controlled; The eye movement data includes eye rotation and eye translation; the anti-shake compensation includes N movement amounts; N is an integer greater than or equal to 1.

[0013] Optionally, the processing module is specifically used for: Based on the reference coordinate system and the mapping matrix, the eye movement data of the first user is converted into first motion data; Based on the first motion data and the proportional gain coefficients corresponding to the N movement amounts, the image stabilization compensation information is obtained; The first motion data includes a first angular velocity, a second angular velocity, and a third angular velocity; the first angular velocity is the angular velocity along the X-axis of the reference coordinate system; the second angular velocity is the angular velocity along the Y-axis of the reference coordinate system; and the third angular velocity is the angular velocity along the Z-axis of the reference coordinate system.

[0014] Optionally, the processing module is specifically used for: The eye movement data of the first user is input into the image stabilization model to obtain the image stabilization compensation information output by the image stabilization model.

[0015] Optionally, the processing module is further configured to: Acquire the second motion data and eye movement data of the electronic device in the reference coordinate system; A mapping matrix is ​​determined based on the second motion data and the eye movement data.

[0016] Optionally, the processing module is specifically used for: Based on a reference coordinate system and a first matrix, the eye movement data is converted into third motion data; wherein, the first matrix is ​​determined by the second motion data and the eye movement data; Based on the third motion data and the proportional gain coefficients corresponding to the N movement amounts, the first compensation information is obtained; Based on the first compensation information, control the movement of the lens assembly; Acquire the first image captured by the lens assembly; Obtain the sharpness difference between the first image and the second image; wherein, the second image is an image captured by the lens assembly after it has moved based on optical image stabilization compensation information; If the sharpness difference is greater than or equal to a first threshold, the first matrix is ​​adjusted and the first image is reacquired; if the sharpness difference is less than the first threshold, the first matrix is ​​determined as a mapping matrix.

[0017] Optionally, the processing module is further configured to: If eye movement data of the second user is obtained, or if the working duration of the lens assembly exceeds a second threshold, a third image and a fourth image captured by the lens assembly are obtained; the shooting interval between the third image and the fourth image is a first duration. The mapping matrix is ​​corrected based on the third and fourth images.

[0018] Optionally, the processing module is further configured to: Obtain the position information of M first feature points in the third image and the position information of M second feature points in the fourth image; M is an integer greater than or equal to 1; Based on the mapping matrix and the position information of the M first feature points, determine the M position information of the M second feature points in the fourth image; The mapping matrix is ​​corrected based on the location information of the M second feature points and the M location information.

[0019] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0020] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0021] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0022] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0023] In this embodiment of the application, the eye movement data of the first user can be used to obtain image stabilization compensation information, and then the lens assembly can be moved according to the image stabilization compensation information to complete the shake compensation and realize eye-motion image stabilization. Attached Figure Description

[0024] Figure 1 This is a schematic flowchart of a stabilization processing method provided in some embodiments of this application; Figure 2 These are schematic diagrams of eye movements provided in some embodiments of this application; Figure 3 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application; Figure 4 This is a schematic diagram of the module structure of the anti-shake processing device provided in some embodiments of this application; Figure 5 This is a schematic diagram of the structure of another electronic device provided in some embodiments of this application; Figure 6 This is a schematic diagram of the structure of another electronic device provided in some embodiments of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0026] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0027] The image stabilization method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0028] like Figure 1 As shown in the figure, an image stabilization method according to an embodiment of this application is executed by an electronic device, the electronic device including a lens assembly, and the method includes: Step 101: Obtain image stabilization compensation information based on the first user's eye movement data. The eye movement data includes eye rotation and eye translation; the image stabilization compensation includes N movement amounts, where N is an integer greater than or equal to 1.

[0029] In other words, by performing this step, the electronic device can obtain image stabilization compensation information for a first user based on their eye movement data. This image stabilization compensation information is used to compensate for shake by controlling the movement of the lens assembly.

[0030] It is important to know that the first user is the user currently using the electronic device. For example, if the electronic device recognizes that user A has unlocked the electronic device, or if it detects that user A has been using the electronic device for a certain period of time, then user A is considered the first user.

[0031] It is also important to know that the shake compensation for the lens assembly is set according to the direction in which the lens assembly can move. In other words, the value of N is determined by the number of directions in which the lens assembly can move. For example, if the lens assembly can move in the length, width and thickness directions of the electronic device, then the value of N is equal to 3. Accordingly, the three movement amounts are the movement amounts of the lens assembly in the length, width and thickness directions.

[0032] Step 102: Control the movement of the lens assembly according to the image stabilization compensation information.

[0033] In this step, the electronic device can control the movement of the lens assembly to perform image stabilization based on the image stabilization compensation information obtained in step 101.

[0034] Thus, through steps 101 and 102, the electronic device executes the image stabilization processing method of this application embodiment, which can obtain image stabilization compensation information using the eye movement data of the first user, and then control the movement of the lens assembly according to the image stabilization compensation information to complete the shake compensation and realize eye-motion image stabilization.

[0035] In addition, electronic devices using the anti-shake processing method of the embodiments of this application can eliminate the need for gyroscopes for anti-shake, thereby reducing device costs.

[0036] Optionally, in this embodiment, the execution of steps 101 and 102 can be triggered by the first input, or it can be understood that the first input is the trigger input for the eye-tracking stabilization function.

[0037] The first input can be triggered by a physical button or a virtual button, or by biometric technology such as voice, touch, infrared, or gesture. Taking a virtual button as an example, a virtual button triggered by the image stabilization function is set on the shooting preview interface; the user's click on this virtual button constitutes the first input.

[0038] Optionally, in this embodiment, the amount of eye rotation can also be understood as eye rotation data, and the amount of eye translation can also be understood as eye translation data.

[0039] Optionally, in this embodiment, eye movement data can be obtained through eye recognition and tracking, for example, as... Figure 2 As shown, when the user starts the rear camera, the front camera captures and recognizes the eye image. When the user views the subject 202 on the shooting preview interface 201, the front camera tracks the movement of the eye to obtain eye movement data.

[0040] Optionally, in this embodiment, based on the standard head coordinate system, the eye rotation can be expressed as R{yaw, pitch}, where yaw represents the angle at which the eyeball deflects left or right from the front direction, and pitch represents the angle at which the eyeball deflects upward or downward from the front direction. Based on the facial coordinate system, the eyeball translation can be expressed as T{x, y}, where x represents the displacement of the eyeball along the X-axis and y represents the displacement of the eyeball along the Y-axis. Example 1: Using eyeball data sampled at 100Hz, at t=0.12s, the angular velocity over the past 0.12 seconds is calculated. Perform discrete integration: ; The calculation shows that the eyeball has rotated a cumulative 0.25 degrees from directly in front or downward in the past 0.12 seconds. Example 2: Using eyeball data sampled at 100Hz, calculate the angular velocity over the past 0.12 seconds at t=0.12s. Perform discrete integration: ; It calculates that the eyeball rotated a total of 0.35 degrees upwards or downwards in the past 0.12 seconds. The direction of rotation can be recorded by adding a positive or negative sign: if the eyeball rotated upwards, it is recorded as +0.25 degrees or +0.35 degrees; if the eyeball rotated downwards, it is recorded as -0.25 degrees or -0.35 degrees.

[0041] Optionally, in this embodiment, the eye movement data may further include a timestamp corresponding to the amount of eye rotation and a timestamp corresponding to the amount of eye translation.

[0042] Optionally, in this embodiment, the eye movement data is real-time data, so as to compensate for the shaking that occurs during the shooting process in a timely manner and improve the timeliness of image stabilization.

[0043] Optionally, in this embodiment, obtaining image stabilization compensation information based on the first user's eye movement data includes: Based on the reference coordinate system and the mapping matrix, the eye movement data of the first user is converted into first motion data; Based on the first motion data and the proportional gain coefficients corresponding to the N movement amounts, the image stabilization compensation information is obtained; The first motion data includes a first angular velocity, a second angular velocity, and a third angular velocity; the first angular velocity is the angular velocity along the X-axis of the reference coordinate system; the second angular velocity is the angular velocity along the Y-axis of the reference coordinate system; and the third angular velocity is the angular velocity along the Z-axis of the reference coordinate system.

[0044] It should be noted that this reference coordinate system corresponds to N movement quantities. Thus, the first angular velocity, second angular velocity, and third angular velocity included in the first motion data can be combined with the proportional gain coefficients (gain) corresponding to the N movement quantities to obtain image stabilization compensation information. In this embodiment, the reference coordinate system is a three-dimensional coordinate system, whose axes can correspond to the length, width, and thickness directions of the electronic device, or to the horizontal east, north, and gravity directions. Specifically, it is determined by the direction of movement supported by the lens assembly.

[0045] In this way, after mapping the eye movement data of the first user to the reference coordinate system through the mapping matrix to obtain the first motion data, the image stabilization compensation information can be obtained from the various angular velocities included in the first motion data and the proportional gain coefficients corresponding to the N movement amounts, so as to complete the image stabilization compensation in the future.

[0046] In some optional embodiments of this application, the first angular velocity Second angular velocity and third angular velocity At this point, N can be at most three, and the proportional gain coefficients corresponding to the N movement amounts can be denoted as gainx, gainy, and gainz. Thus, it can be derived from... The gainx is used to obtain the displacement along the X-axis. The gain is the amount of movement along the Y-axis, obtained by... The gainz parameter is used to obtain the displacement along the Z-axis. For example, the displacement along the X-axis is equal to... Wherein, the maximum stabilization angle of the lens assembly (1.2 degrees) = atan(motor rated stroke (120um) / EFL (5.56mm)), corresponding to a sensor size of 1um / pixel, then gainx = 120um / (1um / pixel) / 1.2 degrees = 100 pixels / degree. When The amount of movement along the X-axis is ×100 pixels / degree = 25 pixels, meaning 25 pixels need to be compensated in the X direction. Electronic devices can control the lens assembly to move 25 pixels in the opposite direction to counteract shake and center the subject in the image. When The amount of movement along the X-axis is ×100 pixels / degree = 35 pixels, which means that 35 pixels need to be compensated in the X direction. Electronic devices can control the lens assembly to move 35 pixels in the opposite direction to counteract the shaking and center the subject of the image.

[0047] It is also important to know that if the lens assembly supports movement in two or one of the three directions, the gain value is 0 in the direction that does not support movement.

[0048] Optionally, in this embodiment, after obtaining the first motion data, a correction matrix can be used to correct the first motion data. Alternatively, the correction matrix can be fused into the mapping matrix so that the transformed first motion data is the corrected data.

[0049] Optionally, in this embodiment, the first motion data, the second motion data, the third motion data, etc., may also include the acceleration of each of the three axes, which can help improve the accuracy of the image stabilization compensation information.

[0050] Optionally, in this embodiment, obtaining image stabilization compensation information based on the first user's eye movement data includes: The eye movement data of the first user is input into the image stabilization model to obtain the image stabilization compensation information output by the image stabilization model.

[0051] Here, the image stabilization model is a pre-defined model that can output corresponding image stabilization compensation information based on the input eye movement data. Thus, in this step, by inputting the first user's eye movement data into the image stabilization model, the image stabilization compensation information output by the model can be obtained, improving processing efficiency.

[0052] It is understandable that the image stabilization model internally performs the following steps: based on a reference coordinate system and a mapping matrix, it converts the first user's eye movement data into first motion data; and based on the first motion data and the proportional gain coefficients corresponding to the N movement amounts, it obtains image stabilization compensation information. The specific implementation is as described above and will not be repeated here.

[0053] It's important to know that this image stabilization model is pre-trained to obtain image stabilization compensation information from eye movement data in order to control the movement of lens components.

[0054] Optionally, in this embodiment, the method further includes: Acquire the second motion data and eye movement data of the electronic device in the reference coordinate system; A mapping matrix is ​​determined based on the second motion data and the eye movement data.

[0055] In other words, the mapping matrix mentioned above needs to be determined based on the second motion data and eye movement data before it can be applied to the process of obtaining image stabilization compensation information to meet the image stabilization requirements. This eye movement data is the eye movement data used for training.

[0056] The second motion data is the same as the first motion data, including the angular velocity along the X-axis, the angular velocity along the Y-axis, and the angular velocity along the Z-axis of the reference coordinate system.

[0057] It should be noted that the eye movement data used to determine the mapping matrix can be understood as training data, which can be pre-collected. Similarly, the second motion data used to determine the mapping matrix can also be understood as training data, which can be obtained by vibrating the electronic device on a vibration table at a fixed frequency and angle before it leaves the factory. For example, obtaining the angular velocity of the gyroscope on the vibration table corresponding to the first axis within a certain time period.

[0058] The eye movement data used as training data can be collected using an infrared eye tracker on a standard simulated human eye. For example, by collecting the displacement of the eyeball along the X and Y axes from time t1 to t2, we can obtain T{x t2 -x t1 ,y t2 -y t1 Furthermore, the collected eye movement data undergoes Kalman smoothing filtering to improve data validity.

[0059] After acquiring the second motion data and the eye movement data used for training, a mapping matching can be performed using the minimum and maximum eye angles and minimum and maximum translations, along with the appropriate coordinate system, scale, and range of the gyroscope. This first matrix, the initial mapping matrix, is then determined. The elements of the mapping matrix, corresponding to the eye rotation and translation, can be divided into rotational and translational elements. Alternatively, each rotational element can be considered a rotational mapping matrix, and each translational element a translational mapping matrix.

[0060] Optionally, in this embodiment, the method further includes: Based on a reference coordinate system and a first matrix, the eye movement data is converted into third motion data; wherein, the first matrix is ​​determined by the second motion data and the eye movement data; Based on the third motion data and the proportional gain coefficients corresponding to the N movement amounts, the first compensation information is obtained; Based on the first compensation information, control the movement of the lens assembly; Acquire the first image captured by the lens assembly; Obtain the sharpness difference between the first image and the second image; wherein, the second image is an image captured by the lens assembly after it has moved based on optical image stabilization compensation information; If the sharpness difference is greater than or equal to a first threshold, the first matrix is ​​adjusted and the first image is reacquired; if the sharpness difference is less than the first threshold, the first matrix is ​​determined as a mapping matrix.

[0061] Here, the third motion data is the same as the first motion data, including the angular velocity along the X-axis, Y-axis, and Z-axis of the reference coordinate system. The first compensation information is the same as the image stabilization compensation information, including N movement amounts. Accordingly, the specific implementation of "converting the eye movement data into third motion data based on the reference coordinate system and the first matrix; obtaining the first compensation information according to the proportional gain coefficients corresponding to the third motion data and the N movement amounts" is the same as the implementation of "converting the first user's eye movement data into first motion data based on the reference coordinate system and the mapping matrix; obtaining the image stabilization compensation information according to the proportional gain coefficients corresponding to the first motion data and the N movement amounts", and will not be repeated here. If N=3, the obtained first compensation information includes movement amounts along the X-axis, Y-axis, and Z-axis. After obtaining the first compensation information, the electronic device can control the lens assembly to move according to each of the movement amounts to counteract the lens assembly's shake. After acquiring the first image captured by the lens assembly, the sharpness difference between the first and second images can be obtained. Then, based on the relationship between the sharpness difference and the first threshold, the first matrix can be adjusted to obtain the required mapping matrix, so as to obtain effective image stabilization compensation information and complete eye-tracking image stabilization.

[0062] It should be noted that the second image was captured by the lens assembly after the lens assembly was moved using the optical image stabilization compensation information generated based on optical image stabilization in that scene, in the scenario where eye motion data 1 occurred. Here, eye motion data 1 is the eye motion data used in the third motion data conversion. The optical image stabilization compensation information includes N movement amounts.

[0063] In this way, after generating the first matrix using the second motion data and the eye movement data used for training, the mapping matrix will be calibrated using image data.

[0064] The first image and the second image are captured by the electronic device at the same location and in the same environment using different image stabilization methods. In some examples, the lens assembly can capture multiple first images and multiple second images within a specific time period. That is, multiple sets of first images and second images corresponding to the same time period are obtained. For these multiple first images and multiple second images, the sharpness difference between the first images and the second images captured at the same time period can be calculated. The obtained multiple sharpness differences are integrated to obtain a value that can more effectively represent the image sharpness difference. For example, the sharpness difference q of K sets of first images and second images is obtained through... The first value is obtained after integration. l Then it is compared with a first threshold. If the first value is greater than or equal to the first threshold, the first matrix is ​​adjusted and the first image is reacquired. If the first value is less than the first threshold, the currently used first matrix is ​​determined as the mapping matrix.

[0065] It should be noted that after reacquiring the first image, the sharpness difference between the first image and the corresponding second image will be reacquired, and then its relationship with the first threshold will be determined again. That is, if the sharpness difference is greater than or equal to the first threshold, the process of "converting the eye movement data into third motion data based on the reference coordinate system and the first matrix; obtaining first compensation information based on the third motion data and the proportional gain coefficients corresponding to the N movement amounts; controlling the movement of the lens assembly based on the first compensation information; acquiring the first image captured by the lens assembly; acquiring the sharpness difference between the first image and the second image; and adjusting the first matrix if the sharpness difference is greater than or equal to the first threshold" will be repeated. The difference is that the eye movement data used to convert the third motion data is different in each repetition; of course, the first matrix is ​​also the first matrix after the previous adjustment.

[0066] Optionally, in this embodiment, adjusting the first matrix can be based on a pre-defined adjustment method. For example, an adjustment object and an adjustment step size can be pre-defined. In each adjustment, the set adjustment object is adjusted according to the adjustment step size to achieve the adjustment of the first matrix. For instance, different adjustment steps r1 and r2 can be set for adjustment object elements 1 and 2. Then, in each adjustment, element 1 is increased by r1, and element 2 is increased by r2. Optionally, the first threshold is close to 0 to ensure image clarity.

[0067] Of course, in this embodiment, the sharpness difference can also be obtained based on other features of the image, such as sharpness, noise, contrast, etc.

[0068] In addition, considering user changes or errors that may occur due to long-term use of physical components within electronic devices, optionally, in this embodiment, the method further includes: If eye movement data of the second user is obtained, or if the working duration of the lens assembly exceeds a second threshold, a third image and a fourth image captured by the lens assembly are obtained; the shooting interval between the third image and the fourth image is a first duration. The mapping matrix is ​​corrected based on the third and fourth images.

[0069] Thus, after acquiring the third and fourth images captured by the lens assembly at a certain time interval, the mapping matrix is ​​corrected using these third and fourth images to improve applicability to different users or reduce errors.

[0070] Optionally, the first duration can be determined by the image frame rate of the lens assembly. For example, assuming an image frame rate of 30Hz, the first duration would be 0.033s.

[0071] Optionally, in this embodiment, correcting the mapping matrix based on the third image and the fourth image includes: Obtain the position information of M first feature points in the third image and the position information of M second feature points in the fourth image; M is an integer greater than or equal to 1; Based on the mapping matrix and the position information of the M first feature points, determine the M position information of the M second feature points in the fourth image; The mapping matrix is ​​corrected based on the location information of the M second feature points and the M location information.

[0072] The first and second feature points can be determined based on the image content. For example, the M first feature points in the third image and the M second feature points in the fourth image are both facial feature points in the image. Furthermore, the i-th first feature point and the i-th second feature point can be understood as related feature points, such as both corresponding to a key location on the face.

[0073] For the i-th first feature point and the i-th second feature point, in the third image i and the fourth image i+1 with a first time interval, first position information P{i} and second position information P{i+1} are obtained. Then, using the mapping matrix and the first position information P{i}, the third position information P'{i+1} of the i-th first feature point in the fourth image is determined. Further, the mapping matrix is ​​corrected using the second position information P{i+1} and the third position information P'{i+1}. Specifically, correcting the mapping matrix includes calculating the position offsets of P{i+1} and P'{i+1}, using these position offsets to determine the correction parameters of the mapping matrix, and then using these correction parameters to correct the mapping matrix, improving its applicability to different users or reducing errors.

[0074] When determining the position information of the second feature point in the fourth image using the mapping matrix, one approach is to directly use the mapping matrix to calculate the position information; another approach is to establish an associated position determination matrix based on the mapping matrix, and obtain the position information from the position determination matrix and the position information of the first feature point.

[0075] Specifically, after obtaining the location information of M second feature points and M location information, a transformation can be performed based on the differences in the location information of the associated first and second feature points to obtain correction parameters. These correction parameters are then used to correct the mapping matrix. Correcting the mapping matrix can be understood as modifying its elements.

[0076] Taking three defined feature points as an example, in the third image, the first position information of the three first feature points is P{i}=[(100,200),(150,200),(125,250)]. In the fourth image, M second feature points are found through feature matching algorithm, and their second position information P{i+1} is obtained. Due to hand tremors and slight changes in the scene, P{i+1}=[(102.5,200.5),(152.5,200.5),(127.5,250.5)]. Assuming that the previously determined mapping matrix can predict that after the second duration, each pixel will be shifted 3 pixels to the right due to pure horizontal jitter, that is, the mapping matrix includes the rotation matrix =[111,111] and the translation matrix =[003,003] (here the matrix uses the 2x3 matrix form of affine transformation). Therefore, based on the first position information, the third position information P'{i+1} = [[(100+3,200),(150+3,200),(125+3,250)]. Then, the difference between the position information of the associated first and second feature points can be calculated using the second and third position information, and this difference can be transformed into a correction parameter to correct the mapping matrix. Optionally, the correction parameter B can be solved using P'{i+1}≈B×P{i+1}, thereby obtaining the corrected mapping matrix.

[0077] It should be understood that in this embodiment, the training of the image stabilization model includes the training of the mapping matrix. The training of the mapping matrix can be achieved using the method described above for determining the mapping matrix, which will not be repeated here. In addition, the training of the image stabilization model may also include the training of the correction matrix, the training of the scaling gain coefficient, and so on.

[0078] It should also be noted that, in this embodiment, after obtaining the image stabilization compensation information, the electronic device can control the movement of the lens assembly via a voice coil motor.

[0079] Optionally, in this embodiment, the structure of the electronic device is as follows: Figure 3As shown, the first unit records the user's eye movement data in real time and transmits the eye movement data and timestamp to the image stabilization device. The second unit records the image data captured by the lens assembly in real time and transmits the image data and timestamp to the image stabilization device. The image data includes, but is not limited to, a first image, a second image, a third image, and a fourth image. The image stabilization device uses the above-described image stabilization method to obtain image stabilization compensation information and uses this information to control the movement of the lens assembly, achieving eye-motion image stabilization. Specifically, the electronic device is equipped with a voice coil motor (VCM). The image stabilization device can generate a control signal based on the image stabilization compensation information, and drive the lens assembly to move by controlling the voice coil motor. The voice coil motor generates precise electromagnetic force by precisely controlling the current, thereby driving the lens assembly to a designated position to counteract the detected eye movement, keeping the optical path stable for the person or scene the eye wants to observe, resulting in clear imaging. In some examples, based on the structure of the lens assembly, it can move in a plane perpendicular to the optical axis or along the optical axis. Of course, the lens assembly can also be controlled to rotate around an axis.

[0080] The image stabilization device uses timestamps to align image data and eye movement data. This allows for processing of image content only at the exposure time, avoiding interference from eye movement data at the exposure time.

[0081] The voice coil motor is equipped with two axial Hall sensors (hallx, hally) to provide the position feedback signals required for control, which is the key feedback for achieving VCM closed-loop control.

[0082] The image stabilization method provided in this application can be executed by an image stabilization device. This application uses an image stabilization device executing the image stabilization method as an example to illustrate the image stabilization device provided in this application.

[0083] like Figure 4 As shown in the figure, an image stabilization processing device 400 according to an embodiment of this application includes: Processing module 410, used for Based on the first user's eye movement data, image stabilization compensation information is obtained; Based on the image stabilization compensation information, control the movement of the lens assembly; The eye movement data includes eye rotation and eye translation; the anti-shake compensation includes N movement amounts; N is an integer greater than or equal to 1.

[0084] The device can obtain image stabilization compensation information using the eye movement data of the first user, and then control the movement of the lens assembly according to the image stabilization compensation information to complete the shake compensation and achieve eye-motion image stabilization.

[0085] Optionally, the processing module is specifically used for: Based on the reference coordinate system and the mapping matrix, the eye movement data of the first user is converted into first motion data; Based on the first motion data and the proportional gain coefficients corresponding to the N movement amounts, the image stabilization compensation information is obtained; The first motion data includes a first angular velocity, a second angular velocity, and a third angular velocity; the first angular velocity is the angular velocity along the X-axis of the reference coordinate system; the second angular velocity is the angular velocity along the Y-axis of the reference coordinate system; and the third angular velocity is the angular velocity along the Z-axis of the reference coordinate system.

[0086] Optionally, the processing module is specifically used for: The eye movement data of the first user is input into the image stabilization model to obtain the image stabilization compensation information output by the image stabilization model.

[0087] Optionally, the processing module is further configured to: Acquire the second motion data and eye movement data of the electronic device in the reference coordinate system; A mapping matrix is ​​determined based on the second motion data and the eye movement data.

[0088] Optionally, the processing module is specifically used for: Based on a reference coordinate system and a first matrix, the eye movement data is converted into third motion data; wherein, the first matrix is ​​determined by the second motion data and the eye movement data; Based on the third motion data and the proportional gain coefficients corresponding to the N movement amounts, the first compensation information is obtained; Based on the first compensation information, control the movement of the lens assembly; Acquire the first image captured by the lens assembly; Obtain the sharpness difference between the first image and the second image; wherein, the second image is an image captured by the lens assembly after it has moved based on optical image stabilization; If the sharpness difference is greater than or equal to a first threshold, the first matrix is ​​adjusted and the first image is reacquired; if the sharpness difference is less than the first threshold, the first matrix is ​​determined as a mapping matrix.

[0089] Optionally, the processing module is further configured to: If eye movement data of the second user is obtained, or if the working duration of the lens assembly exceeds a second threshold, a third image and a fourth image captured by the lens assembly are obtained; the shooting interval between the third image and the fourth image is a first duration. The mapping matrix is ​​corrected based on the third and fourth images.

[0090] Optionally, the processing module is specifically used for: Obtain the position information of M first feature points in the third image and the position information of M second feature points in the fourth image; M is an integer greater than or equal to 1; Based on the mapping matrix and the position information of the M first feature points, determine the M position information of the M second feature points in the fourth image; The mapping matrix is ​​corrected based on the location information of the M second feature points and the M location information.

[0091] The image stabilization device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0092] The image stabilization device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0093] The image stabilization device provided in this application embodiment can achieve... Figures 1 to 3 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0094] Optionally, such as Figure 5As shown, this application embodiment also provides an electronic device 500, including a processor 501 and a memory 502. The memory 502 stores a program or instructions that can run on the processor 501. When the program or instructions are executed by the processor 501, they implement the various steps of the above-described anti-shake processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0095] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0096] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0097] The electronic device 600 includes, but is not limited to, components such as: radio frequency unit 601, network module 602, audio output unit 603, input unit 604, sensor 605, display unit 606, user input unit 607, interface unit 608, memory 609, and processor 610.

[0098] Those skilled in the art will understand that the electronic device 600 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 610 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0099] The electronic device 600 includes a lens assembly. A processor 610 is used for... Based on the first user's eye movement data, image stabilization compensation information is obtained; Based on the image stabilization compensation information, control the movement of the lens assembly; The eye movement data includes eye rotation and eye translation; the anti-shake compensation includes N movement amounts; N is an integer greater than or equal to 1.

[0100] Electronic devices can obtain image stabilization compensation information using the eye movement data of a first user, and then control the movement of the lens assembly according to the image stabilization compensation information to complete the shake compensation and achieve eye-tracking image stabilization.

[0101] It should be understood that, in this embodiment, the input unit 604 may include a graphics processing unit (GPU) 6041 and a microphone 6042. The GPU 6041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 606 may include a display panel 6061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 607 includes at least one of a touch panel 6071 and other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0102] The memory 609 can be used to store software programs and various data. The memory 609 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 609 may include volatile memory or non-volatile memory, or it may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 609 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0103] Processor 610 may include one or more processing units; optionally, processor 610 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 610.

[0104] Optionally, the processor 610 is also used for: Based on the reference coordinate system and the mapping matrix, the eye movement data of the first user is converted into first motion data; Based on the first motion data and the proportional gain coefficients corresponding to the N movement amounts, the image stabilization compensation information is obtained; The first motion data includes a first angular velocity, a second angular velocity, and a third angular velocity; the first angular velocity is the angular velocity along the X-axis of the reference coordinate system; the second angular velocity is the angular velocity along the Y-axis of the reference coordinate system; and the third angular velocity is the angular velocity along the Z-axis of the reference coordinate system.

[0105] Optionally, the processor 610 is also used for: The eye movement data of the first user is input into the image stabilization model to obtain the image stabilization compensation information output by the image stabilization model.

[0106] Optionally, the processor 610 is also used for: Acquire the second motion data and eye movement data of the electronic device in the reference coordinate system; A mapping matrix is ​​determined based on the second motion data and the eye movement data.

[0107] Optionally, the processor 610 is also used for: Based on a reference coordinate system and a first matrix, the eye movement data is converted into third motion data; wherein, the first matrix is ​​determined by the second motion data and the eye movement data; Based on the third motion data and the proportional gain coefficients corresponding to the N movement amounts, the first compensation information is obtained; Based on the first compensation information, control the movement of the lens assembly; Acquire the first image captured by the lens assembly; Obtain the sharpness difference between the first image and the second image; wherein, the second image is an image captured by the lens assembly after it has moved based on optical image stabilization compensation information; If the sharpness difference is greater than or equal to a first threshold, the first matrix is ​​adjusted and the first image is reacquired; if the sharpness difference is less than the first threshold, the first matrix is ​​determined as a mapping matrix.

[0108] Optionally, the processor 610 is also used for: If eye movement data of the second user is obtained, or if the working duration of the lens assembly exceeds a second threshold, a third image and a fourth image captured by the lens assembly are obtained; the shooting interval between the third image and the fourth image is a first duration. The mapping matrix is ​​corrected based on the third and fourth images.

[0109] Optionally, the processor 610 is also used for: Obtain the position information of M first feature points in the third image and the position information of M second feature points in the fourth image; M is an integer greater than or equal to 1; Based on the mapping matrix and the position information of the M first feature points, determine the M position information of the M second feature points in the fourth image; The mapping matrix is ​​corrected based on the location information of the M second feature points and the M location information.

[0110] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described anti-shake processing method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0111] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0112] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described anti-shake processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0113] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0114] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described anti-shake processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0115] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0117] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A stabilization method, performed by an electronic device, characterized in that, The electronic device includes a lens assembly, and the method includes: Based on the first user's eye movement data, image stabilization compensation information is obtained; Based on the image stabilization compensation information, control the movement of the lens assembly; The eye movement data includes eye rotation and eye translation; the anti-shake compensation includes N movement amounts; N is an integer greater than or equal to 1.

2. The method according to claim 1, characterized in that, The step of obtaining image stabilization compensation information based on the first user's eye movement data includes: Based on the reference coordinate system and the mapping matrix, the eye movement data of the first user is converted into first motion data; Based on the first motion data and the proportional gain coefficients corresponding to the N movement amounts, the image stabilization compensation information is obtained; The first motion data includes a first angular velocity, a second angular velocity, and a third angular velocity; the first angular velocity is the angular velocity along the X-axis of the reference coordinate system; the second angular velocity is the angular velocity along the Y-axis of the reference coordinate system; and the third angular velocity is the angular velocity along the Z-axis of the reference coordinate system.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining image stabilization compensation information based on the first user's eye movement data includes: The eye movement data of the first user is input into the image stabilization model to obtain the image stabilization compensation information output by the image stabilization model.

4. The method according to claim 2 or 3, characterized in that, The method further includes: Acquire the second motion data and eye movement data of the electronic device in the reference coordinate system; A mapping matrix is ​​determined based on the second motion data and the eye movement data.

5. The method according to claim 4, characterized in that, The step of determining the mapping matrix based on the second motion data and the eye movement data includes: Based on a reference coordinate system and a first matrix, the eye movement data is converted into third motion data; wherein, the first matrix is ​​determined by the second motion data and the eye movement data; Based on the third motion data and the proportional gain coefficients corresponding to the N movement amounts, the first compensation information is obtained; Based on the first compensation information, control the movement of the lens assembly; Acquire the first image captured by the lens assembly; Obtain the sharpness difference between the first image and the second image; wherein, the second image is an image captured by the lens assembly after it has moved based on optical image stabilization compensation information; If the sharpness difference is greater than or equal to a first threshold, the first matrix is ​​adjusted and the first image is reacquired; if the sharpness difference is less than the first threshold, the first matrix is ​​determined as a mapping matrix.

6. The method according to claim 2 or 3, characterized in that, Also includes: If eye movement data of the second user is obtained, or if the working duration of the lens assembly exceeds a second threshold, a third image and a fourth image captured by the lens assembly are obtained; the shooting interval between the third image and the fourth image is a first duration. The mapping matrix is ​​corrected based on the third and fourth images.

7. The method according to claim 6, characterized in that, The step of correcting the mapping matrix based on the third image and the fourth image includes: Obtain the position information of M first feature points in the third image and the position information of M second feature points in the fourth image; M is an integer greater than or equal to 1; Based on the mapping matrix and the position information of the M first feature points, determine the M position information of the M second feature points in the fourth image; The mapping matrix is ​​corrected based on the location information of the M second feature points and the M location information.

8. A shake-proof processing device, characterized in that, The device includes a processing module for: Based on the first user's eye movement data, image stabilization compensation information is obtained; Based on the image stabilization compensation information, control the movement of the lens assembly; The eye movement data includes eye rotation and eye translation; the anti-shake compensation includes N movement amounts; N is an integer greater than or equal to 1.

9. The apparatus according to claim 8, characterized in that, The processing module is specifically used for: Based on the reference coordinate system and the mapping matrix, the eye movement data of the first user is converted into first motion data; Based on the first motion data and the proportional gain coefficients corresponding to the N movement amounts, the image stabilization compensation information is obtained; The first motion data includes a first angular velocity, a second angular velocity, and a third angular velocity; the first angular velocity is the angular velocity along the X-axis of the reference coordinate system; the second angular velocity is the angular velocity along the Y-axis of the reference coordinate system; and the third angular velocity is the angular velocity along the Z-axis of the reference coordinate system.

10. The apparatus according to claim 8 or 9, characterized in that, The processing module is specifically used for: The eye movement data of the first user is input into the image stabilization model to obtain the image stabilization compensation information output by the image stabilization model.

11. The apparatus according to claim 9 or 10, characterized in that, The processing module is also used for: Acquire the second motion data and eye movement data of the electronic device in the reference coordinate system; A mapping matrix is ​​determined based on the second motion data and the eye movement data.

12. The apparatus according to claim 11, characterized in that, The processing module is specifically used for: Based on a reference coordinate system and a first matrix, the eye movement data is converted into third motion data; wherein, the first matrix is ​​determined by the second motion data and the eye movement data; Based on the third motion data and the proportional gain coefficients corresponding to the N movement amounts, the first compensation information is obtained; Based on the first compensation information, control the movement of the lens assembly; Acquire the first image captured by the lens assembly; Obtain the sharpness difference between the first image and the second image; wherein, the second image is an image captured by the lens assembly after it has moved based on optical image stabilization compensation information; If the sharpness difference is greater than or equal to a first threshold, the first matrix is ​​adjusted and the first image is reacquired; if the sharpness difference is less than the first threshold, the first matrix is ​​determined as a mapping matrix.

13. The apparatus according to claim 9 or 10, characterized in that, The processing module is also used for: If eye movement data of the second user is obtained, or if the working duration of the lens assembly exceeds a second threshold, a third image and a fourth image captured by the lens assembly are obtained; the shooting interval between the third image and the fourth image is a first duration. The mapping matrix is ​​corrected based on the third and fourth images.

14. The apparatus according to claim 13, characterized in that, The processing module is specifically used for: Obtain the position information of M first feature points in the third image and the position information of M second feature points in the fourth image; M is an integer greater than or equal to 1; Based on the mapping matrix and the position information of the M first feature points, determine the M position information of the M second feature points in the fourth image; The mapping matrix is ​​corrected based on the location information of the M second feature points and the M location information.

15. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the anti-shake processing method as described in any one of claims 1-7.

Images