Image stabilization methods and electronic devices
By acquiring the translation and rotation parameters of the lens elements, constructing a mapping relationship, and using the transformation matrix for image compensation, the problem of limited compensation capability of optical image stabilization technology is solved, achieving more efficient image stabilization and sharpness improvement.
Patent Information
- Application Number
- CN202510632556.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing optical image stabilization technologies are limited by hardware structure and have limited compensation capabilities, resulting in unsatisfactory image stabilization effects.
By acquiring the translation and rotation parameters of the lens elements, a preset mapping relationship is constructed, and the image is compensated using a transformation matrix. Combined with calibration data, the image stabilization algorithm is optimized to accurately correct image distortion and blur caused by lens motion.
It improves image stability and clarity, enhances image stabilization, reduces motion blur and distortion, and improves image quality.
Smart Images

Figure CN123123722A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of terminal technology, and in particular to an image stabilization method and electronic device. Background Technology
[0002] In everyday use of electronic devices, hand-held shooting can cause blurry images.
[0003] To address this issue, optical image stabilization technology is employed to compensate for camera shake through hardware structures. This method uses sensors such as gyroscopes to detect shake data, and a drive motor quickly adjusts the position of the lens or sensor to ensure that the imaging optical path remains aligned with the sensor, thereby counteracting the effects of shake.
[0004] However, optical image stabilization is limited by its hardware structure and has limited compensation capabilities, so the stabilization effect may still be unsatisfactory. Summary of the Invention
[0005] This application provides an image stabilization method and electronic device, applicable to the field of terminal technology, to improve the effect of image stabilization.
[0006] In a first aspect, embodiments of this application propose an image stabilization method. The method includes:
[0007] Obtain the first translation parameter of the lens element, which indicates the amount of translation generated by the lens element during the movement;
[0008] Based on the first translation parameter, a first rotation parameter of the lens element is determined, which is used to indicate the amount of rotation generated by the lens element during the movement.
[0009] Acquire the first image based on the lens element;
[0010] For any first pixel in the first image, the position information of the first pixel is compensated according to the first translation parameter and the first rotation parameter to obtain the second image.
[0011] In this embodiment, the rotation amount that the lens element may generate during movement can be quantified based on the first translation parameter. This rotation amount is usually caused by factors such as gravity and inertia affecting the lens element. Thus, by simultaneously using the first translation parameter and the first rotation parameter predicted by the first translation parameter to process the image, image shake can be eliminated more accurately, thereby improving the image stabilization effect.
[0012] In one possible implementation, determining the first rotation parameter of the lens element based on the first translation parameter includes:
[0013] According to the preset mapping relationship, the first adjustment parameter corresponding to the first translation parameter is determined. The first adjustment parameter is used to indicate the degree of rotation caused by the first translation parameter. The preset mapping relationship is used to indicate the correspondence between the translation amount and the adjustment parameter.
[0014] The first translation parameter is adjusted based on the first adjustment parameter to obtain the first rotation parameter, which is proportional to both the first translation parameter and the first adjustment parameter.
[0015] In this implementation, the first rotation parameter is assumed to be directly proportional to the first translation parameter. Specifically, a larger first translation parameter indicates a greater degree of movement of the lens element, thus implying a greater amount of rotation generated during the lens element's movement. Conversely, a smaller first translation parameter indicates a smaller degree of movement of the lens element, thus implying a less amount of rotation generated during the lens element's movement. Based on this, a first adjustment parameter is introduced, and the first rotation parameter is determined according to this parameter and the first translation parameter. This allows for a precise description of the translation and rotational movements that the lens element needs to compensate for during shooting.
[0016] In one possible implementation, the position information of the first pixel is compensated based on the first translation parameter and the first rotation parameter, including:
[0017] Based on the first translation parameter and the first rotation parameter, a first transformation matrix is determined. The first transformation matrix is used to indicate the parameters to be compensated corresponding to the motion state of the lens element. The motion state corresponds to the first translation parameter and the first rotation parameter.
[0018] The position information of the first pixel is compensated based on the first transformation matrix.
[0019] In this implementation, a first transformation matrix is determined based on a first translation parameter and a first rotation parameter. The transformation matrix describes the geometric transformation from the image before stabilization to the image after stabilization. Specifically, the transformation matrix describes how the position of each pixel in the image is transformed from its original position to its compensated new position. By compensating for the position information of the first pixels, motion blur and distortion can be reduced or eliminated, thereby improving the visual quality of the final image. This effectively corrects image distortion or blur caused by lens movement, thus improving image sharpness and stability.
[0020] In one possible implementation, the position information of the first pixel is compensated according to the first transformation matrix, including:
[0021] The position information of the first pixel is compensated by multiplying the first transformation matrix with the position information of the first pixel.
[0022] In this implementation, multiplying the first transformation matrix with the position information of the first pixel can transform the position of each pixel in the image from its original position to a new position after image stabilization, thereby eliminating jitter.
[0023] In one possible implementation, the first translation parameter includes: a first translation component corresponding to the first axis and a second translation component corresponding to the second axis; the first rotation parameter includes: a first rotation component corresponding to the first axis, a second rotation component corresponding to the second axis and a third rotation component corresponding to the third axis.
[0024] The first transformation matrix is determined based on the first translation parameter and the first rotation parameter, including:
[0025] Based on the first translation parameter, a first submatrix is determined. The elements in the first submatrix corresponding to the first axis are calculated based on the first translation component, and the elements in the first submatrix corresponding to the second axis are calculated based on the second translation component.
[0026] Based on the first rotation parameter, a second sub-matrix is determined. The elements in the second sub-matrix corresponding to the first axis are calculated based on the first rotation component, the elements in the second sub-matrix corresponding to the second axis are calculated based on the second rotation component, and the elements in the second sub-matrix corresponding to the third axis are calculated based on the third rotation component.
[0027] The first transformation matrix is determined based on the first submatrix and the second submatrix.
[0028] In this implementation, by accurately calculating the translation and rotation components on each axis, the transformation matrix can more effectively compensate for image jitter caused by camera motion, thereby improving image stability and accuracy.
[0029] In one possible implementation, the method further includes:
[0030] The electronic device is vibrated based on at least one preset amplitude, so as to collect at least one set of calibration data corresponding to each preset amplitude during the vibration process. The calibration data includes at least one of the following: first video, gyroscope data, second translation parameters and frame information, and at least one preset amplitude corresponds to its respective translation parameters.
[0031] The calibration process is performed based on at least one set of calibration data corresponding to at least one preset amplitude to obtain a preset mapping relationship. The preset mapping relationship includes adjustment parameters corresponding to multiple translation parameters, and the multiple translation parameters correspond to at least one preset amplitude.
[0032] In this implementation, by acquiring data at a known preset amplitude, the adjustment parameters corresponding to each of the multiple translation parameters can be precisely calibrated. The calibration process aims to reveal the device's behavioral characteristics at different amplitudes, thereby optimizing the device's stabilization algorithm and improving its performance in practical applications. Through precise calibration, the device can better compensate for and correct errors caused by vibration, improving image stability and the accuracy of data acquisition.
[0033] In one possible implementation, calibration processing is performed based on at least one set of calibration data corresponding to at least one preset amplitude to obtain a preset mapping relationship, including:
[0034] For any one of the at least one preset amplitudes, multiple selectable adjustment parameters are preset;
[0035] For any one of the multiple selectable adjustment parameters, the second rotation parameter corresponding to the second translation parameter is determined based on the selectable adjustment parameter and the second translation parameter contained in the calibration data of the preset amplitude.
[0036] Based on the second rotation parameter and the second translation parameter, the corresponding second transformation matrix is determined. The second transformation matrix is used to indicate the parameters to be compensated corresponding to the motion state of the lens element during the vibration process. The motion state corresponds to the second translation parameter and the second rotation parameter.
[0037] For any frame of the third image in the first video, the position information of each pixel in the third image is compensated based on the second transformation matrix to obtain the fourth image;
[0038] The error is determined based on the expected image and the fourth image corresponding to the candidate adjustment parameter, and the candidate adjustment parameter with the smallest error is determined as the adjustment parameter corresponding to the translation parameter corresponding to the preset amplitude.
[0039] In this implementation, multiple candidate adjustment parameters are set for each preset amplitude. Based on these parameters and the calibration data of the preset amplitude, geometric transformations are performed on the image to obtain a stable image after image stabilization for each candidate adjustment parameter. Then, the adjustment parameter that minimizes the error between the stabilized image and the expected image is used as the adjustment parameter corresponding to the translation parameter of that preset amplitude. This allows the system to accurately determine the adjustment parameter based on the translation parameter in practical applications, thereby determining the first rotation parameter, effectively reducing errors in the image compensation process and improving the accuracy of compensation.
[0040] Secondly, embodiments of this application provide an image stabilization device, which can be an electronic device, or a chip or chip system within an electronic device. The image stabilization device may include a display unit and a processing unit.
[0041] When the image stabilization device is an electronic device, the display unit can be a display screen. The display unit is used to perform the display step so that the electronic device implements an image stabilization method described in the first aspect or any possible implementation of the first aspect.
[0042] When the image stabilization device is an electronic device, the processing unit may be a processor. The image stabilization device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an image stabilization method described in the first aspect or any possible implementation thereof.
[0043] When the image stabilization device is a chip or chip system within an electronic device, the processing unit can be a processor. The processing unit executes instructions stored in the storage unit to cause the electronic device to implement an image stabilization method described in the first aspect or any possible implementation of the first aspect. The storage unit can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).
[0044] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory for storing code instructions, and the processor for running the code instructions to perform the methods described in the first aspect or any possible implementation of the first aspect.
[0045] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.
[0046] Fifthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.
[0047] Sixthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.
[0048] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).
[0049] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description
[0050] Figure 1 A schematic diagram of the interface of an electronic device provided in an embodiment of this application;
[0051] Figure 2 A schematic diagram of lens movement provided for an embodiment of this application;
[0052] Figure 3 This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of this application;
[0053] Figure 4 This is a schematic diagram of the software structure of a terminal device provided in an embodiment of this application;
[0054] Figure 5 A schematic diagram of the image stabilization method provided in the embodiments of this application;
[0055] Figure 6 A schematic diagram illustrating a scenario of simulated electronic device jitter provided in an embodiment of this application;
[0056] Figure 7 A schematic diagram illustrating the correspondence between preset amplitude and translation parameters provided in the embodiments of this application;
[0057] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0058] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:
[0059] 1. OIS
[0060] Optical image stabilization (OIS) compensates for displacement caused by hand tremors or equipment vibration by moving the lens assembly or image sensor. It operates in real-time during shooting; when the sensor detects shaking, it immediately drives the lens or sensor to perform physical compensation. It is highly effective at compensating for low-frequency shaking (such as hand tremors), but its compensation range is limited by its mechanical structure.
[0061] 2. EIS
[0062] Electronic image stabilization (EIS) compensates for the effects of jitter by cropping image edges and using algorithms to transform the image. It is performed during image or video processing, typically as a supplement after image stabilization (OIS). It is particularly effective at compensating for high-frequency jitter (such as fast movement), but it sacrifices some image resolution.
[0063] 3. Other terms
[0064] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.
[0065] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0066] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.
[0067] 4. Electronic equipment
[0068] The electronic devices in this application embodiment may include handheld devices with image processing functions, vehicle-mounted devices, etc. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving vehicles, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, wearable devices, terminal devices in 5G networks, or future evolution of public land mobile communication networks. Terminal devices in a network (PLMN), etc., are not limited to this in the embodiments of this application.
[0069] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.
[0070] Furthermore, in this embodiment of the application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.
[0071] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.
[0072] In this embodiment, the electronic device or various network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.
[0073] To better understand the technical solution of this application, the relevant technologies involved in this application will be further described in detail below.
[0074] With the widespread use of smartphones, users increasingly rely on their phones to shoot videos and record important moments in life. However, due to the inherent instability of handheld devices, camera shake often occurs during shooting, resulting in blurry or unstable video footage. To solve this problem, image stabilization technology has emerged. This technology uses a combination of hardware and software to detect and compensate for device movement in real time, significantly improving video clarity and stability, and providing users with a better shooting experience. This not only enhances video quality but also makes it easier for users to capture wonderful moments in life.
[0075] The following is combined Figure 1 This section introduces schematic diagrams of electronic device interfaces. Figure 1This is a schematic diagram of the interface of an electronic device provided in an embodiment of this application. The description uses a mobile phone as an example.
[0076] like Figure 1 As shown, this interface can be understood as the camera interface of an electronic device. This interface may include one or more of the following: a camera control 101, a camera switching control 102, a preview area 103, or multiple function controls. The camera application may also include modes other than the shooting mode, such as supporting shooting mode, night scene mode, video recording mode, and HDR mode. The preview area displays a preview image, which can be an image captured by the camera.
[0077] When users take photos with their mobile phones, hand tremors or camera shake can both cause blurry images. This is especially noticeable when the subject is in motion, significantly reducing the user's shooting experience.
[0078] Based on the above introduction, the implementation of image stabilization methods will be described below.
[0079] One approach involves using OIS (Optical Image Stabilization) technology to achieve image stabilization. Specifically, when the device detects shaking, the OIS system can quickly adjust the position of the lens or image sensor by driving a motor to compensate for the movement caused by hand tremors or device vibration, thereby counteracting the effects of shaking.
[0080] Traditional motors typically operate in a single mode. For example, in implementing optical image stabilization, the processor uses sensors such as gyroscopes to detect device shake in different directions, and then drives the motor to move the lens in the x and y axes to compensate for the light shift caused by device shake, thereby ensuring stable and clear imaging.
[0081] However, due to the limitations of optical image stabilization in hardware structure, its compensation capability is limited, and the stabilization effect may still be unsatisfactory.
[0082] To improve image stabilization performance, a collaborative approach combining OIS and EIS can be employed. The EIS algorithm crops the edges of the image and dynamically transforms the image to counteract the effects of camera shake. In the collaborative implementation of OIS and EIS, the EIS algorithm can dynamically adjust the cropping area for each frame based on the motor motion position data fed back from the OIS system.
[0083] In actual implementation, the driver integrated circuit (DriverIC) module of the OIS system is responsible for reading the drive data used to control the movement of the OIS motor and determining the direction and displacement of the OIS motor based on this data. This drive data may include the position data that the OIS motor needs to reach, calculated by the processor based on factors such as the current shooting scene and image stabilization requirements, as well as parameters such as the motor's speed and acceleration, which are used to determine how the OIS motor moves from the current position to the target position.
[0084] In addition, the DriverIC module also reads feedback information from the OIS motor, such as the motor's current actual position and motion state. This feedback data is used to achieve closed-loop control, enabling the DriverIC module to continuously adjust the drive signal according to the actual situation, ensuring that the OIS motor accurately reaches the target position and maintains a stable operating state. By continuously reading and processing this feedback information, the DriverIC module can correct the motor's motion deviation in real time, improving the control accuracy and stability of the OIS system. The EIS algorithm can obtain the motor's position data (displacement) read by the Driver IC module through the API (Application Programming Interface).
[0085] After the EIS algorithm acquires the position data fed back by the OIS system, it can perform geometric transformations on each frame of the image based on the homography matrix (H matrix) to achieve jitter compensation. The H matrix describes the transformation relationship between the image before and after image stabilization. The formula for calculating the H matrix can be found in Formula 1 below:
[0086]
[0087] In Formula 1, the submatrix This is the camera's intrinsic parameter matrix, used to convert the camera coordinate system to the pixel coordinate system. Here, f represents the camera focal length, and c... x It is the coordinate of the principal point of the image (the intersection of the optical axis and the image plane) on the x-axis in the pixel coordinate system, c y This is the y-coordinate of the principal point of the image (the intersection of the optical axis and the image plane) in the pixel coordinate system. The pixel coordinate system has its origin at the top-left corner of the image plane and uses pixels as the unit. Typically, c... x This represents the coordinates of the pixel center along the horizontal axis of the image, and its value is half the number of pixels along the image width. y This represents the coordinates of the pixel center along the vertical axis of the image, and its value is half the number of pixels along the height axis of the image. The camera coordinate system has the camera's optical center as its origin. The x-axis and y-axis of the camera coordinate system are parallel to the x-axis and y-axis of the image coordinate system, respectively. The camera's optical axis is the Z-axis, and it follows the right-hand rule.
[0088] submatrix Let be the extrinsic parameter matrix of the camera, where x1 refers to the amount of rotation compensation required along the x-axis calculated by the gyroscope after the electronic device shakes, y1 refers to the amount of rotation compensation required along the y-axis, and z1 refers to the amount of rotation compensation required along the z-axis. In essence, the gyroscope can detect the angle of shaking of the terminal device (around the three axes, namely x, y, and z), and calculate the distance that the lens module needs to compensate for based on the angle, allowing the lens to counteract the shaking of the terminal device through reverse movement. The values of x1, y1, and z1 are all in the world coordinate system.
[0089] submatrix This is the camera's intrinsic parameter matrix, used to indicate the translation compensation amount of the principal point in the image after rotation. Where o x o y These represent the translations of the lens element along the x and y axes in the world coordinate system, obtained through the API interface. The unit is typically micrometers (μm). During calculation, since this is done in pixel coordinates, the translation amount also needs to be factored in. x and o y Convert it into the corresponding pixel coordinate value.
[0090] Among them, o x o y The data represents physical translations, while the H-matrix is based on the concept of pixels. Since physical quantities and pixels belong to different measurement systems, in practical applications, physical quantities cannot be directly applied to pixel-based H-matrix calculations; a conversion between the two is necessary. Taking a 1-micrometer physical displacement as an example, a calibration method is typically used to determine the corresponding number of pixels. Specifically, by accurately calibrating the device, a correspondence between physical quantities and pixels can be established. For example, the calibration result shows that a 1-micrometer physical displacement corresponds to 10 pixels. Since this is not the focus of this paper, this step will not be discussed in detail.
[0091] -(c x -o x This is used to indicate the coordinates of the optical center of the principal point of the image along the x-axis of the imaging plane after jitter occurs. y -o y This is used to indicate the coordinates of the optical center of the principal point of the image along the y-axis of the imaging plane after jitter occurs.
[0092] The H matrix is used to indicate the transformation relationship between the image before and after image stabilization. The H matrix can be used to perform geometric transformations on the current frame image to eliminate jitter. In this process, the EIS algorithm can use the H matrix to calculate the position of each pixel in the new image to correct for motion or jitter.
[0093] Specifically, for the position coordinates of any pixel in a frame of an image, the H matrix is applied for transformation. The calculation formula can be found in Formula 2 below:
[0094]
[0095] In Formula 2, (u, v) is a pixel position in the current frame image, and (u、, v、) are the corresponding pixel positions in the transformed stable image. This process can be understood as processing the position information of each pixel in the current frame image. The position information refers to the pixel's coordinates. The parameter s is the image scaling ratio.
[0096] During image transformation, when calculating the new position of each pixel, the pixel coordinates (u, v, ...) may not fall exactly on integer grid points but rather on decimal places, or some pixels may be mapped to locations that did not exist in the original image. In such cases, interpolation algorithms can be used to estimate the new pixel values. The purpose of interpolation is to calculate the color value of each transformed pixel. Common interpolation methods include nearest neighbor interpolation, bilinear interpolation, and cubic interpolation.
[0097] Furthermore, during image transformation, some areas of the image may be stretched or moved beyond the image boundaries, or blank areas may appear in the image (e.g., blank edges may appear after translation or rotation), making it impossible to calculate pixel values. In such cases, the image can be cropped (removing invalid areas) or padded (filling blank areas through interpolation to avoid image fragmentation) to maintain image integrity.
[0098] It's important to understand that in the collaborative implementation of OIS and EIS, because the EIS algorithm relies on accurate lens position data for image compensation, if there's a deviation in the position read back by the DriverIC module, the algorithm will be unable to accurately calculate the image offset, leading to a significant decrease in image stabilization performance. Therefore, this process requires the displacement data fed back by the OIS system to be highly consistent with the actual movement position of the motor.
[0099] In actual implementation, when the DriverIC module drives the motor, it involves two stages: feedforward control and feedback control. Feedforward control is mainly responsible for determining the motor's direction of motion and target position, and sending corresponding control commands to the motor to drive it to the target position.
[0100] During feedforward control, the system lacks strict closed-loop control over the motor's final actual position, driving the motor towards the target position only through an open-loop mechanism. After the motor reaches a position, the system can read back the current position via a position sensor (feedback control phase) to form a closed-loop correction.
[0101] However, several factors can cause discrepancies between the actual and read-back positions in the data transmission link. These factors include, but are not limited to: noise interference in the link, limitations in read-back accuracy, the physical characteristics of the motor itself, the influence of external gravity, and the user's handheld device posture. This deviation can further affect the processing performance of the EIS algorithm, ultimately potentially leading to unsatisfactory image stabilization.
[0102] For example, during actual compensation, the motor may undergo compound motion (such as translation + rotation) due to inertia or gravity. However, due to the limited ability of current sensors to analyze multidimensional motion parameters, the Driver IC module can only acquire displacement data of the translation component and cannot acquire the rotation component. In this case, the displacement data acquired by the EIS algorithm deviates from the complete displacement corresponding to the actual movement position of the motor. This leads to inaccurate compensation vectors generated by the anti-shake algorithm, thereby reducing anti-shake performance.
[0103] The following is combined Figure 2 This issue will be introduced. Figure 2 This is a schematic diagram of lens movement provided in an embodiment of this application.
[0104] like Figure 2 As shown in (a), assume there is a lens element and a photosensitive element, with the lens element driven by a motor. A Cartesian coordinate system is established with the top-left corner of the image plane as the origin, the horizontal direction parallel to the image as the x-axis (positive direction to the right in the image), and the vertical direction parallel to the image as the y-axis (positive direction downwards in the image). The following explanation assumes the lens only moves along the y-axis.
[0105] Reference Figure 2 In (a), ideally, the photosensitive plane is translated along the y-axis by o. y When the distance is reached, the motor drives the lens to also shift. y After reaching a certain distance, the point projected from the optical center of the lens onto the photosensitive plane remains unchanged, and the image remains unaffected. At this point, lens movement only involves translation.
[0106] However, in reality, due to factors such as inertia and gravity, the motor will experience rotational motion during the compensation process. This rotational motion causes a change in the projection point of the lens's optical center onto the photosensitive plane, thus leading to changes in the image. (Refer to...) Figure 2 In (b), the projection point of the optical center of the lens element onto the photosensitive plane has shifted compared to the corresponding point under ideal conditions. The amount of displacement that needs to be compensated for must include both the translation along the y-axis and the displacement along the y-axis. y This also includes the rotation amount R. y .
[0107] Theoretically, the data used for subsequent image stabilization processing in the EIS algorithm should fully reflect the... y With R y The overall motion situation. However, in reality, the motion data fed back to the EIS algorithm by the OIS system only includes o y And lacking R y This information will affect the image stabilization effect of the EIS algorithm.
[0108] To address the problems described above, this application proposes the following technical concept: A mapping relationship is established between the translation and rotation amounts generated by the lens element during its movement, enabling the prediction of the possible rotation amount of the lens element during movement based on the translation amount fed back by the OIS system. Then, an H matrix is constructed based on the translation amount fed back by the OIS system and the predicted rotation amount to implement the EIS image stabilization algorithm, thereby improving the image stabilization effect.
[0109] The image stabilization method of this application can be executed by an electronic device equipped with a camera, or by a chip, chip system, or processor that supports the implementation of the image stabilization method in the electronic device. It can also be executed by a logic module or software that can implement all or part of the functions of the electronic device; this application does not impose specific limitations in this regard. The image stabilization method of this application will be described in detail below using an electronic device as the execution subject as an example.
[0110] Electronic devices can be, for example, terminal devices. The following section will first combine... Figure 3 and Figure 4 A brief introduction to the terminal equipment.
[0111] For example, Figure 3 This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of this application.
[0112] Figure 3 This is a schematic diagram of the terminal device provided in an embodiment of this application. The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a gyroscope sensor, a magnetometer, and an accelerometer.
[0113] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device. In other embodiments of this application, the terminal device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0114] The processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors. In one implementation, for example, the processor may execute the image stabilization method provided in this application.
[0115] The terminal device implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.
[0116] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel.
[0117] Terminal devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.
[0118] Camera 193 is used to capture still images or videos.
[0119] A gyroscope sensor can be used to determine the motion posture of a terminal device. In some embodiments, the gyroscope sensor can determine the angular velocity of the terminal device around three axes (i.e., the x, y, and z axes). A gyroscope sensor can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor detects the angle of the terminal device's shake, calculates the distance the lens module needs to compensate based on the angle, and allows the lens to counteract the shake by moving in the opposite direction, thus achieving image stabilization. Gyroscope sensors can also be used in navigation and motion-sensing gaming scenarios.
[0120] The magnetic sensor includes a Hall effect sensor. The terminal device can use the magnetic sensor to detect the opening and closing of the flip cover. In some embodiments, when the terminal device is a flip phone, it can detect the opening and closing of the flip cover using the magnetic sensor. Based on the detected opening and closing state of the cover or the flip cover, features such as automatic unlocking of the flip cover can be configured.
[0121] An accelerometer can detect the magnitude of acceleration in various directions (typically three axes) of a terminal device. When the terminal device is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the attitude of the terminal device and is applied to applications such as screen orientation switching and pedometers.
[0122] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations performed on different applications (such as taking photos, playing audio, etc.).
[0123] The software system of a terminal device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc. This application uses the layered architecture Android system as an example to illustrate the software structure of the terminal device.
[0124] For example, Figure 4 This is a schematic diagram of the software structure of a terminal device provided in an embodiment of this application.
[0125] like Figure 4 As shown, the layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the system may include an application layer, an application framework layer, an Android runtime and system libraries, a hardware abstraction layer (HAL), and a kernel layer. It should be noted that this application uses the Android system as an example; however, the solution can also be implemented in other operating systems (such as HarmonyOS, iOS, etc.) as long as the functions implemented by each module are similar to those in the embodiments of this application.
[0126] The application layer can include a series of application packages.
[0127] like Figure 4 As shown, the application package may include applications such as camera, gallery, calendar, maps, phone, music, settings, email, video, and social media. Of course, the application layer may also include other application packages, such as third-party applications like payment apps, shopping apps, banking apps, and social media apps; this application does not limit this.
[0128] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.
[0129] like Figure 4 As shown, the application framework layer may include a window manager, content provider, resource manager, view system, notification manager, etc.
[0130] The Android runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.
[0131] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.
[0132] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.
[0133] The system library can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.
[0134] The HAL layer is a wrapper around Linux kernel drivers, providing interfaces to the upper layers and shielding them from the implementation details of the lower-level hardware.
[0135] The HAL layer may include a Wi-Fi HAL, an audio HAL, a camera service (Camera HALServer) unit, and a software code library. In this embodiment, for example, a camera HAL and a camera algorithm library may be deployed in the HAL layer. The camera HAL is primarily responsible for interacting between the camera hardware and upper-layer applications, performing various operations such as device initialization, parameter configuration, image data acquisition and processing, and image stabilization to achieve camera functionality and ensure image quality. The camera algorithm library may include, for example, running code and data implementing the image stabilization method provided in this embodiment, used to perform a series of image algorithm processing on the acquired images to obtain the final output image.
[0136] The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, and a sensor driver. In this embodiment, for example, a motor driver, a gyroscope driver, and a camera driver can be deployed at the kernel layer. The camera driver is used to drive the camera to capture images. The motor driver is used to drive the motor to change the position of the lens. The gyroscope driver is used to detect jitter information in the electronic device. The Hall effect sensor is used to detect the position of the lens.
[0137] The technical solutions of the embodiments of this application and how the technical solutions of the embodiments of this application solve the above-mentioned technical problems will be described in detail below with reference to the accompanying drawings and specific examples. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0138] The technical solution of this application relates to an image stabilization method, which will be described below. The image stabilization process will be explained in conjunction with specific embodiments. Figure 5 The implementation of image stabilization methods is introduced. Among them, Figure 5 This is a schematic diagram of an image stabilization method provided in an embodiment of this application. The process of the image stabilization method may include, for example, the following steps:
[0139] S501. Obtain the first translation parameter of the lens element. The first translation parameter is used to indicate the amount of translation generated by the lens element during the movement.
[0140] Based on the above description, it can be determined that after the sensor device detects jitter, the Driver IC module first drives the voice coil motor (VCM) through a closed-loop control system to perform optical compensation movement of the lens. During this process, the Driver IC module also reads the first translation parameters of the lens elements driven by the OIS motor in the vertical direction (x / y axis). The first translation parameters include: the first translation component corresponding to the first axis and the second translation component corresponding to the second axis.
[0141] Wherein, the first axis is the x-axis in the world coordinate system, and the second axis is the y-axis in the world coordinate system. The first translation component can be the distance the lens element moves along the x-axis, and the second translation component can be the distance the lens element moves along the y-axis.
[0142] Then, the EIS algorithm can be used to compensate the first image captured by the lens element.
[0143] S502. Based on the first translation parameter, determine the first rotation parameter of the lens element. The first rotation parameter is used to indicate the amount of rotation generated by the lens element during the movement.
[0144] Based on the above introduction, it can be determined that due to factors such as inertia and gravity, the motor's motion may exhibit multiple modes (translation or rotation). However, due to the insufficient analytical capabilities of current sensors for multidimensional motion parameters, only translational displacement data can be acquired, while rotational displacement data cannot be obtained. This leads to a discrepancy between the displacement data acquired by the EIS algorithm and the complete displacement corresponding to the actual movement position of the motor, resulting in the algorithm's inability to accurately compensate for lens shake and ultimately causing poor image stabilization performance.
[0145] In this embodiment, to obtain a more accurate complete displacement corresponding to the actual movement position of the motor, the rotational amount generated during the motor's movement can be predicted, for example. In one implementation, it can be assumed that the characteristics of the motors in each electronic device are the same, and that the rotational error generated by the motor after translating a certain distance is constant. Specifically, if the motor translation is greater, it means that the electronic device vibrates more violently and its stability is worse, and the undetected rotational amount generated by the motor movement may also be greater. Conversely, if the motor translation is smaller, it means that the electronic device vibrates less and its stability is better, and the undetected rotational amount generated by the motor movement may also be smaller.
[0146] Therefore, for example, the first rotation parameter can be determined based on the first translation parameter. The first rotation parameter includes: the first rotation component corresponding to the first axis, the second rotation component corresponding to the second axis, and the third rotation component corresponding to the third axis. The first axis can be the x-axis in the world coordinate system, the second axis can be the y-axis in the world coordinate system, and the third axis can be the z-axis in the world coordinate system. The first rotation parameter is proportional to the first translation parameter.
[0147] You can refer to this. Figure 5To understand this, in one implementation, a first adjustment parameter can be introduced, and a first translation parameter can be adjusted based on the first adjustment parameter to obtain a first rotation parameter. The first adjustment parameter indicates the degree of rotation caused by the first translation parameter. The first rotation parameter is proportional to both the first translation parameter and the first adjustment parameter. In another implementation, the first adjustment parameter corresponding to the first translation parameter can be determined based on a preset mapping relationship.
[0148] Specifically, based on a preset mapping relationship, the adjustment parameters corresponding to the first translation component and the second translation component can be determined. For example, assume the current first translation component o... x The value is 3 micrometers. Based on the preset mapping relationship, the first translation component o can be determined. x The corresponding adjustment parameter is 0.0015.
[0149] In one implementation, OIS calibration technology can be used to establish a mapping relationship between translation parameters and adjustment parameters. The mapping relationship between translation parameters and adjustment parameters will not be explained here.
[0150] The determination of the first rotation parameter will be introduced below.
[0151] In one implementation, for example, a first rotation component can be determined based on a first translation component and its corresponding adjustment parameters. A second rotation component can be determined based on a second translation component and its corresponding adjustment parameters. Furthermore, a third rotation component can be determined based on the first rotation component.
[0152] For example, in one implementation, the formula for calculating the first rotation component can be referenced as Formula 3 below:
[0153] Δx=o x *gain x Formula 3
[0154] In Formula 3, Δx is the first rotational component, o x For the first translation component, gain x These are the adjustment parameters corresponding to the first translation component.
[0155] In one implementation, the formula for calculating the second rotation component can be referenced, for example, Formula 4 below:
[0156] Δy=o y *gain y Formula 4
[0157] In Formula 4, Δy is the second rotational component, o y For the second translation component, gain yThis is the adjustment parameter corresponding to the second translation component.
[0158] Furthermore, in one implementation, the formula for calculating the third rotational component can be referenced, for example, Formula 5 below:
[0159] Formula 5: Δz = ±Δx
[0160] In Formula 5, Δx is the first rotational component. The sign of Δx can be determined through calibration.
[0161] S503, Acquire the first image based on the lens element.
[0162] S504. For any first pixel in the first image, the position information of the first pixel is compensated according to the first translation parameter and the first rotation parameter to obtain the second image.
[0163] In this embodiment, after obtaining the first translation parameter and the first rotation parameter, the first transformation matrix can be determined. The first transformation matrix is used to indicate the parameters to be compensated corresponding to the motion state of the lens element, and the motion state corresponds to the first translation parameter and the first rotation parameter. (See reference...) Figure 5 By understanding this, the first transformation matrix can be determined based on the gyroscope data, the first translation parameter, and the first rotation parameter.
[0164] Based on Formula 1 above, it can be understood that a sub-matrix can be constructed based on the first translation parameter to compensate for the translational motion of the lens element. Furthermore, a sub-matrix can be constructed based on gyroscope data to compensate for the rotation of the lens element. However, when compensating for the rotation of the lens element using gyroscope data, the rotation caused by factors such as gravity and inertia is not taken into account.
[0165] In one implementation, a second sub-matrix is constructed based on a first rotation parameter and gyroscope data to compensate for the rotation of the lens element.
[0166] For example, in one implementation, the formula for calculating the first transformation matrix can be found in Formula Six below:
[0167]
[0168] In Formula 6, the submatrix Let f be the intrinsic parameter matrix of the camera, where f represents the camera focal length and c x It is the coordinate of the principal point of the image (the intersection of the optical axis and the image plane) on the x-axis in the pixel coordinate system, c y It is the coordinate of the principal point of the image (the intersection of the optical axis and the image plane) on the y-axis in the pixel coordinate system.
[0169] First submatrix This is the camera's intrinsic parameter matrix, used to indicate the translation compensation amount of the principal point of the image after rotation. x o y These represent the translation amounts of the lens element in the world coordinate system along the x and y axes, obtained through the API interface.
[0170] Second submatrix This is the camera's extrinsic parameter matrix, used to indicate the amount of rotation compensated for by the lens element during movement when camera shake occurs. It can be understood that x1, y1, and z1 refer to the rotation amounts to be compensated for by the electronic device on the three axes (x, y, and z axes) in the world coordinate system, calculated by the gyroscope. Δx, Δy, and Δz are the first rotation parameters predicted based on the first translation parameters. These first rotation parameters indicate the amount of rotation generated by the lens element during movement.
[0171] When performing image stabilization, the rotational amount Rx used for image compensation corresponds to the compensation amount on the photosensitive plane, not directly to the rotation of the electronic device (i.e., x1, y1, and z1). It's understandable that after the electronic device rotates, the lens element may rotate in the opposite direction due to factors such as force and inertia. This reverse rotation of the lens element is represented by Δx, Δy, and Δz. Therefore, the actual rotational amount that needs compensation should be the electronic device's rotation minus the reverse rotation of the lens element. Thus, x1-Δx, y1-Δy, and z1-Δz can be used to indicate the actual rotational amount that needs compensation. For example, assuming the current rotation angle of the electronic device is 3° and the reverse rotation angle of the lens element is 0.5°, then the actual angle that needs compensation on the photosensitive plane is 2.5°.
[0172] After obtaining the first transformation matrix, it can be applied to any pixel in the first image captured by the lens element to achieve image stabilization. This part can be understood by referring to the above introduction.
[0173] In this embodiment, the rotation amount that the lens element may generate during movement can be quantified based on the first translation parameter. This rotation amount is usually caused by factors such as gravity and inertia affecting the lens element. In this way, by using the first translation parameter and the first rotation parameter predicted by the first translation parameter simultaneously to process the image, image jitter can be eliminated more accurately.
[0174] In the implementation process, a first adjustment parameter is introduced, and a first rotation parameter is determined based on this parameter and a first translation parameter. This allows for a precise description of the translation and rotation movements that the lens elements need to compensate for during the shooting process.
[0175] Based on the above-described embodiments, the following is combined with... Figure 6 and Figure 7 The determination of the first adjustment parameter will be explained. Among them, Figure 6 This is a schematic diagram illustrating a scenario of simulated electronic device jitter provided in an embodiment of this application. Figure 7 This is a schematic diagram illustrating the mapping relationship between the preset amplitude and translation parameters provided in the embodiments of this application.
[0176] Referring to the above description of S502, it can be determined that in this embodiment, it is necessary to determine the first adjustment parameter corresponding to the first translation parameter according to the preset mapping relationship. Therefore, the determination of the preset mapping relationship between the translation parameter and the adjustment parameter will be explained below.
[0177] In this embodiment, the vibration of the electronic device can be controlled based on a preset amplitude to simulate the shaking of the electronic device in a real-world scenario. Then, a preset mapping relationship between the translation parameters and the adjustment parameters is determined based on the anti-shake effect of the electronic device. The following section combines... Figure 6 This section introduces a scenario simulating electronic device vibration, using a mobile phone as an example.
[0178] like Figure 6 As shown, in the current experimental scenario, the mobile phone can be placed on a vibration table. The vibration table can simulate various vibration environments, such as vibration states of different frequencies and amplitudes. By simulating these complex vibration conditions, the anti-shake effect of the mobile phone when encountering vibrations in actual use can be tested.
[0179] The calibration object on the right side of the screen serves as a key reference point throughout the entire test. While the phone is continuously vibrating under the control of the vibration table, it continuously captures video data with the calibration object as the shooting target.
[0180] In the specific implementation process, for example, multiple preset amplitudes can be preset, and then the electronic device can be controlled to vibrate based on at least one preset amplitude, so as to collect at least one set of calibration data corresponding to each preset amplitude during the vibration process. The calibration data may include at least one of the following: first video, gyroscope data, second translation parameters, and frame information. For example, the preset amplitudes can be set to: 0.05°, 0.1°, 0.15°, ..., 1.5°, 1.55°, ..., 3°.
[0181] The first video consists of video frames captured by the lens element of the electronic device, which will be used for subsequent image stability analysis. The gyroscope data comprises the angular velocity and angle information of the electronic device output by the gyroscope, reflecting the actual vibration conditions. The second translation parameter is the actual driving force of the OIS system, i.e., the displacement (unit: micrometers) of the lens or sensor driven by the OIS system; this data reflects the OIS's vibration compensation action. Frame information, including the timestamp and exposure time of each frame, can be used for the synchronous analysis of the first video and gyroscope data.
[0182] Then, calibration processing can be performed based on at least one set of calibration data corresponding to at least one preset amplitude to obtain a preset mapping relationship between translation parameters and adjustment parameters.
[0183] It is understandable that when the vibration table control electronics vibrate based on any preset amplitude, the lens element can compensate for the shaking by moving in the opposite direction. The translation parameter of the lens element during this reverse movement is o. x or o y Therefore, based on any preset amplitude, the translation parameters (i.e., o) generated by the lens element during its movement can be determined. x or o y ).
[0184] It can also be understood that, based on the preset amplitude and the image stabilization effect of the electronic device under that preset amplitude, the correspondence between the preset amplitude and the adjustment parameters can be determined. Therefore, the translation parameters (i.e., o) generated by the lens elements during movement can be used as a basis for determining the relationship between the preset amplitude and the adjustment parameters. x or o y The correspondence between the translation parameters and adjustment parameters generated by the lens elements during the movement is used to determine the preset mapping relationship.
[0185] The following is combined Figure 7 First, we will introduce the correspondence between the preset amplitude and the translation parameters generated by the lens elements during the movement.
[0186] like Figure 7 As shown, assume there is a lens element, a photosensitive plane, and a subject. The distance from the lens element to the photosensitive plane is the image distance. Assume the electronic device is vibrating along the y-axis (i.e., rotating around the y-axis), with a preset amplitude of 3° (the maximum angle of rotation around the y-axis is 3°). During vibration, the lens element of the electronic device can, for example, rotate in the opposite direction along the y-axis to compensate for the shake. At this time, the mapping point of the optical center of the lens element on the photosensitive plane will be translated along the y-axis. This translation parameter is o. y For example, when the maximum angle of rotation of the electronic device around the y-axis is 3°, the maximum angle of rotation of the lens element in the opposite direction can also be approximated as 3°.
[0187] Reference Figure 7 The translation parameter o can be obtained from the image distance and the tangent of the lens element rotation angle θ. y , that is o y The formula for calculating θ is image distance * tan(θ). Corresponding to... Figure 7 In the middle, o y Image distance * tan(3°). When the electronic device vibrates along the x-axis, the preset amplitude and translation parameters are...x The mapping relationship is similar, and will not be elaborated further here.
[0188] Then, the correspondence between the preset amplitude and the adjustment parameters can be introduced.
[0189] In determining the preset mapping relationship between preset amplitude and adjustment parameters, the electronic device can be vibrated on two axes (x and y axes) in the world coordinate system according to at least one preset amplitude, so as to determine the adjustment parameters corresponding to at least one preset amplitude on each of the two axes.
[0190] The processing method for any one of the at least one preset amplitudes is similar. The following explanation will take any one preset amplitude as an example.
[0191] In one implementation, a preset amplitude at a mixed frequency (e.g., 0.05-5 Hz) can be set on the vibration table. The electronic device is then vibrated according to the preset amplitude at the mixed frequency of 0.05-5 Hz to obtain the adjustment parameters calibrated at that preset amplitude.
[0192] Taking a preset amplitude of 3° and a vibration direction along the x-axis as an example, the electronic device is fixed on a vibration table, and the amplitude is set to 3° at a mixed frequency of 0.05-5 Hz to simulate handheld shooting. During this process, a set of calibration data is collected, which may include the first video, gyroscope data, the second translation parameters, and frame information.
[0193] Then, multiple candidate adjustment parameters can be preset, and the stabilized video sequence can be calculated based on the calibration data and any one of the candidate adjustment parameters. In one implementation, for example, the value range of the candidate adjustment parameters can be manually set first based on experience. Then, the value of the candidate adjustment parameter can be selected within this value range with a certain step size. For example, the value range of the candidate adjustment parameter can be set to [0.0017, 0.0022], and the candidate adjustment parameter can be selected within this range with a step size of 0.0001. Then, the multiple candidate adjustment parameters can be, for example, 0.0017, 0.0018, 0.0019, 0.0020, 0.0021, 0.0022.
[0194] After obtaining the stabilized video sequence corresponding to the candidate adjustment parameter, for example, each frame in the stabilized video sequence can be compared with its corresponding expected image to calculate the error.
[0195] It's understandable that for a given candidate adjustment parameter, its corresponding stabilized video sequence can be calculated. Therefore, by iterating through each candidate adjustment parameter, the stabilized video sequence corresponding to that parameter can be obtained. Then, for each candidate adjustment parameter, the error between each frame in the stabilized video sequence and the expected image is calculated. The image error can be calculated, for example, based on the image's pixel information. Finally, the candidate adjustment parameter with the smallest error is determined as the adjustment parameter corresponding to that preset amplitude. The adjustment parameter corresponding to each preset amplitude can be determined in this way.
[0196] The following describes the process of calculating the stabilized video sequence based on a single candidate adjustment parameter. The processing method for any one of multiple candidate adjustment parameters is similar; we will use any single candidate adjustment parameter as an example below. For instance, let's take a preset amplitude of 3° and a vibration direction along the x-axis.
[0197] In this process, the path of the electronic device during movement can be estimated using gyroscope data. This path refers to the camera pose corresponding to each frame of the first video. This pose is the camera's rotation angle along the x, y, and z axes, respectively. Then, based on specific optimization algorithms (such as filtering algorithms), a smoother path can be generated by optimizing the pose path estimated by the gyroscope.
[0198] For each frame in the first video, the pose in the optimized smooth path serves as a virtual reference pose. When the camera pose corresponding to that image is the corresponding pose in the optimized smooth path, the image after image stabilization can be obtained.
[0199] In the specific implementation process, each frame of the first video can be geometrically transformed using the second transformation matrix, converting it into an image corresponding to the optimized pose. The second transformation matrix indicates the parameters to be compensated corresponding to the motion state of the lens element during vibration, with the motion state corresponding to the second translation parameters and the second rotation parameters.
[0200] Specifically, the second transformation matrix can be determined based on Formula 6 above, where x1, y1, and z1 refer to the rotation amounts compensated by the gyroscope on the three axes (x, y, and z axes) in the world coordinate system, respectively. Δx, Δy, and Δz are the first rotation parameters predicted based on the first translation parameters.
[0201] It's important to understand that in this scenario, the vibration direction is set to the x-axis (i.e., the electronic device only rotates around the x-axis). Theoretically, the gyroscope has no compensated rotation on the y and z axes. However, in actual operation, when the electronic device rotates around the x-axis, it may also generate slight rotations in the y and z axes. Therefore, there will actually be some compensated rotation on the y and z axes. It's worth noting, however, that the rotation on the y and z axes (Δy and Δz) is relatively small and can be ignored in applications where high precision is not required or in specific scenarios.
[0202] Therefore, in one implementation, when the electronic device rotates around a single x-axis, the second transformation matrix can be referenced, for example, as in Formula 7:
[0203]
[0204] In Formula 7, Δx1 is the first rotational component corresponding to the first axis in the second rotational parameter. From Formula 3, the calculation formula for the first rotational component is Δx1 = o. x1 *gain x1 , where o x1 It is the first translation component in the second translation parameter included in the calibration data of the preset amplitude. x1 These are the optional adjustment parameters corresponding to the preset amplitude of 3° in the x-axis direction.
[0205] Subsequently, the second transformation matrix can be obtained based on the second rotation parameters and the second translation parameters. By multiplying the second transformation matrix by any third image in the first video frame, the fourth image, i.e., the stabilized image corresponding to the third image, can be calculated. By processing all video frames in this manner, a completely new video sequence, i.e., the stabilized video sequence, can be generated.
[0206] Based on the above method, it can be understood that the adjustment parameter corresponding to each preset amplitude in the x-axis direction can be determined. Then, the adjustment parameter corresponding to any amplitude between any two preset amplitudes can be calculated using bilinear interpolation.
[0207] Bilinear interpolation is a method for interpolating discrete data in two-dimensional space to estimate the value of an unknown point among known data points. It calculates the value of the target point by weighted averaging the values of its four nearest neighbors. A detailed explanation of bilinear interpolation will not be provided here.
[0208] Based on the above introduction, it can be determined that, based on the preset mapping relationship between the preset amplitude and the adjustment parameter on the x-axis, and the mapping relationship between the preset amplitude and the translation parameter on the x-axis, the preset mapping relationship between the translation component and the adjustment parameter on the x-axis can be obtained.
[0209] In one implementation, after obtaining the preset mapping relationship between the translation component on the x-axis and the adjustment parameter, these can be used, for example, to determine the preset mapping relationship between the translation component on the y-axis and the adjustment parameter.
[0210] For example, in one implementation, the preset amplitude can be 3°, with the vibration direction along the y-axis. The electronic device is fixed on a vibration table, and the amplitude is set to 3° at a mixed frequency of 0.05-5 Hz to simulate handheld shooting. During this process, a set of calibration data is collected, which may include a first video, gyroscope data, a second translation parameter, and frame information.
[0211] The following steps can be performed as described above. Multiple candidate adjustment parameters can be preset, and the stabilized video sequence can be calculated based on the calibration data and any one of these candidate parameters. Furthermore, the error corresponding to the stabilized video sequence for any candidate adjustment parameter can be calculated. Finally, the candidate adjustment parameter with the smallest error can be determined as the adjustment parameter corresponding to the preset amplitude.
[0212] However, in this case, the calculation method for the second transformation matrix differs from the description above. For example, in one implementation where the electronic device rotates around the y-axis, the formula for calculating the second transformation matrix can be found in Formula Eight:
[0213]
[0214] In Formula 8, x1, y1, and z1 refer to the rotational amounts calculated by the gyroscope on the three axes (x, y, and z axes) in the world coordinate system, respectively. Δx2 is the first rotational component corresponding to the first axis in the second rotational parameters, and Δy2 is the second rotational component corresponding to the second axis in the second rotational parameters.
[0215] From Formula 3, we can determine that the formula for calculating the first rotational component is Δx² = o. x2 *gain x2 , where o x2 It is the first translation component in the second translation parameter included in the calibration data of the preset amplitude. x2 Let o be the first translation component in the x-axis direction. x2 The corresponding adjustment parameters can be obtained from the mapping relationship between the translation components on the x-axis and the adjustment parameters. This can be understood as the gain mentioned above... x1 It can be any one of several pre-set selectable adjustment parameters. And gain x2 The first translation component obtained after calibration. x2This corresponds to a specific candidate adjustment parameter, which is selected from the multiple candidate adjustment parameters mentioned earlier, gain. x1 Selected from [the list].
[0216] From Formula 4, we can determine that the formula for calculating the second rotational component is Δy2 = o y2 *gain y2 , where o y2 It is the second translation component in the second translation parameter included in the calibration data of the preset amplitude. y2 These are the optional adjustment parameters corresponding to the preset amplitude of 3° in the y-axis direction.
[0217] In one implementation, after obtaining the preset mapping relationship between the translation component on the x-axis and the adjustment parameter, and the preset mapping relationship between the translation component on the y-axis and the adjustment parameter, they can be used, for example, to determine the positive and negative relationship between the third rotation component Δz and the first rotation component Δx.
[0218] In one implementation, the electronic device can be controlled to vibrate along the z-axis with a specific amplitude to determine the positive or negative relationship between Δz and Δx. For example, a preset amplitude of 3° can be used, with the vibration direction along the z-axis. The electronic device is fixed on a vibration table, and the amplitude is set to 3° at a mixed frequency of 0.05-5 Hz to simulate handheld shooting. During this process, a set of calibration data is collected, which may include a first video, gyroscope data, a second translation parameter, and frame information.
[0219] Then, the error of the stabilized video sequence can be calculated for both cases where Δz is Δx and Δz is -Δx. Afterwards, based on the sign of Δz and Δx when the error is minimized, the sign of Δz and Δx in practical applications can be determined (i.e., the calculation formula for the third rotation component shown in Formula 5).
[0220] For example, assuming Δz = Δx, the calculated error of the stabilized video sequence is 0.01. When Δz = -Δx, the calculated error is 0.04. Therefore, it can be determined that Δz equals Δx when the error is minimized, and thus the formula for calculating the third rotation component in practical applications is Δz = Δx. The specific implementation of this process is similar to the description above.
[0221] However, it's important to understand that the calculation method for the second transformation matrix differs from the methods described above in this case. For example, in one implementation where the electronic device rotates around the z-axis, the formula for calculating the second transformation matrix can be found in Formula Nine:
[0222]
[0223] In Formula 9, x1, y1, and z1 refer to the rotational amounts compensated by the gyroscope on the three axes (x, y, and z axes) in the world coordinate system, respectively. Δx3 is the first rotational component corresponding to the first axis in the second rotational parameters, Δy3 is the second rotational component corresponding to the second axis in the second rotational parameters, and Δz3 is the third rotational component corresponding to the third axis in the second rotational parameters.
[0224] From Formula 3, we can determine that the formula for calculating the first rotational component is Δx3 = o x3 *gain x3 , where o x3 It is the first translation component in the second translation parameter included in the calibration data of the preset amplitude. x3 Let o be the first translation component in the x-axis direction. x3 The corresponding adjustment parameters can be obtained from the mapping relationship between the translation components on the x-axis and the adjustment parameters. Specifically, gain x3 The first translation component obtained after calibration. x3 This corresponds to a specific candidate adjustment parameter, which is selected from the multiple candidate adjustment parameters mentioned earlier, including the gain parameter. x1 Selected from [the list].
[0225] From Formula 4, we can determine that the formula for calculating the second rotational component is Δy3=o. y3 *gain y3 , where o y3 It is the second translation component in the second translation parameter included in the calibration data of the preset amplitude. y3 Let o be the translation component in the y-axis direction. y3 The corresponding adjustment parameters can be obtained from the mapping relationship between the translation components on the y-axis and the adjustment parameters. This can be understood as the gain mentioned above... y2 It can be any one of several pre-set selectable adjustment parameters. And gain y3 The second translation component obtained after calibration. y3 This corresponds to a specific candidate adjustment parameter, which is selected from the multiple candidate adjustment parameters mentioned earlier, gain. y2 Selected from [the list].
[0226] Formula 5 shows that the formula for calculating the third rotational component is Δz3 = Δx3, or Δz3 = -Δx3.
[0227] In this embodiment, multiple selectable adjustment parameters are set for any preset amplitude, and the most suitable selectable adjustment parameter is determined based on the anti-shake effect of the multiple selectable adjustment parameters. This can more accurately obtain the adjustment parameters corresponding to the translation parameters of the preset amplitude, thereby effectively reducing errors in the image compensation process and improving the accuracy of compensation.
[0228] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.
[0229] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0230] The image stabilization method according to the embodiments of this application has been described above. The apparatus for performing the above method provided in the embodiments of this application is described below. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced by each other, and the related apparatus provided in the embodiments of this application can perform the steps in the above image stabilization method.
[0231] The image stabilization method provided in this application can be applied to electronic devices with image processing capabilities. Electronic devices include terminal devices, and the specific device form of the terminal device can be referred to the above-described related information, which will not be repeated here.
[0232] In one implementation, this application provides an electronic device. Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.
[0233] like Figure 8 As shown, the electronic device 800 includes a processor 801 and a memory 802; the memory 802 stores computer execution instructions; the processor 801 executes the computer execution instructions stored in the memory 802, causing the electronic device 800 to perform the above-described method.
[0234] When the memory 802 is set up independently, the electronic device also includes a bus 803 for connecting the memory 802 and the processor 801.
[0235] This application provides a chip. The chip includes a processor, which is used to call a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.
[0236] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.
[0237] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.
[0238] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.
[0239] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0240] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.
Claims
1. An image anti-shake method, characterized in that, An electronic device comprising a motor and a lens element, wherein the motor is used to drive the lens element to move and adjust its position during image stabilization processing; The method includes: Obtain a first translation parameter of the lens element, wherein the first translation parameter is used to indicate the amount of translation generated by the lens element during the movement; Based on the first translation parameter, a first rotation parameter of the lens element is determined, wherein the first rotation parameter is used to indicate the amount of rotation generated by the lens element during the movement; Acquire a first image based on the lens element; For any first pixel in the first image, the position information of the first pixel is compensated according to the first translation parameter and the first rotation parameter to obtain the second image.
2. The method of claim 1, wherein, Determining the first rotation parameter of the lens element based on the first translation parameter includes: According to the preset mapping relationship, the first adjustment parameter corresponding to the first translation parameter is determined. The first adjustment parameter is used to indicate the degree of rotation caused by the first translation parameter. The preset mapping relationship is used to indicate the correspondence between the translation amount and the adjustment parameter. The first translation parameter is adjusted based on the first adjustment parameter to obtain the first rotation parameter, wherein the first rotation parameter is proportional to both the first translation parameter and the first adjustment parameter.
3. The method according to claim 2, characterized in that, The step of compensating the position information of the first pixel based on the first translation parameter and the first rotation parameter includes: Based on the first translation parameter and the first rotation parameter, a first transformation matrix is determined. The first transformation matrix is used to indicate the parameters to be compensated corresponding to the motion state of the lens element. The motion state corresponds to the first translation parameter and the first rotation parameter. The compensation process is performed on the position information of the first pixel based on the first transformation matrix.
4. The method according to claim 3, characterized in that, The compensation process for the position information of the first pixel based on the first transformation matrix includes: The compensation process is performed on the position information of the first pixel by multiplying the first transformation matrix with the position information of the first pixel.
5. The method according to claim 3 or 4, characterized in that, The first translation parameter includes: a first translation component corresponding to the first axis and a second translation component corresponding to the second axis; the first rotation parameter includes: a first rotation component corresponding to the first axis, a second rotation component corresponding to the second axis, and a third rotation component corresponding to the third axis; Determining the first transformation matrix based on the first translation parameter and the first rotation parameter includes: Based on the first translation parameter, a first sub-matrix is determined, wherein the elements in the first sub-matrix corresponding to the first axis are calculated based on the first translation component, and the elements in the first sub-matrix corresponding to the second axis are calculated based on the second translation component; Based on the first rotation parameter, a second sub-matrix is determined. The elements in the second sub-matrix corresponding to the first axis are calculated based on the first rotation component. The elements in the second sub-matrix corresponding to the second axis are calculated based on the second rotation component. The elements in the second sub-matrix corresponding to the third axis are calculated based on the third rotation component. The first transformation matrix is determined based on the first submatrix and the second submatrix.
6. The method according to any one of claims 2-5, characterized in that, The method further includes: The electronic device is controlled to vibrate based on at least one preset amplitude, so as to collect at least one set of calibration data corresponding to each of the at least one preset amplitude during the vibration process. The calibration data includes at least one of the following: first video, gyroscope data, second translation parameters and frame information, and the at least one preset amplitude corresponds to its respective translation parameters. The calibration process is performed based on at least one set of calibration data corresponding to each of the at least one preset amplitude to obtain the preset mapping relationship. The preset mapping relationship includes adjustment parameters corresponding to each of the multiple translation parameters, and the multiple translation parameters correspond to the at least one preset amplitude respectively.
7. The method according to claim 6, characterized in that, The calibration process, based on at least one set of calibration data corresponding to each of the at least one preset amplitude, to obtain the preset mapping relationship, includes: For any one of the at least one preset amplitudes, multiple selectable adjustment parameters are preset; For any one of the multiple candidate adjustment parameters, a second rotation parameter corresponding to the second translation parameter is determined based on the candidate adjustment parameter and the second translation parameter included in the calibration data of the preset amplitude. Based on the second rotation parameter and the second translation parameter, a corresponding second transformation matrix is determined. The second transformation matrix is used to indicate the parameters to be compensated corresponding to the motion state of the lens element during the vibration process. The motion state corresponds to the second translation parameter and the second rotation parameter. For any frame of the third image in the first video, the position information of each pixel in the third image is compensated based on the second transformation matrix to obtain the fourth image; The error is determined based on the expected image corresponding to the candidate adjustment parameter and the fourth image, and the candidate adjustment parameter with the smallest error is determined as the adjustment parameter corresponding to the translation parameter corresponding to the preset amplitude.
8. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 7.
9. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 7.
11. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 7.
