Optical image stabilization device and optical image stabilization method

Abstract

Disclosed in embodiments of the present application are an optical image stabilization device and an optical image stabilization method. The optical image stabilization device may comprise a motor driver and an image sensor physically connected to the motor driver; the image sensor is used for reporting the working state of the image sensor to the motor driver; the working state comprises one or more of an exposure state, a read state, and an inter-frame idle state; the motor driver is used for monitoring the working state of the image sensor and adjusting the driving state of the motor driver to a target driving state on the basis of the working state; and the driving state comprises a driving mode and a driving frequency. The present application can improve the imaging quality and reduce the power consumption.

Description

Optical anti-shake device and optical anti-shake method

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 26, 2023, with application number 202310762980.8, and priority to the Chinese patent application entitled “An Optical Anti-Shake Device and Optical Anti-Shake Method”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image imaging technology, and in particular to an optical image stabilization device and an optical image stabilization method. Background Art

[0003] When taking handheld photos, a number of factors can prevent the phone from being completely stable. To improve the quality of the video, anti-shake processing is required. Common anti-shake methods include optical image stabilization (OIS) and electronic image stabilization (EIS). Electronic image stabilization (EIS) is a software-based post-processing algorithm, while optical image stabilization (OIS) corrects for shaking by causing relative displacement of the optical lens or image sensor. The phone's built-in detection device detects hand shaking, and an algorithm calculates the displacement required to offset the shake. This control then controls the motor, causing the lens or image sensor to move relative to each other, achieving anti-shake.

[0004] In optical image stabilization (OIS), a motor moves the lens or image sensor. This movement requires a relatively high current, and the average current flowing through the motor over time determines the motor's position. This current is typically provided by the phone's power management unit (PMU), with the motor driver controlling the current intensity, pattern, and drive frequency.

[0005] In the prior art, see Figure 1A, which shows the current path within the driver. The motor driver controls the motor's current intensity based on anti-shake requirements, and a device such as a MOS tube or transistor is typically located along the current path. When supplying current to the motor, see Figure 1B, which shows the equivalent current path within the driver. At this point, the device along the current path is equivalent to an on-resistance, Ron, resulting in wasted power consumption. Furthermore, the motor driver's motor is an automated closed-loop system. The higher the motor's frequency, the better the closed-loop system performance and the better the response to OIS displacement, but this also increases power consumption.

[0006] Summary of the Invention

[0007] Embodiments of the present application provide an optical image stabilization device and an optical image stabilization method, so as to reduce the power consumption of the optical image stabilization device while ensuring image quality.

[0008] In a first aspect, an embodiment of the present application provides an optical image stabilization device, which may include a motor driver and an image sensor physically connected to the motor driver; the image sensor is used to report its own working status to the motor driver; the working status includes one or more of an exposure state, a readout state, and an inter-frame idle state; the motor driver is used to monitor the working status of the image sensor and adjust its own driving state to a target driving state based on the working status; the driving state includes a driving mode and a driving frequency.

[0009] The embodiment of the present application provides an optical image stabilization device, which mainly includes the design of a structure and drive adjustment strategy for an optical device, wherein the structure includes a motor driver and an image sensor that are physically connected, wherein the image sensor directly reports its different states during operation, such as exposure, readout, or idle state between frames, to the motor driver through the physical connection, and the motor driver can adjust its own drive state based on the different working states of the image sensor, specifically including different drive modes and different drive frequencies, so as to better regulate the motor, greatly reduce power consumption and improve motor performance while ensuring imaging quality. In summary, the embodiment of the present application provides a new low-power management solution for the motor driver, which adopts an image sensor (Sensor) component to link with a motor driver (Driver), and the motor driver accurately obtains the working state of the image sensor, and accurately controls the drive mode and drive frequency of the motor according to the different working states of the image sensor, thereby reducing power consumption while ensuring shooting quality.

[0010] In one possible implementation, the driving mode includes one or more of linear driving, pulse width modulation (PWM) driving, and dynamic linear driving. In the embodiment of the present application, the driving mode may include multiple modes, such as a linear driving mode without electromagnetic radiation interference but with high power consumption, a dynamic linear driving mode without electromagnetic radiation interference but with low power consumption, and a PWM driving mode with electromagnetic radiation but with lower power consumption.

[0011] In one possible implementation, the optical image stabilization device further includes a motor; the motor driver is further used to drive the motor in the target drive state to push the image sensor or lens to perform a target displacement. Since the essence of optical image stabilization is to drive the motor through the motor driver, the motor can push the image sensor or lens to perform a target displacement to offset the shaking displacement of the human hand, thereby ultimately achieving optical image stabilization. That is, a method of correcting shaking by causing a relative displacement of the image sensor (Sensor). For example, the shaking information of the human hand is obtained through the built-in detection device of the mobile phone, and the displacement required to offset the anti-shake is calculated through an algorithm, thereby controlling the motor to push the lens or image sensor to perform a relative displacement to complete the anti-shake.

[0012] In one possible implementation, when the operating state is an exposure state, the target drive state includes a PWM drive mode and a first high-speed drive frequency. When the image sensor is in the exposure period of the imaging phase, it has high requirements for optical image stabilization but is insensitive to electromagnetic radiation. Specifically, the image sensor is sensitive to hand shake during the exposure phase, and maintaining a high drive frequency during this phase is beneficial for improving image quality. Furthermore, since the motor driver generates varying degrees of power consumption and causes varying levels of electromagnetic radiation interference under different drive modes, for example, when the motor driver uses a pulse-width modulation (PWM) drive mode, the motor driver's current output is a periodic square wave. Thus, the output current has a certain frequency, generating electromagnetic radiation that interferes with the image sensor (potentially causing image quality degradation). However, this approach can significantly reduce power consumption. Furthermore, since the electromagnetic radiation affects the image sensor's readout circuit (i.e., the image sensor in a phase not including the readout phase is not affected by electromagnetic radiation), the image sensor in the exposure-only state is insensitive to electromagnetic radiation and is therefore suitable for a PWM drive mode. Therefore, in the embodiment of the present application, when the working state of the image sensor is in the exposure state of the imaging stage, the driving mode of the motor driver can be adjusted to a pulse width modulation (PWM) driving mode (meeting low electromagnetic radiation sensitivity while reducing power consumption) and a high frequency state (meeting high anti-shake requirements) to ensure high imaging quality and low power consumption at the same time.

[0013] In one possible implementation, when the operating state is simultaneously in an exposure state and a readout state, and the target drive state includes a dynamic linear drive mode and a first drive frequency in a high-speed state, the image sensor has high requirements for optical image stabilization and is sensitive to electromagnetic radiation during the exposure and readout periods of the imaging phase. Specifically, image sensors are sensitive to hand shake during the exposure and readout phases, so maintaining a high drive frequency at this time is beneficial for improving image quality. Furthermore, because motor drivers generate varying degrees of power consumption and cause varying levels of electromagnetic radiation interference under different drive modes, for example, when the motor driver is driven in a dynamic linear drive mode, the current supplied to the motor is always the same as the average current required to maintain the motor's displacement. After optical image stabilization is enabled, the MOS transistor or transistor within the motor driver remains on, the voltage at the power supply management terminal remains constant, and the power consumption on Ron is (Vin-VO)*I. In this mode, there is no electromagnetic radiation interference with peripheral devices, and power consumption is lower than with linear drive. However, since electromagnetic radiation can affect the image sensor's readout circuit (i.e., an image sensor in a phase including readout will be affected by electromagnetic radiation), the image sensor, which is simultaneously in the exposure and readout phases, is sensitive to electromagnetic radiation and is therefore not suitable for PWM drive. Furthermore, since linear drive consumes significant power, dynamic linear drive is suitable. Therefore, in the embodiment of the present application, when the working state of the image sensor is in the exposure and readout state of the imaging stage, the driving mode of the motor driver can be adjusted to a dynamic linear driving mode (meeting high electromagnetic radiation sensitivity while reducing power consumption) and a high frequency state (meeting high anti-shake requirements) to improve imaging quality.

[0014] In one possible implementation, when the operating state is a readout state, the target drive state includes a dynamic linear drive mode and a second drive frequency in a low-speed state. When the image sensor is only in a readout period, it does not require high optical image stabilization, but is sensitive to electromagnetic radiation. Specifically, the image sensor does not require high optical image stabilization during the readout phase, so maintaining a low drive frequency at this time helps reduce power consumption. Furthermore, because the motor driver generates different levels of power consumption and causes different levels of electromagnetic radiation interference under different driving modes, for example, when the motor driver is driven in a dynamic linear drive mode, the current intensity supplied to the motor is always the same as the average current intensity required to maintain the motor's displacement. After optical image stabilization is enabled, the MOS transistor or transistor inside the motor driver remains in an on state, the voltage at the power supply management terminal remains constant, and the power consumption on Ron is (Vin-VO)*I. In this mode, there is no electromagnetic radiation interference with peripheral devices, and the power consumption is lower than that of linear drive. However, since electromagnetic radiation can affect the readout circuit of the image sensor (i.e., the image sensor in the readout phase will be affected by electromagnetic radiation), the image sensor in the exposure and readout phases is sensitive to electromagnetic radiation and is not suitable for PWM drive. Furthermore, since linear drive consumes a lot of power, dynamic linear drive is suitable. Therefore, in this embodiment of the present application, when the image sensor is operating only in the readout state, the motor driver's drive mode can be adjusted to a dynamic linear drive mode (meeting high electromagnetic radiation sensitivity while reducing power consumption) and a low-speed frequency state (meeting low anti-shake requirements while reducing power consumption), thereby simultaneously ensuring high image quality and low power consumption. Furthermore, it will be understood that in this embodiment of the present application, the first drive frequency is greater than the second drive frequency.

[0015] In one possible implementation, when the operating state is an inter-frame idle state, the target drive state includes a PWM drive mode and a second drive frequency in a low-speed state. When the image sensor is in the inter-frame idle state, it has low requirements for optical image stabilization and is insensitive to electromagnetic radiation. Specifically, the image sensor does not require high optical image stabilization during the readout phase, and maintaining a low drive frequency at this time helps reduce power consumption. Furthermore, since motor drivers generate varying degrees of power consumption and cause varying levels of electromagnetic radiation interference under different drive modes, for example, when the motor driver uses a pulse-width modulation (PWM) drive mode, the motor driver's current output is a periodic square wave. Thus, the output current has a certain frequency, generating electromagnetic radiation that interferes with the image sensor (potentially causing image quality degradation). However, this approach significantly reduces power consumption. Furthermore, since the electromagnetic radiation affects the image sensor's readout circuit (i.e., the image sensor in a state not including readout is not affected by electromagnetic radiation), the image sensor in the inter-frame idle state is insensitive to electromagnetic radiation and is therefore suitable for PWM drive. Therefore, in an embodiment of the present application, when the working state of the image sensor is in the inter-frame idle stage (i.e., the frame interval stage), the driving mode of the motor driver can be adjusted to a PWM driving mode (meeting low electromagnetic radiation sensitivity while reducing power consumption) and a low-speed frequency state (meeting low anti-shake requirements while reducing power consumption) to ensure high imaging quality and low power consumption at the same time.

[0016] In one possible implementation, the image sensor is specifically configured to: report the exposure status to the motor driver at time t2 before its own exposure start time T2; and the motor driver is specifically configured to, after receiving the exposure status reported by the image sensor at time t2, complete switching the drive state to the target drive state before the exposure time T2.

[0017] In an embodiment of the present application, the image sensor can report its own working status to the motor driver in advance through status output control, so that the motor driver can prepare for switching the driving mode and driving frequency in advance. The reason is that it takes a certain amount of time to switch from one driving state to another. In this way, the motor driver can finally complete the driving mode switching when the image sensor actually enters the corresponding working state, thereby making the driving state of the motor driver and the working state of the image sensor more accurately matched, so as to better achieve high imaging quality and low power consumption.

[0018] In one possible implementation, the image sensor is specifically configured to: report the readout status to the motor driver at time t1 before its own start of readout time T1; and the motor driver is specifically configured to, after receiving the readout status reported by the image sensor at time t1, complete switching the drive state to the target drive state before the readout time T1.

[0019] In an embodiment of the present application, the image sensor can report its own working status to the motor driver in advance through status output control, so that the motor driver can prepare for switching the driving mode and driving frequency in advance. The reason is that it takes a certain amount of time to switch from one driving state to another. In this way, the motor driver can finally complete the driving mode switching when the image sensor actually enters the corresponding working state, thereby making the driving state of the motor driver and the working state of the image sensor more accurately matched, so as to better achieve high imaging quality and low power consumption.

[0020] In a possible implementation, the image sensor is connected to the motor driver via an input / output (IO) interface; the image sensor is specifically configured to report its own working status to the motor driver via the IO interface.

[0021] In an embodiment of the present application, the image sensor and the motor driver are physically connected through an IO interface, and the image sensor can report its own working status to the motor driver in real time through the IO interface, for example, whether it is currently in an exposure state, an exposure and readout state, or an idle state between frames, so that the subsequent motor driver can adjust its own driving mode and driving frequency according to the working state of the image sensor, thereby accurately driving the motor to better achieve high imaging quality and low power consumption.

[0022] In a second aspect, embodiments of the present application provide an optical image stabilization method that can be applied to an optical image stabilization device, the optical image stabilization device including a motor driver and an image sensor physically connected to the motor driver. The method may include:

[0023] Reporting the working status of the image sensor to the motor driver via the image sensor; the working status includes one or more of an exposure state, a readout state, and an inter-frame idle state;

[0024] The motor driver monitors the working state of the image sensor and adjusts its own driving state to a target driving state based on the working state; the driving state includes a driving mode and a driving frequency.

[0025] In a possible implementation, the driving mode includes one or more of linear driving, pulse width modulation (PWM) driving, and dynamic linear driving.

[0026] In a possible implementation, the optical image stabilization device further includes a motor; and the method further includes:

[0027] The motor is driven by the motor driver in the target driving state to push the image sensor or the lens to perform target displacement.

[0028] In a possible implementation, when the working state is an exposure state, the target driving state includes a PWM driving mode and a first driving frequency of a high-speed state.

[0029] In a possible implementation, when the working state is simultaneously in an exposure state and a readout state, the target driving state includes a dynamic linear driving mode and a first driving frequency in a high-speed state.

[0030] In a possible implementation, when the working state is in a readout state, the target driving state includes a dynamic linear driving mode and a second driving frequency in a low-speed state.

[0031] In a possible implementation, when the working state is in an inter-frame idle state, the target driving state includes a PWM driving mode and a second driving frequency in a low-speed state.

[0032] In a possible implementation, reporting the working status of the image sensor to the motor driver includes: reporting the exposure status to the motor driver at time t2 before the image sensor starts exposure at time T2;

[0033] The motor driver monitors the working state of the image sensor and adjusts its own driving state to a target driving state based on the working state, including: after receiving the exposure state reported by the image sensor at time t2, the motor driver completes the change of the driving state to the target driving state before the exposure time T2.

[0034] In a possible implementation, reporting the working status of the image sensor to the motor driver includes: reporting the read status of the image sensor to the motor driver at time t1 before the image sensor starts reading the status at time T1;

[0035] The motor driver monitors the working state of the image sensor and adjusts its own driving state to a target driving state based on the working state, including: after receiving the readout state reported by the image sensor at time t1, the motor driver completes the change of the driving state to the target driving state before the readout time T1.

[0036] In a possible implementation, the image sensor is connected to the motor driver via an input / output (IO) interface; and reporting the working status of the image sensor to the motor driver includes:

[0037] The image sensor reports its own working status to the motor driver through the IO interface.

[0038] In a third aspect, an embodiment of the present application provides a camera device, which may include an optical anti-shake device, a motor, an inertial sensor IMU and a motor position detector provided by any one of the implementation methods of the first aspect above; wherein the IMU is used to detect the shake information of the device; the motor driver is used to drive the anti-shake displacement of the motor based on the shake information; and the motor position detector is used to correct the anti-shake displacement of the motor.

[0039] In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: an optical image stabilization device and a processor provided by any one of the implementation methods of the first aspect above; the processor is used to send a shooting instruction to the image sensor in the optical image stabilization device; the image sensor is used to enter a corresponding working state according to the received shooting instruction; the working state includes one or more of an exposure state, a readout state, and an inter-frame idle state.

[0040] In a possible implementation, the electronic device further includes: a memory for storing program instructions; the program instructions are executed by the processor.

[0041] In a fifth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein,

[0042] The memory is used to store program code, and the processor is used to call the program code stored in the memory to perform the following steps:

[0043] Reporting the working status of the image sensor to the motor driver via the image sensor; the working status includes one or more of an exposure state, a readout state, and an inter-frame idle state;

[0044] The motor driver monitors the working state of the image sensor and adjusts its own driving state to a target driving state based on the working state; the driving state includes a driving mode and a driving frequency.

[0045] In a sixth aspect, an embodiment of the present application provides a system on chip (SOC), which may include a motor driver and an IO interface; the motor driver is connected to the image sensor via the IO interface;

[0046] The IO interface is used for the image sensor to report its own working status to the motor driver; the working status includes one or more of an exposure state, a readout state, and an inter-frame idle state;

[0047] The motor driver is used to monitor the working state of the image sensor through the IO interface and adjust its own driving state to a target driving state based on the working state; the driving state includes a driving mode and a driving frequency.

[0048] In a possible implementation, the driving mode includes one or more of linear driving, pulse width modulation (PWM) driving, and dynamic linear driving.

[0049] In a possible implementation, the motor driver is further connected to a motor; and the motor driver is further configured to drive the motor in the target driving state to push the image sensor or lens to perform target displacement.

[0050] In a possible implementation, when the working state is an exposure state, the target driving state includes a PWM driving mode and a first driving frequency of a high-speed state.

[0051] In a possible implementation, when the working state is simultaneously in an exposure state and a readout state, the target driving state includes a dynamic linear driving mode and a first driving frequency in a high-speed state.

[0052] In a possible implementation, when the working state is in a readout state, the target driving state includes a dynamic linear driving mode and a second driving frequency in a low-speed state.

[0053] In a possible implementation, when the working state is in an inter-frame idle state, the target driving state includes a PWM driving mode and a second driving frequency in a low-speed state.

[0054] In one possible implementation, the IO interface is specifically used to: the image sensor reports the exposure status to the motor driver at time t2 before its own exposure start time T2; the motor driver is specifically used to, after receiving the exposure status reported by the image sensor through the IO interface at time t2, complete the switching of the drive state to the target drive state before the exposure time T2.

[0055] In one possible implementation, the IO interface is specifically configured to: cause the image sensor to report a readout status to the motor driver at time t1 before its own readout start time T1; and cause the motor driver to complete switching the drive state to the target drive state before the readout time T1 after receiving the readout status reported by the image sensor through the IO interface at time t1.

[0056] In a seventh aspect, the present application provides a semiconductor chip, which may include the SOC provided by any one of the implementation methods in the sixth aspect above.

[0057] In an eighth aspect, the present application provides a semiconductor chip, which may include: the SOC provided by any one of the implementation methods in the sixth aspect above, an internal memory coupled to the SOC, and an external memory.

[0058] In a ninth aspect, the present application provides an optical image stabilization system-on-chip (SOC), comprising the SOC provided by any one of the implementations of the sixth aspect, and internal and external memories coupled to the SOC. The optical image stabilization SOC may consist of a chip or may include a chip and other discrete components.

[0059] In a tenth aspect, the present application provides a chip system, which may include the optical image stabilization device provided by any one of the implementations of the first aspect. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary or relevant to the operation of the optical image stabilization device. The chip system may be composed of a chip or may include a chip and other discrete components.

[0060] In an eleventh aspect, the present application provides an optical image stabilization device having the function of implementing the optical image stabilization method provided by any one of the implementations of the second aspect. This function can be implemented via hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions.

[0061] In a twelfth aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store program code. When the program code is executed by a processor, it implements the optical image stabilization method provided by any one of the implementation methods in the above-mentioned second aspect.

[0062] In a thirteenth aspect, an embodiment of the present application provides a computer program, which includes instructions. When the instructions are executed by a processor, the processor executes the optical image stabilization method provided by any one of the implementation methods in the above-mentioned second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0064] Figure 1A shows the current path inside the driver.

[0065] Figure 1B shows the equivalent current path inside the driver.

[0066] FIG. 2A is a schematic diagram of linear voltage regulation corresponding to a linear driving method in the prior art.

[0067] FIG2B is a schematic diagram of dynamic linear voltage regulation corresponding to the dynamic linear driving method in the prior art.

[0068] FIG. 2C is a schematic diagram of a driver current output corresponding to a pulse width modulation driving method in the prior art.

[0069] FIG. 2D is a schematic diagram of another driver current output corresponding to a pulse width modulation driving method in the prior art.

[0070] FIG. 2E is a schematic diagram of driver current output corresponding to a combination of PWM and linear drive in the prior art.

[0071] FIG3A is a schematic diagram of a scenario of an anti-shake application for mobile phone photography provided by an embodiment of the present invention.

[0072] FIG3B is a schematic diagram of a drone optical image stabilization scenario provided by an embodiment of the present invention.

[0073] FIG3C is a schematic diagram of a scenario of intelligent vehicle-mounted optical image stabilization provided by an embodiment of the present invention.

[0074] FIG4A is a schematic diagram of the hardware structure of an optical image stabilization device provided in an embodiment of the present application.

[0075] FIG4B is a schematic diagram of the hardware structure of another optical image stabilization device provided in an embodiment of the present application.

[0076] FIG4C is a schematic diagram of the hardware structure of another optical image stabilization device provided in an embodiment of the present application.

[0077] FIG4D is a schematic diagram of the hardware structure of another optical image stabilization device provided in an embodiment of the present application.

[0078] FIG5A is a schematic diagram of a working range of an image sensor provided in an embodiment of the present application.

[0079] FIG5B is a schematic diagram of an image sensor status notification and switching timing provided by an embodiment of the present application.

[0080] FIG5C is a schematic diagram of a working range of a motor driver provided in an embodiment of the present application.

[0081] FIG5D is a schematic diagram of the coordinated operation of an image sensor and a motor driver provided in an embodiment of the present application.

[0082] FIG6 is a flow chart of an optical image stabilization method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] The embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0084] The terms "first," "second," "third," and "fourth," etc., in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, rather than to describe a specific order. In addition, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0085] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0086] As used in this specification, the terms "component," "module," "system," and the like are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of illustration, both an application running on a computing device and a computing device can be a component. One or more components can reside in a process and / or an execution thread, and a component can be located on a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures stored thereon. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component on a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0087] First, some terms in this application are explained to facilitate understanding by those skilled in the art.

[0088] (1) Charge-coupled device (CCD) can be called CCD image sensor, which can convert optical images into digital signals. There are many neatly arranged capacitors on the CCD that can sense light and convert images into digital signals. The function of CCD is like film, but it converts image pixels into digital signals. The tiny photosensitive substances implanted on the CCD are called pixels. The more pixels a CCD contains, the higher the image resolution it provides. CCD cameras composed of them are small in size, light in weight, unaffected by magnetic fields, and resistant to vibration and impact, and are widely used.

[0089] (2) Complementary Metal-Oxide-Semiconductor (CMOS) image sensors are a typical solid-state imaging sensor. CMOS image sensors are typically composed of a pixel array, row driver, column driver, timing control logic, analog-to-digital (AD) converter, data bus output interface, and control interface. These components are typically integrated on the same silicon chip. Their operating process can generally be divided into reset, photoelectric conversion, integration, and readout.

[0090] (3) Optical Image Stabilizer (OIS), also known as optical image stabilization, corrects "optical axis deviation" through the floating lens of the lens.

[0091] (4) Electronic Image Stabilization (EIS) refers to the use of force to increase the CCD sensitivity parameters in digital cameras while speeding up the shutter speed, analyzing the image captured on the CCD, and then using the edge image to compensate for the shakiness. Electronic image stabilization is actually a technology that compensates for shakiness by reducing image quality. This technology attempts to strike a balance between image quality and image shakiness. Compared with optical image stabilization, this technology is much cheaper (in fact, it can be achieved by only making some adjustments to the internal software of ordinary digital cameras), but the effect is also less effective.

[0092] (5) Pulse width modulation (PWM) is an analog control method that modulates the bias of the transistor base or MOS tube gate according to the corresponding load changes to achieve changes in the transistor or MOS tube conduction time, thereby changing the output of the switching regulated power supply. This method can keep the output voltage of the power supply constant when the operating conditions change. It is a very effective technology for controlling analog circuits using digital signals from a microprocessor.

[0093] (6) Exposure area (exposure) and readout area (ReadOut). The image sensor (Sensor) exposes row by row starting with the first row. After one row cycle is completed, the second row begins to be exposed. Similarly, after N-1 rows, the Nth row begins to be exposed. After the first row is exposed, the data is read out. It takes one row cycle time (including the row blanking time, i.e., H_Blank) to read out a row. After the first row is completely read out, the second row begins to be read out. Similarly, after the N-1 row is read out, the Nth row begins to be read out, and so on, until the entire image is completely read out. Among them, the exposure area (exposure) and the readout area (ReadOut) will overlap in time.

[0094] First, to facilitate understanding of the embodiments of this application, we further analyze and propose the technical problems that this application specifically aims to solve. In the prior art, there are a variety of technical solutions for managing motor driver current and drive frequency. The following are four commonly used technical solutions. Specifically:

[0095] Solution 1, linear drive:

[0096] Please refer to Figure 2A, which shows a linear voltage regulation diagram corresponding to a conventional linear drive method. The current supplied to the motor is always the same as the average current required to maintain the motor's displacement. When optical image stabilization is enabled, the MOS transistor or transistor within the driver remains on, and the voltage at the PMU remains constant. The power consumption on Ron is (Vin - V0) * I. This method eliminates electromagnetic radiation interference with surrounding devices, but consumes significant power.

[0097] Option 2, dynamic linear drive:

[0098] Please refer to Figure 2B, which shows a schematic diagram of dynamic linear voltage regulation corresponding to a dynamic linear drive method in the prior art. The current supplied to the motor is always the same as the average current required to maintain the motor's displacement. When optical image stabilization is enabled, the MOS transistor or transistor within the driver remains on. The voltage at the PMU terminal changes with the average motor current, maintaining a relatively small difference between the two voltages. This method eliminates electromagnetic radiation interference with surrounding devices and reduces power consumption compared to linear drive.

[0099] Solution 3: Pulse Width Modulation (PWM) drive:

[0100] Please refer to Figures 2C and 2D. Figure 2C is a schematic diagram of the driver current output corresponding to a pulse width modulation drive method in the prior art, and Figure 2D is a schematic diagram of another driver current output corresponding to a pulse width modulation drive method in the prior art. The current output of the driver is a periodic square wave. By changing the duty cycle or frequency of the square wave, the average current of the motor is adjusted to achieve position change. This method can significantly reduce power consumption. However, because the output current has a certain frequency, it generates electromagnetic radiation, which interferes with surrounding devices such as image sensors and causes image quality degradation.

[0101] Solution 4: Combination of PWM and linear drive:

[0102] Please refer to Figure 2E, which is a schematic diagram of the driver current output corresponding to the combination of PWM and linear drive in the prior art. Linear drive is used during periods when peripheral devices are sensitive to electromagnetic radiation, and PWM drive is used during periods when they are not. This current approach relies on sensor output frames or indirect output signals to determine or predict periods of electromagnetic radiation sensitivity, resulting in suboptimal power consumption and inability to completely avoid spatial radiation. Furthermore, the power consumption of the linear drive stage is relatively high.

[0103] In summary, the four aforementioned drive schemes each have their own drawbacks, failing to strike a balance between image quality and power consumption, leading to issues such as excessive power consumption and poor image quality. Therefore, the technical problems to be addressed by this application may include the following: providing a method for managing motor driver current and drive frequency that reduces internal power consumption while avoiding impacts on other components, such as the image sensor. This approach, in other words, reduces power consumption while ensuring image quality.

[0104] The optical image stabilization device in this application can be applied to a variety of electronic devices with optical image stabilization shooting functions, such as smartphones, smart glasses, smart wearable devices, augmented reality devices, digital cameras, video cameras, portable shooting equipment, optical equipment, drones, smart bracelets, robots, etc., which are not listed here one by one. The following examples list the scenarios in which the optical image stabilization device in this application can be used, which may include the following three scenarios.

[0105] Scenario 1: Using the optical image stabilization device in this application to achieve the anti-shake function of mobile phone photography:

[0106] Please refer to Figure 3A, which is a schematic diagram of a scenario for a mobile phone camera anti-shake application according to an embodiment of the present invention. In this application scenario, when the touch screen of a mobile phone detects a user's camera operation, the mobile phone may display the exemplary user camera interface 31 shown in Figure 3A in response to the camera operation. Please refer to Figure 3A, which illustrates exemplary user camera interfaces 31 and 32 of a "Camera" application on an electronic device such as a smartphone. User camera interface 31 may include a photo viewing control 301, a photo control 302, a camera switching control 303, and a shooting interface 304. Furthermore, user camera interface 31 may also include an optical image stabilization control 305. When a user desires to take an anti-shake photo (including taking a photo or video), they may tap on optical image stabilization control 305 (user camera interface 31) to enable the optical image stabilization function, as shown in "Optical image stabilization is on" in user camera interface 32. This means that the mobile phone can implement optical image stabilization for the user through the relevant functions of any of the optical image stabilization devices provided in this application, thereby improving the quality of the photo. Optionally, the bottom of the user photographing interface 31 or the user photographing interface 32 may further include a list of shooting mode options, for example, may include: night scene mode options, portrait mode options, photo mode options, video mode options and more shooting mode options.

[0107] Scenario 2: Using the optical image stabilization device in this application to realize the shooting function of the drone's shooting system:

[0108] Please refer to Figure 3B, which is a schematic diagram of a scenario of optical image stabilization of a drone provided by an embodiment of the present invention. The drone in this application scenario may include a fuselage, a power system, a flight control system, a link system, a mission payload, etc. Among them, the drone's shooting system can not only perform aerial photography, but also can be used for forest fire prevention, earthquake investigation, nuclear radiation detection, border patrol, emergency rescue, crop yield estimation, pipeline inspection, wildlife monitoring in protected areas, military reconnaissance, etc. However, during the aerial photography process, it is often difficult for the drone to ensure the stability of the image due to high-altitude airflow, resistance and the vibration of the aircraft itself. The drone can improve the shooting quality of the drone through the relevant functions of any one of the optical image stabilization devices provided in this application. For example, the payload of an aerial photography drone generally includes the following: a camera, a gimbal, and an image transmission system, wherein the camera can include any one of the optical image stabilization devices provided in this application to improve the shooting quality of the drone.

[0109] Scenario 3: Using the optical image stabilization device in this application to realize the shooting function of the smart car-mounted shooting system:

[0110] Please refer to Figure 3C, which is a schematic diagram of a scenario of intelligent in-vehicle optical image stabilization provided by an embodiment of the present invention. The in-vehicle intelligent system in this application scenario may include an intelligent driving system, an intelligent interconnection system, an intelligent monitoring system, etc., wherein the in-vehicle monitoring system is a system that uses technologies such as computer vision, the Internet of Things and artificial intelligence to monitor the internal and external environment of the vehicle and provide real-time alarms and data recording. It can help drivers predict potential dangers and improve driving safety. The monitoring system may include at least cameras, radars, infrared sensors, etc. For example, safety monitoring: monitor the vehicle and the surrounding environment through cameras and sensors inside and outside the vehicle, and provide instant alarms to help drivers avoid potential dangers. For another example, behavior analysis: by analyzing the movement of people and objects inside and outside the vehicle, the intelligent in-vehicle monitoring system can help identify abnormal behaviors, such as driver fatigue, distraction, drinking, etc. The cameras inside and outside the vehicle can include any one of the optical image stabilization devices provided in this application to improve the shooting quality of the in-vehicle scene.

[0111] It is understood that the application scenarios in Figures 3A, 3B, and 3C are merely exemplary implementations of the embodiments of the present invention, and the application scenarios in the embodiments of the present invention include, but are not limited to, the above application scenarios. The optical image stabilization device or method in this application can also be applied to application scenarios such as real-time shooting in augmented reality glasses, robot vision systems, or professional digital cameras. Other scenarios and examples will not be listed or elaborated one by one.

[0112] Please refer to Figure 4A, which is a schematic diagram of the hardware structure of an optical image stabilization device provided in an embodiment of the present application. The optical image stabilization device 40 can be located in any electronic device, such as a computer, a mobile phone, a tablet and other types of devices. The hardware structure of the optical image stabilization device 40 can specifically be a chip or a chipset or a circuit board equipped with a chip or a chipset. The chip or chipset or the circuit board equipped with a chip or a chipset can work under the necessary software drive. In the optical image stabilization device 40 shown in Figure 4A, it can include a motor driver 402 and an image sensor 401 physically connected to the motor driver 402; wherein,

[0113] The image sensor 401 is used to report its own working status to the motor driver 402; the working status includes one or more of the exposure state, readout state, and inter-frame idle state. In one possible implementation, please refer to Figure 4B, which is a schematic diagram of the physical connection between an image sensor and a motor driver provided in an embodiment of the present application. The image sensor 401 is connected to the motor driver 402 through an input and output IO interface; the image sensor 401 is specifically used to generate readout and exposure signals based on the status output control module, and report its own working status to the motor driver 402 through the IO interface. Among them, the image sensor 401 utilizes the photoelectric conversion function of the photoelectric device to convert the light image on the photosensitive surface into an electrical signal that is proportional to the light image. Optionally, the image sensor 401 can be a charge coupled device (CCD), that is, a CCD image sensor, or the image sensor 401 can also be a complementary metal oxide semiconductor (CMOS) phototransistor, that is, a CMOS image sensor. Exemplarily, image sensor 401 converts optical signals into electrical signals, which are then transmitted to an image signal processor (ISP) for conversion into digital image signals. The ISP then outputs the digital image signals to a digital signal processor (DSP) for processing. Ultimately, the DSP converts the digital image signals into image signals in standard RGB, YUV, or other formats. Optionally, image sensor 401 may be within a camera. For example, image sensor 401 may be part of a camera module, which may include not only image sensor 401 but also a lens, a motor, a motor position detector, and the like.

[0114] The motor driver 402 is configured to monitor the operating state reported by the image sensor 401 and adjust its own driving state to a target driving state based on the operating state; the driving state includes a driving mode and a driving frequency. In one possible implementation, the driving mode includes one or more of linear driving, pulse width modulation (PWM) driving, and dynamic linear driving. Exemplarily, the motor driver 402 is a device that controls the speed and direction of the motor. The principle is to control the speed and direction of the motor by controlling the output voltage and current. Specifically, the motor driver 402 converts the input power supply voltage into a voltage and current suitable for the motor, and drives the rotation of the motor through the power output in the control circuit.

[0115] Further, referring to FIG4C , FIG4C is a schematic diagram of the hardware structure of another optical image stabilization device provided in an embodiment of the present application. The optical image stabilization device may further include one or more of a motor 403 , an inertial sensor (IMU) 404 , and a motor position detector 405 .

[0116] Motor 403. Since the essence of optical image stabilization is ultimately to drive motor 403 through motor driver 402, motor 403 can push image sensor 401 or lens to achieve target displacement to offset the shaking displacement of the human hand, thereby achieving optical image stabilization. In other words, the image sensor 401 is relatively displaced to correct the shaking. In the optical image stabilization device provided in the embodiment of the present application, motor 403 is a component that causes the lens or image sensor 401 to displace, and can specifically be an electric motor or an engine. Exemplarily, its working principle can be that the starter rotor rotates when an energized coil is forced to rotate in a magnetic field, and the pinion on the rotor drives the engine flywheel to rotate. In one possible implementation, the optical image stabilization device also includes a motor; the motor driver is also used to drive the motor in the target drive state to push the image sensor or lens to achieve target displacement. Optionally, the motor 403 is a motor inside the camera module. Exemplarily, the motor 403 can be a voice coil motor (VCM), the main principle of which is to control the stretching position of the spring sheet by changing the DC current of the coil inside the motor in a permanent magnetic field, thereby driving the up and down movement. Alternatively, the motor 403 can also be a linear memory alloy (SMA) motor, the main principle of which is to drive the relevant components to move by the combined force generated by the power-on contraction of multiple SMA wires. Alternatively, the motor 403 can also be a piezoelectric motor, the principle of which is that under the action of an alternating electric field, piezoelectric ceramics produce movement phenomena such as expansion and contraction, and when a certain type of vibration and ultrasonic vibration is excited in the elastomer, the surface of the elastomer uses friction to push the object in contact with it to move. It can be understood that the motors in the embodiments of the present application can also be of other types, which are not listed here one by one. Setting the drive frequency refers to setting the frequency of the motor displacement, for example, setting a displacement once every 10ms, and setting the target displacement rate every 1ms.

[0117] The inertial sensor (IMU) 404 can be a detection device for measuring the three-axis angular velocity and acceleration of an object. For example, the inertial sensor 404 may include an accelerometer: detecting the acceleration signals of the three independent axes of the carrier coordinate system; a gyroscope: detecting the angular velocity signal of the carrier relative to the navigation coordinate system; a magnetometer: using algorithms such as Kalman or complementary filtering to provide the user with pitch angles, roll angles and heading angles with absolute references. In the embodiment of the present application, the IMU can sense the shaking direction and the amount of shaking of the optical anti-shake device (i.e., the shaking information of the human hand) to determine in which direction the motor is pushed and how much the displacement is. Furthermore, the inertial sensor 404 can obtain the shaking information of the human hand, and calculate the displacement required to offset the anti-shake through the algorithm, so that the motor driver 402 can control the motor 403 to push the image sensor 401 or the lens to cause relative displacement to complete the anti-shake.

[0118] The motor position detector 405 is built into the optical image stabilization device 40 (e.g., a mobile phone) to correct the motor displacement, i.e., correct the motor displacement error during the image stabilization process, thereby accurately determining the motor's image stabilization displacement. Optionally, the motor is a position detector within the camera module.

[0119] Further, referring to FIG4D , FIG4D is a schematic diagram of the hardware structure of another optical image stabilization device provided in an embodiment of the present application. The optical image stabilization device may further include one or more of a processor 400, a lens 406, a power management unit (PMU) 407, and a battery 408.

[0120] The processor 400 may include one or more processing units. For example, the processor 400 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0121] Lens 406 may be part of a camera module. An object is projected through lens 406 to generate an optical image that is then projected onto image sensor 401. Furthermore, motor driver 402 may drive motor 403, thereby enabling motor 403 to move image sensor 401 or lens 406 to offset hand movement, thereby achieving optical image stabilization.

[0122] The power management unit (PMU) 407 may include power management functions and charging management functions, that is, it integrates power management and charging management functions. The power management unit 407 can be used to connect to the battery 408, the processor 400, and the motor driver 402. The power management unit 407 receives input from the battery 408 and provides power to the processor 400, the motor driver 402, the image sensor 401, etc. In the optical image stabilization device of the embodiment of the present application, since a relatively large current is required in the process of pushing the lens 406 or the image sensor 401, and the average current flowing through the motor 403 over a period of time determines the position of the motor, the current can be provided by the power management unit 407, and the motor driver 402 controls the current intensity and mode and driving frequency. Exemplarily, the power management unit 407 can also be used to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), etc. In some other embodiments, the power management unit 407 can also be set in the processor 400.

[0123] The embodiment of the present application provides an optical image stabilization device, which mainly includes the design of a structure and drive adjustment strategy for an optical device, wherein the structure includes a motor driver and an image sensor that are physically connected, wherein the image sensor directly reports its different states during operation, such as exposure, readout, or idle state between frames, to the motor driver through the physical connection, and the motor driver can adjust its own drive state based on the different working states of the image sensor, specifically including different drive modes and different drive frequencies, so as to better regulate the motor, greatly reduce power consumption and improve motor performance while ensuring imaging quality. In summary, the embodiment of the present application provides a new low-power management solution for the motor driver, which adopts an image sensor (Sensor) component to link with a motor driver (Driver), and the motor driver accurately obtains the working state of the image sensor, and accurately controls the drive mode and drive frequency of the motor according to the different working states of the image sensor, thereby reducing power consumption while ensuring shooting quality.

[0124] It should be understood that the structures illustrated in the above embodiments do not constitute specific limitations on the optical image stabilization device 40. In other embodiments of the present application, the optical image stabilization device 40 may include more or fewer components than illustrated, or may combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0125] As shown in Figure 5A, Figure 5A is a schematic diagram of the working range of an image sensor provided in an embodiment of the present application. Specifically, the image sensor (Sensor) is divided into an exposure area (exposure), a readout area (ReadOut), and an idle area between frames in the working range, wherein the exposure area (exposure) and the readout area (ReadOut) may overlap in time. In an embodiment of the present application, when the driver Driver obtains its working status from the image sensor (Sensor), it can accurately control the driving mode and driving frequency of the motor. Based on the working range of the image sensor shown in Figure 5A, the working state of the image sensor can be divided into the following four states: only in the exposure state, in the exposure state and the readout state at the same time, only in the readout state, and in the idle state between frames. The following is an illustrative description of the four driving schemes of the motor driver in the embodiment of the present application based on the above four working states of the image sensor.

[0126] (1) Driving scheme 1, the image sensor is only in the exposure state:

[0127] When the current working state of the image sensor is in the exposure state, the motor driver needs to adjust its own driving state to the target state, and the target driving state includes a PWM driving mode and a first driving frequency in a high-speed state.

[0128] When the image sensor is in the exposure period of the imaging phase, it has high requirements for optical image stabilization, but is insensitive to electromagnetic radiation. Specifically, the image sensor is sensitive to human hand shake during the exposure phase, and maintaining a high drive frequency at this time is beneficial to improving image quality. In addition, because the motor driver generates different levels of power consumption and causes different electromagnetic radiation interference under different driving modes, for example, when the motor driver is driven in a pulse width modulation (PWM) driving mode, since the current output of the motor driver is a periodic square wave, its output current has a certain frequency and thus generates electromagnetic radiation, which will interfere with the image sensor (potentially causing image quality degradation). However, this method can significantly reduce power consumption. Since electromagnetic radiation affects the readout circuit of the image sensor (i.e., the image sensor in the phase that does not include readout is not affected by electromagnetic radiation), the above-mentioned image sensor that is only in the exposure state is insensitive to electromagnetic radiation and is suitable for the PWM driving mode. Therefore, in the embodiment of the present application, when the working state of the image sensor is in the exposure state of the imaging stage, the driving mode of the motor driver can be adjusted to a pulse width modulation (PWM) driving mode (meeting low electromagnetic radiation sensitivity while reducing power consumption) and a high frequency state (meeting high anti-shake requirements) to ensure high imaging quality and low power consumption at the same time.

[0129] In one possible implementation, the image sensor is specifically configured to: report the exposure status to the motor driver at time t2 before its own exposure start time T2; and the motor driver is specifically configured to, after receiving the exposure status reported by the image sensor at time t2, complete switching the drive state to the target drive state before the exposure time T2.

[0130] For example, see Figure 5B, which illustrates an image sensor status notification and switching timing diagram according to an embodiment of the present application. When the driver receives the sensor's operating status from its I / O interface, it determines whether to switch from linear drive to PWM, or from PWM to dynamic linear drive, and also determines whether the drive motor frequency should switch from high speed to low speed, or from low speed to high speed. Switching both the drive mode and the drive frequency requires a certain amount of time. The sensor is sensitive to electromagnetic radiation during the readout phase. If the switch is initiated after receiving the actual readout time, the sensor will be subject to electromagnetic interference during the switching time. During the exposure phase, the sensor is sensitive to hand shake and, therefore, prefers to maintain a high drive frequency. If the switch is initiated from low speed to high speed after the actual exposure time, the optical image stabilization effect will be affected during the switching time. To minimize electromagnetic radiation during the drive mode switching process and maintain optical image stabilization performance when switching the drive frequency, the sensor can output status information ahead of its actual state transition time. Because the switching times of different drivers vary, the advance time can be configured externally.

[0131] In an embodiment of the present application, the image sensor can report its own working status to the motor driver in advance through status output control, so that the motor driver can prepare for switching the driving mode and driving frequency in advance. The reason is that it takes a certain amount of time to switch from one driving state to another. In this way, the motor driver can finally complete the driving mode switching when the image sensor actually enters the corresponding working state, thereby making the driving state of the motor driver and the working state of the image sensor more accurately matched, so as to better achieve high imaging quality and low power consumption.

[0132] (2) Driving scheme 2: The image sensor is in both exposure and readout states at the same time:

[0133] When the current working state of the image sensor is simultaneously in the exposure state and the readout state, the motor driver needs to adjust its own driving state to the target state, which includes a dynamic linear driving mode and a first driving frequency in a high-speed state.

[0134] When the image sensor is in the exposure and readout period of the imaging stage, it has high requirements for optical image stabilization and is sensitive to electromagnetic radiation. Specifically, image sensors are sensitive to hand shake during the exposure and readout phases, so maintaining a high drive frequency at this time is beneficial for improving image quality. Furthermore, because motor drivers generate varying degrees of power consumption and cause varying levels of electromagnetic radiation interference under different drive modes, for example, when the motor driver is driven in a dynamic linear drive mode, the current supplied to the motor is always the same as the average current required to maintain the motor's displacement. After optical image stabilization is enabled, the MOS transistor or transistor within the motor driver remains on, the voltage at the power supply management terminal remains constant, and the power consumption on Ron is (Vin-VO)*I. In this mode, there is no electromagnetic radiation interference with peripheral devices, and power consumption is lower than with linear drive. However, since electromagnetic radiation can affect the image sensor's readout circuit (i.e., an image sensor in a phase including readout will be affected by electromagnetic radiation), the image sensor, which is simultaneously in the exposure and readout phases, is sensitive to electromagnetic radiation and is therefore not suitable for PWM drive. Furthermore, since linear drive consumes significant power, dynamic linear drive is suitable. Therefore, in the embodiment of the present application, when the working state of the image sensor is in the exposure and readout state of the imaging stage, the driving mode of the motor driver can be adjusted to a dynamic linear driving mode (meeting high electromagnetic radiation sensitivity while reducing power consumption) and a high frequency state (meeting high anti-shake requirements) to improve imaging quality.

[0135] (3) Driving scheme three, the image sensor is only in the readout state:

[0136] When the current working state of the image sensor is in the readout state, the motor driver needs to adjust its own driving state to a target state, which includes a dynamic linear driving mode and a second driving frequency in a low-speed state.

[0137] When the image sensor is only in the readout period, it does not require high optical image stabilization (OIS) but is sensitive to electromagnetic radiation. Specifically, the image sensor does not require high OIS during the readout phase, so maintaining a low drive frequency during this phase helps reduce power consumption. Furthermore, because motor drivers generate different levels of power consumption and cause different levels of electromagnetic radiation interference under different driving modes, for example, when the motor driver is driven in a dynamic linear drive mode, the current supplied to the motor is always the same as the average current intensity required to maintain the motor's displacement. When OIS is enabled, the MOS transistors or transistors within the motor driver remain on, the voltage at the power supply management terminal remains constant, and the power consumption on Ron is (Vin - VO) * I. In this mode, there is no electromagnetic radiation interference with peripheral devices, and power consumption is lower than with linear drive. However, since electromagnetic radiation can affect the image sensor's readout circuitry (i.e., an image sensor in a phase including readout is affected by electromagnetic radiation), image sensors in both exposure and readout periods are sensitive to electromagnetic radiation and are therefore not suitable for PWM drive. Furthermore, since linear drive consumes a lot of power, dynamic linear drive is suitable. Therefore, in this embodiment of the present application, when the image sensor is operating only in the readout state, the motor driver's drive mode can be adjusted to a dynamic linear drive mode (meeting high electromagnetic radiation sensitivity while reducing power consumption) and a low-speed frequency state (meeting low anti-shake requirements while reducing power consumption), thereby simultaneously ensuring high image quality and low power consumption. Furthermore, it will be understood that in this embodiment of the present application, the first drive frequency is greater than the second drive frequency.

[0138] In one possible implementation, the image sensor is specifically configured to: report a readout state to the motor driver at time t1 before the image sensor starts reading out; and the motor driver is specifically configured to, after receiving the readout state reported by the image sensor at time t1, complete switching the drive state to the target drive state before the readout time T1. In this embodiment of the present application, the image sensor can use state output control to report its operating state to the motor driver in advance, so that the motor driver can prepare to switch the drive mode and drive frequency in advance. This is because switching from one drive state to another requires a certain amount of time. In this way, the motor driver can complete the drive mode switch when the image sensor actually enters the corresponding operating state, thereby achieving a more accurate match between the motor driver's drive state and the image sensor's operating state, thereby better achieving high imaging quality and low power consumption.

[0139] (4) Driving scheme 4, the image sensor is in the idle state between frames:

[0140] When the current working state of the image sensor is in the inter-frame idle state, the motor driver needs to adjust its own driving state to a target state, which includes a PWM driving mode and a second driving frequency in a low-speed state.

[0141] When the image sensor is in the inter-frame idle phase, it does not require high optical image stabilization and is insensitive to electromagnetic radiation. Specifically, the image sensor does not require high optical image stabilization during the readout phase, and maintaining a low drive frequency at this time is beneficial for reducing power consumption. In addition, because the motor driver generates different levels of power consumption and causes different electromagnetic radiation interference under different driving modes, for example, when the motor driver is driven in a pulse width modulation (PWM) driving mode, since the motor driver's current output is a periodic square wave, its output current has a certain frequency and thus generates electromagnetic radiation, which interferes with the image sensor (potentially causing image quality degradation). However, this method can significantly reduce power consumption. Since electromagnetic radiation affects the readout circuit of the image sensor (i.e., the image sensor in the phase that does not include readout is not affected by electromagnetic radiation), the image sensor in the inter-frame idle state is insensitive to electromagnetic radiation and is suitable for PWM driving. Therefore, in an embodiment of the present application, when the working state of the image sensor is in the inter-frame idle stage (i.e., the frame interval stage), the driving mode of the motor driver can be adjusted to a PWM driving mode (meeting low electromagnetic radiation sensitivity while reducing power consumption) and a low-speed frequency state (meeting low anti-shake requirements while reducing power consumption) to ensure high imaging quality and low power consumption at the same time.

[0142] Based on the structure of the optical image stabilization device 40 provided in the above embodiment of the present application, the following exemplarily describes a specific implementation method in which the motor driver in the device 40 adjusts the driving state according to the working state fed back by the image sensor:

[0143] Please refer to Figure 5C, which is a schematic diagram of a motor driver working range provided in an embodiment of the present application. After the motor driver (Driver) obtains the actual working state of the image sensor (Sensor), it determines the driving mode and driving frequency of the motor according to different working states. In Figure 5C, the image sensor (Sensor) is divided into an exposure area (exposure), a readout area (ReadOut), and an inter-frame idle area in the working range, wherein the exposure area (exposure) and the readout area (ReadOut) may overlap in time. Optical image stabilization (OIS) sensitive means high requirements for optical image stabilization, and OIS insensitive means low requirements for optical image stabilization; Sensor insensitive means insensitive to electromagnetic radiation, and Sensor sensitive means sensitive to electromagnetic radiation. Vertical block (VB) refers to the frame interval. Motor anti-shake means that the motor is in a high-speed state and the corresponding motor driver's driving frequency is in a high-speed state; motor following means that the motor is in a low-speed state and the corresponding motor driver's driving frequency is in a high-speed state. PWM refers to pulse width modulation drive, Linear refers to dynamic linear drive, the readout frame of the readout area (ReadOut) corresponds to the dynamic linear drive mode between the start of frame (SOF) and the end of frame (EOF), while other working intervals correspond to pulse width modulation drive mode. Specifically,

[0144] (1) The image sensor is only in the exposure period, which is the imaging stage. It has high requirements for optical image stabilization (OIS), but is not sensitive to electromagnetic radiation, that is, the sensor is insensitive. At this time, pulse width modulation (PWM) can be used to drive the motor. In order to improve the imaging quality, the motor needs to turn on the anti-shake function, that is, the motor is in a high-speed state. Correspondingly, the motor drive frequency must be in a high-speed state.

[0145] (2) The image sensor device is in the exposure (Exposure) and readout (ReadOut) time periods at the same time. It is in the imaging stage, which has high requirements for optical image stabilization (OIS), that is, OIS is sensitive, and it is also in the electromagnetic radiation sensitive stage (Sensor is sensitive). At this time, the motor is driven by dynamic linear drive. In order to improve the imaging quality, the motor needs to turn on the anti-shake function, that is, the motor is in a high-speed state. Correspondingly, the motor drive frequency must be in a high-speed state.

[0146] (3) The image sensor device is only in the ReadOut time period and is not in the imaging stage. It does not have high requirements for optical image stabilization (OIS is not sensitive), but is sensitive to electromagnetic radiation (Sensor is sensitive). At this time, dynamic linear voltage regulation is used to drive the motor. In order to reduce power consumption, the motor only needs to turn on the follow-up function, that is, the motor can be in a low-speed state. Correspondingly, the motor drive frequency is in a low-speed state.

[0147] (4) The image sensor device is neither in Exposure nor ReadOut period, which is the idle area between frames of the sensor. At this time, it is not sensitive to electromagnetic radiation (Sensor is not sensitive) and is not in the imaging stage (OIS is not sensitive). The motor can be driven by PWM. The motor only needs to turn on the follow-up function, that is, the motor can be in a low-speed state. Correspondingly, the motor drive frequency is in a low-speed state.

[0148] To further reduce power consumption, the present embodiment of the present invention replaces the traditional PWM and linear voltage regulation switching scheme with a PWM and dynamic linear voltage regulation switching scheme. To further reduce power consumption, the motor drive switches to a low speed and enters a follow state from the actual readout moment, rather than entering an IDLE state at the beginning of an idle moment.

[0149] Please refer to FIG. 5D , which is a schematic diagram of the coordinated operation of an image sensor and a motor driver according to an embodiment of the present application. The following exemplarily describes a specific implementation method in which the motor driver in the device 40 adjusts the driving state according to the working state fed back by the image sensor:

[0150] 1. Measure the time t1 when the motor driver switches from PWM to dynamic linear drive, and the time t2 when the dynamic linear drive switches to PWM.

[0151] 2. t1 and t2 can be configured as advance times in the image sensor.

[0152] 3. When taking a photo, the image sensor notifies the motor driver in advance to enter the exposure and readout states, with the lead time being t2 and t1 respectively.

[0153] 4. The image sensor outputs an exposure status t2 before its actual exposure start time. The motor driver receives this status and adjusts the drive frequency from low to high. The end of the switch is synchronized with or before the actual exposure time. After receiving the exposure end status, the motor driver switches from high-speed mode to low-speed mode.

[0154] 5. The image sensor outputs a readout status t1 before its actual readout start time. The driver receives this status and switches from PWM to dynamic linear drive. The end of the switch is synchronized with or before the actual readout time to prevent electromagnetic radiation from lingering after the actual readout time. Upon receiving the readout end status, the driver switches from dynamic linear mode to PWM mode.

[0155] Compared with the prior art, in order to reduce power consumption in the readout stage, the embodiment of the present application switches from a linear drive mode to a dynamic linear drive mode.

[0156] Referring to FIG. 6 , FIG. 6 is a schematic flow chart of an optical image stabilization method provided in an embodiment of the present application. This method may be applied to the optical image stabilization device described in FIG. 4A , FIG. 4B , or FIG. 4C . The optical image stabilization device includes a motor driver and an image sensor physically connected to the motor driver. The optical image stabilization method may include method flow steps S601 to S602.

[0157] Step S601: reporting the working status of the image sensor to the motor driver via the image sensor; the working status includes one or more of an exposure state, a readout state, and an inter-frame idle state;

[0158] Step S602: monitoring the working state of the image sensor through the motor driver, and adjusting its own driving state to a target driving state based on the working state; the driving state includes a driving mode and a driving frequency.

[0159] In a possible implementation, the driving mode includes one or more of linear driving, pulse width modulation (PWM) driving, and dynamic linear driving.

[0160] In a possible implementation, the optical image stabilization device further includes a motor; and the method further includes:

[0161] The motor is driven by the motor driver in the target driving state to push the image sensor or the lens to perform target displacement.

[0162] In a possible implementation, when the working state is an exposure state, the target driving state includes a PWM driving mode and a first driving frequency of a high-speed state.

[0163] In a possible implementation, when the working state is simultaneously in an exposure state and a readout state, the target driving state includes a dynamic linear driving mode and a first driving frequency in a high-speed state.

[0164] In a possible implementation, when the working state is in a readout state, the target driving state includes a dynamic linear driving mode and a second driving frequency in a low-speed state.

[0165] In a possible implementation, when the working state is in an inter-frame idle state, the target driving state includes a PWM driving mode and a second driving frequency in a low-speed state.

[0166] In a possible implementation, reporting the working status of the image sensor to the motor driver includes: reporting the exposure status to the motor driver at time t2 before the image sensor starts exposure at time T2;

[0167] The motor driver monitors the working state of the image sensor and adjusts its own driving state to a target driving state based on the working state, including: after receiving the exposure state reported by the image sensor at time t2, the motor driver completes the change of the driving state to the target driving state before the exposure time T2.

[0168] In a possible implementation, reporting the working status of the image sensor to the motor driver includes: reporting the read status of the image sensor to the motor driver at time t1 before the image sensor starts reading the status at time T1;

[0169] The motor driver monitors the working state of the image sensor and adjusts its own driving state to a target driving state based on the working state, including: after receiving the readout state reported by the image sensor at time t1, the motor driver completes the change of the driving state to the target driving state before the readout time T1.

[0170] In a possible implementation, the image sensor is connected to the motor driver via an input / output (IO) interface; and reporting the working status of the image sensor to the motor driver includes:

[0171] The image sensor reports its own working status to the motor driver through the IO interface.

[0172] It should be noted that the various steps of the optical image stabilization method described in the embodiments of the present application can be found in the relevant descriptions of the device embodiments described in Figures 3A to 5D above, and will not be repeated here.

[0173] The embodiment of the present application further provides a system on chip (SOC), which may include a motor driver and an IO interface; the motor driver is connected to the image sensor via the IO interface;

[0174] The IO interface is used for the image sensor to report its own working status to the motor driver; the working status includes one or more of an exposure state, a readout state, and an inter-frame idle state;

[0175] The motor driver is used to monitor the working state of the image sensor through the IO interface and adjust its own driving state to a target driving state based on the working state; the driving state includes a driving mode and a driving frequency.

[0176] In one possible implementation, the present embodiment does not specifically limit the type of communication interface between the motor driver and the image sensor. Different communication interfaces can be used for connection and communication based on different application scenarios or transmission requirements. For example, the motor driver can query a register using an interface protocol to obtain the operating status of the image sensor.

[0177] In a possible implementation, the driving mode includes one or more of linear driving, pulse width modulation (PWM) driving, and dynamic linear driving.

[0178] In a possible implementation, the motor driver is further connected to a motor; and the motor driver is further configured to drive the motor in the target driving state to push the image sensor or lens to perform target displacement.

[0179] In a possible implementation, when the working state is an exposure state, the target driving state includes a PWM driving mode and a first driving frequency of a high-speed state.

[0180] In a possible implementation, when the working state is simultaneously in an exposure state and a readout state, the target driving state includes a dynamic linear driving mode and a first driving frequency in a high-speed state.

[0181] In a possible implementation, when the working state is in a readout state, the target driving state includes a dynamic linear driving mode and a second driving frequency in a low-speed state.

[0182] In a possible implementation, when the working state is in an inter-frame idle state, the target driving state includes a PWM driving mode and a second driving frequency in a low-speed state.

[0183] In one possible implementation, the IO interface is specifically used to: the image sensor reports the exposure status to the motor driver at time t2 before its own exposure start time T2; the motor driver is specifically used to, after receiving the exposure status reported by the image sensor through the IO interface at time t2, complete the switching of the drive state to the target drive state before the exposure time T2.

[0184] In one possible implementation, the IO interface is specifically configured to: cause the image sensor to report a readout status to the motor driver at time t1 before its own readout start time T1; and cause the motor driver to complete switching the drive state to the target drive state before the readout time T1 after receiving the readout status reported by the image sensor through the IO interface at time t1.

[0185] It should be noted that the functions of the SOC described in the embodiments of the present application can be found in the relevant descriptions in the device embodiments described in Figures 3A to 5D above, and will not be repeated here.

[0186] An embodiment of the present application further provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program includes part or all of the steps of any one of the above method embodiments.

[0187] An embodiment of the present application further provides a computer program, which includes instructions. When the computer program is executed by a computer, the computer can perform some or all of the steps of any one of the above method embodiments.

[0188] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0189] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0190] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0191] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, the functional units in the embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0193] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to execute all or part of the steps of the above-mentioned methods of each embodiment of the present application. Among them, the aforementioned storage medium may include: U disk, mobile hard disk, magnetic disk, optical disk, read-only memory (Read-Only Memory, abbreviated: ROM) or random access memory (Random Access Memory, abbreviated: RAM) and other media that can store program codes.

[0194] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An optical image stabilization device, characterized in that: including a motor driver, and an image sensor physically connected to the motor driver; The image sensor is used to report its own working state to the motor driver; the working state includes one or more of an exposure state, a readout state, and an inter-frame idle state; The motor driver is used to monitor the working state of the image sensor and adjust its own driving state to a target driving state based on the working state; the driving state includes a driving mode and a driving frequency.

2. The device according to claim 1, characterized in that The driving mode includes one or more of linear driving, pulse width modulation (PWM) driving, and dynamic linear driving.

3. The device according to claim 1 or 2, characterized in that: The optical image stabilization device further includes a motor; and the motor driver is further configured to drive the motor in the target driving state to drive the image sensor or the lens to cause target displacement.

4. The device according to any one of claims 1 to 3, characterized in that: When the working state is in the exposure state; the target driving state includes a PWM driving mode and a first driving frequency in a high-speed state.

5. The device according to any one of claims 1 to 3, characterized in that: When the working state is in an exposure state and a readout state at the same time; the target driving state includes a dynamic linear driving mode and a first driving frequency in a high-speed state.

6. The device according to any one of claims 1 to 3, characterized in that: When the working state is in the readout state; the target driving state includes a dynamic linear driving mode and a second driving frequency in a low-speed state.

7. The device according to any one of claims 1 to 3, characterized in that: When the working state is in an inter-frame idle state; the target driving state includes a PWM driving mode and a second driving frequency in a low-speed state.

8. The device according to any one of claims 1 to 5, characterized in that: The image sensor is specifically used for: Reporting the exposure status to the motor driver at time t2 before the exposure start time T2; The motor driver is specifically configured to complete switching the driving state to the target driving state before the exposure time T2 after receiving the exposure state reported by the image sensor at time t2.

9. The device according to any one of claims 1-3 and 5-6, characterized in that: The image sensor is specifically used for: Reporting the read status to the motor driver at time t1 before the time T1 at which the motor driver starts reading; The motor driver is specifically configured to complete switching the driving state to the target driving state before the reading time T1 after receiving the readout state reported by the image sensor at time t1.

10. The device according to any one of claims 1 to 9, characterized in that: The image sensor is connected to the motor driver via an input / output (IO) interface; The image sensor is specifically used to report its own working status to the motor driver through the IO interface.

11. An optical image stabilization method, characterized in that: The method is applied to an optical image stabilization device, wherein the optical image stabilization device comprises a motor driver and an image sensor physically connected to the motor driver; the method comprises: Reporting the working status of the image sensor to the motor driver through the image sensor; the working status includes one or more of an exposure state, a readout state, and an inter-frame idle state; The working state of the image sensor is monitored by the motor driver, and its own driving state is adjusted to a target driving state based on the working state; the driving state includes a driving mode and a driving frequency.

12. The method according to claim 11, characterized in that The driving mode includes one or more of linear driving, pulse width modulation (PWM) driving, and dynamic linear driving.

13. The method according to claim 11, characterized in that The optical image stabilization device further includes a motor; and the method further includes: The motor is driven by the motor driver in the target driving state to push the image sensor or the lens to perform target displacement.

14. The method according to any one of claims 11 to 13, characterized in that: When the working state is in the exposure state; the target driving state includes a PWM driving mode and a first driving frequency in a high-speed state.

15. The method according to any one of claims 11 to 14, characterized in that: When the working state is in an exposure state and a readout state at the same time; the target driving state includes a dynamic linear driving mode and a first driving frequency in a high-speed state.

16. The method according to any one of claims 11 to 14, characterized in that: When the working state is in the readout state; the target driving state includes a dynamic linear driving mode and a second driving frequency in a low-speed state.

17. The method according to any one of claims 11 to 14, characterized in that: When the working state is in an inter-frame idle state; the target driving state includes a PWM driving mode and a second driving frequency in a low-speed state.

18. The method according to any one of claims 11 to 15, characterized in that: The reporting of the working status of the image sensor to the motor driver includes: reporting the exposure status to the motor driver at time t2 before the image sensor starts exposure time T2; The method of monitoring the working state of the image sensor by the motor driver and adjusting its own driving state to a target driving state based on the working state includes: after receiving the exposure state reported by the image sensor at time t2, the motor driver completes switching the driving state to the target driving state before the exposure time T2.

19. The method according to any one of claims 11-13 and 15-16, characterized in that: The reporting of the working status of the image sensor to the motor driver includes: reporting the readout status to the motor driver at time t1 before the image sensor starts to read out time T1; The method of monitoring the working state of the image sensor by the motor driver and adjusting its own driving state to a target driving state based on the working state includes: after receiving the readout state reported by the image sensor at time t1, the motor driver completes changing the driving state to the target driving state before the readout time T1.

20. The method according to any one of claims 11 to 19, characterized in that: The image sensor is connected to the motor driver via an input / output (IO) interface; and the image sensor reports its own working status to the motor driver, including: The image sensor reports its own working status to the motor driver through the IO interface.

21. A camera device, characterized in that: The optical image stabilization device, a motor, an inertial sensor IMU and a motor position detector are included in any one of claims 1 to 10; wherein: The IMU is used to detect the vibration information of the device; The motor driver is used to drive the anti-shake displacement of the motor based on the shake information; The motor position detector is used to correct the anti-shake displacement of the motor.

22. An electronic device, characterized in that: include: The optical image stabilization device and processor according to any one of claims 1 to 10; The processor is used to send a shooting instruction to the image sensor in the optical image stabilization device; The image sensor is used to enter a corresponding working state according to a received shooting instruction; the working state includes one or more of an exposure state, a readout state, and an inter-frame idle state.

23. The electronic device according to claim 22, characterized in that: Also includes: The memory is used to store program instructions; the program instructions are executed by the processor.

24. An electronic device, characterized in that: include: a processor and a memory; wherein The memory is used to store program codes, and the processor is used to call the program codes stored in the memory to execute the following steps: Reporting the working status of the image sensor to the motor driver through the image sensor; the working status includes one or more of an exposure state, a readout state, and an inter-frame idle state; The working state of the image sensor is monitored by the motor driver, and its own driving state is adjusted to a target driving state based on the working state; the driving state includes a driving mode and a driving frequency.

25. A system on chip SOC, characterized in that: It includes a motor driver and an IO interface; the motor driver is connected to the image sensor through the IO interface; The IO interface is used for the image sensor to report its own working state to the motor driver; the working state includes one or more of an exposure state, a readout state, and an inter-frame idle state; The motor driver is used to monitor the working state of the image sensor through the IO interface, and adjust its own driving state to a target driving state based on the working state; the driving state includes a driving mode and a driving frequency.

26. A computer-readable storage medium, characterized in that: The computer-readable medium is used to store program codes, and when the program codes are executed by a processor, the method according to any one of claims 11 to 20 is implemented.

27. A computer program, characterized in that The computer program comprises instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 11-20.

Images