Methods for controlling the OIS system

Abstract

This invention provides a method for controlling the optical image stabilization (OIS) system of an imaging device. [Solution] A first time series sample and a second time series sample are obtained from the motion signal of the motion sensor of the imaging device. The motion signal indicates the movement of the imaging device, and the first time series spans a longer period of motion signal than the second time series. A first time series sample and a second time series sample are obtained. Based on the frequency analysis of the first time series, the current dominant frequency of the motion signal is identified. A model function based on the sine and / or cosine functions and the current dominant frequency is fitted to the second time series. To predict consecutive samples, the fitted model function is extrapolated past the last sample of the second time series. The setpoint of the OIS system is identified by identifying the setpoint based on the predicted consecutive samples.

Description

[Technical Field]

[0001] The present invention broadly relates to a method for controlling an optical image stabilization (OIS) system, an imaging device for carrying out such a method, and a computer program product. [Background technology]

[0002] Optical image stabilization (OIS) is used in cameras to mitigate the effects of camera movement (especially vibration) on captured image frames. For example, camera movement can cause image frames to become blurry, and / or, when capturing video, camera movement between video frames can result in unstable video (i.e., "inter-frame jitter").

[0003] If the camera is mounted on a support structure such as a wall, ceiling, or pole (which is common in video surveillance applications), camera vibration can be caused by shaking of the camera and / or the support structure due to collisions with other objects or exposure to other external forces such as wind. If the camera is handheld, camera vibration can be caused by the camera user's unstable hands.

[0004] The basic principle of lens-based OIS is to activate a movable lens element within the camera block to counteract camera vibrations. The goal is compensation. An OIS system can compensate for vibrations by moving a movable lens element based on detected camera movement, thereby keeping the image on the camera's image sensor stable. In sensor-based OIS, instead of moving a movable lens element, the OIS system may activate the image sensor based on detected movement. This approach is sometimes called "sensor-based image stabilization" (SIS) and is considered a type of OIS in this disclosure.

[0005] A common feature of the OIS approaches described above is the activation of physical elements with a certain inertia (e.g., lenses and sensors). Therefore, it takes some time to activate the moving elements, and thus compensation typically lags behind the motion to some extent. The amount of lag depends on specific parameters of the OIS system (e.g., the responsiveness of the OIS system and actuators, the inertia of the moving elements, the vibration frequency, etc.). As an illustrative and non-limiting example, a typical lag in a state-of-the-art OIS system can be in the range of approximately 5 to 10 milliseconds. This can translate into residual displacement of the image on the image sensor after OIS (sometimes called "OIS error"), resulting in blurring of (one or more) image frames and / or movement between (one or more) image frames. The amount of residual displacement tends to be more pronounced at higher vibration frequencies (e.g., above 5 Hz). [Overview of the project]

[0006] From the above, the object of the present invention is to provide an improved approach for controlling an optical image stabilization (OIS) system, thereby reducing residual movement and thus enabling the provision of a more effective OIS system. Further and alternative objects will be understood from the following.

[0007] According to a first aspect of the present invention, a method for controlling the OIS system of an imaging device is provided, the method comprising the following: The method involves acquiring a first time series sample and a second time series sample derived from the motion signal of the motion sensor of an imaging device, wherein the motion signal indicates the movement of the imaging device, and the first time series covers a longer period of the motion signal than the second time series. Based on the frequency analysis of the first time series, the current dominant frequency of the motion signal is identified, and Fitting a model function based on the sine and / or cosine functions and the current dominant frequency to a second time series, To predict consecutive samples, the fitted model function is extrapolated beyond the last sample of the second time series, and By identifying setpoints based on predicted consecutive samples, The goal is to identify the configuration points of the OIS system.

[0008] This allows the setpoint for controlling the position of the movable element of the OIS system to be identified from a series of samples predicted based on a first and second time series. This allows the OIS system to be controlled to compensate for the successive (i.e., next or future) motion of the imaging device (i.e., by acting on the movable element based on the setpoint). Therefore, the OIS system can be controlled to compensate for successive motion in advance, thereby reducing the delay between the compensation performed by the OIS system and the actual motion of the imaging device (e.g., camera). This makes it possible to reduce OIS errors.

[0009] Whether the imaging device is mounted to a support structure (e.g., a wall or ceiling) or used handheld, vibrations of the imaging device tend to manifest as oscillations with one or more dominant frequencies. These dominant frequencies tend to correlate with the natural frequencies of the coupling system formed by the imaging device and the support structure, or by the imaging device and the user. The model function is not static, but can be adapted to the dominant (e.g., natural) frequency / vibration of the oscillation, since this current dominant frequency is then identified from a first time series sample.

[0010] Since the first time series spans a longer period than the second time series, the first and second time series can represent the "(longer) history" and "(shorter) history" of the motion signal, respectively.

[0011] The first time series / long history can extend to a period suitable for capturing the long-term characteristics of the motion signal. Therefore, accurate frequency estimation may be possible by identifying the current dominant frequency based on the long-term history of the sample.

[0012] On the other hand, the second time series / short-term history may extend to a period suitable for capturing the short-term characteristics of the motion signal, such as transition characteristics. Therefore, predictions of consecutive samples may, to a smaller extent, take into account the long-term history of the motion signal. This can contribute to faster and more accurate predictions by giving more weight to the history of the more recent signal.

[0013] Furthermore, it has been found that by fitting a model function, particularly one based on the sine and / or cosine functions, to a second time series of data in combination with the current dominant frequency, accurate predictions of consecutive samples become possible with relatively simple and low computational complexity.

[0014] In some embodiments, the oldest (earliest) sample of the first time series is earlier than the oldest sample of the second time series, and the last sample of the first time series is simultaneous with or earlier than the last sample of the second time series. Thus, the last sample of the second time series can be at least as recent (i.e., latest) as the last sample of the first time series. The fact that the last sample of the first time series is earlier than the last sample of the second time series means that successive samples can be predicted at a higher rate than the current dominant frequency. That is, successive samples can be predicted in response to the acquisition of a new current sample without waiting for an updated identification of the current dominant frequency that takes the new current sample into account. Conversely, this means that the first time series and the current dominant frequency do not need to be updated as frequently as the second time series and the prediction of successive samples. The current dominant frequency is identified from the long-term history of the sample and further reflects the long-term characteristics of the motion signal; therefore, it is predicted that the current dominant frequency may change relatively slowly over time and thus does not need to be updated as frequently as predictions for consecutive samples.

[0015] In some embodiments, the motion signal indicates rotational motion. Therefore, the motion signal can be obtained from a sensor capable of detecting the rotational motion (e.g., as angular velocity) of an imaging device / motion sensor, such as a gyroscope. Vibration of an imaging device due to shaking tends to cause larger fluctuations in orientation / angle than in the linear position of the imaging device. Therefore, effective OIS (Optical Image Stabilization) may be possible by identifying setpoints based on samples derived from the motion signal indicating rotational motion.

[0016] In some embodiments, the motion signal represents angular velocity, and the sample is derived by integrating the motion signal. Thus, the sample can define an "orientation sample" that indicates the instantaneous orientation of the imaging device. Therefore, prediction of successive samples (successive orientation samples) can be performed within the angular domain.

[0017] In some embodiments, the motion sensor includes a gyroscope. The gyroscope can provide a motion signal that indicates rotational motion with relatively little noise.

[0018] In some embodiments, samples of a first time series are acquired at a first sampling rate, and samples of a second time series are acquired at a second sampling rate. The first and second sampling rates may be equal or different.

[0019] In some embodiments, the first sampling rate of the first time series is lower than the second sampling rate of the second time series. This allows predictions and setpoints to be based on a set of samples updated more frequently than the current dominant frequency. This can further contribute to the speed and accuracy of predictions. Furthermore, it may be possible for consecutive samples to be predicted at a higher rate than the current dominant frequency is identified / updated. Moreover, since the current dominant frequency reflects the long-term history of the motion signal, it can also be reliably identified from a set of samples that are updated less frequently.

[0020] In some embodiments, fitting a model function to a second time series involves using orthogonal-triangular matrix decomposition to identify a least-squares fit of the model function to the second time series, and identifying a least-squares fit involves selecting a pair of orthogonal and triangular matrices from a plurality of such pairs. Each pair is pre-calculated for the model function based on sine and / or cosine functions and their respective frequencies, and the selected pair of orthogonal and triangular matrices is a pair from a plurality of orthogonal and triangular matrices calculated for the frequencies closest to the identified current dominant frequency.

[0021] Therefore, instead of performing least-squares fitting in real time for each prediction of consecutive samples, the method may select a pair of orthogonal and triangular matrices from a plurality of pre-calculated pairs of matrices. Each pre-calculated pair of matrices is pre-calculated for each frequency.

[0022] In some embodiments, the matrix decomposition is QR, RQ, QL, or LQ decomposition.

[0023] In some embodiments, the method further, Sequentially updating the second time series with new samples derived from the motion signal, and This includes sequentially identifying updated configuration points in the OIS system, where each updated configuration point is: Fitting a model function based on the sine and / or cosine functions and the current dominant frequency to the last updated second time series, To predict updated consecutive samples, extrapolate the fitted model function beyond the last sample or the last updated second time series, and By identifying setpoints based on predicted, updated, consecutive samples, Identified.

[0024] Therefore, updated settings for the OIS system can be identified sequentially in conjunction with the acquisition of new samples.

[0025] As described above, the rate at which the current dominant frequency and the predictions for consecutive samples are updated may differ. Therefore, once each updated setpoint is identified, the “current dominant frequency” on which the model function is based may be the same current dominant frequency used to identify one or more preceding updated setpoints, or an updated current dominant frequency, i.e., the current dominant frequency updated since the identification of the most recent updated setpoint. In other words, identifying each updated setpoint may involve fitting a model function based on sine and / or cosine functions and the last (i.e., most recently) identified current dominant frequency to the last updated second time series.

[0026] In some embodiments, the method further, The first time series is sequentially updated with new samples derived from the motion signal, and This further includes sequentially identifying the updated current dominant frequency based on the last updated sample of the first time series, In that case, each model function used to predict each updated future sample is based on the last updated current dominant frequency.

[0027] Therefore, the updated current dominant frequency used by the OIS system to identify the updated setpoint can be identified sequentially. The updated current dominant frequency can be identified at a rate lower than or equal to the rate at which the first time series is updated. Typically, the updated current dominant frequency can be identified at a rate lower than the rate at which the updated setpoint is identified.

[0028] In some embodiments, the method further includes identifying variations in the identified current dominant frequency sequence (i.e., identified as described above). In this case, the length of the subsequently updated second time series is based on that variation, thereby decreasing in response to higher variations rather than lower variations.

[0029] This allows the model function used to subsequently identify the updated setpoints to be fitted to fewer samples in response to higher fluctuations rather than lower fluctuations. Therefore, when the dominant vibration frequency in the motion signal is stable, a longer first time series can be used to improve the reliability of the prediction. Conversely, when the dominant vibration frequency in the motion signal is not very stable, a shorter second time series can be used to improve the reliability of the prediction.

[0030] In some embodiments, the first and second time series are updated at the first and second sampling rates, respectively. In this case, the first sampling rate is less than or equal to the second sampling rate. Typically, the setpoints to be updated may be specified at the same rate as the second time series is updated, i.e., the same rate as the second sampling rate.

[0031] In some embodiments, identifying the current dominant frequency includes the following: Based on a frequency analysis of a sample of the first time series, determine whether the first time series contains any frequency components with amplitudes exceeding an amplitude threshold. In response to identifying at least one frequency component having an amplitude exceeding a frequency threshold, the current dominant frequency is set to a frequency among the dominant frequency components of the at least one identified frequency component. This involves setting the current dominant frequency to a predetermined default frequency, in accordance with the fact that no frequency components with amplitudes exceeding the amplitude threshold were identified.

[0032] Therefore, if a dominant frequency cannot be reliably detected within the first time series sample, the "current dominant frequency" may be set to a default value of a given frequency.

[0033] In some embodiments, the OIS system is configured to compensate for vibrations of the imaging device by controlling the position of the movable elements of the OIS system based on identified setpoints (each setpoint being updated sequentially).

[0034] Therefore, the setpoint can be provided to the OIS system as a control signal. In that case, the OIS system can actuate the movable element according to the setpoint.

[0035] Identifying setpoints in an OIS system may involve applying transformations to successive samples and mapping them to setpoints representing the positions of the moving elements. Thus, predictions may be performed in a first coordinate system within the domain of motion signals (e.g., the angular domain) and mapped to a second coordinate system representing the positions of the moving elements.

[0036] According to a second embodiment, an imaging device is provided that includes the following: A motion sensor configured to output a motion signal indicating the instantaneous movement of an imaging device. An OIS system configured to compensate for the movement of the imaging device, and A processing device configured to perform the method of the first embodiment or any multiple embodiments thereof.

[0037] According to a third aspect, a non-temporary computer program product is provided which includes a portion of computer program code. The portion of computer program code is configured, when executed by a processing device, to perform the method of the first aspect or any multiple embodiments thereof.

[0038] In general, any embodiments, features, effects, or advantages described in relation to the first aspect apply similarly to the second and third aspects.

[0039] The above, as well as further objects, aspects, embodiments, features, and effects of this disclosure, can be better understood through the following illustrative and non-limiting detailed description with reference to the accompanying drawings. In the drawings, unless otherwise noted, similar elements are given the same reference numerals. [Brief explanation of the drawing]

[0040] [Figure 1] This is a block diagram of an imaging device according to one embodiment. [Figure 2] This is a block diagram of an imaging device according to another embodiment. [Figure 3] A block diagram of an OIS system according to one embodiment is shown in more detail. [Figure 4] This is a flowchart of the method for controlling the OIS system. [Figure 5] This flowchart shows the steps to identify the settings of the OIS system. [Figure 6] This is an illustrative diagram showing a sample of the orientation of the first time series. [Figure 7] This is one example of the frequency-converted domain representation of the first time-series orientation sample in Figure 6. [Figure 8] This is an illustrative diagram showing the orientation sample of the second time series and the prediction of consecutive orientation samples. [Figure 9] Figure 3 is a block diagram of a further embodiment of the prediction system. [Figure 10] Figure 3 is a block diagram of a further embodiment of the prediction system. [Figure 11] This flowchart shows the steps to identify the currently dominant frequency. [Modes for carrying out the invention]

[0041] Figure 1 is a schematic block diagram of the imaging device 100. The imaging device 100 may be a camera capable of capturing still images and / or video. For example, a useful application of the image stabilization approach of this disclosure is an imaging device 100 taking the form of a monitoring or surveillance camera with video capture capabilities, such as a network surveillance camera (e.g., an Internet Protocol (IP) camera). Thus, the imaging device 100 can be adapted for fixed installation by being mounted on a support structure such as a building structure (e.g., a wall, ceiling, roof, lighting pole, mast, etc.) or other suitable structure for monitoring a site. However, the image stabilization approach of this disclosure is also applicable to imaging devices suitable for gripping or body-worn imaging and / or mounting on a camera tripod. For the sake of brevity and without loss of generality, the imaging device 100 may be referred to as camera 100 below.

[0042] Camera 100 comprises an optical system 114 and an image sensor 122. The optical system 114 comprises a system of optical elements such as one or more lenses 116, 118, 120. Some of the optical elements shown in Figure 1 are merely non-limiting examples, and fewer or more lenses and / or other optical elements are also possible. During the imaging process, camera 100 can monitor the field by providing a sequence of field images or frames F by using the image sensor 122 to capture images or frames F formed on the image sensor 122 by the optical system 114. In the case of a video capture process, the image frames F may be captured at a predetermined or variable frame rate suitable for a given monitoring application. The sequence of image frames F may, in particular, form a sequence of image frames for a video sequence of the field. The captured image frames may be provided to a downstream image processing pipeline that undergoes typical image processing steps such as demosaicing and encoding before transmission and / or storage. Each of the post-processing steps may be of a type known in the art and are therefore not described further herein.

[0043] As described above, camera 100 movements, such as vibrations caused by camera shaking during the imaging process, can impair the quality of not only individual image frames but also sequences of image frames in a video. To compensate for such camera movements, camera 100 implements optical image stabilization (OIS) as presented below.

[0044] Camera 100 includes a motion sensor 102 configured to output a motion signal m. The motion signal m indicates instantaneous movement of the motion sensor 102 (and therefore of camera 100). Camera 100 includes an OIS system 104 configured to compensate for the movement of camera 100 based on the motion signal m. The OIS system 104 includes a prediction system 106, an OIS controller 108, a driver 110, and a movable element. In the illustrated embodiment, the OIS system 104 is configured for lens-based OIS. In this case, the movable element is a movable optical element of an optical system 114, exemplified herein by a lens 118 which is a movable lens. As can be understood, the movable element may also be formed by a group of movable lenses of the optical system 114, or any other optical element. The prediction system 106 is configured to identify a control signal c which takes the form of a setpoint of the OIS controller 108. The prediction system 106 is described in more detail below. The OIS controller 108 is configured to control the position of the movable lens 118 according to a control signal / setpoint c. The OIS controller 108 is configured to generate an actuation signal u to the driver 110 to actuate the movable lens 118 based on the setpoint c. The driver 110 is appropriately configured to actuate the movable lens 118 according to the actuation signal u, thereby compensating for the vibration motion of the camera 100.

[0045] The driver 110 may include, for example, one or more voice coil motor (VCM) actuators, or other suitable conventional high-speed actuators, such as comb-type drive units or piezo actuators. The OIS system 104 could typically compensate for movement along a set of axes of compensation axes, such as two or more. The driver 110 may appropriately include, for each compensation axis of the OIS system 104, a respective actuator (e.g., a VCM) for operating the movable lens 118 to provide compensation along the compensation axis. Thus, the driver 110 may include actuators (e.g., VCMs) for shifting the position of the movable lens 118. In this specification, position may refer to location (i.e., linear position) and / or rotation (i.e., angle / tilt of the lens / multiple lenses). For example, the driver 110 may include actuators for translating the movable lens 118 in a plane perpendicular to the optical axes of the optical system 114. The driver 110 may further or alternatively include actuators for rotating the movable lens 118 with respect to the optical axes. For example, the OIS system 104 may be configured to move the movable lens 118 along two lateral directions in a plane. This allows the OIS system 104 to compensate for changes in the pitch and yaw (as defined below) of the camera 100. Other approaches to controlling the position of the movable lens 118 are also possible; for example, the movable lens 118 can be moved along a curved path (e.g., a parabola) to simultaneously change the location and angle of the movable lens 118. However, these are just examples, and other approaches to operating the movable lens or other movable optical elements are possible.

[0046] The motion sensor 102 may be any type of sensor capable of sensing (for example, around or along) at least one sensing axis, and may output a motion signal m indicating the motion detected for each sensing axis. The motion sensor 102 may be configured to detect motion along each of the set of compensation axes of the OIS system 104. Conveniently, the motion sensor 102 may include a corresponding set of sensing axes, and the set of sensing axes may be arranged to align with the set of compensation axes. Thus, the motion sensor 102 may output a motion signal m indicating the instantaneous value of each motion component corresponding to each compensation axis.

[0047] The motion sensor 102 may be configured to detect rotational and / or linear motion and to output a motion signal m indicating the detected rotation and / or linear motion. The motion sensor 102 may include one or more gyroscopes, one or more accelerometers, or other suitable types of inertial measuring units (IMUs). As used herein, the terms “gyroscope” and “accelerometer” may refer to gyroscopes and accelerometers having one or more sensing axes. For example, a “single” gyroscope or accelerometer may, at the physical / hardware level, include several separate gyroscope or accelerometer sensors, each configured to detect motion along its respective sensing axis. Thus, a “two-axis gyroscope” may actually include two separate gyroscope sensors, each configured to detect angular velocity around its respective axis (e.g., pitch and yaw). Similarly, a “three-axis accelerometer” may include three separate accelerometer sensors, each configured to detect acceleration along its respective axis (e.g., three orthogonal axes with fixed orientations relative to the camera 100). When two or more sensors and / or types of detection techniques are used, data fusion can be used to combine the individual motion signals from each sensor into a motion signal m that indicates the movement of one or more detection axes of the motion sensor 102.

[0048] For example, motion sensor 102 may be configured to detect rotational motion as the angular velocity (i.e., the rate of change in orientation / rotation) of camera 100 / motion sensor 102 and output a corresponding motion signal m indicating the detected angular velocity. Motion sensor 102 may be configured to detect angular velocity along one or more axes, e.g., pitch, yaw, and / or roll. Hereinafter, pitch may refer to the pitch angle of the optical axis (i.e., line of sight) of camera 100 in the vertical plane. Yaw may refer to the yaw angle of the optical axis of camera 100 in the horizontal plane. Hereinafter, roll may refer to the roll angle of camera 100 around the optical axis of camera 100. Angular velocity may be conveniently detected using a gyroscope. For example, a two-axis gyroscope may be configured to detect the angular velocity of the pitch and yaw angles of camera 100. A three-axis gyroscope may be configured to detect the angular velocity of the pitch, yaw, and roll angles of camera 100. Rotational motion can also be detected using a pair of sensing axes of a two-axis (or more) accelerometer. The acceleration detected along the pair of sensing axes can be merged into a scalar value representing the angular velocity around an axis orthogonal to the pair of sensing axes (e.g., integrated and transformed by trigonometric transformation). This transformation can be performed by the on-sensor calculation block of the motion sensor 102 or the off-sensor calculation block of the camera 100. More generally, any sensor configuration that enables the detection of rotational motion (e.g., a gyroscope and / or accelerometer) can be used. For example, the motion sensor 102, which combines a gyroscope and an accelerometer, may use the gyroscope to detect rotational motion around a first sensing axis and the accelerometer to detect rotational motion around a second sensing axis.

[0049] As described above, vibrational motion tends to cause greater fluctuations in rotation than in linear translation of the camera 100. Therefore, having a motion sensor 102 configured to detect at least rotational motion enables more sensitive detection of vibrational motion, and thus more effective image stabilization. Accordingly, the following describes several embodiments of an OIS system that compensates for motion based on a motion signal indicating rotational motion (e.g., angular velocity). However, the following description can also be applied in a manner corresponding to several embodiments of an OIS system that compensates for motion based on a motion signal indicating linear translation along one or more axes (e.g., as linear velocity or linear acceleration).

[0050] Regardless of the specific embodiment of the motion sensor 102, the motion sensor 102 may be configured to output a motion signal m as either a digital motion signal or an analog motion signal. If the motion sensor 102 outputs an analog motion signal m, it may be sampled by an analog-to-digital converter (ADC) in camera 100, which is located upstream of the OIS system 104 and connected to the analog output of the motion sensor 102. Thus, the analog motion signal can be AD-converted (e.g., for each component) into a digital signal containing a time series of operating samples provided as input to the OIS system 104. If the motion sensor 102 outputs a digital motion signal, the motion sensor 102 may include an internal ADC and therefore perform an internal AD-conversion of the analog motion signal before it is output via the digital output of the motion sensor 102. Thus, the motion signal m may be output as a digital signal containing a time series of operating samples provided as input to the OIS system 104 (e.g., for each component).

[0051] In the illustrated embodiment, the prediction system 106 is shown to receive a motion signal m directly from the motion sensor 102. However, the motion signal m may typically undergo A / D conversion (if the motion sensor 102 has an analog output) and / or filtering (for example, by a filtering step including integration and / or a low-pass filter of the motion signal m) before being received by the prediction system 106.

[0052] In the illustrated embodiment, the OIS controller 108 of the OIS system 104 is implemented as a closed-loop controller. Thus, the OIS system 104 further comprises a position sensor 112 configured to detect the instantaneous position of the movable lens 118 (e.g., the linear position and / or angle / tilt of the lens / multiple lenses) and provide a corresponding position signal v as a feedback signal to the OIS controller 108. Thus, the position sensor 112 may be located in the feedback path of the OIS controller 108 and configured to output a feedback signal indicating the position of the movable lens 118. The position sensor 112 may include, for example, a Hall effect sensor, e.g., one for measuring the position of the movable lens 118 along each respective compensation axis. However, several other embodiments of the position sensor 112 may also be optical sensors or giant magnetoresistance (GMR) effect sensors, etc.

[0053] Figure 2 is a block diagram of an alternative embodiment of the imaging device 100, which includes a sensor-based OIS system 104' instead of the lens-based OIS system 104 as in Figure 1. Thus, the OIS system 104' of Figure 2 is configured to provide OIS by controlling the position of the image sensor 122. The driver 110 of the OIS system 104' may be implemented using a set of actuators, such as a VCM actuator, to control the position of the image sensor 122 in the imaging plane and / or the tilt angle of the image sensor 122, similar to the driver 110 of the OIS system 104 in Figure 1. The OIS system 104' may further implement closed-loop control and include a position sensor 112 (realized, for example, by a Hall sensor and / or an optical sensor) which may provide a position signal v indicating the instantaneous position of the image sensor 122 as a feedback signal to the OIS controller 108. The description of Figure 1 applies similarly to Figure 2, and to avoid excessive repetition, please refer to the above for the description of the elements with corresponding numbers.

[0054] Although several alternative embodiments have been presented herein for simplification, it is also possible to implement OIS using a combination of lens-based and sensor-based OIS. For example, a movable optical element (e.g., corresponding to the movable lens 118 in Figure 1) and an image sensor 122 may be located within a common camera module. In that case, OIS can be achieved by controlling the position and / or angle of the camera module (i.e., as a single unit).

[0055] Figure 3 provides a more detailed block diagram of one embodiment of the OIS system 104. While Figure 3 shows the lens-based OIS system 104 corresponding to Figure 1, the description also applies to the image sensor-based OIS shown in Figure 2, and to combinations of lens-based and sensor-based OIS.

[0056] Without loss of generality, the OIS system 104 in Figure 3 will be described with reference to a motion sensor 202 implemented by a gyroscope. Furthermore, a single detection and compensation axis (e.g., pitch or yaw) will be mentioned. Thus, for the purposes of the following description, it is assumed that the motion sensor 202 outputs a motion signal ω indicating the angular velocity of the change in orientation of the motion sensor 202 around its detection axis (e.g., the rate of change in pitch or yaw).

[0057] The OIS system 104 includes a closed-loop control system 221 comprising an OIS controller 108. The OIS controller 108 can be implemented by a PID controller. In principle, a simpler embodiment of the OIS controller 108 is also possible, such as a PI controller. However, considering the fast response typically required for effective OIS, it is usually beneficial to use each of the P-, I-, and D- components.

[0058] Block 224 represents the controlled system of the control system 221, and, referring to Figure 1, may represent the movable lens 118 and the driver 110 that acts on the movable lens 118. In the case of an image sensor-based OIS, such as the OIS system 104' in Figure 2, block 224 may instead represent the image sensor 122 and the driver 110 that acts on the image sensor 122. The control parameter (i.e., process variable) of the control system 221 is the position of the movable element, denoted as s. This position is measured by the position sensor 110 (e.g., a Hall sensor) as described above and provided as a feedback signal v. The feedback signal v is subtracted from a setpoint c received from the prediction system 106 to generate an error signal e for the OIS controller 108.

[0059] The steps performed by the prediction system 106 to generate a setpoint (i.e., a given moment in time) are summarized in the flowcharts in Figures 4 and 5.

[0060] In step S301 of method 300 in FIG. 4, the prediction system 106 obtains samples θ of a first time series and a second time series derived from the motion signal ω of the motion sensor 202. The first time series and the second time series are, respectively, hereinafter θ L and θ S as denoted. The first time series θ L is stored in a first buffer 210 of the prediction system 106, hereinafter referred to as the "long-term history buffer 210" in this specification. The second time series θ S is stored in a second buffer 214 of the prediction system 106, hereinafter referred to as the "short-term history buffer 214" in this specification. If necessary to simplify the explanation, the motion sample ω obtained at a given time instant t = t i can be denoted hereinafter as ω(t i ). In this specification, the term i is an integer index for a given time instant. Thereby, t i = i * Δ t where Δ t is the sampling interval of the motion signal / motion sample ω, and t0 is an arbitrary reference point in time. Correspondingly, the orientation sample derived from the motion sample ω(t i ) obtained at a given time instant t = t i can be denoted as θ(t i ). Further, the first time series and the second time series at a given time instant t = t i can be denoted as θ L (t i ) and θ S (t i ) respectively. That is, at a given time instant t = t i respectively, θ L (t i ) indicates the first time series stored in the long-term history buffer 210, and θ S (t i) represents a second time series stored in the short-term history buffer 214. In this disclosure, the term “sampling interval” (interchangeably “sampling period”) is used in the ordinary sense of the word to refer to the time interval or period (i.e., the reciprocal of the sampling rate) between sampling moments. For ease of explanation, the first time series θ is used below. L and the second time series θ S Furthermore, assuming that the sampling rates of the motion signals ω are equal, and thereby using this notation, θ L , θ S It is assumed that this can refer to each time instant and each sample of ω. However, different sampling rates are also possible, as will be explained further below.

[0061] First time series θ L and the second time series θ S The sample θ is derived by passing an angular velocity sample of the motion signal ω through a filtering step 204, according to the illustrated embodiment. The filtering step 204 includes an integrator 206 that integrates the angular velocity sample over time to generate a time-series orientation sample θ. The orientation sample θ may also be called an angle sample. To reduce the sensitivity of the motion signal ω to noise, the integrator 206 may be a leakage integrator. Thus, t=t i+1 =t i +Δ t The updated orientation / angle sample θ(t) in can be calculated according to the following formula: θ(t i +Δ t )=C*(θ(t i )+ω(t i +Δ t )*Δ t ) Here, Δ tω is the sampling interval of the motion signal, and C is the "leakage" integral. The integral C can be set to a value in the range of 0.99 to 0.9999, for example, as a non-limiting example. The specific value may be a design choice made considering factors such as the amount of noise in the motion signal ω and the desired responsiveness of the OIS system 104. The filtering stage 204 may further include a low-pass filter 208, as shown in the figure. The low-pass filter 208 is shown herein as a post-processing step of the integral 206, but the low-pass filter may be performed alternatively or even before the integral 206. In any case, the low-pass filter 208 may further suppress noise and thus reduce noise sensitivity.

[0062] Figures 6 and 8 show, respectively, a given time instant t=t i The first time series sample θ can be stored in the long-term history buffer 210 and the short-term history buffer 214, respectively. L and the second time series sample θ S This shows a schematic representation of the first time series θ. L The length of the long-term history buffer 210 is T samples. The second time series θ S The length of the short-term history buffer 214 is W samples. The first time series θ L This is the second time series θ S This extends over a longer period for the motion signal ω and sample θ. Therefore, the first time series θ L However, while it forms a sample θ of long-term history, it also forms a second time series θ S This forms the sample θ of the short-term history.

[0063] First time series θ L This can extend for a period of at least 0.5 seconds of motion signal ω, for example, about 1-2 seconds. The first time series θ L This can therefore extend over a suitable period of motion signal ω to allow for a reliable estimation of its current dominant frequency (described below). For example, assuming a motion signal ω frequency of at least a few Hz, the first time series θ LTherefore, to facilitate accurate frequency estimation, the second time series θ can be several (typically multiple) periods of the motion signal ω. S Depending on the expected range of the compensated operating frequency and / or the amount of memory available in the short-term history buffer 214, this can extend to a shorter duration of the motion signal ω, for example, 100 milliseconds or less, 50 milliseconds or less, 25 milliseconds or less, or even 10 milliseconds or less.

[0064] In the assumed frequency range of the motion signal ω, the second time series θ S However, this can therefore extend to a smaller number of periods of the motion signal ω, for example, a single period, or just a fraction of a period. In Figure 8, the second time series θ S However, for example, it extends to about half the period of the motion signal ω (which has the same fundamental frequency as the corresponding time series orientation sample θ). S The advantage is that the sampling rate can be relatively high for the latest portion of the motion signal ω, thus facilitating accurate regression and prediction (described below) while limiting the total amount of data to process and store. A relatively short second time series θ S A further advantage is that regression and prediction can respond more quickly to transient changes in the motion signal ω.

[0065] Figures 6 and 8 show the first time series θ L and the second time series θ S However, it is indicated that sampling is performed at the same sampling rate. However, the sample θ of the first time series L Generally, the first sampling rate is used, and the second time series sample θ is used. S This can be acquired at a second sampling rate. The first and second sampling rates can be, for example, in the range of 0.1 to 10 kHz, respectively. This allows the first time series θ L and the second time series θ SThese allow for the capture of long-term and short-term characteristics of the motion signal ω caused by vibration frequencies that can occur across a wide range of typical imaging scenarios. The first and second sampling rates may be equal or different. The first sampling rate may be lower than the second sampling rate. As described above and as can also be understood under the further description below, the regression and prediction of consecutive samples performed by the prediction system 106 may be based on a set of samples that are updated more frequently than the current dominant frequency estimate. The advantage of using a lower sampling rate for the first time series is that fewer samples can be used to extend to longer periods, thus reducing the amount of data that needs to be stored and processed. For example, the first and second sampling rates may correspond to lower and higher frequencies within the frequency range described above, respectively. In one embodiment, the first sampling rate may be in the range of 0.2 to 1 kHz, while the second sampling rate may be in the range of 2 to 8 kHz. In both cases, the first and second sampling rates can be lower than the sampling rate of the motion signal ω. For example, the first and second sampling rates can each be a part (e.g., a fraction) of the sampling rate of the motion signal ω.

[0066] As shown in Figures 6 and 8, the first time series θ L The oldest (earliest) sample θ L (t i-T+1 ) is given time moment t i In this case, the second time series θ S The oldest sample θ S (t i-W+1 This is faster than the first time series θ. L (t i ) and the second time series θ S (t i Each of the last samples of ) is a simultaneous sample, i.e., the same sample ω of the motion signal.i can be derived from. However, in a plurality of other scenarios, the last sample of the first time series can also be earlier than the last sample of the second time series. In particular, this is when the first sampling rate of the first time series θ L is lower than the second sampling rate of the second time series θ S .

[0067] Each of the long-term history buffer 210 and the short-term history buffer 214 can be implemented as a respective first-in first-out (FIFO) buffer. Thus, assuming that the long-term history buffer 210 is filled with a sample θ of the first time series at time instant t i-1 at L (t i-1 ), when a new sample θ(t i ) is obtained at time instant t i , the first time series θ L (t i-1 ) can be updated with the new sample θ by discarding the oldest (first) sample θ of the first time series θ L (t i-1 ), and the new sample θ can be added as the latest (last) sample θ to the remaining samples of the first time series θ i (t L ) to form the updated first time series θ i (t L ). This is applied in correspondence with the update of the samples of the second time series θ i-1 stored in the short-term history buffer 214. S

[0068] Referring back to FIG. 4, the prediction system 106 then proceeds to step S302, where the dominant current frequency f = f(t i ) of the motion signal ω is determined based on the frequency analysis of the first time series θ L (t i ) stored in the long-term history buffer 210 at time instant t i) is identified. Frequency analysis involves applying a frequency domain transform, such as a Fast Fourier Transform (FFT) or other discrete-time Fourier transform or frequency transform, to a first time series θ L The current dominant frequency f can be identified as the highest amplitude component in the frequency domain representation (i.e., in the spectrum). An amplitude threshold can be used to filter out components so as to limit the number of amplitude components to be analyzed to those with amplitudes exceeding the amplitude threshold. This may typically correspond to the fundamental frequency of the motion signal ω. An example of the spectrum thus obtained is shown in FIG. 7 together with the display of the peak corresponding to the current dominant frequency f. It is advantageous that the identification can be conditioned on a predetermined frequency range. That is, the current dominant frequency f can be identified as the highest amplitude component within a predetermined frequency range of the spectrum. Thereby, for example, based on prior knowledge, peaks in amplitude at frequencies that are not expected to be associated with the motion compensated by the OIS can be ignored in the analysis. For example, the frequency range can be limited to frequencies of 50 Hz or less, such as 30 Hz or less. The frequency range may also include a lower limit value (e.g., 0.1 to 1 Hz) for excluding DC components and low-speed fluctuation components that do not correspond to the operating frequencies to be compensated. Once identified, the current dominant frequency f can be stored in an output buffer (not shown) of the frequency analysis block 212 for use as an input to the prediction block 216 further described below. The first time series θ L is sequentially updated with new samples θ(t i+1 ), θ(t i+2 ), etc., so the frequency analysis block 212 can sequentially identify the corresponding updated current dominant frequencies f(t i+1 ), f(t i+2 ), etc., and store them in its output buffer as the most recently updated current dominant frequency f.

[0069] In step S303, the prediction system 106 identifies a setpoint (e.g., a new / updated / next setpoint) that will move forward and form a control signal c for the OIS system 104. Identifying the setpoint c involves several substeps, which will be further illustrated with reference to the flowchart in Figure 5.

[0070] The prediction system 106 includes a prediction block 216. The prediction block 216 performs a regression step S304 and a prediction step S305. In step S304, the prediction block 216 uses the last updated second time series θ S (For example, the second time series θ currently stored in the short-term history buffer 214) S The model function y=y(t) is fitted to the second time series θ. In step S305, the prediction block 216 performs the fitting of the model function y=y(t) to the second time series θ. S The consecutive samples θ' (for example, the last sample θ(t i To predict future or subsequent samples following the first time series, we extrapolate the fitted model function y past the last sample of the second time series.

[0071] The regression and prediction steps are schematically shown in Figure 8. Therefore, as shown in the figure, the model function y is given a time instant t=t i In this case, the second time series θ S (t i The fitted model function y is then y p (Interchangeable "predicted sample y p To identify consecutive samples denoted as '', consecutive future time moments t=t i It is extrapolated to +δ. As shown in Figure 8, a favorable selection of future time moments (i.e., given by the prediction interval δ) is y p (t i ) = y(t i+1 ) such that t=t i +δ=t i+1 =t i +Δ t This is possible. In other words, the predicted sample P is the second time series θ. SThe predicted consecutive (i.e., future) samples (i.e., θ) S (t i+1 ) of θ i+1 ) may correspond to. However, other choices for the prediction interval δ are also possible. The optimal choice may depend on factors such as the step response of the OIS system 104 and the mass of the movable elements (e.g., lens 118 or image sensor 122). Generally, a prediction interval δ that is too short may cause the OIS system 104 to reach the corresponding setpoint (as defined below) before further predicted samples are identified by the prediction block 216. This allows the driver 110 to identify the next predicted sample (e.g., y p (t i+1 It may go into a dormant state until it becomes available. In that case, the forecast may bottleneck the compensation amount. Conversely, a forecast period δ that is too long may cause the OIS system 104 to target a setpoint that does not accurately reflect the dynamics of camera vibration.

[0072] The model function y is based on the sine and / or cosine functions and the current dominant frequency f. An example of the model function y is as follows: y(t,f)=x0+x1·sin(2π·f·t)+x2·cos(2π·f·t) (Formula 1) Here, f represents the last updated current dominant frequency identified by frequency analysis block 212 (for example, as currently stored in the output buffer of frequency analysis block 212). Below, the updated current dominant frequency is given by time instant t=t i It is assumed that this has been identified. In that case, the current dominant frequency in Equation 1 is f = f(t i It is given by ).

[0073] The model function of Equation 1 is the second time series θ S After fitting, the consecutive samples y p (t i ) = y(t i+1 ) can be identified as follows: That is, yp =x0+x1·sin(2π·f·(t i +δ))+x²·cos(2π·f·(t i +δ)) (Equation 2) Here, the prediction interval δ is, for example, Δ t It can be equal to.

[0074] As described above, by basing the model function specifically on a sine and / or cosine function combined with the current dominant frequency f, accurate prediction of consecutive directional samples becomes possible with a relatively simple model and little computational complexity. This can be understood by considering the following: the vibration of camera 100 can be modeled as a harmonic oscillator for the purpose of OIS, which is a reasonable good approximation. In this case, the vibration frequency corresponds to the natural frequencies of camera 100 and the object holding camera 100 (e.g., the support structure to which camera 100 is mounted, or the user holding camera 100 by hand).

[0075] In some scenarios, the oscillation can be approximated as a simple harmonic oscillator. In other scenarios, a more accurate approximation of the oscillation can be a damped harmonic oscillator. However, the longer first time series sample θ L To estimate the current dominant frequency f from a shorter second time series sample θ S Fitting the model function to the first and / or second time series sample θ L , θ S By updating the model fit as new samples are added, Equation 1 can be used even when the characteristics of the vibrational motion change over time. This is because the prediction system 106 is continuously updated over time for changes in the dominant vibrational frequency and / or amplitude of the vibrational motion.

[0076] The model fitting performed in step S304 can be equivalent to solving a system of linear equations. This system of equations can be expressed as a matrix by the following formula: Ax = b (Equation 3) Here, A is the coefficient matrix given by the following matrix based on equation 1. That is, TIFF2026103817000002.tif35170

[0077] x is a vector containing the set of fitting parameters in Equation 1. That is, TIFF2026103817000003.tif26170b is the sample θ of the second time series. S It is a vector that contains . That is, TIFF2026103817000004.tif32170

[0078] In this specification, index t j=0…m The second time series θ in equation 6 is currently stored in the short buffer 214. S This corresponds to the indices of multiple samples of the sample. Therefore, m = W-1. Thus, the matrix A defined in equation 4 is the sample θ of the second time series. S It can also be used when it is updated (the sample θ of the second time series) S As long as the length does not change, the following second time series sample θ S (See the explanation for changing the length of ). The fitted parameter x can be estimated using linear regression, preferably the least squares method. An effective approach to finding the least squares solution to Equation 3 is QR factorization. In this case, the coefficient matrix A is decomposed into an orthogonal matrix Q and an upper triangular matrix R. That is,

[0079] A = QR (Equation 7)

[0080] Therefore, the fitting parameter x can be estimated according to the following formula: Rx=Q T b (Equation 8) Use back substitution. (Consecutive samples y) p (t i ) can then be predicted according to Equation 2, using the estimated fitting parameter x and the current dominant frequency f.

[0081] The least-squares fit of the model function can be determined in a similar manner using other matrix decomposition algorithms, particularly other orthogonal-triangular matrix decompositions such as RQ decomposition, QL decomposition, or LQ decomposition.

[0082] In step S306, the prediction system 106 proceeds to the predicted consecutive samples y identified by the prediction block 216. p (t i Based on ), a given time moment t i The following setting point c is identified. Setting point c is therefore the predicted sample y p (t i ) is derived from. In the illustrated embodiment, the setpoint c is identified by the angle-position block 218. The purpose of angle-position block 218 is to predict y p (t i The method involves applying an angle-position function to convert y given as the angle. In this embodiment, y p (t i ) is converted to a corresponding position value for a movable element, for example, a movable lens 118 or an image sensor 122. More specifically, the conversion is predicted y p (t i ) maps to a setpoint c that represents the position of the movable element (y p (t i ) represents the predicted orientation of camera 100). The specific form of the angle-position function is the design of the OIS system 104, predicted y p (t i The transformation may depend on the position of the movable element relative to the pivot point of the angular displacement shown by ), the geometric relationship between the detection axis and the compensation axis, etc. Typically, the transformation is performed using the predicted sample y p (t iThis can be achieved by multiplying by a predetermined conversion coefficient (constant). As part of an OIS system, a suitable approach for converting angular displacement measured by a motion sensor (e.g., a gyroscope) to linear position of a movable compensating element measured by a position sensor (e.g., a Hall sensor) is known in the art and can therefore be implemented by the angle-position function block 218.

[0083] The amount of compensation that needs to be applied in response to a given change in the orientation of camera 100 (i.e., the required translation of the movable element of the OIS system 104) further depends on the focal length of the camera's optical system (e.g., optical system 114). If the optical system has a zoom lens, the transformation performed by the angle-position block 218 may further take into account the current zoom level L of the optical system. As shown in Figure 3, the current zoom level L may be provided by the zoom level block 220. In one computationally efficient embodiment, the angle-position block 218 may retrieve a gain value from a predetermined lookup table (e.g., stored in the memory of the OIS system 104) that associates each of several zoom level entries with a predetermined gain value. The retrieved gain value may be a predetermined gain value associated with the (e.g., nearest) zoom level entry corresponding to the current zoom level L. The angle-position block 218 may then multiply the position value (e.g., calculated according to Equation 9) by the retrieved gain value. The result may be output to the control system 221, which includes the OIS controller 108, as the next setpoint c.

[0084] The setpoint c input to the control system 221 is summed with the inverted position signal v (the negative of the position signal v, i.e., -v) output by the position sensor 112 to generate an error e representing the tracking error of the control system 221, as shown in block 222 in Figure 3. More specifically, the error e represents the tracking error with respect to the position of the movable element. The error e forms an input to the OIS controller 108. The OIS controller 108 responds by generating an actuation signal u to a driver (e.g., driver 110) to actuate the movable element (e.g., lens 118 or sensor 122) of the controlled system 224. For example, if the OIS controller 108 is a PID controller, the actuation signal u may be identified as the sum of the P-, I-, and D- components based on the error e. Any other suitable conventional approach for generating the actuation signal u based on the error e in a closed-loop controller may be used.

[0085] In the above embodiment, the prediction system 106 performs an angle-position conversion to identify a setpoint c relating to the setpoint of the movable element's position. However, several other embodiments are possible. For example, the angle-position block 218 may be replaced by a position-angle block in the feedback path of the closed-loop control system 221, which converts the position value v output by the position sensor 112 into the corresponding angle. The input error signal e to the OIS controller 108 in this case represents the tracking error of the control system 221 in the angle domain. Accordingly, the OIS controller 108 may perform an angle-position conversion to generate an actuation signal u.

[0086] The above discloses a regression based on orthogonal-triangular matrix decomposition (e.g., QR decomposition). Thereafter, orthogonal (e.g., Q) and triangular (e.g., R) matrices can be identified during execution time each time a new current dominant frequency f is made available by the frequency analysis block 212. Figure 9 is a block diagram of a further embodiment of the prediction block 216 of the prediction system 106 of Figure 3. In this case, the regression is instead based on pre-calculated orthogonal and triangular matrices. Figure 9 will be explained with particular reference to QR decomposition, but similar approaches may be used for other types of orthogonal-triangular matrix decompositions such as RQ, QL, or LQ decomposition.

[0087] Prediction block 216 is a set of multiple QR matrix pairs Q0, R0;…;Q N ,R N It includes a lookup table 2162 that stores the following. Each pair of QR matrices contains the sine function and / or cosine function and their respective frequencies f. p0 ,…,f pN Based on this, the model function is pre-calculated. More specifically, each QR matrix pair may correspond to the QR decomposition of the coefficient matrix A of Equation 4, specified for each value of frequency f. For example, Q0 and R0 are, p0 This can be the QR decomposition of the coefficient matrix A calculated for the frequency f, where Q1 and R1 are the frequencies f p1 This can be the QR decomposition of the coefficient matrix A calculated for Q N and R N Until Q N and R N The same applies to each QR matrix containing . Regardless of the specific form of the coefficient matrix, each matrix pair can therefore be pre-calculated for each (dominant) frequency and associated with that frequency. p0 ,…,f pNThese frequencies may be distributed within a predetermined frequency range. The frequency range may extend to a range of frequencies that are expected to occur in a typical imaging scenario and that the OIS system 104 can compensate for. The frequency range may be the same frequency range described above with reference to, for example, the frequency analysis block 212 and Figure 7 (e.g., 50 Hz or less or 30 Hz or less). The number of frequencies may depend on the available memory for the lookup table 2162, the desired accuracy of the regression, etc.

[0088] The prediction block 216 further includes a selector block 2164. The selector block 2164 selects the current dominant frequency f (e.g., f = f(t)) identified by the frequency analysis block 212. i The time instant t=t given by ) i Based on the input in ), multiple QR matrix pairs Q0,R0;…;Q N ,R N The selector block 2164 selects the frequency f closest to the currently dominant frequency f identified. p0 ,…,f pN Multiple QR matrix pairs Q0, R0;…;Q calculated for N ,R N Select a QR matrix pair from the options provided.

[0089] The selected QR matrix pair is received as input by the fitting and extrapolation block 2166. Accordingly, the fitting and extrapolation block 2166 estimates the fitting parameter x from Equation 8 by using the selected QR matrix pair. Subsequently, the fitting and extrapolation block 2166 uses the estimated fitting parameter x and the current dominant frequency f to generate a sequence of samples y according to Equation 2. p (t i Identify the prediction for ).

[0090] Figure 10 is a block diagram of a further embodiment of the prediction system 106 of Figure 3. As described above, each given time moment t iThe currently dominant frequency f identified can be stored in the output buffer of the frequency analysis block 212. In Figure 10, such an output buffer is represented by block 2122. The output buffer 2122 stores the history, i.e., time series, of previously identified currently dominant frequencies f identified at each time instant. In Figure 10, the length of the output buffer 2122 is U, meaning that the output buffer 2122 maintains the history of U previously identified currently dominant frequencies f.

[0091] The output buffer block 2122 can be implemented as a FIFO buffer. Therefore, the output buffer 2122 is at time instant t i-1 In this case, the time series f(t) of U previously identified current dominant frequencies f. i-1 Assuming that it is satisfied, time instant t i In this new current dominant frequency f(t i When you obtain the time series f(t i-1 ) is the oldest (first) frequency f(t i By discarding -U, the new frequency f(t i ) may be updated with a new frequency f(t i ) is the updated time series f(t) of the current dominant frequency. i To form a time series f(t i-1 The newest (last) frequency f(t) for the remaining frequencies of ) i ) can be added as such.

[0092] The (updated) time series f(t) of the current dominant frequency i ) is a time series f(t i The variation is received as input by the frequency variation calculation block 2124, which identifies the variation of the time series f(t). i This can be calculated as the variance of ). Based on the identified fluctuations, the frequency fluctuation calculation block 2124 adapts the length W of the short-term history buffer 214 by setting the length W to a smaller value for higher fluctuations and a larger value for lower fluctuations.

[0093] As mentioned above, regression therefore responds to higher variability than to lower variability, resulting in a shorter second time series θ. S This can be based on the following. The advantage can be understood by considering that lower fluctuations in the dominant vibration frequency may mean that the motion signal changes in a more stable way over time. Therefore, a longer second time series θ S This can be used to improve the accuracy of model fitting. Conversely, when the dominant vibration frequency in the motion signal is more unstable, a shorter second time series θ S This may indicate more transient behavior within the motion signal ω, meaning that the motion signal ω is more unstable over time. Therefore, a shorter time series θ S This allows regression to adapt to such changes more quickly. The frequency variation calculation block 2124 can, for example, compare the variation (e.g., variance) against a variation threshold and set the length W to a first value depending on whether the variation is less than (or equal to) the threshold, and to a second value higher than the first value depending on whether the variation is greater than the threshold. The first and second values ​​are, for example, a second time series θ for different quantities of frequency variation. S This may be a predetermined value determined by testing the performance of OIS compensation for different lengths. In another embodiment, the frequency calculation block 2124 may determine the length W as a function of increasing frequency variation in multiple steps (e.g., three, four, or more steps). Thus the second time series θ S The length can change in a more gradual way than is dependent on frequency variations.

[0094] In any case, the first time series θ S After deciding to decrease the length W, in order to obtain the updated second time series θ with the updated decreased length W, the second time series θ S By discarding some of the oldest samples θ, we obtain a second time series θ S The length W may be updated. For example, if the time instant t=t iin Δ W Assuming that it decreases by only a certain amount, the second time series θ S Multiple samples θ(t i-W+1 ) from θ(t i-W+1+ΔW-1 ) may be discarded, here, Δ W This represents a decrease in length W. Conversely, the second time series θ S After deciding to increase the length W, the updated second time series θ of the updated increased length W is obtained. S To obtain this, several new samples θ derived from the motion signal ω are used in the second time series θ. S By sequentially adding to it, the second time series θ S It can be updated. For example, if length W is given by time instant t=t i in Δ W Assuming it increases by only a certain amount, the second time series θ S The next sample θ(t i+1 ) from θ(t i+ΔW ) can be updated.

[0095] The embodiment of the prediction system 106 shown in Figure 10 can be combined with the embodiment of the prediction block 216 described with reference to Figure 9. In this case, the lookup table 2162 is the second time series θ S For each of several lengths W, there are multiple QR matrix pairs Q0, R0;…;Q N ,R N(or other types of orthogonal and triangular matrix pairs, such as RQ, QL, or LQ matrix pairs) can be modified to store these. That is, the number of rows in the Q and R matrices of each QR matrix pair associated with length W can be W. Thus, selector block 2164 may receive length W as input from frequency variation calculation block 2124 and select a QR matrix from a plurality of QR matrix pairs associated with length W that are further associated with the frequency closest to the identified current dominant frequency f. For example, lookup table 2162 may store a first plurality of QR matrix pairs for a first value of W and a second plurality of QR matrix pairs for a second value of W. Selector block 2164 may perform a QR selection from the first plurality of QR matrix pairs depending on whether the input length W is equal to the first value, and may perform a QR selection from the second plurality of QR matrix pairs depending on whether the input length W is equal to the second value.

[0096] Referring again to Figures 3 and 4, under some scenarios, the sample θ of the first time series L It is conceivable that there may be clearly distinguishable frequency components within the signal. In such cases, the frequency analysis block 212 may be unable to identify the currently dominant frequency f in step S302. To address such cases, the prediction system 106 may perform the following variation of step S302, as shown in the flowchart of Figure 11.

[0097] In step S3021, the frequency analysis block 212 analyzes the first time series θ L To determine whether it contains any frequency components with amplitudes exceeding the amplitude threshold A, a sample θ of the first time series is used. L A frequency analysis is performed. This frequency analysis is performed to obtain the spectrum shown in Figure 7, as described above, using the first time series θ L This may include applying a frequency-domain transformation (e.g., FFT).

[0098] Depending on the identification of one such frequency component, the frequency analysis block 212 proceeds and, in step S3022, identifies the current dominant frequency as the frequency of the thus identified frequency component, namely f = f1, where f1 is the frequency of the identified frequency component. If at least two such frequency components are identified, the frequency analysis block 212 may identify the current dominant frequency as the frequency of the dominant (i.e., highest amplitude) frequency component among the at least two identified frequency components, namely f = f n And here, f n This is the frequency of the dominant frequency component.

[0099] In the event that no such frequency component could be identified, the frequency analysis block 212 identifies the current dominant frequency as the default frequency in step S3023. That is, f=f D That is the case.

[0100] The amplitude threshold A can be determined by examining the spectra acquired for various vibrational motions of camera 100 and by identifying a threshold level at which a stable amplitude peak within the frequency range of interest can be identified under a suitable wide range of scenarios. The default frequency can be predetermined, for example, based on prior knowledge of one or more target use scenarios of camera 100, as a frequency that is expected to provide some degree of OIS compensation. For example, a choice that has proven useful under a relatively wide range of scenarios in which camera 100 is mounted on a supporting building structure is a default frequency in the range of 5 Hz to 20 Hz, e.g., 10 Hz.

[0101] Various processes and blocks involved in controlling an OIS system described herein, such as OIS system 104, can be implemented in both hardware and software. In a software implementation, the imaging device (e.g., camera 100) may include a processing device implemented in the form of one or more processors, such as one or more central processing units. Computer program code instructions stored in a (non-temporary) computer-readable medium, such as non-volatile memory, associated with this processing device cause the processing device to execute method steps for controlling the OIS system. Several examples of non-volatile memory include read-only memory, flash memory, ferroelectric RAM, magnetic computer storage devices, and optical discs. In a hardware implementation, the processing device may instead be implemented by a dedicated circuit configured to perform method steps for controlling the OIS system. This circuit may take the form of one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs) or one or more field-programmable gate arrays (FGPAs). It will be understood that combinations of hardware and software implementations are also possible. This means that some method steps may be implemented by dedicated circuits, while others may be implemented in software.

[0102] Those skilled in the art will understand that the present invention is not in any way limited to the embodiments described above. Conversely, many modifications and variations are possible within the scope of the appended claims.

[0103] For example, the model function in Equation 1 is based on both sine and cosine functions, but it is also possible to define an equivalent model function based on either the sine or cosine function. In this case, the frequency term 2π·f·t of the sine or cosine function can be supplemented by a phase shift.

[0104] Furthermore, the above has primarily focused on identifying a single dominant frequency for a motion signal. However, the frequency analysis block 212 in Figure 3 is capable of identifying two or more dominant frequencies. For example, as shown in Figure 7, the frequency analysis block 212 can identify a "second" dominant frequency f' in addition to a "first" dominant frequency f. The first and second dominant frequencies f, f' are the two highest amplitude components of the frequency domain representation. Therefore, the model function in Equation 1 can be modified to include, for example, a further term for the second dominant frequency f'. For example, y(t,f)=x0+x1·sin(2π·f·t)+x2·cos(2π·f·t)+x3·sin(2π·f'·t)+x4·cos(2π·f'·t) (Formula 1')

[0105] This may inevitably involve corresponding modifications to Equations 2 and 4 (e.g., introducing two additional columns and two additional rows corresponding to further fitting parameters x3 and x4). This approach can be extended to identify three or more dominant frequencies. Access to more dominant frequencies may allow for more accurate predictions of consecutive samples, but it may introduce further computational complexity, both in frequency analysis and during regression. Thus, the number of dominant frequencies can generally be a trade-off between accuracy and computational complexity. In a typical embodiment, one or two dominant frequencies are considered to strike a good balance between accuracy and computational complexity.

[0106] Furthermore, the description with reference to the drawings has primarily referred to a single sensing axis of the motion sensor 202 implemented by a gyroscope (e.g., pitch or yaw) and a single corresponding compensation axis of the OIS system 104. However, the above description can be applied in corresponding ways to any of the other multiple embodiments of the motion sensor 102 described above, and to any number of sensing and compensation axes. For example, the multiple embodiments and multiple examples described above with reference to the OIS system 104 in Figure 3 can be performed for each sensing axis of the motion sensor (e.g., both axes of a two-axis gyroscope) and each corresponding compensation axis of the OIS system (e.g., translation of the movable lens along the two transverse axes in a plane perpendicular to the optical axis of the optical system 114). For example, in step S301 of method 300 in Figure 4, samples of the first and second time series can be derived from each sensing axis of the motion signal (e.g., pitch and yaw). Correspondingly, step S302 can be applied individually to each of the first time series to identify the current dominant frequency for each sensing axis of the motion signal. Furthermore, step S303 may include identifying the setpoint for each of the corresponding compensation axes. The setpoint for each axis may be identified by performing each of steps S304 to S306 individually for each sensing axis. That is, each model function based on the sine and / or cosine functions identified for each first time series may be fitted to each second time series, and consecutive samples may be predicted by extrapolating each fitted model function. The setpoint for each axis may then be identified based on each consecutive sample. As can be understood, the OIS system 104 may in this case include each driver 110 and each position sensor 112 for each compensation axis.

Claims

1. A method for controlling the optical image stabilization (OIS) system (104, 104') of an imaging device (100), S301) Acquires a first time series sample and a second time series sample derived from the motion signal of the motion sensor (102, 202) of the imaging device (100), wherein the motion signal indicates the movement of the imaging device (100), the first time series extends over a longer period of the motion signal than the second time series, the oldest sample of the first time series is earlier than the oldest sample of the second time series, and the last sample of the first time series is simultaneous with or earlier than the last sample of the second time series. Based on the frequency analysis of the first time series, the current dominant frequency of the motion signal is identified (S302), and To fit a model function based on the sine function and / or cosine function and the current dominant frequency to the second time series (S304), In order to predict consecutive samples, the fitted model function is extrapolated beyond the last sample of the second time series (S305), and By identifying a setpoint based on the predicted consecutive samples (S306), A method comprising identifying the setting point of the OIS system (104) (S303).

2. The method according to claim 1, wherein the motion signal indicates rotational motion.

3. The method according to claim 2, wherein the motion signal represents angular velocity, and the sample is derived by integrating the motion signal.

4. The method according to claim 2 or 3, wherein the motion sensor includes a gyroscope (202).

5. The method according to any one of claims 1 to 4, wherein fitting the model function to the second time series comprises identifying a least-squares fit of the model function to the second time series using orthogonal-triangular matrix decomposition, wherein identifying the least-squares fit comprises selecting a pair of orthogonal and triangular matrices from a plurality of pairs of orthogonal and triangular matrices, each pair being pre-calculated for the model function based on a sine function and / or cosine function and its respective frequency, and the selected pair of orthogonal and triangular matrices is the pair among the plurality of pairs of orthogonal and triangular matrices calculated for the frequency closest to the identified current dominant frequency.

6. The method according to claim 5, wherein the orthogonal-triangular matrix decomposition is a QR, RQ, QL, or LQ decomposition.

7. The second time series is sequentially updated with new samples derived from the motion signal, and The process further includes sequentially identifying the updated setting points of the OIS system (104, 104'), where each updated setting point is To fit a model function based on the sine and / or cosine functions and the current dominant frequency to the last updated second time series, To predict the updated consecutive samples, extrapolate the fitted model function beyond the last sample or the last updated second time series, and By identifying the updated setpoint based on the predicted updated consecutive samples, The method according to any one of claims 1 to 6, as specified.

8. The first time series is sequentially updated with new samples derived from the motion signal, and The process further includes sequentially identifying the updated current dominant frequency based on the last updated sample of the first time series, The method according to claim 7, wherein each of the model functions used to predict each updated future sample is based on the currently dominant frequency that was last updated.

9. The method of claim 8, further comprising identifying variations in the identified current dominant frequency sequence, wherein the length of a subsequently updated second time series is based on the variations such that it decreases in response to higher variations rather than to lower variations.

10. The method according to claim 8 or 9, wherein the first time series and the second time series are updated at a first sampling rate and a second sampling rate, respectively, and the first sampling rate is less than or equal to the second sampling rate.

11. Identifying the current dominant frequency is, Based on the frequency analysis of the sample of the first time series, it is determined whether the first time series contains any frequency component having an amplitude exceeding an amplitude threshold (S3021). In response to identifying at least one frequency component having an amplitude exceeding the amplitude threshold, the current dominant frequency is set to the dominant frequency component among the at least one identified frequency component (S3022), and The method according to any one of claims 1 to 10, further comprising setting the current dominant frequency to a predetermined default frequency (S3023) in accordance with the fact that no frequency component having an amplitude exceeding the amplitude threshold has been identified.

12. The method according to any one of claims 1 to 11, wherein the OIS system (104, 104') is configured to compensate for the movement of the imaging device (100) by controlling the position of the movable elements (118, 122) of the OIS system (104, 104') based on the setpoint, and identifying the setpoint of the OIS system (104, 104') includes applying a transformation (218) to the consecutive samples to map the consecutive samples to the setpoint representing the position of the movable elements (118, 122).

13. An imaging device (100), Motion sensors (102, 202) configured to output a direction signal indicating the instantaneous movement of the imaging device (100), Optical image stabilization (OIS), system (104, 104'), configured to compensate for the movement of the imaging device (100), and An imaging device (100) comprising a processing device configured to perform the method according to any one of claims 1 to 12 for controlling the OIS system (104).

14. A non-temporary computer-readable medium including a portion of computer program code, wherein the portion of computer program code is configured to, when executed by a processing device, perform the method according to any one of claims 1 to 12.

Images