Methods for performing digital image stabilization

Abstract

This invention provides a method for performing digital image stabilization (DIS) on a sequence of video frames captured by an image capture device. [Solution] The method involves capturing a sequence of video frames, acquiring a sequence of motion values, acquiring a sequence of position values ​​sampled from a position signal, determining residual motion values, and performing DIS, which includes detecting image features in a reference video frame, tracking image features in further video frames, determining frame motion data based on the displacement of the image features, and generating a stabilized sequence of video frames based on the frame motion data, wherein the size of the tracking window for tracking image features in further video frames is set based on the sequence of residual motion values.

Description

[Technical Field]

[0001] The present invention generally relates to a method for performing digital image stabilization (DIS), an image processor, an image capture device, and a computer program product for carrying out such a method. [Background technology]

[0002] Image stabilization (IS) is used in cameras to reduce the effects of camera motion, particularly vibration, on captured image frames. For example, camera motion can result in blurred image frames, and / or, when capturing video, camera motion between video frames can result in unstable video.

[0003] When a camera is mounted on a support structure such as a wall, ceiling, pole, or other camera support (as is common in video surveillance applications), camera vibration may be caused by shaking of the camera and / or support structure due to collision with another object or exposure to other external forces such as wind. If the camera is handheld, camera vibration may be caused by the camera user's unstable hands.

[0004] Optical image stabilization (OIS) includes lens-based image stabilization and sensor-based image stabilization. The basic principle of lens-based OIS is to actuate a movable lens element of the camera's optical system to compensate for camera vibrations. The OIS system can compensate for vibrations by moving the movable lens element based on the sensed motion of the camera, thereby keeping the image stable on the camera's image sensor. In sensor-based OIS, instead of moving the movable lens element, the OIS system may actuate the image sensor based on the sensed motion. This approach is sometimes called "sensor-based image stabilization" (SIS) and is considered a type of OIS in this disclosure.

[0005] A common characteristic of the OIS approaches described above is that they involve activating physical elements (e.g., lenses or sensors) that have a certain inertia. Therefore, activating moving elements takes some time, and thus compensation usually delays the motion to some extent. The amount of delay depends on specific parameters of the OIS system (e.g., responsiveness of the OIS system and actuators, inertia of the moving elements, frequency of vibration, etc.). As an illustrative, non-limiting example, a typical delay in a modern OIS system can be in the range of about 5–10 ms. This can be translated into some OIS-induced residual motion of the image on the image sensor. Such OIS-induced residual motion can, among other things, result in motion of the image between image frames and thus produce unstable video. The amount of residual motion tends to be more pronounced for higher frequency vibrations (e.g., above 5 Hz). [Overview of the Initiative]

[0006] In view of the foregoing, the object of the present invention is to provide an improved approach for performing image stabilization, in particular including digital image stabilization (DIS) for a sequence of video frames, which enables effective compensation even when residual motion after OIS is present in order to reduce video instability. Further and alternative issues can be understood from the following.

[0007] According to a first aspect of the present invention, a method is provided for performing digital image stabilization (DIS) on a sequence of video frames captured by an image capture device, wherein the image capture device comprises a motion sensor configured to output a motion signal indicating the motion of the image capture device, an optical image stabilization (OIS) system configured to compensate for the motion of the image capture device by controlling the position of a movable element of the OIS system based on the motion signal, and a position sensor configured to output a position signal indicating the instantaneous position of the movable element, and the method is as follows: This involves capturing a sequence of video frames while simultaneously obtaining a sequence of motion values ​​sampled from the motion signal, so that each video frame is associated with its respective motion value. This involves capturing a sequence of video frames and obtaining a sequence of position values ​​sampled from the position signal, so that each video frame is associated with its respective position value. For each video frame, the residual motion value is determined to obtain a sequence of residual motion values ​​for the sequence of video frames, wherein each residual motion value is determined based on the motion and position values ​​associated with each video frame, and the residual motion value represents the residual motion of the image capture device that is not compensated by the OIS system when capturing each video frame. This involves performing digital image stabilization (DIS) on a sequence of video frames, Detecting image features within a reference video frame of a sequence of video frames, Tracking image features within each further video frame in a sequence of video frames, Determining frame motion data based on the displacement of image features within each further video frame in a sequence of video frames, Performing DIS, which includes generating a stabilized sequence of video frames based on frame motion data. Includes, The size of the tracking window for tracking image features within each further video frame is set based on a sequence of residual motion values.

[0008] The method according to the first embodiment combines DIS with OIS to enable effective but computationally efficient stabilization of the video frame sequence.

[0009] The residual motion values ​​determined according to this method are calculated in real time and can therefore provide an instantaneous estimate of the residual motion after OIS (hereinafter referred to as "OIS error"). The residual motion values ​​can then be used to adapt the tracking window, and thus the amount of pixel data being analyzed, in order to track image features. This enables the use of accurate and sophisticated computer vision-based approaches to determine frame motion data used by DIS, such as interframe motion, i.e., approaches that are computationally expensive to apply to the entire image region of a video frame. Otherwise, in practice, such approaches can be computationally too expensive to perform on the fly using the available processing resources in image capture devices with limited on-device processing resources, such as typical surveillance cameras. Therefore, the method of the first embodiment enables an efficient yet lightweight approach to performing DIS.

[0010] This method allows for a more accurate estimation of the amount of compensation required to stabilize a sequence of video frames than can typically be achieved with conventional gyro-based electronic image stabilization (EIS).

[0011] Furthermore, by combining OIS and DIS in this way, image stabilization may use a larger cropping size when generating a sequence of stabilized video frames, which means that less pixel data needs to be discarded from each video frame.

[0012] While residual motion values ​​are determined based on the respective signals output by the motion and position sensors, it should be noted that the DIS subsequently applied to the video frame sequence does not directly depend on the measurement signals from these sensors, but rather on the analysis of the pixel data of the video frames in the sequence.

[0013] A residual motion value based on the size of a tracking window for tracking a given image feature within a given further video frame may be referred to herein as a “representative residual motion value.” Various embodiments for determining or selecting a representative residual motion value for a given video frame are described herein.

[0014] In some embodiments, determining each residual motion value includes determining a corresponding orientation value that indicates the instantaneous orientation estimate of the image capture device at the time each video frame was captured, based on the motion values ​​associated with each video frame, and determining the residual motion value based on the orientation value and the position value.

[0015] The vibrational motion of an image capture device due to shaking tends to produce greater fluctuations in orientation / angle than in the linear position of the image capture device. Therefore, determining residual motion values ​​based on instantaneous orientation estimates may allow for a more accurate and sensitive estimation of OIS error at each moment.

[0016] In some embodiments, residual motion values ​​are determined based on the difference between azimuthal and positional values ​​when mapped to a common coordinate system. Therefore, one or both of the azimuthal and positional values ​​may be transformed to be expressed in common units to facilitate determining a more appropriate estimate of the OIS error at each moment. For example, a transformation may be applied to the azimuthal value to determine the corresponding positional value. Alternatively, a transformation may be applied to the positional value to determine the corresponding azimuthal value. Alternatively, each transformation may be applied to the azimuthal and positional values ​​respectively to determine the corresponding values ​​in the common coordinate system.

[0017] In some embodiments, the motion signal indicates rotational motion. Therefore, the motion signal can be obtained from a sensor capable of detecting the rotational motion (e.g., as angular velocity) of the image capture device / motion sensor, such as a gyroscope. As mentioned above, sensing the orientation / angle of the image capture device leads to a more sensitive measurement of vibrational motion, and thus, in addition to effective OIS, can enable a more accurate and sensitive estimation of the OIS error.

[0018] In some embodiments, the motion signal represents angular velocity, and the azimuth value is derived by integrating the motion signal. Thus, the azimuth value can be obtained by integrating the motion signal.

[0019] In some embodiments, the motion sensor includes a gyroscope. The gyroscope can provide motion signals indicating rotational motion, particularly as angular velocity, with relatively low noise.

[0020] In some embodiments, tracking image features involves using optical flow analysis, which is selectively applied to pixels within the tracking window of each subsequent video frame. Optical flow analysis allows for the accurate estimation of the displacement of image features within each subsequent video frame. By selectively applying optical flow analysis to the tracking window, i.e., by limiting the optical analysis to the tracking window, the amount of pixel data that needs to be processed during tracking can be reduced.

[0021] In some embodiments, the size of the tracking window within each further video frame is set based on the residual motion value associated with the further video frame. Thus, the size of the tracking window can be updated based on the instantaneous residual motion value associated with each respective video frame.

[0022] In some embodiments, the size of the tracking window is configured to increase with increasing residual motion values. For larger OIS errors, larger displacements of image features between video frames are expected. Therefore, the size of the tracking window may be increased to allow tracking of larger displacements of image features. Conversely, for smaller OIS errors, smaller displacements of image features between video frames may be expected, allowing for a reduction in the tracking window, and thus the amount of pixel data that needs to be analyzed within the video frame (e.g., using optical flow analysis).

[0023] In some embodiments, the tracking window is set to a first size in response to the residual motion value being less than a threshold, and to a second size greater than the first size in response to the residual motion value being greater than a threshold. Thus, the size of the tracking window can be changed in a favorable manner between the two sizes based on a computationally efficient threshold comparison.

[0024] In some embodiments, the position of the tracking window for tracking image features within each further video frame is determined based on the position of the tracking window in the preceding video frame of the video sequence (i.e., each video frame preceding each further video frame) and the residual motion values ​​associated with each further video frame.

[0025] Therefore, residual motion values ​​can be further used to update the position of the tracking window between consecutive video frames. This can improve the robustness of feature tracking by enabling tracking of image features even for large displacements that would otherwise cause the image features to move out of the tracking window. This may be particularly useful for lower-performance OIS systems and / or when the OIS system is not sufficiently calibrated.

[0026] In some embodiments, the sequence of motion values ​​is a sequence of first motion values ​​sampled from a first motion signal of a first sensing axis of a motion sensor, and the method further includes acquiring a sequence of second motion values ​​sampled from a second motion signal of a second sensing axis of a motion sensor, while capturing a sequence of video frames, such that each video frame is further associated with its respective second motion value. The sequence of position values ​​is a sequence of first position values ​​sampled from a first position signal of a first sensing axis of a position sensor, and the method further includes acquiring a sequence of second position values ​​sampled from a second position signal of a second sensing axis of a position sensor while capturing a sequence of video frames, such that each video frame is further associated with its respective second position value, and the method For each video frame, the method involves determining a residual motion vector to obtain a sequence of residual motion vectors for the sequence of video frames, wherein each residual motion vector is determined based on a first motion value and a second motion value and a first position value and a second position value associated with each video frame, and each residual motion vector includes a first component having a first residual motion value and a second component having a second residual motion value, where the first and second residual motion values ​​represent the residual motion of the image capture device along a first and second compensation axis of the OIS system that is not compensated by the OIS system when capturing each video frame.

[0027] Therefore, the motion of the image capture device and the position of the movable elements of the OIS system can be sensed along each of the two sets of sensing axes, and then it becomes possible to estimate the residual motion, i.e., the OIS error, in two dimensions.

[0028] The first residual motion value of the first component of the sequence of residual motion vectors may correspond to or define the sequence of residual motion values ​​described above (which may be called the “sequence of first residual motion values”), and the second residual motion value of the second component of the sequence of residual motion vectors may correspond to or define the sequence of second residual motion values.

[0029] Once the residual motion vector is determined, the size of the tracking window in each subsequent video frame may be set based on the first and second residual motion values ​​of the first and second components of the sequence of residual motion vectors. For example, the size of the tracking window in any given subsequent video frame may be set based on the magnitude of the residual motion vector associated with the given subsequent video frame, or the maximum values ​​of the magnitudes of the first and second components.

[0030] In embodiments where the tracking window position is updated, the position of the tracking window for tracking image features within each subsequent video frame may be determined based on the position of the tracking window in the preceding video frame of the video sequence and the residual motion vector associated with each subsequent video frame. Thus, the position of the tracking window in a given video frame may be updated relative to the preceding video frame according to the first and second components of its associated residual motion vector.

[0031] Here, the first and second components of the residual vector can be mapped along the horizontal and vertical axes of the image sensor, respectively. Therefore, the first component of the residual vector associated with a given video frame can be used to shift the position of the tracking window relative to its position in the preceding video frame, along the horizontal axis of the image sensor. Correspondingly, the second component of the residual vector associated with a given video frame can be used to shift the position of the tracking window relative to its position in the preceding video frame, along the vertical axis of the image sensor. The horizontal and vertical axes of the image sensor can correspond to (i.e., align with) the X and Y axes of the video frame, respectively.

[0032] In some embodiments, the image feature is a first image feature detected in a reference video frame, and performing DIS on a sequence of video frames is: Further detection of at least a second image feature within a reference video frame of a sequence of video frames, Tracking the first and second image features within each further video frame of a sequence of video frames, This includes determining frame motion data based on the respective displacements of a first image feature and a second image feature within each further video frame in a sequence of video frames, The size of each tracking window for tracking the first and second image features within each further video frame is set based on a sequence of residual motion values.

[0033] Therefore, DIS may be based on tracking (at least) first and second image features detected within a reference image frame, the first and second image features being tracked individually within each further video frame using their respective tracking windows. Frame motion data may be based, for example, on the average of the respective displacements of the first and second image features within each further video frame.

[0034] In some embodiments, generating a sequence of stabilized video frames based on motion data includes applying an image transformation to each video frame in the sequence, where the image transformation is based on the motion data. For example, the image transformation may include trimming the video frames.

[0035] In some embodiments, the movable element is a movable optical element of the OIS system. Thus, the OIS system can be configured for lens-based OIS.

[0036] In some embodiments, the movable element is an image sensor. Thus, the OIS system can be configured for sensor-based OIS.

[0037] In some embodiments, the OIS system includes a closed-loop controller configured to generate OIS control signals for controlling the position of a movable element and to use the position signals as feedback signals. Thus, a position sensor may be placed within the feedback path of the closed-loop controller and configured to output a feedback signal indicating the position of the movable element.

[0038] In some embodiments, the position sensor includes a Hall effect sensor.

[0039] According to a second aspect, a method is provided for performing digital image stabilization on a sequence of video frames captured by an image capture device, the image capture device comprising: a motion sensor configured to output a motion signal indicating the motion of the image capture device; an OIS system configured to compensate for the motion of the image capture device by controlling the position of a movable element of the optical image stabilization (OIS) system based on the motion signal; and a position sensor configured to output a position signal indicating the instantaneous position of the movable element, the method being provided. This involves capturing a sequence of video frames while simultaneously obtaining a sequence of motion values ​​sampled from the motion signal, and This involves capturing a sequence of video frames and obtaining a sequence of position values ​​sampled from a position signal, wherein each position value corresponds to each motion value in a sequence of motion values, and as a result, the sequence of motion and position values ​​defines a sequence of motion-position pairs. The sampling rate of the motion and position sequences exceeds the frame rate of the video frame sequence, and as a result, each video frame is associated with a subset of motion-position pairs, and each motion-position pair is associated with a subset of pixel rows in each video frame, and this method For each video frame, a subset of residual motion values ​​is determined, wherein each residual motion value in each subset of residual motion values ​​associated with each video frame is determined based on the motion and position values ​​of motion-position pairs associated with each subset of pixel rows in each video frame, and as a result, a subset of residual motion values ​​is determined that represents the residual motion of the image capture device, which is not compensated by the OIS system, when capturing each subset of pixel rows in each video frame. This involves performing digital image stabilization on a sequence of video frames, Detecting image features within a reference video frame of a sequence of video frames, Tracking image features within each further video frame in a sequence of video frames, Determining frame motion data based on the displacement of image features within each further video frame in a sequence of video frames, Performing digital image stabilization, including generating a sequence of stabilized video frames based on frame motion data. It further includes, The size of the tracking window for tracking image features within a given further video frame is set based on representative residual motion values ​​of a subset of residual motion values ​​associated with the given further video frame.

[0040] In some embodiments, a representative residual motion value is the respective residual motion value associated with a subset of pixel rows in a given video frame having the same set of pixel row indices as a subset of pixel rows in a preceding video frame containing image features.

[0041] In some embodiments, the representative residual motion value is the maximum value of a subset of residual motion values ​​associated with a given further video frame.

[0042] According to a third aspect, a method is provided for performing digital image stabilization on a sequence of video frames captured by an image capture device, the image capture device comprising: a motion sensor configured to output a first motion signal and a second motion signal indicating the motion of the image capture device along a first sensing axis and a second sensing axis of a motion sensor, respectively; an OIS system configured to compensate for the motion of the image capture device by controlling the position of a movable element of an optical image stabilization (OIS) system based on the first motion signal and the second motion signal; and a position sensor configured to output a first position signal and a second position signal indicating the instantaneous position of a movable element along a first sensing axis and a second sensing axis of the position sensor, respectively, the method is as follows: The method involves obtaining a sequence of motion vectors, wherein each motion vector is associated with its respective video frame, and the sequence of motion vectors includes a first motion value and a second motion value sampled from a first motion signal and a second motion signal, respectively, while capturing a sequence of video frames. Obtaining a sequence of position vectors, wherein each position vector includes a first position value and a second position value sampled from a first position signal and a second position signal, respectively, while capturing a sequence of video frames, such that each position vector is associated with its respective video frame. Determining residual vectors for each video frame in order to obtain a sequence of residual vectors for the sequence of video frames, wherein each residual vector includes a first residual motion value based on a first motion and position value of the motion and position vectors, and a second residual motion value based on a second motion and position value of the motion and position vectors, and is determined based on the motion and position vectors associated with each video frame to represent the residual motion of the image capture device that is not compensated by the OIS system when capturing each video frame. This involves performing digital image stabilization on a sequence of video frames, Detecting image features within a reference video frame of a sequence of video frames, Tracking image features within each further video frame in a sequence of video frames, Determining frame motion data based on the displacement of image features within each further video frame in a sequence of video frames, Performing digital image stabilization, including generating a sequence of stabilized video frames based on frame motion data. Includes, The first and second dimensions of the tracking window for tracking further image features within video frames are set based on the first and second residual motion values ​​of the residual vectors in the sequence of residual vectors, respectively.

[0043] In some embodiments, the position of the tracking window for tracking image features within each given further video frame is determined based on the position of the tracking window in a video frame preceding the given further video frame and the residual motion vector associated with the given further video frame.

[0044] In some embodiments, the sampling rate of the motion and position sequences exceeds the frame rate of the sequence of video frames, such that each video frame is associated with a subset of motion and position vector pairs, and each pair of motion and position vectors in the subset of motion and position vector pairs associated with each video frame is associated with a subset of pixel rows in each video frame. Determining a subset of residual vectors for each video frame, wherein each residual vector in each subset of residual vectors associated with each video frame is determined based on the motion vector and position vector of a pair of motion vectors and position vectors associated with each subset of pixel rows of each video frame, The first and second dimensions of the tracking window for tracking image features within each given further video frame are set based on residual vectors associated with a subset of pixel rows in a given further video frame that has the same set of pixel row indices as the subset of pixel rows in a preceding video frame containing the image features.

[0045] In some embodiments, the position of the tracking window for tracking image features in a given further video frame is determined based on the position of the tracking window in a video frame preceding the given further video frame and a residual vector associated with a subset of pixel rows in the given further video frame having the same set of pixel row indices as the subset of pixel rows in the preceding video frame containing the image features.

[0046] According to a fourth aspect, an image capture device is provided, and the image capture device is A motion sensor configured to output a motion signal indicating the movement of an image capture device, An OIS system configured to compensate for the motion of an image capture device by controlling the position of the movable elements of the OIS system based on motion signals, A position sensor configured to output a position signal indicating the position of a movable element, The system comprises a processing device configured to perform a method of the first embodiment or any embodiment thereof.

[0047] According to a third aspect, a computer program product is provided which includes a portion of computer program code configured to perform the method of the first aspect or any embodiment thereof when executed by a processing device.

[0048] In general, any embodiments, features, effects, or advantages described in relation to the first, second, and third embodiments apply in correspondence with the fourth and fifth embodiments.

[0049] The above and additional issues, aspects, embodiments, features, and effects of this disclosure can be better understood through the following illustrative and non-limiting detailed description with reference to the accompanying drawings. In the drawings, unless otherwise specified, similar reference numerals are used for similar elements. [Brief explanation of the drawing]

[0050] [Figure 1] This is a block diagram of an image capture device in one implementation configuration. [Figure 2] This is a block diagram of an image capture device in a further implementation form. [Figure 3] This is a more detailed block diagram of an IS system combining OIS and DIS in one implementation configuration. [Figure 4] This is a flowchart showing how to perform a DIS (Distributed Information System). [Figure 5] This is a flowchart showing the substeps of the method in Figure 4. [Figure 6a] This diagram schematically illustrates image feature detection and tracking referencing a sequence of video frames. [Figure 6b] This diagram schematically illustrates image feature detection and tracking referencing a sequence of video frames. [Figure 6c] This diagram schematically illustrates image feature detection and tracking referencing a sequence of video frames. [Figure 7a] This diagram schematically illustrates a further example of image feature tracking, where, in addition to the size of the tracking window, the position of the tracking window is updated between video frames. [Figure 7b] This diagram schematically illustrates a further example of image feature tracking, where, in addition to the size of the tracking window, the position of the tracking window is updated between video frames. [Figure 8a] This figure schematically illustrates a further example of image feature tracking, where each video frame is associated with a subset of residual motion values. [Figure 8b] This figure schematically illustrates a further example of image feature tracking, where each video frame is associated with a subset of residual motion values. [Modes for carrying out the invention]

[0051] Figure 1 is a schematic block diagram of the image capture device 100. The image capture device 100 may be a video camera. For example, a useful application of the image stabilization approach of this disclosure is an image capture device 100 in the form of a monitoring camera or surveillance camera having video capture capabilities, such as a network surveillance camera (e.g., an Internet Protocol (IP) camera). Thus, the image capture device 100 may be adapted for fixed installation by being mounted on a support structure such as a building structure (e.g., a wall, ceiling, roof, lighting pole, mast, etc.) or other suitable structure for monitoring a scene. However, the image stabilization approach of this disclosure is also applicable to image capture devices suitable for handheld or body-worn image capture and / or mounting on a camera tripod. For brevity, the image capture device 100 may be referred to as camera 100 below without loss of generality.

[0052] Camera 100 includes an optical system 114 and an image sensor 122. The optical system 114 comprises a system of optical elements such as one or more lenses 116, 118, 120. The number of optical elements shown in Figure 1 is merely an example and fewer and more lenses and / or other optical elements are possible. During image capture operation, camera 100 may monitor a scene by using the image sensor 122 to capture video frames F imaged onto the image sensor 122 by the optical system 114, thereby providing a sequence of video frames F of the video of the scene. The video frames F may be captured at a predetermined or variable frame rate suitable for a given monitoring application. The video frames F may be provided to a downstream video processing pipeline and undergo typical video processing operations such as demosaicing and encoding before transmission and / or storage. Each of these examples of post-processing operations may be of a type known in the art and therefore not described further herein.

[0053] As described above, camera motion, such as vibration caused by shaking during image capture, can impair the quality of individual frames and the quality of the video image frame sequence. To compensate for such camera motion, camera 100 is equipped with an image stabilization (IS) system 101 that performs optical image stabilization (OIS), as described below.

[0054] Camera 100 includes a motion sensor 102 configured to output a motion signal m. The motion signal m indicates the instantaneous motion of the motion sensor 102, and therefore of the camera 100. IS system 101 includes an OIS system 104 configured to compensate for the motion of camera 100 based on the motion signal m. OIS system 104 includes a setpoint controller 106, an OIS controller 108, a driver 110, and a movable element. In the illustrated example, OIS system 104 is configured for lens-based OIS, and the movable element is a movable optical element of an optical system 114, exemplified here by a lens 118, which is a movable lens. However, as can be understood, the movable element may also be formed by a group of movable lenses of an optical system 114, or some other optical element. The setpoint controller 106 is configured to determine a control signal c in the form of a setpoint of the OIS controller 108. The setpoint controller 106 may also be called a setpoint generator or setpoint block. The setpoint controller 106 is described in more detail below. The OIS controller 108 is configured to control the position of the movable lens 118 in response to a control signal / setpoint c. The OIS controller 108 is configured to generate an actuation signal u to the driver 110 to actuate the movable lens 118 based on the setpoint c. The driver 110 is then configured to actuate the movable lens 118 according to the actuation signal u, thereby compensating for the vibration motion of the camera 100.

[0055] The driver 110 may comprise, for example, one or more voice coil motor (VCM) actuators, or other suitable conventional high-speed actuators such as comb drives or piezo actuators. The OIS system 104 can typically compensate for motion along a set of compensation axes, such as two or more. Thus, the driver 110 may comprise a corresponding actuator (e.g., a VCM) for each compensation axis of the OIS system 104 to actuate the movable lens 118 to provide compensation along the compensation axis. Thus, each compensation axis of the OIS system 104 can be associated with a corresponding axis of motion of the movable lens 118. The driver 110 may comprise, for example, an actuator (e.g., a VCM) for shifting the position of the movable lens 118. Position here may refer to location (i.e., linear position) and / or rotation (i.e., angle / tilt of one or more lenses). For example, the driver 110 may comprise an actuator for translating the movable lens 118 in a plane intersecting the optical axis of the optical system 114. The driver 110 may additionally or alternatively include an actuator for rotating the movable lens 118 relative to the optical axis. For example, the OIS system 104 may be configured to move the movable lens 118 along two lateral directions in a plane. Thereafter, the OIS system 104 can compensate for changes in the pitch and yaw (defined below) of the camera 100. Other approaches are possible for controlling the position of the movable lens 118, such as moving the movable lens 118 along a curved path (e.g., a parabola) to simultaneously achieve changes in the position and angle of the movable lens 118. However, these are just a few examples, and other approaches are possible for operating movable lenses or other movable optical elements.

[0056] The motion sensor 102 can be any type of sensor capable of sensing motion with respect to at least one sensing axis (for example, around or along the sensing axis) and outputting a motion signal m indicating the motion sensed for each sensing axis. The motion sensor 102 may be configured to sense motion along each of the set of compensation axes of the OIS system 104. Conveniently, the motion sensor 102 may have a corresponding set of sensing axes, and the set of sensing axes may be arranged to align with the set of compensation axes. Thus, the motion sensor 102 may output a motion signal m indicating the instantaneous value of each motion component corresponding to each compensation axis.

[0057] The motion sensor 102 may be configured to sense rotational motion and / or linear motion and to output a motion signal m indicating the sensed rotational motion and / or linear motion. The motion sensor 102 may comprise one or more gyroscopes, one or more accelerometers, or other suitable types of inertial measuring units (IMUs). As used herein, the terms “gyroscope” and “accelerometer” may refer to gyroscopes and accelerometers having one or more sensing axes. For example, a “single” gyroscope or accelerometer may comprise several individual gyroscope or accelerometer sensors, each configured at the physical / hardware level to sense motion around its respective sensing axis. Thus, a “two-axis gyroscope” may comprise two individual gyroscope sensors, each configured to sense angular velocity (e.g., pitch and yaw) around its respective axis. A “three-axis gyroscope” may comprise three individual sensors, each configured to sense angular velocity (e.g., pitch, yaw, and roll) around its respective axis. Similarly, a "three-axis accelerometer" may comprise three individual accelerometers, each configured to sense acceleration along its respective axis (for example, three orthogonal axes having a fixed orientation relative to camera 100). If two or more types of sensors and / or sensing techniques are used, data fusion may be used to combine the individual motion signals from each sensor into a motion signal m indicating motion along one or more sensing axes of motion sensor 102.

[0058] For example, motion sensor 102 may be configured to sense rotational motion as the angular velocity (i.e., rate of change of orientation / rotation) of camera 100 / motion sensor 102 and to output a corresponding motion signal m indicating the sensed angular velocity. Motion sensor 102 may be configured to sense angular velocity around one or more axes, such as pitch, yaw, and / or roll. Pitch may be used here to refer to the pitch angle of the optical axis (i.e., line of sight) of camera 100 in the vertical plane. Yaw may refer to the yaw angle of the optical axis of camera 100 in the horizontal plane. Roll may refer to the roll angle of camera 100 around its optical axis. Angular velocity can conveniently be sensed using a gyroscope. For example, a two-axis gyroscope may be configured to sense the angular velocity of the pitch and yaw angles of camera 100. A three-axis gyroscope may be configured to sense the angular velocity of the pitch, yaw, and roll angles of camera 100. Rotational motion can also be sensed using a pair of sensing axes of a two-axis (or more) accelerometer. The accelerations sensed along a pair of sensing axes can be merged (e.g., integrated and transformed by trigonometric function transformations) into scalar values ​​representing angular velocity around axes orthogonal to the pair of sensing axes. This transformation can be performed by the on-sensor calculation block of the motion sensor 102 or by the off-sensor calculation block of the camera 100. More generally, any sensor configuration that enables the sensing of rotational motion (e.g., a gyroscope and / or accelerometer) can be used. For example, the motion sensor 102, which combines a gyroscope and an accelerometer, may use the gyroscope to sense rotational motion around a first sensing axis and the accelerometer to sense rotational motion around a second sensing axis.

[0059] As described above, vibrational motion tends to generate greater fluctuations in rotation of the camera 100 than in linear translation. Therefore, having a motion sensor 102 configured to sense at least rotational motion may enable more sensitive detection of vibrational motion and, consequently, more effective image stabilization. Accordingly, the following description mainly refers to an implementation of an OIS system that compensates for motion based on a motion signal indicating rotational motion (e.g., angular velocity). However, the following description can also be applied in a manner that accommodates an implementation of an OIS system that compensates for motion based on a motion signal indicating linear motion (e.g., linear velocity or linear acceleration).

[0060] Regardless of the specific implementation of the motion sensor 102, it may be configured to output a motion signal m as either a digital or analog motion signal. If the motion sensor 102 outputs an analog motion signal m, it may be sampled by an analog-to-digital converter (ADC) in a camera 100 located upstream of the OIS system 104 and connected to the analog output of the motion sensor 102. Thus, the analog motion signal can be AD-converted to a digital signal containing (for example, for each component) time-series motion values ​​(i.e., "motion samples") provided as input to the OIS system 104. If the motion sensor 102 outputs a digital motion signal m, it may have an internal ADC and therefore perform an internal AD-conversion of the analog motion signal before it is output via the digital output of the motion sensor 102. Thus, the motion signal m can be output as a digital signal containing (for example, for each component) time-series motion values ​​(i.e., "motion samples") provided as input to the OIS system 104.

[0061] In the illustrated example, the setpoint controller 106 is shown to receive the motion signal m directly from the motion sensor 102. However, the motion signal m may typically undergo A / D conversion (if the motion sensor 102 has an analog output) and / or filtering (for example, by a filtering step including integration and / or low-pass filtering of the motion signal m) before being received by the setpoint controller 106.

[0062] In the illustrated example, the OIS controller 108 of the OIS system 104 is implemented as a closed-loop controller. Thus, the OIS system 104 further includes a position sensor 112 configured to sense the instantaneous position of the movable lens 118 (e.g., the linear position and / or angle / tilt of one or more lenses) and provide a corresponding position signal ν as a feedback signal to the OIS controller 108. Thus, the position sensor 112 may be located in the feedback path of the OIS controller 108 and configured to output a feedback signal indicating the position of the movable lens 118. The position sensor 112, like the motion sensor 102, may have one or more sensing axes and therefore be configured to sense / measure the position of the movable lens 118 (or more generally, the movable element) with respect to each of its sensing axes. Thus, for each sensing axis of the position sensor 112, the position sensor 112 may provide a respective position signal indicating the instantaneous position of the movable element / movable lens 118 with respect to the sensing axis. The position sensor 112 may include, for example, a Hall effect sensor that includes one Hall sensor element for measuring the position of the movable lens 118 along each respective sensing axis.

[0063] Figure 2 is a block diagram of an alternative implementation of the image capture device 100, which includes a sensor-based OIS system 104' instead of the lens-based OIS system 104 as shown in Figure 1. Thus, the OIS system 104' in Figure 2 is configured to provide OIS by controlling the position of the image sensor 122. The driver 110 of the OIS system 104', similar to the driver 110 of the OIS system 104 in Figure 1, may be implemented using a set of actuators, such as a VCM actuator, to control the position of the image sensor 122 on the imaging plane and / or the tilt angle of the image sensor 122. The OIS system 104' may further implement closed-loop control and include a position sensor 112 (realized by, for example, a Hall sensor and / or an optical sensor) to provide a position signal ν indicating the instantaneous position of the image sensor 122 as a feedback signal to the OIS controller 108. The description of Figure 1 applies similarly to Figure 2 in other respects, and therefore, to avoid excessive repetition, refer to the above for the description of the elements of the corresponding number.

[0064] For simplicity, an alternative implementation is shown here, but it is also possible to implement OIS using a combination of lens-based and sensor-based OIS. For example, the movable optical element (e.g., corresponding to lens 118 in Figure 1) and the image sensor 122 may be located within a common camera module, and the OIS may be implemented by controlling the position and / or angle of the camera module, i.e., as a single unit.

[0065] The IS system 101 shown in Figures 1 and 2, respectively, further comprises a digital image stabilization (DIS) system or module 124 (in addition to the OIS systems 104 and 104', respectively). The DIS system 124 may apply post-processing to the captured video frames F to compensate for residual motion remaining after compensation by the OIS system 104 and to output a sequence of stabilized video frames F'. The DIS system 124 may be included in the video processing pipeline described above. The DIS system 124 may typically be implemented early or at least early in the pipeline so that subsequent post-processing can benefit from the stabilization achieved by the DIS. The DIS performed by the DIS system 124 is based on tracking one or more image features across the sequence of video frames F and applying image transformations to the video frames F so that the tracked image features are kept stable within the image regions of consecutive video frames. According to this disclosure, in order to facilitate computationally efficient DIS and reduce the amount of pixel data to be processed, the DIS system 124 provides an estimate of the instantaneous OIS error by considering the residual motion value r and adapting the size of the tracking window used to track each image feature based on the residual motion value r. Implementation forms of the DIS and the DIS system 124 are described further below.

[0066] Figure 3 provides a more detailed block diagram of the IS system 101, with a particular focus on the OIS system 104. While Figure 3 shows the lens-based OIS system 104 corresponding to Figure 1, the explanation applies equally to the image sensor-based OIS shown in Figure 2, as well as the combined lens and sensor-based OIS.

[0067] Without loss of generality, the OIS system 104 in Figure 3 is described with reference to a motion sensor 202 implemented by a gyroscope. A single sensing and compensation axis, e.g., pitch or yaw, is further referenced. Thus, for the purposes of the following description, it is assumed that the motion sensor 202 outputs a motion signal ω indicating the angular velocity (e.g., rate of change of pitch or yaw) of the change in orientation of the motion sensor 202 around its sensing axis. It is further assumed that the sensing axis of the motion sensor 202 is aligned with the compensation axis of the OIS system 104 such that the motion of the motion sensor 202 relative to its sensing axis can be compensated by the corresponding motion / actuation of a movable element (e.g., lens 118 or image sensor 122) relative to its motion axis.

[0068] The OIS system 104 includes a closed-loop control system 210 comprising an OIS controller 108. The OIS controller 108 can be implemented as a PID controller. In principle, simpler implementations of the OIS controller 108, such as a PI controller, are also possible. However, given the fast response typically required for effective OIS, it is usually beneficial to use the P, I, and D components, respectively.

[0069] Block 224 represents the controlled system of the control system 210, and referring to Figure 1, it may represent the movable lens 118 (e.g., the movable lens) and the driver 110 that acts on the movable lens 118. In the case of an image sensor-based OIS, such as the OIS system 104' in Figure 2, block 224 may instead represent the image sensor 122 and the driver 110 that acts on the image sensor 122. The controlled parameter (i.e., process variable) of the control system 210 is the position of the movable lens 118, denoted as s. The position is measured by the position sensor 112 (e.g., a Hall sensor), as described above, and provided as a feedback signal ν. The feedback signal v is subtracted from a setpoint c received from the setpoint controller 106 to generate an error signal e for the OIS controller 108.

[0070] A general description of the operations performed by the OIS system 104 to execute OIS during an active state is provided below while referring to FIG. 3. These operations may be particularly performed during the capture of a video sequence of video frames F.

[0071] The OIS system 104 sequentially acquires a time series (i.e., a sequence) of motion values (i.e., “angular velocity samples” or “motion samples”) of the motion signal ω from the motion sensor 202. For convenience, hereinafter, it is assumed that the motion signal ω is a digital motion signal, and thus, the motion samples and the motion signal can be referred to using the same label ω. If necessary to facilitate the explanation, for a given time point t = t i acquired at (i.e., sampled from the motion signal ω at the sampling time point), the motion sample ω can be denoted as ω(t i ). Here, the parameter i is an integer index for a given time / sampling time point such that t i = i × Δ t and Δ t is the sampling interval of the motion signal / motion sample ω, and t0 is an arbitrary reference time point. Correspondingly, the time series of the motion sample ω acquired by the OIS system 104 at a given time point t = t i can be denoted as ω(t i ). The term “sampling interval” (interchangeably “sampling period”) is used in the ordinary sense of the word in the present disclosure to refer to the time interval or time period between sampling time points, i.e., the reciprocal of the sampling rate. The sampling rate of the sample ω (e.g., the sampling rate of the gyro) may be in the range, for example, from several kHz to 10 kHz or higher.

[0072] The time series ω of the samples may optionally be stored in a buffer (not shown individually) of the OIS system 104. The buffer may be implemented, for example, as a first-in-first-out (FIFO) buffer. Thus, when the buffer is at time point t i-1 [ the time series ω(t i-1Assuming that the conditions are met, at time t i So a new sample ω(t i When you obtain the time series ω(t i-1 ) is a time series ω(t i-1 By discarding the oldest (first) sample ω of ) a new sample ω(t i ) may be updated with new sample ω(t i ) is the latest (last) sample ω(t i ) as a time series ω(t i-1 The remaining samples are added to the updated / current time series ω(t) of the motion samples. i ) can form.

[0073] The motion sample ω acquired by the OIS system 104 then passes through a filtering stage 204 located upstream of the setpoint controller 106. The filtering stage 204 comprises an integrator 206 and a low-pass filter 208 that integrate and filter the motion / angular velocity sample ω over time, respectively, to generate a time series of orientation values ​​(i.e., "orientation sample" or "angle sample"). The time series of orientation samples output by the filtering stage 204 may be denoted as θ below, while individual orientation samples ("angle samples") may be denoted as θ. Similar to the description of the motion sample ω, the motion sample ω(t i The azimuth sample derived from ) is θ(t i ) can be shown to mean that θ(t i ) is the time of the motion signal ω / sampling time t i The azimuth samples obtained for this are shown. Correspondingly, given time / sampling time t=t i The time series of the azimuth sample θ obtained by the OIS system 104 is θ(t i ) can be shown as follows. The orientation values / samples can be generated at the same rate as the sampling rate of the motion signal ω, such that the time series ω of the motion samples and the time series θ of the orientation samples have equal sampling rates (i.e., the sampling interval between each of their samples is the same for both time series).

[0074] To reduce the sensitivity of the motion signal ω to noise, integrator 206 is implemented as a leakage integrator. Therefore, integrator 206 operates according to the following equation, time t = t i+1 =t i +Δ t It is possible to calculate updated azimuth / angle samples for [the specified location]. θ(t i +Δ t ) = C × (θ(t i )+ω(t i +Δ t )×Δ t ) (Formula 1) Here, Δ t ω is the sampling interval of the motion signal ω, and C is the “leakage” integral. The integral C can be set to a value in the range of 0.99 to 0.9999, for example, as a non-restrictive example. The specific value may be a design choice made considering factors such as the amount of noise in the motion signal ω and the desired responsiveness of the OIS system 104. The filtering stage 204 may further include a low-pass filter 208, as shown in the figure. The low-pass filter 208 is shown here as a post-processing step for the integral 206, but the low-pass filtering may be performed alternatively or additionally before the integral 206. In either case, the low-pass filter 208 can further suppress noise and therefore reduce noise sensitivity.

[0075] The setting value controller 106 is located downstream of the filtering stage 204 to sequentially acquire the azimuth sample θ (e.g., integrated and typically low-pass filtered) of the time series θ of the azimuth sample output by the filtering stage 204. The setting value controller 106 may optionally include internal buffers (not shown individually), which may be implemented by, for example, a FIFO buffer. For the purposes of this explanation, the setting value controller 106 may be assumed to acquire the azimuth sample θ at the same sampling rate as the motion sample ω is acquired from the motion sensor 202. However, it is also possible to configure the setting value controller 106 to acquire the azimuth sample θ from the filtering stage 204 at a lower rate. That is, the setting value controller 106 may perform downsampling of the time series θ of the azimuth sample by a fraction of its sampling rate (e.g., 1 / 2 or 1 / 4). Generally, the sampling rate of the setting value controller 106 may depend on factors such as the amount of memory available for buffering the azimuth sample θ, the rate at which the setting value c is updated in the control loop of the OIS system 104 to provide the desired response, and the processing speed of the processor circuit implementing the setting value controller 106.

[0076] The change between a pair of consecutive samples θ in a time series (for example, θ(t i-1 ) and θ(t i )) is, t i-1 and t i This indicates the angular displacement of camera 100 between (i.e., around the sensing axis of motion sensor 202). Thus, in response to acquiring a sample θ (e.g., a new / updated / next sample θ), the setpoint controller 106 determines an updated setpoint (e.g., a new / updated / next setpoint) that forms a control signal for the OIS system 104 based on the acquired sample θ. Various implementation forms of the setpoint controller 106 are possible.

[0077] For example, the setting value controller 106 may implement a function between angle and position to convert the acquired sample θ, which in this example is an angle, into a corresponding position value for a movable element, such as the movable lens 118 or the image sensor 122. More specifically, the function between angle and position may map the sample θ (representing the instantaneous orientation of the camera 100) to a setting value c representing the position of the movable element (e.g., the lens 118 or the sensor 122). Thus, the orientation sample θ can be mapped to the coordinate system of the position value v output by the position sensor 112. The specific form of the function between angle and position depends on the design of the OIS system 104, the position of the movable element relative to the pivot point of the angular displacement indicated by the sample θ, the geometric relationship between the sensing axis and the compensation axis, and so on. This conversion can usually be achieved by multiplying the orientation sample θ by a predetermined conversion coefficient (constant). As part of an OIS system, a suitable approach for converting angular displacement measured by a motion sensor (e.g., a gyroscope) into the linear position of a movable compensation element measured by a position sensor (e.g., a Hall sensor) is known in the art and can therefore be implemented by a setpoint controller 106.

[0078] The amount of compensation that needs to be applied in response to a given change in the orientation of camera 100 (i.e., the required translation of the movable elements of the OIS system 104) further depends on the focal length of the camera's optical system (e.g., optical system 114). Therefore, if the optical system has a zoom lens, the function between angle and position may further take into account the current zoom level L of the optical system. The current zoom level L may be provided by a zoom level block 107, as shown in Figure 3. For a computationally efficient implementation, the setpoint controller 106 may obtain a gain value from a predetermined lookup table (e.g., stored in the memory of the OIS system 104) that associates each of several zoom level entries with a predetermined gain value. The obtained gain value may be a predetermined gain value associated with the (e.g., nearest) zoom level entry corresponding to the current zoom level L. The setpoint controller 106 may, accordingly, multiply the position value given by the function between angle and position by the obtained gain value. The result may be output as the next setpoint c to the control system 210 comprising the OIS controller 108.

[0079] In the above example, the setpoint c is determined by applying a function between angle and position to the azimuth sample θ obtained from the filtering stage 204, but a more sophisticated implementation of the setpoint controller 106 is also possible. For example, the setpoint controller 106 could determine the next azimuth sample θ(t i+1 To estimate the direction sample value θ(t), a Kalman filter or other predictive filter can be implemented. i+1 The θ can then be transformed using a function between angle and position as described above, and the transformed value can be output as the setpoint c. It is also possible to first apply a function between angle and position to the acquired sample θ, and then apply a prediction filter to the transformed sample to determine the setpoint c.

[0080] The input of a setpoint c to the control system 210 is summed with the inverted position signal v (the negative of the position signal v, i.e., -v) output by the position sensor 112, as shown in block 222 of Figure 3, to generate an error e representing the tracking error of the control system 210. This error e represents the tracking error with respect to the position of the movable element (i.e., instantaneous tracking error). The error e forms an input to the OIS controller 108, which in response generates an actuation signal u to cause a driver (e.g., driver 110) to actuate the movable element of the controlled system 224 (e.g., lens 118 or sensor 122). For example, if the OIS controller 108 is a PID controller, the actuation signal u may be determined as the sum of the P, I, and D components based on the error e. Any other suitable conventional approach for generating the actuation signal u based on the error e in a closed-loop controller may be used.

[0081] Similar to the notation introduced above for motion signals and motion samples ω, position signals and samples of position signals can be referred to using the same label v. Furthermore, if necessary for the sake of clarity, given a time t=t i The position sample v obtained (i.e., sampled from the position signal ω at the sampling time) is given by the following v(t i ) can be shown to mean that, given time t=t i The time series v of the position sample acquired / sampled from the position signal is v(t i ) can be shown as follows.

[0082] In the above example, the setpoint controller 106 performs a conversion between angle and position to determine the setpoint c with respect to the setpoint of the movable element's position. However, other implementations are possible. For example, the setpoint controller 106 may alternatively be configured to output the setpoint c in the angular domain of the azimuth sample θ. Alternatively, a block between angle and position may be provided in the feedback path of the closed-loop control system 210 to convert the position sample / position value v output by the position sensor 112 to the corresponding angle. In this case, the error signal e input to the OIS controller 108 represents the tracking error of the control system 210 in the angular domain. The OIS controller 108 may accordingly perform a conversion between angle and position to generate an actuation signal u.

[0083] As described above, the IS system 101 comprises a DIS system 124 that supplements the OIS with DIS, in addition to the OIS system 104 or 104'. The DIS is based on a residual motion value r (hereinafter interchangeably referred to as "residual value r", "residual sample r", or simply "residual r") which provides an estimate of the instantaneous OIS error. To determine the residual r, the IS system 101 comprises a residual calculation block 230 (hereinafter interchangeably referred to as "residual block 230"), as shown in Figure 3. Hereinafter, an implementation form of method 300 for controlling the DIS system 124 based on the residual r determined by the residual block 230 is disclosed with further reference to the flowcharts in Figures 4 and 5. It should be noted that the steps of method 300 described below are performed while the camera 100 is capturing a sequence F of video frames and the OIS system 104 is active and therefore actively performing OIS to compensate for the vibrational motion of the camera 100 based on the motion signal ω.

[0084] In step S301, the IS system 101 acquires a sequence / time series of motion values / samples ω from the motion sensor 202. The motion samples ω pass through the filtering stage 204 to derive the corresponding directional samples θ, as described above. Thus, the directional samples θ are derived from the motion signal ω by integrating (and optionally low-pass filtering) the motion / angular velocity samples ω. In step S302, the IS system 101 acquires a sequence / time series of position values / samples v sampled from the position sensor 112. The directional samples θ and position samples v acquired in steps S301 and S302 are sequentially input to the residual block 230. Next, in step S303, the residual block 230 determines a sequence / time series of residuals r based on the respective time series of samples θ and position samples v.

[0085] Figure 4 shows steps S301, S302, and S303 in succession, but it should be understood that these steps are performed in parallel, and further in parallel with the capture of the sequence of video frames F. Thus, the azimuthal sample θ and positional sample v are acquired in parallel so that each video frame is associated with at least one respective azimuthal sample θ and positional sample v. Thus, the residual block 230 can determine at least one residual r for each video frame so that each video frame is associated with at least one residual r. To say that a sample or value (such as an azimuthal sample θ, a positional sample v, or a residual value r) is "associated" with each video frame or a given video frame means, as used herein, that the value / sample is temporally aligned with the video frame. More specifically, the value / sample is at a time / sampling point t that overlaps with or coincides with the recording / capture of the video frame. i When a value / sample is obtained for a given video frame, it can be considered to be temporally aligned with that frame.

[0086] Each residual r is determined based on the respective pairs of azimuthal sample θ and positional sample v associated with the same video frame. More specifically, each pair of azimuthal sample θ and positional sample v may refer to temporally corresponding or temporally aligned pairs of samples θ and v, i.e., each pair of samples θ and v acquired simultaneously from the respective signals so that they reflect the state of the camera 100 and OIS system 104 at the same time, or at least substantially the same time within the limitations of the temporal resolution defined by the sampling rates of the motion sensor 202 and the position sensor 112.

[0087] To simplify the explanation, below, the sampling rates of the time series of the motion sample ω, orientation sample θ, and position sample v are equal, and furthermore, each of their samples is the motion sample ω(t i For each time / sampling point of ), the corresponding orientation sample θ(t i ) and position sample v(t i It is assumed that the samples are approximately aligned in time, such that ) exist. Therefore, the residual r is given by the motion sample ω(t i ) and position sample v(t i Based on ), a given time point / sampling time t i It is decided about this, and accordingly, r(t i) can be shown as follows. Typically, the sampling rates of the time series ω, θ, and v may exceed the frame rate of the sequence of video frames F (otherwise, the OIS system 104 may not be able to compensate for the vibrational frequencies that cause blurring of individual video frames). Thus, each video frame can typically be associated with each subset of the motion samples ω, each subset of the orientation samples θ, and each subset of the position samples v. However, to further facilitate understanding of the general principles of the DIS approach according to this disclosure, a further simplification of assumptions is made, such that the sampling rates of the time series ω, θ, and v are equal to the frame rate of the sequence of video frames F, and as a result, one motion sample ω, one orientation sample θ, and one position sample v can be associated with each video frame, and therefore one residual r can be determined for each video frame. However, the following explanation is also applicable to a scenario where the sampling rates of the time series ω, θ, and v exceed the frame rate of the sequence of video frames F, but the residual block 230 downsamples the time series θ and v and obtains downsampled counterparts of the time series θ and v, respectively, at a sampling rate equal to the frame rate of the sequence of video frames, so that the residual block 230 obtains one azimuthal sample θ and one positional sample v for each video frame, and the residual block 230 can determine one residual r for each video frame. These assumptions are intended to facilitate explanation and understanding, but the DIS system 124 may be better suited to compensating for lower vibration frequencies than those compensated by the OIS system 104, and therefore may be sufficient in any case to determine the residual r at the frame rate of the video sequence F.

[0088] Given the above assumptions, the residual block 230 sequentially determines new residuals r, thereby providing a time series or residual r over time, with each residual r associated with its respective video frame. To facilitate the determination of residuals r representing or indicating the OIS error at a given point in time, the residual block 230 includes an angle-position block 232. The angle-position block 232 performs a transformation similar to the angle-position function described above, with reference to the setpoint controller 106. Thus, the angle-position block 232 maps the azimuthal sample θ to the coordinate system of the position sample v output by the position sensor 112. The output of the angle-position block 232, i.e., the mapped representation of a given azimuthal sample θ, is shown as v' in Figure 3. The residual r is then determined (for example, by the difference block 234) as the difference between the mapped position sample v' and its associated (temporally aligned) position value v. In Figure 3, the residual r is determined by subtracting v from v', and vice versa. Generally, the magnitude of the residual r is sufficient for the purpose of utilizing it during DIS, as will be explained below. Therefore, the residual r can be determined as the absolute value of the difference between v and v'. Furthermore, in Figure 3, the residual block 230 implements a function between angle and position to map the azimuthal sample θ to the coordinate system of the position sample v, but the residual block 230 can also implement a function between angle and position to map the position sample v to the corresponding angle, i.e., to the coordinate system of the azimuthal sample θ. Moreover, it is possible to map both the azimuthal sample θ and the position sample v using their respective transformations adapted so that the azimuthal sample θ and the position sample v are mapped to some other common coordinate system, and the residual r can be determined by determining the difference between the mapped azimuthal and position samples in the common coordinate system. For example, the azimuthal sample θ and the position sample v can each be mapped to their respective pixel coordinates such that each residual represents the instantaneous OIS error in pixel units.

[0089] The explanation here primarily focuses on determining the residual r based on the azimuth sample θ and position sample v (derived from the angular velocity motion sample ω). However, it is intended that the residual r can also be determined in a similar manner in IS systems that utilize accelerometers to provide motion signals. Thus, the linear position sample can be derived from the linear acceleration sample (for example, by performing a double integral of the linear acceleration sample in the filtering stage 204). Therefore, the residual r can be determined for each respective video frame based on the derived linear position sample and the position sample v obtained from the position sensor 112 and associated with each video frame. The angle-position function used by the residual block 230 described above is replaced in this case by some other suitable transformation adapted to map the linear position sample and position sample v to a common coordinate system.

[0090] Returning to Figure 3, the residual r determined by the residual block 230 is provided to the DIS system 124, as shown, and the DIS system 124 performs DIS on the sequence of video frames F in step S304, as will be further described below. Step S304 includes several substeps, which will be further described with reference to the flowcharts in Figures 5 and 6a-6c, which schematically show each of the video frames 400-1, 400-2, and 400-3 of the sequence of video frames F. Video frame 400-1 defines a reference video frame, while video frames 400-2 and 400-3 define each of the further video frames in the sequence of video frames. The reference video frame 400-1 may here be the first video frame of a subsequence of video frames within a context window of video frames (for example, defined by several video frames) in which one or more image features are tracked as part of the DIS, as will be described below.

[0091] In step S3041, the DIS system 124 detects, i.e., identifies, an image feature 402 in the reference video frame 400-1, as shown in Figure 6a. The image feature 402 is schematically shown here as a single feature. The feature may correspond, for example, to a corner detected in the video frame 400-1. However, the image feature 402 is not limited to a corner and may be an edge, a blob, etc., depending on the type of feature detection algorithm. For example, the image feature 402 may be detected using an edge detection algorithm, a corner detection algorithm (e.g., Shi-Tomasis corner detection or Harris corner detection), Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), or some other suitable feature detection algorithm. As will be further explained below, step S3041 is not limited to detecting a single image feature (e.g., a single feature), but may typically include detecting two or more image features that are tracked individually, and image feature 402 may be called the first image feature 402. Feature detection may be performed by feature detection blocks (not shown individually) of the DIS system 124. Feature detection may be applied to the entire image area of ​​the reference video frame, or selectively to the region of interest. Since DIS usually involves cropping, the region of interest may be, for example, centered within the reference video frame and matching or smaller in size to the size of the cropped area.

[0092] In step S3042, the DIS system 124 proceeds to track the (first) image features 402 in further video frames, as shown for further video frames 400-2 and 400-3 in Figures 6b and 6c. An image feature 402 detected in reference video frame 400-1 (e.g., a corner or other feature) may be tracked accordingly across further consecutive video frames, including video frames 400-2, 400-3 (and further consecutive video frames in the context window). Feature tracking may be performed by feature tracking blocks of the DIS system 124 (not shown individually). Image features 402 may be tracked using any conventional suitable feature tracking algorithm. Examples include object trackers such as Kalman filtering-based object trackers, and feature tracking algorithms based on optical flow analysis such as the Lucas-Kanade method. What is common when applying any of these feature tracking algorithms in this method is that the tracking algorithm is limited to performing feature tracking within a tracking window associated with the detected image features 402. Furthermore, the size of the tracking window is set based on the residual r associated with the video frame, which is determined by the residual block 230.

[0093] Figure 6a schematically shows the tracking window 404-1 containing image feature 402. For simplicity, the tracking window 404-1 is a square with a size defined by width W1. Here, we assume that the reference video frame 400-1 is the first video frame, so the tracking window 404-1 has an initial size W=W initIt can be set to W=W. Note that the size of the tracking window 404-1 relative to video frame 404 is not drawn to a constant scale and usually covers a considerably smaller area of ​​video frame 404 than illustrated. Appropriate initial sizes for the tracking window 404-1 may be 8x8 pixels, 16x16 pixels, 32x32 pixels, 64x64 pixels, or 128x128 pixels, as some non-limiting examples. Non-square rectangular tracking windows 404-1 are also possible. In general, the initial size of the tracking window 404-1 may depend on factors such as the resolution of the video frame, available memory, and processing resources. Fixed initial size of tracking window 404-1 W=W init Alternatively, the size of the tracking window 404-1 may also be based on the residual r=r1 associated with the reference video frame 400-1 such that W=W(r1), as described below for further video frames 400-2, 400-3. In either case, the tracking window 404-1 may be positioned within the video frame 404-1 so as to surround the image feature 402. The position of the tracking window 404-1 may be determined, for example, by setting the center of the tracking window 404-1 to align with the position of the image feature 402. For example, if the image feature 402 is a feature such as a corner, the center of the tracking window 404-1 may be set to the pixel coordinates of the image feature 402. For more general and / or complex image features, the center of the tracking window 404-1 may be set to the pixel coordinates of the center or centroid of the image feature 402.

[0094] In Figure 6b, feature tracking proceeds by applying a tracking algorithm to track image features 402 within a tracking window 404-2 of further video image frames 400-2. The size of the tracking window 404-2 is set here based on the residual r=r2 associated with video frame 400-2 such that W2=W(r2), where W is a function of the residual r. Various forms of the function W are possible. Typically, the function W is defined such that the size of the tracking window (e.g., width) increases with increasing residual r (e.g., with increasing magnitude of residual r). In a simple implementation, the function W may be a step function that sets the tracking window to a first size in response to residual r being below a threshold, and sets the tracking window to a second size larger than the first size in response to residual r being above a threshold, for example, the following equation: TIFF2026113414000002.tif10170

[0095] A more refined size function may provide a mapping between the residual r and multiple sizes based on a linear equation such as the following: W(r) = max{W0 + A × r, W min} (Formula 3) Here, W0 and A are some appropriate selection of coefficients, and W min is a predetermined minimum tracking window size. The scaling factor A can be understood, for example, as reflecting the reliability of the residual r in accurately capturing the magnitude of the OIS error. The scaling factor A can be determined by considering factors such as the overall performance and speed of the OIS system 104, the noise of the motion samples (e.g., gyro samples ω), and calibration errors. To give an integer window size, it may be convenient to apply rounding operations to the linear equation (e.g., by rounding to the nearest integer, or by using some other rounding function such as a floor function or ceiling function). Equation 3 gives the tracking window size if the size is a predefined maximum size W maxIt can be further restricted to ensure that it does not exceed a certain value. Furthermore, while the above example adapts the size by setting the width W of a square tracking window, it is also possible to define the function W(r) to adapt both the height and width of a rectangular tracking window. Note that these examples are just a few non-restrictive examples, and other definitions of the function W(r) are possible.

[0096] Regardless of the specific form of the function W(r), the position of the tracking window 404-2 may be set to the same coordinates as the tracking window 404-1 of a previously processed video frame, which in the case of video frame 400-2 is the reference video frame 400-1. Once a tracking window 404-2 is defined for video frame 400-2, feature tracking may be applied to pixels within the tracking window 404-2.

[0097] For example, image feature 402 can be tracked using optical flow analysis. Therefore, optical flow analysis is selectively applied to pixels within the tracking window 404-2 of video frame 400-2. That is, optical flow analysis can estimate the optical flow of pixels within the tracking window 404-2 relative to the preceding video frame 400-1 (e.g., reference video frame 400-1). The position of image feature 402 in video frame 400-2 can then be estimated by updating the pixel coordinates of image feature 402 in the preceding video frame 400-1 according to the estimated optical flow, for example, by adding the optical flow vector estimated by optical flow analysis to the coordinates of image feature 402 in the preceding video frame 400-1. For image feature 402 containing two or more pixels (e.g., a blob), it is possible to update the pixel coordinates of each pixel of image feature 402. However, it is also possible to update only representative pixel coordinates of image feature 402, such as the pixel coordinates of the center or centroid of image feature 402. To further reduce computational complexity, the pixel coordinates of image feature 402 can be updated using the average optical flow determined for pixels within the tracking window 404-2.

[0098] A similar approach can be applied to continue tracking image features 402 within subsequent video frames 400-3. When the tracking operation is applied to further video frames 400-3, the size of the tracking window can be set, for example, according to equation 2 or 3, based on the residual r=r3 associated with video frame 400-3. Furthermore, the position of the tracking window 404-3 can be set to the same coordinates as the tracking window 404-2 of the previously processed video frame 400-2. Thus, both the size and position of the tracking window can be updated for each subsequent video frame 400-2, 400-3 and beyond. However, variations of this approach are also possible.

[0099] For example, the size of the tracking window may be updated for each further video frame 400-2, 400-3 based on their associated residuals r2 and r3, but the position of the tracking window is fixed to the position determined for tracking window 404-1 within reference video frame 400-1. Furthermore, it is not necessary to update the size of the tracking window for each further video frame 400-2, 400-3, etc. Instead, the size of the tracking window may be updated only every other frame, every four frames, or even less frequently. Also, the size of the tracking window for all video frames in the context window may be set to W based on the residual r=r1 of reference video frame 400-1. j It is also possible to determine that =W(r1), where j is the index across video frames in the context window.

[0100] The tracking operation described above, with reference to video frames 400-2 and 400-3, may be repeated until image feature 402 is tracked in all further video frames within the context window. At this stage, steps S3041 and S3042 may be repeated for further subsequences of video frames in a new context window.

[0101] As described above, the tracking window 404-1 can already be determined for the reference video frame 400-1. However, since feature tracking does not need to be applied to the reference video frame 400-1, it is not necessary to define the tracking window 404-1 again for the reference video frame 400-1. Instead, the tracking window can be initially initialized for a first further video frame following the reference video frame 400-1, where the image feature 402 detected in the reference video frame 400-1 should be tracked. In the illustrated example, this is the further video frame 400-2. In this case as well, the size of the tracking window 404-2 can be configured based on its associated residual r=r2, for example, so that W2=W(r2). The position of the tracking window 404-2 can be determined so that its center aligns with the center or centroid of the image feature 402 in the reference video frame 400-1. That is, the center coordinates of the tracking window 404-2 can be set to the coordinates of the image feature 402 in the reference video frame 400-1.

[0102] Tracking the image feature 402 in step S3042 allows for the estimation of the displacement d of the image feature 402 between consecutive video frames in the context window. Figure 6b schematically shows the displacement d of feature 402 in video frame 400-2 (filled circle) relative to the position of feature 402 in the preceding video frame 400-1 (unfilled circle). Figure 6c schematically shows the displacement d of feature 402 in video frame 400-3 (filled circle) relative to the position of feature 402 in the preceding video frame 400-2 (dashed outline circle). Note that the displacement d is shown schematically only and is not drawn to scale. For example, in the case of a sequence of video frames captured at a frame rate of 25 frames per second or higher, the displacement d between a pair of consecutive frames caused by camera vibrations not compensated by the OIS system 104 may be as small as a few pixels. However, in the case of large amplitude camera shakes, a considerably large displacement d may occur frame by frame. In Figures 6b and 6c, the displacements d of image feature 402 in video frames 400-2 and 400-3 are shown relative to the position of image feature 402 in the preceding video frame, but the displacements d of image feature 402 in video frames 400-2 and 400-3 can also be determined relative to the position of image feature 402 in reference video frame 400-1. In either case, the determined set of displacements d determined for the video frames (e.g., in the context window) can be collected as displacement data.

[0103] Therefore, in step S3043, the DIS system 124 proceeds to determine frame motion data based on displacement data, which includes the respective displacements d for each image feature 402 in each further video frame 400-2, 400-3 of the context window. Determining frame motion data may include determining inter-frame motion data from the displacement data that shows the estimated inter-frame motion of the camera 100.

[0104] Determining the frame motion data (used to form a stabilized video sequence in step S3044 described later) to increase the likelihood that it reflects motion associated with the vibrational motion of camera 100, which is desirable to be compensated for by DIS, may involve filtering displacement data, including displacement d. For example, the displacement data may be subjected to high-pass or band-pass filtering to extract frequency components of interest from the displacement d over a sequence of video frames F.

[0105] In the above, the steps of image feature detection (S3041) and image feature tracking (S3042) are described with reference to a single first image feature 402. However, step S3041 may include detecting two or more image features within reference video frame 400-1, and therefore step S3042 may include tracking each of the detected image features across further video frames 400-2, 400-3. This is schematically illustrated in Figures 6a-6c by a second image feature 408. Thus, in step S3041, a first image feature 402 (e.g., a first feature corresponding to a first corner) and a second image feature 408 (e.g., a second feature corresponding to a second corner) are detected within reference video frame 400-1. The first and second image features 402, 408 may be located, for example, within a region of interest in reference video frame 400-1, or in spaced-out pixel regions within different, non-overlapping regions of interest in reference video frame 400-1. For example, the first and second image features 402, 408 may correspond to the respective spaced objects shown in reference video frame 400-1. The first and second image features 402, 408 are, in particular, given a size (e.g., W init , W large or W maxThey can be spaced apart so that they cannot be accommodated within the same tracking window. Thus, as shown in Figure 6a, the second image feature 408 is located within tracking window 406-1, which does not overlap with tracking window 404-1. Otherwise, the description of tracking window 404-1 above applies to tracking window 406-1 accordingly.

[0106] In step 3042, the DIS system 124 proceeds to track the first image and the second image features 402 and 408, respectively, across further video frames 400-2 and 400-3, as shown in Figures 6b and 6c. The tracking is performed independently for each of the first image and the second image features 402 and 408. Thus, similar to the tracking of the first image feature 402 described above, the tracking of the second image feature 408 is performed similarly within the respective tracking windows 406-2 and 406-3 of the further video frames 400-2 and 400-3. The approach for setting the size of the tracking windows 404-2 and 404-3 described above can be applied in a similar manner to setting the size of the tracking windows 406-2 and 406-3.

[0107] Based on tracking the second image feature 408, a corresponding displacement d' can be determined for the second image feature 408. Displacements d and d' may each be collected as displacement data. Thus, in step S3043, the DIS system 124 proceeds to determine frame motion data based on displacement data, including each of the respective displacements d and d', for the first and second image features 402 and 408 in each further video frame 400-2, 400-3 of the context window. The advantage of determining frame motion data based on displacement data derived from tracking two or more image features is that statistics can be applied to determine reliable frame motion data. For example, determining frame motion data may involve averaging the respective displacements d and d' determined for each image feature 402 and 408 in each further video frame 400-2, 400-3. The averaged displacement data may then be filtered (e.g., high-pass or band-pass) as described above to obtain filtered final frame motion data that will form the basis for subsequent DIS.

[0108] As can be understood, a similar approach may be applied to a set of image features containing any number of image features, such as two, three, or more, in step S3042, each image feature may be tracked individually using its own tracking window, and in step S3043, frame motion data may be determined based on the respective displacement of each tracked image feature, for example, by averaging the respective displacements of each tracked image feature for each further video frame 400-2, 400-3. For example, step S3041 may include detecting the N strongest corners in reference video frame 400-1, and thus in step S3042, we proceed to track each of the N corners individually.

[0109] In step S3044, the DIS system 124 proceeds to generate a sequence of stabilized video frames F' based on frame motion data. Various image processing-based techniques for performing digital image stabilization on a sequence of video frames based on frame motion data showing frame-by-frame motion are known in the art themselves. As one non-limiting example, the DIS may generate a sequence of stabilized video frames F' by applying an image transform to a sequence of video frames 400-1, 400-2, 400-3 in the form of image trimming, where the trimming position within each video frame 400-1, 400-2, 400-3 is shifted between consecutive video frames according to frame motion data. By shifting the trimming position, a tracked set of image features 402, 408 can be kept stable within the image area of ​​the trimmed video frame. However, trimming is only one example of an image transform suitable for DIS, and other more complex types of DIS transforms, such as DIS transforms that compensate for roll and / or skew, may be implemented additionally or alternatively.

[0110] In the above, Method 300 is described with reference to a single sensing axis of motion sensor 202 and a position sensor 112 that determines the position of a movable element (e.g., lens 118) relative to a single sensing axis corresponding to / associated with a compensation axis of OIS system 104. However, the present disclosure is also applicable to implementations that utilize two or more sensing axes and compensation axes, such as two sensing axes of motion sensor 202 (e.g., pitch and yaw) and two corresponding sensing axes of position sensor 112 and compensation axes of the OIS system. For example, in such an implementation, motion sensor 202 may provide respective motion signals (first motion signal and second motion signal) indicating angular velocity around each sensing axis, each corresponding to a single motion signal ω. Thus, Method 300 includes, in step S301, obtaining a sequence of first motion values ​​sampled from a first motion signal of a first sensing axis (e.g., pitch) of motion sensor 202 and a sequence of second motion values ​​sampled from a second motion signal of a second sensing axis (e.g., yaw) of motion sensor 202. In other words, in step S301, a sequence of motion vectors may be acquired, each motion vector including a first motion value and a second motion value sampled from a first motion signal and a second motion signal, respectively, and each motion vector is associated with a respective video frame. The first and second motion values ​​of each motion vector may here refer to a corresponding, in particular, time-aligned pair of first and second motion values. Each azimuthal signal (first azimuthal signal and second azimuthal signal) may be derived from each sequence of motion signals / motion values ​​(e.g., by integration and optionally by filtering). Thus, each azimuthal vector may be derived for each motion vector. Correspondingly, the position sensor 112 may include a first sensing axis and a second sensing axis, each providing a position signal (first position signal and second position signal) indicating the position of a movable element (lens 118 or image sensor 122) with respect to each sensing axis of the position sensor 112.Accordingly, method 300 may include, in step S302, obtaining a sequence of first position values ​​sampled from a first position signal of a first sensing axis of the position sensor 112 (e.g., the first axis of the Hall sensor) and a sequence of second position values ​​sampled from a second position signal of a second sensing axis of the position sensor 112 (e.g., the second axis of the Hall sensor). In other words, in step S302, a sequence of position vectors may be obtained, each position vector comprising a first position value and a second position value sampled from the first position signal and the second position signal, respectively, and each position vector associated with each video frame. The first and second position values ​​of each motion vector may here refer to corresponding, in particular, temporally aligned pairs of first and second position values. Furthermore, each motion vector and position vector associated with each respective video frame may here refer to corresponding, in particular, temporally aligned pairs of motion vectors and position vectors associated with each respective video frame. The residual block 230 may be adapted accordingly to determine a sequence of residual motion vectors (hereinafter referred to as residual vectors) for a sequence of video frames (for example, in step S303 of method 300), where each residual motion vector is determined based on a first motion value and a second motion value, as well as a first position value and a second position value, associated with each video frame. That is, the residual motion vector associated with a given video frame is determined based on the position vector and motion vector associated with the given video frame. Thus, each residual motion vector may include a first component having a first residual motion value and a second component having a second residual motion value, where the first and second residual motion values ​​represent the residual motion (i.e., OIS error) of the camera 100 along the first and second compensation axes of the OIS system 104, respectively.

[0111] When the first and second sensing axes of the motion sensor 202 and the first and second sensing axes of the position sensor 112 are aligned with respect to each other, the first position value of the first component of the residual motion vector associated with a given video frame may be determined based on the first motion value and first position value associated with the given frame. Correspondingly, the second position value of the second component of the residual motion vector may be determined based on the second motion value and second position value associated with the given frame. The residual block 230 may apply a function between the respective angles and positions, or any of the other mappings described above, to the first motion value and / or position value and the second motion value and / or position value, respectively, so as to map the first motion value and position value to a common coordinate system and the second motion value and position value to a common coordinate system. Thereafter, the first and second residuals are determined by determining the difference between the respective mapped azimuth and position values.

[0112] The residual motion vector can be used in different ways. For example, the residual block 230 may determine a representative residual value for each video frame based on the values ​​of the first and second components of the residual vector associated with each video frame. The representative residual value may be determined, for example, as the maximum magnitude of the first and second components of the residual vector. Thus, the residual block 230 may determine a representative residual value for each video frame as the magnitude of the residual vector.

[0113] Alternatively, the residual block 230 may determine a residual vector for each video frame and output it to the DIS system 124. Thus, the DIS system 124 may use the components of the residual vector (i.e., the first residual motion value and the second residual motion value) during the tracking step S3042 to set a first dimension (e.g., width) of the tracking window for a given video frame based on the first residual of the residual vector, and a second dimension (e.g., height) of the tracking window based on the second residual of the residual vector. The first and second dimensions may each be determined using their respective functions in the same form as described above, for example with reference to function W, and may be analogous to equation 2 or 3. This example assumes that the first and second components of the residual vector correspond to or align with the horizontal and vertical axes of the image sensor 122, respectively, and therefore correspond to or align with the X and Y axes of the video frame, i.e., define the width and height dimensions of the video frame. More generally, the residual block 230 or DIS system 124 may apply a coordinate transformation (e.g., a 2D rotation transformation) to the first and second components of the residual vector so that the residual vector is mapped to the X and Y axes of each video frame.

[0114] In a further example, the residual vector can also be used to update the position of the tracking window during the process of tracking one or more image features across the sequence of video frames F described above. Figures 7a and 7b illustrate such an approach with reference to consecutive first and second video frames 500-1, 500-2. The first and second video frames 500-1, 500-2 here can be any pair of consecutive video frames in a sequence (e.g., a subsequence) of video frames F. The first video frame 500-1 may correspond to reference video frame 400-1 in Figure 6a, but can also correspond to any further consecutive video frames in sequence F. The first video frame 500-1 is given by the first residual vector r = r1 = (r 11 ,r12 ) is associated with, and here, r 11 is the first component (for example, the motion residual with respect to the yaw axis and X dimension of video frame 500-1), and r 12 is the second component of the residual vector r1 (for example, the motion residual with respect to the pitch axis and Y dimension of video frame 500-1). The second video frame 500-2 is the second residual vector r = r2 = (r 21 ,r 22 ) is associated with, and here, r 21 is the first component (for example, the motion residual with respect to the yaw axis and X dimension of video frame 500-2), and r 22 is the second component of the residual vector r2 (for example, the motion residual with respect to the pitch axis and Y dimension of video frame 500-2). The first and second components of the first and second residual vectors r1 and r2 can be determined in the same way as described with reference to the preceding example. Figure 7a shows a single image feature 502 being tracked for simplicity. The position of the center of the tracked window 504-1 in video frame 500-1 (for example, pixel coordinates) is given by wp1 = (w x1 ,w y1 ) Image feature 502 is shown here, as an example, slightly displaced with respect to the center of the tracking window 504-1.

[0115] As described above with reference to step S3042, the DIS system 124 proceeds to track the image feature 502 in the second video frame 500-2. Here, instead of setting the position of the tracking window 504-2 in the second video frame 500-2 to the coordinates of the tracking window 504-1 in the first video frame 500-1, the DIS system 124 determines the updated position of the tracking window 504-2 based on its associated second residual vector r2. More specifically, the updated position of the tracking window 504-2 is determined by calculating the tracking window displacement vector Δwp2 and determining the updated position of the tracking window 504-2 by adding the displacement vector Δwp2 to the position wp1 of the tracking window 504-1, for example, as shown in the following equation. wp2=(w x2 ,w y2 ) = wp1 + Δwp2 (Equation 4)

[0116] The displacement vector Δwp2 is determined based on the second residual vector r2 associated with the second video frame 400-2 and the scaling factor, and is expressed, for example, as follows: Δwp2=(Δw x2 ,Δw y2 ) = 1 / R s ×r2 (formula 5) Here, R s R is the scaling factor. s The scaling coefficient R can be a predetermined scaling coefficient, usually greater than 1, such that the displacement vector Δwp2 is determined as a fraction of the residual vector r2. s For example, the scaling factor R can be understood as reflecting the reliability of the residual vector r with respect to accurately capturing the direction of the OIS error. sThis can be determined by considering factors such as the overall performance and speed of the OIS system 104, the noise of the motion sample (e.g., gyro sample ω), and calibration errors. After determining the updated position wp2 of the tracking window 504-2, feature tracking can be applied to the pixels within the tracking window 504-2 as described above.

[0117] Optionally, the sizes of the tracking windows 504-1, 504-2 within the first and second video frames 500-1, 500-2 may also be updated individually for the first and second dimensions (e.g., width and height) as described above, or collectively based on the respective residual vectors r1, r2 associated with the video frames 500-1, 500-2, based on the respective magnitudes of their associated residual vectors, e.g., |r1| and |r2|, as shown in Figures 7a and 7b. In the latter case, the sizes of the tracking windows 504-1, 504-2 may be determined, for example, according to Equation 2 or Equation 3, using r1 = |r1| for video frames 500-1 and 500-2.

[0118] By tracking the image feature 502 in the second video frame 500-2, the displacement d (shown as a vector in Figure 7b) of the image feature 502 between the first video frame 500-1 and the second video frame 500-2 can be estimated. The displacement d is collected as displacement data and may then be used to determine the frame motion data as described above with reference to step S3043.

[0119] This method can be further extended in a corresponding manner for any further consecutive frames in the sequence of video frames F. The position (and size) of the tracking window may be updated for each consecutive frame as described above, or at a less frequent frequency, such as every other frame, every four frames, or less.

[0120] It should be noted that the approach described with reference to Figures 7a and 7b can be applied to any selected number of detected and tracked image features.

[0121] Various operations and blocks involved in controlling the IS systems described herein, such as IS system 101 including OIS system 104 or 104' and DIS system 124, can be implemented in both hardware and software. In a software implementation, an image capture device, such as a camera 100, may include a processing device implemented in the form of one or more processors, such as one or more central processing units, which causes the processing device to execute method steps for controlling the IS system in relation to computer program code instructions stored in a (non-temporary) computer-readable medium such as non-volatile memory. Examples of non-volatile memory include read-only memory, flash memory, ferroelectric RAM, magnetic computer storage devices, optical discs, and the like. In a hardware implementation, the processing device may instead be implemented by a dedicated circuit configured to implement method steps for controlling the IS system. This circuit may be in the form of one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs) or one or more field-programmable gate arrays (FPGAs). It is also possible to have a combination of hardware and software implementation forms. Please understand that this means some method steps may be implemented in dedicated circuits, while other method steps may be implemented in software.

[0122] The steps of Method 300 described above with reference to Figures 4 and 5, namely step S304 and substeps S3041-S3041, are well-suited for implementing on-device DIS in edge devices with limited processing resources, such as surveillance cameras. By limiting feature tracking to a tracking window with a dynamically adjusted size (based on real-time estimated OIS error), the amount of pixel data that needs to be analyzed to facilitate feature tracking can be limited. Furthermore, basing the DIS on frame motion data derived from tracking image features across captured video frames enables accurate and effective DIS. Further advantages of this method are described above.

[0123] Those skilled in the art will understand that the present invention is by no means limited to the examples described above. Conversely, many modifications and variations are possible within the scope of the appended claims.

[0124] For example, in the illustrated exemplary implementation described above, it was assumed that one motion sample ω, one orientation sample θ, one position sample v, and one residual r were associated with each video frame (for example, for each sensing axis of the motion sensor and position sensor). However, in the intended implementation of the Method and IS System 101, the sampling rates of the motion and position values ​​would typically exceed the frame rate of the sequence of video frames, and as a result, the sequence of motion values ​​and the sequence of position values ​​may each contain a subset of the motion and position values ​​for each video frame, respectively. Thus, each video frame may be associated with a subset of the motion values ​​in the sequence of motion values, and further, with a subset of the position values ​​in the sequence of position values. Therefore, the Method may include determining a subset of residual motion values ​​for each video frame based on the subset of motion values ​​and the subset of position values ​​associated with that video frame. More specifically, each position value may correspond to each motion value in a sequence of motion values ​​(e.g., temporally aligned) so that a sequence of motion and position values ​​defines a sequence of corresponding motion-position pairs (e.g., temporally aligned motion-position pairs). Thus, each given video frame may be associated with each subset of motion-position pairs, and each motion-position pair is associated with each subset of pixel rows in a given video frame. Next, each residual motion value may be associated with each subset of pixel rows in a given video frame and may be determined based on the motion and position values ​​of the (corresponding, temporally aligned) motion-position pairs associated with each subset of pixel rows in a given video frame. Thus, each residual motion value may indicate or represent the residual motion of the image capture device, which is not compensated by the OIS system, when capturing each subset of pixel rows in each video frame. Various approaches exist for utilizing such subsets of residual motion values.

[0125] For purposes of illustration, FIGS. 8a and 8b show how a subset {r 11 , r 12 , r 13} (the "first subset of residual motion values") of residual motion values can be associated with a first video frame 600-1 (e.g., corresponding to a reference video frame), and how a subset {r 21 , r 22 , r 23} (the "second subset of residual motion values") of residual motion values can be associated with a second / further video frame 600-2, etc. The first subset {r 11 , r 12 , r 13} of residual motion values is associated with respective subsets of pixel rows 600-1a, 600-1b, 600-1c of the first video frame 600-1. The second subset {r 21 , r 22 , r 23} of residual motion values is associated with respective subsets of pixel rows 600-2a, 600-2b, 600-2c of the second video frame 600-2. The respective subsets of pixel rows 600-1a, 600-1b, 600-1c, and 600-2a, 600-2b, 600-2c of the first video frame 600-1 and the second video frame 600-2 are subsets of pixel rows having respective sets of corresponding (i.e., the same) pixel row indices, as shown in FIGS. 8a and 8b. That is, the subset of pixel row 600-1a of the first video frame 600-1 has the same set of pixel row indices as the subset of pixel row 600-2a of the second video frame 600-2, the subset of pixel row 600-1b of the first video frame 600-1 has the same set of pixel row indices as the subset of pixel row 600-2b of the second video frame 600-2, and the subset of pixel row 600-1c of the first video frame 600-1 has the same set of pixel row indices as the subset of pixel row 600-2c of the second video frame 600-2.

[0126] As used herein, the term “pixel row index” should be understood as a subset or range of pixel row coordinates within a video frame, i.e., a subset or range of vertical pixel coordinates (e.g., the y-coordinate along the height dimension of the video frame). Thus, several subsets of pixel rows in a given video frame divide the given video frame into a corresponding number of non-overlapping pixel regions along the vertical dimension (e.g., the y-dimension), each pixel region spanning the horizontal dimension (e.g., the x-dimension) of the given video frame.

[0127] For ease of explanation, each video frame 600-1, 600-2 is associated with three subsets of residual motion values ​​in Figures 8a and 8b. Therefore, the number of subsets of pixel rows in each video frame 600-1, 600-2 is 3. This means that each subset of pixel rows in a given video frame 600-1, 600-2 corresponds (for example, across) one-third of all pixel rows in the given video frame 600-1, 600-2 (at least approximately). Next, this corresponds to a scenario where the frame rate of the sequence of motion-position pairs is (at least about) three times the frame rate of the sequence of video frames. Thus, the number and vertical dimension of the subsets of pixel rows in each video frame depend on the relative frame rates of the sequence of motion-position pairs and the sequence of video frames. For example, each residual motion value can be associated with any subset of 8 pixel rows, 4 pixel rows, or 2 pixel rows in each given video frame. In some cases, each subset of a pixel row may correspond to a single pixel row, meaning that each residual motion value can be associated with any pixel row in a given video frame.

[0128] In some embodiments, a representative (i.e., single) residual motion value representative of a video frame may be determined, for example, as the mean or maximum value of a subset of residual motion values associated with the video frame, in order to obtain a sequence of representative residual motion values. By way of illustration, the representative residual motion value for the first video frame 600-1 may be determined as the maximum (or mean) value of a subset of residual motion values {r 11 ,r 12 ,r 13}, and the representative residual motion value for the second video frame 600-2 may be determined as the maximum (or mean) value of a second subset of residual motion values {r 21 ,r 22 ,r 23}. Thus, during DIS, the size of the tracking windows 604-1, 606-1, 604-2, 606-2 used to track a given image feature 602, 608 within a given video frame 600-1, 600-2 may be set based on the representative residual motion value for the given video frame 600-1, 600-2, e.g., the maximum (or mean) residual determined for the video frames 600-1, 600-2.

[0129] In some embodiments, representative residual motion values ​​for a given further video frame and a given tracked image feature may instead be the respective residual motion values ​​associated with a subset of pixel rows (e.g., one or more pixel rows) of a given further video frame having the same set of pixel row indices (e.g., one or more pixel row indices) as the subset of pixel rows containing the image feature in the video frame preceding the given video frame. Thus, the size of the tracking window for tracking a given image feature in a given (e.g., each given) further video frame may be set based on the respective residual motion values ​​(only) associated with the subset of pixel rows of a given further video frame having the same set of pixel row indices as the subset of pixel rows of the preceding video frame containing the image feature. The video frame preceding a given video frame is typically the video frame immediately preceding the given video frame.

[0130] Referring to Figures 8a and 8b, the size of the tracking window 604-2 used to track the image feature 602 in the second video frame 600-2 is such that the subset of pixel rows 600-2a in video frame 600-2 has the same set of pixel row indices as the subset of pixel rows 600-1a in the preceding video frame 600-1 containing the image feature 602, and therefore the size of the tracking window 604-2 used to track the image feature 602 in the second video frame 600-2 is such that subset of pixel rows 600-2a in video frame 600-2a has the same set of pixel row indices as the subset of pixel rows 600-1a in the preceding video frame 600-1 containing the image feature 602, and therefore the size of the tracking window 604-2 used to track the image feature 602 in the second video frame 600-2 is such that the size of the tracking window 21 It can be set based on the following. Therefore, the size W1 of the tracking window 604-2 in video frame 600-2 is W1 = W(r 21 ) is set to ). If the position of image feature 602 in the preceding video frame 600-1 is in another subset of pixel rows, such as 600-1b, then the size W1 of the tracking window 604-2 in video frame 600-2 is set to r in this case. 22However, since this is the residual motion value associated with a subset of pixel row 600-2b that has the same set of pixel row indices as the subset of pixel row 600-1b of the preceding video frame 600-1 which contains image feature 602, W1 = W(r 22 The size of the second tracking window 606-2 used to track the second image feature 608 is set to W2 = W(r 23 To set the size of the tracking windows 604-1, 606-1, used to track each of the image features 602, 608 in the first video frame 600-1, a corresponding approach may be applied. As further illustrated, assuming that video frame 600-1 is preceded by another video frame (e.g., a reference video frame), the sizes of the tracking windows 604-1, 606-1, used to track each of the image features 602, 608 in the first video frame 600-1, may be set in a corresponding manner. In each of these examples, the size of the tracking window may be set using one of the examples of function W described above, such as the function of Equation 2 or Equation 3.

[0131] It may be beneficial to combine any of the above approaches with determining frame motion data that includes each subset of pixel row motion data associated with each subset of pixel row motion data for each further video frame. This would allow the DIS to generate a sequence of stabilized video frames by, for example, applying motion compensation transforms individually to each subset of pixel row based on its associated subset of pixel row motion data, in addition to the trimming transforms described above.

[0132] The above describes an approach to resizing the tracking window based on residual motion values ​​associated with each subset of pixel rows, referring to scalar residual motion values. Note that the corresponding approach can be applied to residual motion vectors ("residual vectors"). Thus, each given video frame may be associated with each subset of motion-position vector pairs, and each motion-position vector pair in each subset of motion-position vector pairs associated with each video frame is associated with each subset of pixel rows in each video frame. Therefore, each subset of residual vectors may be determined for each video frame, and each residual vector in each subset of residual vectors associated with each video frame is determined based on the motion-position vector pairs associated with each subset of pixel rows in each video frame. Thus, each residual vector is associated with the same subset of pixel rows as the motion-position vector pair on which the residual vector is based (i.e., the motion-position vector pair from which the residual vector is derived). Therefore, the size of the tracking window for tracking image features in a given further video frame can be set based on a representative residual vector for the given further video frame. The representative residual vector may be, for example, a residual vector from a subset of residual vectors associated with the given further video frame having the largest magnitude. Alternatively, the representative residual vectors for a given further video frame and a given tracked image feature may be the respective residual vectors associated with a subset of pixel rows in the given further video frame having the same set of pixel row indices as the subset of pixel rows containing the given tracked image feature in the video frame preceding the given video frame. In this case, the size of the tracking window may be set based, for example, on the magnitude of the representative residual vector (for example, only that).Alternatively, the first dimension of the tracking window (e.g., width) may be set based on the first residual motion value (first component) (e.g., only) of the representative residual vector, and the second dimension of the tracking window (e.g., height) may be set based on the second residual motion value (second component) (e.g., only) of the representative residual vector. As described above with reference to Figures 7a and 7b, it is further possible to update the position of the tracking window used to track a given image feature based on the representative vector.

Claims

1. A method for performing digital image stabilization on a sequence of video frames captured by an image capture device, wherein the image capture device comprises a motion sensor configured to output a motion signal indicating the motion of the image capture device, an OIS system configured to compensate for the motion of the image capture device by controlling the position of a movable element of the optical image stabilization (OIS) system based on the motion signal, and a position sensor configured to output a position signal indicating the instantaneous position of the movable element, the method being: The process involves capturing a sequence of video frames while simultaneously obtaining a sequence of motion values ​​sampled from the motion signal. The process involves capturing a sequence of video frames and obtaining a sequence of position values ​​sampled from the position signal, wherein each position value corresponds to each motion value in the sequence of motion values, and as a result, the sequence of motion values ​​and the sequence of position values ​​define a sequence of motion-position pairs. The sampling rates of the motion value sequence and the position value sequence exceed the frame rate of the video frame sequence, and as a result, each video frame is associated with each subset of motion value-position value pairs, and each motion value-position value pair is associated with each subset of pixel rows of each video frame, and the method Determining a subset of residual motion values ​​for each video frame, wherein each residual motion value in each subset of residual motion values ​​associated with each video frame is determined based on the motion value and position value of the motion value-position value pair associated with each subset of pixel rows of each video frame, thereby determining a subset of residual motion values ​​that are associated with each subset of pixel rows and represent the residual motion of the image capture device not compensated by the OIS system when capturing each subset of pixel rows of each video frame. Performing digital image stabilization on the sequence of video frames, To detect image features within the reference video frame of the aforementioned sequence of video frames, Tracking the image features within each further video frame of the sequence of video frames, Determining frame motion data based on the displacement of the image features in each further video frame within the sequence of video frames, Performing digital image stabilization, which includes generating a sequence of stabilized video frames based on the aforementioned frame motion data. It further includes, A method in which the size of a tracking window for tracking the image features in a given further video frame is set based on representative residual motion values ​​of the subset of residual motion values ​​associated with the given further video frame.

2. The method according to claim 1, wherein the representative residual motion values ​​are the respective residual motion values ​​associated with the subset of pixel rows of a given further video frame having the same set of pixel row indices as the subset of pixel rows of a preceding video frame containing the image features.

3. The method according to claim 1, wherein the representative residual motion value is the maximum value of the subset of residual motion values ​​associated with the given further video frame.

4. The method according to claim 1, wherein determining each residual motion value includes determining a corresponding orientation value indicating the estimated instantaneous orientation of the image capture device based on the motion value associated with the respective subset of the pixel rows of the respective video frame, and determining the residual motion value based on the orientation value and the position value.

5. The method according to claim 4, wherein the residual motion value is determined based on the difference between the azimuth value and the position value when mapped to a common coordinate system.

6. The method according to claim 4, wherein the motion signal represents angular velocity, and the orientation value is derived by integrating the motion signal.

7. The method according to claim 1, wherein tracking the image features comprises using optical flow analysis, the optical flow analysis being selectively applied to pixels within the tracking window of each further video frame.

8. The method according to claim 1, wherein the size of the tracking window for tracking the image features in each given further video frame is set based on the representative residual motion values ​​of the subset of pixel rows of the given further video frame.

9. The method according to claim 1, wherein the size of the tracking window is configured to increase with an increase in the typical residual motion value.

10. The method according to claim 1, wherein the movable element is a movable optical element of the optical image stabilization system, or the movable element is an image sensor of the image capture device.

11. The method according to claim 1, wherein the position of the tracking window for tracking the image features in each given further video frame is determined based on the position of the tracking window in a preceding video frame of the video sequence and the representative residual motion value associated with the given further video frame.

12. The aforementioned image feature is a first image feature detected within the reference video frame, and performing the digital image stabilization on the sequence of video frames is: Further detection of at least a second image feature within the reference video frame of the sequence of video frames, Tracking the first image feature and the second image feature within each further video frame of the sequence of video frames, This includes determining the frame motion data based on the respective displacements of the first image feature and the second image feature in each further video frame within the sequence of video frames, The size of each tracking window for tracking the first and second image features within a given further video frame is set based on representative residual motion values ​​of the subset of residual motion values ​​associated with the given further video frame, The representative residual motion values ​​are the respective residual motion values ​​associated with the subset of pixel rows of a given further video frame having the same set of pixel row indices as the subset of pixel rows of a preceding video frame containing each of the image features, or The method according to claim 1, wherein the representative residual motion value is the maximum value of the subset of residual motion values ​​associated with the given further video frame.

13. The method according to claim 1, wherein the motion sensor includes a gyroscope and / or the position sensor includes a Hall effect sensor.

14. A method for performing digital image stabilization on a sequence of video frames captured by an image capture device, wherein the image capture device comprises a motion sensor configured to output a first motion signal and a second motion signal indicating the motion of the image capture device along a first sensing axis and a second sensing axis of the motion sensor, respectively; an optical image stabilization (OIS) system configured to compensate for the motion of the image capture device by controlling the position of a movable element of the OIS system based on the first motion signal and the second motion signal; and a position sensor configured to output a first position signal and a second position signal indicating the instantaneous position of the movable element along a first sensing axis and a second sensing axis of the position sensor, respectively, wherein the method is: The method involves obtaining a sequence of motion vectors, wherein each motion vector is associated with a video frame, and the sequence of motion vectors includes a first motion value and a second motion value sampled from the first motion signal and the second motion signal, respectively, while capturing a sequence of video frames, such that each motion vector is associated with a video frame. Obtaining a sequence of position vectors, wherein each position vector includes a first position value and a second position value sampled from the first position signal and the second position signal, respectively, while capturing a sequence of video frames, such that each position vector is associated with its respective video frame. For each video frame, the residual vector is determined in order to obtain a sequence of residual vectors for the sequence of video frames, wherein each residual vector is: Each residual vector includes a first residual motion value based on the first motion value of the motion vector and the first position value of the position vector, and a second residual motion value based on the second motion value of the motion vector and the second position value of the position vector, and represents the residual motion of the image capture device that is not compensated by the OIS system when capturing each of the video frames. Determining a residual vector based on the motion vector and position vector associated with each of the aforementioned video frames, Performing digital image stabilization on the sequence of video frames, To detect image features within the reference video frame of the aforementioned sequence of video frames, Tracking the image features within each further video frame of the sequence of video frames, Determining frame motion data based on the displacement of the image features in each further video frame within the sequence of video frames, Performing digital image stabilization, which includes generating a sequence of stabilized video frames based on the aforementioned frame motion data. Includes, A method wherein the first and second dimensions of a tracking window for tracking the image features within the further video frames are set based on the first and second residual motion values ​​of the residual vectors of the sequence of residual vectors, respectively.

15. The method according to claim 14, wherein the position of the tracking window for tracking the image features in each given further video frame is determined based on the position of the tracking window in a video frame preceding the given further video frame and the residual vector associated with the given further video frame.

16. The sampling rate of the sequence of motion and position values ​​exceeds the frame rate of the sequence of video frames such that each video frame is associated with a subset of motion and position vector pairs, and each of the motion and position vector pairs in the subset of motion and position vector pairs associated with each video frame is associated with a subset of the pixel rows of each video frame, and the method Determining a subset of residual vectors for each video frame, wherein each residual vector in the respective subset of residual vectors associated with each video frame is determined based on the motion vector and position vector of the motion vector-position vector pair associated with the respective subset of pixel rows of the video frame, The method according to claim 14, wherein the first and second dimensions of the tracking window for tracking the image feature in each given further video frame are set based on the residual vector associated with the subset of pixel rows of the given further video frame having the same set of pixel row indices as the subset of pixel rows of the preceding video frame containing the image feature.

17. The method according to claim 16, wherein the position of the tracking window for tracking the image features in the given further video frame is determined based on the position of the tracking window in the video frame preceding the given further video frame and the residual vector associated with the subset of pixel rows of the given further video frame.

18. It is an image capture device, A motion sensor configured to output a motion signal indicating the motion of the image capture device, The OIS system is configured to compensate for the motion of the image capture device by controlling the position of the movable elements of the optical image stabilization (OIS) system based on the motion signal, A position sensor configured to output a position signal indicating the position of the movable element, The processing device is configured to perform a method for digital image stabilization of a sequence of video frames captured by the image capture device, and the method is: The process involves capturing a sequence of video frames while simultaneously obtaining a sequence of motion values ​​sampled from the motion signal. The process involves capturing a sequence of video frames and obtaining a sequence of position values ​​sampled from the position signal, wherein each position value corresponds to each motion value in the sequence of motion values, and as a result, the sequence of motion values ​​and the sequence of position values ​​define a sequence of motion-position pairs. The sampling rates of the motion value sequence and the position value sequence exceed the frame rate of the video frame sequence, and as a result, each video frame is associated with each subset of motion value-position value pairs, and each motion value-position value pair is associated with each subset of pixel rows of each video frame, and the method Determining a subset of residual motion values ​​for each video frame, wherein each residual motion value in each subset of residual motion values ​​associated with each video frame is determined based on the motion value and position value of the motion value-position value pair associated with each subset of pixel rows of each video frame, thereby determining a subset of residual motion values ​​that are associated with each subset of pixel rows and represent the residual motion of the image capture device not compensated by the OIS system when capturing each subset of pixel rows of each video frame. Performing digital image stabilization on the sequence of video frames, To detect image features within the reference video frame of the aforementioned sequence of video frames, Tracking the image features within each further video frame of the sequence of video frames, Determining frame motion data based on the displacement of the image features in each further video frame within the sequence of video frames, Performing digital image stabilization, which includes generating a sequence of stabilized video frames based on the aforementioned frame motion data. It further includes, An image capture device in which the size of a tracking window for tracking the image features within a given further video frame is set based on representative residual motion values ​​of the subset of residual motion values ​​associated with the given further video frame.

19. The image capture device according to claim 18, wherein the representative residual motion values ​​are the respective residual motion values ​​associated with the subset of pixel rows of a given further video frame having the same set of pixel row indices as the subset of pixel rows of a preceding video frame containing the image features.

20. The image capture device according to claim 18, wherein the representative residual motion value is the maximum value of the subset of residual motion values ​​associated with the given further video frame.

Images