Image stabilization of a video stream captured by a pannable and / or tiltable video camera

Abstract

To provide a method for image stabilization of a video stream captured by a pannable and / or tiltable video camera, a video camera and a storage medium.SOLUTION: A method comprises: generating (S202) a motor position signal Y1 of a pan / tilt motor; generating (S204) a gyro signal Y2 of a gyroscopic sensor; and generating (S206) a reference signal from a predetermined movement curve of the pan / tilt motor, the reference signal is a reference on how a pan / tilt operation is made without shaking of the video camera. The method further comprises: generating (S206) the reference signal; from the motor position signal Y1 and the gyro signal Y2 generating (S208) a combined signal Y according to: Y=F1*Y1+F2*Y2, where F1 is a low pass filter and F2 is a high pass filter; and performing (S210) image stabilization on the video stream based on a difference between the combined signal Y and the reference signal.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to image stabilization of video streams, in particular of video streams captured by a video camera during pan / tilt of the video camera. [Background technology]

[0002] During the capture of a video stream, the camera capturing the video stream may be shaken, for example, by wind, passing traffic, etc. As a result, the images of the video stream are often unclear and difficult to observe. Image stabilization, particularly electronic image stabilization (EIS), effectively minimizes the effects of vibration and shaking to give clear and accurate image frames of the video stream. During EIS, one or more gyroscope sensors of the camera continuously detect the movement and vibration of the camera. Based on the output from the one or more gyroscope sensors, the image frames of the video stream can be adjusted to ensure a stable image stream.

[0003] However, during pan / tilt movements of the pan / tilt video camera, the EIS is typically switched off. Thus, a need exists for enabling image stabilization of a video stream captured using a pan / tilt video camera that is panning and / or tilting during video camera capture. Summary of the Invention

[0004] In view of the above, it is an object of the present invention to provide image stabilization of a video stream captured by a pan / tilt video camera that is panning and / or tilting during video camera capture.

[0005] It is an object to mitigate, alleviate or eliminate one or more of the above identified shortcomings and disadvantages in the art singly or in any combination, and to solve at least the above-mentioned problems.

[0006] According to a first aspect, there is provided a method for image stabilization of a video stream captured by a pannable and / or tiltable video camera, the method comprising: generating a motor position signal Y1 of a pan / tilt motor of the video camera; generating a gyro signal Y2 of a gyroscope sensor of the video camera; generating a reference signal from a predetermined movement curve of the pan / tilt motor of the video camera, the reference signal being a reference for how the pan / tilt movement of the video camera would be performed without shaking of the video camera; generating a combined signal Y from the motor position signal Y1 and the gyro signal Y2, where Y=F1 * Y1+F2 * generating a combined signal Y according to Y2, where F1 is a low pass filter and F2 is a high pass filter; and performing image stabilization on the video stream based on a difference between the combined signal Y and a reference signal.

[0007] The method offers the possibility for improved image stabilization of image frames of a video stream captured during panning and / or tilting of a video camera, since a low pass filter filters the motor position signal and a high pass filter filters the gyro signal to achieve a less noisy combined signal indicative of the shaking of the video camera during panning and / or tilting. Furthermore, a reference signal can be used to indicate the pan and / or tilt movement of the video camera. The method allows image stabilization induced by both the shaking of the video camera and the recoil effect induced by the motor.

[0008] For example, a single-pole infinite impulse response (IIR) filter can be used, which can be used to implement both low-pass and high-pass filtering.

[0009] The high pass filter can be considered to be complementary to the low pass filter, such that F2=1-F1.

[0010] According to a second aspect, there is provided a non-transitory computer readable storage medium having stored thereon instructions for performing the method according to the first aspect when executed on a video camera having processing capability.

[0011] According to a third aspect, there is provided a video camera comprising: an image sensor configured to capture image frames of a video stream; a pan / tilt motor configured to pan and / or tilt the video camera; a gyroscope sensor configured to detect shaking of the video camera; and control circuitry. The control circuitry comprises: a motor position signal generating function configured to generate a motor position signal Y1 of the pan / tilt motor; a gyro signal generating function configured to generate a gyro signal Y2 related to shaking of the video camera detected by the gyroscope sensor; a reference signal generating function configured to generate a reference signal from a predetermined movement curve of the pan / tilt motor, the reference signal being a reference for how the pan / tilt movement of the video camera would be performed without shaking of the video camera; and a combined signal Y from the motor position signal Y1 and the gyro signal Y2, where Y=F1. * Y1+F2 *a signal combination function configured to generate a combined signal Y, where F1 is a low pass filter and F2 is a high pass filter according to Y2; and an image stabilization function configured to perform image stabilization on image frames of the video stream based on a difference between the combined signal Y and a reference signal.

[0012] The above-stated features of the method also apply to the second and third aspects, when applicable, and to avoid undue repetition, reference is made thereto.

[0013] Further scope of applicability of the present invention will become apparent from the detailed description set forth hereinafter. It should be understood, however, that the detailed description and specific examples, while indicating preferred embodiments of the invention, are given by way of illustration only, since various changes and modifications within the scope of the invention will become apparent to those skilled in the art from this detailed description.

[0014] Thus, it is understood that the invention is not limited to the particular component parts of the described device or the particular acts of the described method, as such devices and methods may vary. It is also understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It should be noted that, as used in this specification and the appended claims, the articles "a," "an," "the," and "said" are intended to mean that there are one or more of an element, unless the context clearly dictates otherwise. Thus, for example, reference to "a unit" or "the unit" can include several devices and the like. Furthermore, the terms "comprising," "including," "containing," and similar terms do not exclude other elements or steps.

[0015] The above and other aspects of the present invention will now be described in more detail with reference to the accompanying drawings, which should not be considered limiting, but instead are used for illustration and understanding.

[0016] As shown in the figures, the sizes of layers and regions may be exaggerated for illustrative purposes, and thus are provided to show general structure. Like reference numbers refer to like elements throughout. [Brief description of the drawings]

[0017] [Figure 1] FIG. [Diagram 2] 1 is a block scheme of a method for image stabilization of a video stream captured by a pannable and / or tiltable video camera. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which presently preferred embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided for thoroughness and completeness, and to fully convey the scope of the invention to those skilled in the art.

[0019] 1 shows a video camera 100. The video camera 100 is typically a monitoring camera. The video camera 100 may be a digital video camera, in particular a networked digital video camera. The video camera 100 is configured to monitor a scene. Typically, this is done by capturing a video stream of the scene. The video stream is captured using an image sensor 102.

[0020] The video camera 100 is a pannable and / or tiltable video camera. Thus, while capturing a video stream of a scene, the video camera 100 can be panned and / or tilted. The panning and / or tilting of the video camera 100 is performed using a pan and / or tilt motor 104. The video camera 100 can have a dedicated pan motor and a dedicated tilt motor. The pan and / or tilt motor 104 can be a step motor. The pan and / or tilt motor 104 can be controlled by a pan / tilt control signal. To perform the panning and / or tilting, the pan / tilt control signal is sent to the pan and / or tilt motor 104. The pan / tilt control signal commands the pan and / or tilt motor 104 to move according to a pre-defined movement curve of the pan / tilt motor. The pre-defined movement curve can be a pre-known acceleration curve. Thus, for a given pan / tilt movement, a target speed is set and the pan and / or tilt motor 104 is set to accelerate to this speed according to the pre-known acceleration curve. The pan / tilt motor 104 will accelerate as closely as possible according to the acceleration curve and the target speed. However, the actual speed may differ slightly from the predetermined travel curve. One example would be that the pan / tilt motor 104 receives a go-to-target position command. The pan / tilt motor 104 is configured to first accelerate to a set speed, then maintain the set speed, and then decelerate to a smooth stop. However, sometimes undesirable deviations from the pre-known acceleration curve occur, especially when the pan / tilt motor 104 stops. When the pan / tilt motor 104 stops, the video camera 100 can start to wobble around the target position before coming to a complete stop. This is due to the "springiness" of the pan / tilt motor 104 and / or the mechanical drive system of the pan / tilt motor 104. Such wobbling around the target position can be compensated for using image stabilization according to the present invention.

[0021] The video camera 100 may include one or more motor position sensors 105. The one or more motor position sensors 105 may be implemented as one or more encoders. The motor position may be measured by the one or more motor position sensors 105. This may be done especially during panning and / or tilting of the video camera 100.

[0022] The video camera 100 may further include one or more gyroscope sensors 106. The one or more gyroscope sensors 106 are configured to detect shaking of the video camera 100. The one or more gyroscope sensors 106 may be configured to continuously detect shaking of the video camera 100.

[0023] Video camera 100 further comprises control circuitry 110. Control circuitry 110 is configured to perform overall control of the functions and operations of video camera 100. Control circuitry 110 may include a processor 111, such as a central processing unit (CPU), microcontroller, or microprocessor. Processor 111 is configured to execute program code stored in memory 120 to perform the functions and operations of video camera 100.

[0024] The memory 120 can be one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, a random access memory (RAM), or another suitable device. In a typical arrangement, the memory 120 can include a non-volatile memory for long-term data storage and a volatile memory that serves as a system memory for the control circuitry 110. The memory 120 can exchange data with the control circuitry 110 through a data bus. There can also be associated control lines and an address bus between the memory 120 and the control circuitry 110.

[0025] The functions and operations of video camera 100 may be embodied in the form of executable logic routines (e.g., lines of code, software programs, etc.) that are stored on a non-transitory computer-readable medium (e.g., memory 120) of video camera 100 and executed by control circuitry 110 (e.g., using processor 111). Furthermore, the functions and operations of video camera 100 may be standalone software applications or may form part of a software application that performs further tasks related to video camera 100. The functions and operations described may be considered as the way in which a corresponding device is configured to perform them. Similarly, while the functions and operations described may be implemented in software, such functionality may also be implemented by dedicated hardware or firmware, or a combination of hardware, firmware, and / or software.

[0026] The control circuitry 110 can be configured to perform a pan / tilt function 121. The pan / tilt function 121 is configured to generate pan / tilt control signals that are used to control the panning and / or tilting of the video camera 100.

[0027] The control circuit 110 can be configured to execute a motor position signal generating function 122. The motor position signal generating function 122 is configured to generate a motor position signal Y1. The motor position signal Y1 is a signal related to the position of the pan / tilt motor 104. The motor position signal Y1 allows to log reference points for the video camera 100 at different times, i.e., which pan / tilt position the video camera is in during a period of time. The motor position signal can be measured during panning and / or tilting of the video camera 100. The motor position signal generating function 122 can be configured to generate a motor position signal from a motor position measured by one or more motor position sensors 105. The position signal can be related to the rotation of the pan / tilt motor 104 or the rotation of a gear of the pan / tilt motor 104. Alternatively or in combination, the motor position signal generating function 122 can be configured to generate a motor position signal from a pan / tilt control signal generated by the pan / tilt function 121, especially if the pan / tilt motor 104 is a step motor. The motor position signal is a signal that indicates the time-dependent pan / tilt position of the video camera 100 .

[0028] The control circuit 110 can be configured to perform a gyro signal generation function 123. The gyro signal generation function 123 is configured to generate a gyro signal Y2. The gyro signal Y2 is related to the shaking of the video camera detected by the one or more gyroscope sensors 106.

[0029] The gyroscope sensor(s) 106 do not have a fixed reference point. The gyroscope sensor 106 measures inertial forces, motion relative to "standing still". By itself, the gyroscope sensor 106 cannot answer questions such as "where are we now". The motor position sensor(s) 105, on the other hand, know their position relative to a rigid body, e.g., the encoder ring. Thus, the motor position sensor(s) 105 have a fixed reference point. The motor position signal, unlike the gyro signal, cannot detect when the entire video camera 100 is vibrating. The motor position signal is a measure of the motion of the video camera 100 relative to a fixed reference point.

[0030] The control circuit 110 is configured to execute a reference signal generation function 124. The reference signal generation function 124 is configured to generate a reference signal from a predetermined movement curve of the pan / tilt motor 104. The reference signal is a reference for how the pan / tilt movement of the video camera 100 would be performed without shaking of the video camera 100.

[0031] The control circuit 110 is configured to perform a signal combination function 125. The signal combination function 125 is configured to generate a combined signal Y from the motor position signal Y1 and the gyro signal Y2. The combined signal Y is expressed as Y=F1 * Y1+F2 *Y2, where F1 is a low-pass filter and F2 is a high-pass filter. For example, a single-pole infinite impulse response IIR filter can be used. Such a filter can be used to perform both low-pass and high-pass filtering. By forming a combined signal, improved image stabilization can be performed. Motor position signals are typically contaminated by high frequency noise. High frequency noise may be introduced into the motor position signal by quantization errors. Gyro signals are typically contaminated by low frequency noise. Because the gyro signal (a measure of angular velocity) must be integrated to obtain position, and because the gyro signal will always contain some noise, the noise will add up and become non-negligible over time. Gyro signals typically contain a DC offset as well. Thus, when there is no motion, the gyro signal will give a small DC signal due to the DC offset. The DC offset is not constant and will vary slowly over time, for example with temperature. This will also contribute to low frequency noise. Thus, low frequency noise may be introduced into the gyro signal by errors due to sensor imperfections as well as temperature changes that are integrated over time. By applying a low pass filter to the motor position signal and a high pass filter to the gyro signal and combining the two signals, the noise of each can be at least partially filtered out to produce a more accurate signal indicative of the shaking of the video camera during panning and / or tilting of the video camera.

[0032] The control circuit 110 is configured to execute an image stabilization function 126. The image stabilization function 126 is configured to perform image stabilization on image frames of the video stream. The image stabilization is based on a difference between the combined signal Y and a reference signal, e.g., the reference signal minus the combined signal. The image stabilization is preferably electronic image stabilization EIS. EIS is commonly known to those skilled in the art and will not be discussed in detail in this disclosure.

[0033] With reference to Figure 2, a method 200 for image stabilization of a video stream captured by a pannable and / or tiltable video camera is discussed. Method 200 is based on the insight made by the inventors that improved image stabilization of image frames captured by video camera 100 can be achieved by generating a combined signal from a low-pass filtered motor position signal and a high-pass filtered gyro signal, and generating a reference signal from a predetermined travel curve of pan / tilt motor 104 used to pan and / or tilt video camera 100.

[0034] Some of all steps of method 200 may be performed by functions of video camera 100 described above. However, it is also noted that some or all of the steps of method 200 may be performed by similar functions performed in other devices. Method 200 includes the following steps. The steps may be performed in any suitable order.

[0035] Generating S202 a motor position signal Y1 of the pan / tilt motor 104 of the video camera 100. The motor position signal Y1 is a signal related to the position of the pan / tilt motor 104. The motor position signal Y1 allows to log reference points for the video camera 100 at different times, i.e., in which pan / tilt position the video camera is in during a period of time. The motor position signal is a signal that can be measured during panning and / or tilting of the video camera 100. The motor position signal can be generated from a motor position measured by one or more motor position sensors 105. Alternatively or in combination, the motor position signal can be generated from a pan / tilt control signal. The motor position signal is a signal indicative of the time-dependent pan / tilt position of the video camera 100.

[0036] generating S204 a gyro signal Y2 of a gyroscope sensor 106 of the video camera 100; the gyro signal Y2 relating to shaking of the video camera as detected by one or more gyroscope sensors 106;

[0037] Generate S206 a reference signal from a predetermined movement curve of the pan / tilt motor 104 of the video camera 100. The reference signal is a reference for how the pan / tilt movement of the video camera 100 would be performed without shaking of the video camera 100.

[0038] Generate a combined signal Y from the motor position signal Y1 and the gyro signal Y2 (S208). The combined signal Y is expressed as Y=F1 * Y1+F2 * Y2, where F1 is a low pass filter and F2 is a high pass filter.

[0039] S210 performing image stabilization on the video stream based on the difference between the combined signal Y and the reference signal. The image stabilization is preferably electronic image stabilization (EIS). EIS is commonly known to those skilled in the art and will not be discussed in detail in this disclosure.

[0040] Those skilled in the art will recognize that the present invention is by no means limited to the preferred embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims.

[0041] For example, the high pass filter can be the complement of the low pass filter, so F2 can be equal to 1-F1.

[0042] Additionally, variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

Claims

1. A method for image stabilization of a video stream, comprising: Capturing a video stream by a pannable and / or tiltable video camera (100) during pan / tilt of said video camera (100); generating a motor position signal Y1 of the pan / tilt motor (104) of the video camera (S202); generating (S204) a gyro signal Y2 of a gyroscope sensor (106) of the video camera (100), the gyro signal Y2 being an integral of measurements from the gyroscope sensor (106); generating (S206) a reference signal from a predetermined movement curve of the pan / tilt motor (104) of the video camera (100), the reference signal being a reference for how the pan / tilt movement of the video camera (100) would be performed without shaking of the video camera (100); From the motor position signal Y1 and the gyro signal Y2, Y=F1 ＊ Y1+F2 ＊ generating a combined signal Y according to F1 and F2 (S208), where F1 is a low-pass filter and F2 is a high-pass filter; and performing (S210) electronic image stabilization on the video stream captured by the pannable and / or tiltable video camera (100) during pan / tilt of the video camera (100) based on a difference between the combined signal Y and the reference signal; A method comprising:

2. 2. The method of claim 1, wherein the high pass filter is complementary to the low pass filter such that F2=1-F1.

3. The method of claim 1, wherein the motor position signal is a signal indicating a time-dependent pan / tilt position of the video camera.

4. A non-transitory computer readable storage medium having stored thereon instructions for performing the method of any one of claims 1 to 3 when executed on a video camera having processing capabilities.

5. A video camera, an image sensor (102) configured to capture image frames of a video stream; a pan / tilt motor (104) configured to pan and / or tilt the video camera; a gyroscope sensor (106) configured to detect shaking of the video camera; and a control circuit section (110), a motor position signal generating function (122) configured to generate a motor position signal Y1 of the pan / tilt motor (104); a gyro signal generating function (123) configured to generate a gyro signal Y2 related to a shake of the video camera detected by the gyro sensor (106), the gyro signal Y2 being an integral of a measurement from the gyro sensor (106); a reference signal generation function (124) configured to generate a reference signal from a predetermined movement curve of the pan / tilt motor (104), the reference signal being a reference for how a pan / tilt movement of the video camera would be performed without shaking of the video camera; From the motor position signal Y1 and the gyro signal Y2, Y=F1 ＊ Y1+F2 ＊ a signal combining function (125) configured to generate a combined signal Y according to F1, F2, F3, F4, F5, F6, F7, F8, F9, F10, F11, F12, F13, F14, F15, F16, F17, F18, F19, F20, F21, F22, F23, F24, F25, F26, F27, F28, F29, F30, F31, F32, F33, F34, F35, F36, F37, F38, F39, F40, F41, F42, F43, F44, F45, F46, F47, F48, F49, F50, F51, F52, F53, F54, F55, F56, F57, F60, F61, F62, F63, F6 an image stabilization function (126) configured to perform image stabilization on image frames of a video stream captured by the video camera during pan / tilt of the video camera based on a difference between the combined signal Y and the reference signal; The video camera is configured to:

6. 5. The video camera of claim 4, wherein the high pass filter is complementary to the low pass filter such that F2=1-F1.

7. The video camera of claim 5, wherein the motor position signal is a signal indicative of a time-dependent pan / tilt position of the video camera.

Images