Image stabilization of video stream captured by pannable and / or tiltable video camera
Patent Information
- Application Number
- JP2021195471
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-12-07
- Filing Date
- 2021-12-01
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-12-01
AI Technical Summary
Existing video cameras with pan and tilt capabilities often experience image instability due to vibrations and shaking, leading to unclear video streams, and existing electronic image stabilization (EIS) systems fail to effectively stabilize images during these movements.
A method involving low-pass and high-pass filtering of motor position and gyro signals to generate a combined signal for image stabilization, using a single pole infinite impulse response (IIR) filter to correct for shaking and recoil effects during pan and tilt operations.
Improves image stabilization by filtering noise from both motor position and gyro signals, effectively stabilizing video frames during pan and tilt movements, reducing unwanted deviations and enhancing image clarity.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to image stabilization of a video stream, particularly a video stream captured by a video camera during panning / tilting of the video camera.
Background Art
[0002] During the capture of a video stream, the camera capturing the video stream may shake due to, for example, wind, passing traffic, etc. As a result, the image of the video stream is often unclear and difficult to observe. Image stabilization, particularly electronic image stabilization EIS (Electronic Image Stabilization), effectively minimizes the effects of vibration and shake in order to provide a clear and accurate image frame of the video stream. During EIS, one or more gyroscope sensors of the camera continuously detect the movement and vibration of the camera. Based on the output from one or more gyroscope sensors, the image frames of the video stream can be adjusted to ensure a stable image stream.
[0003] However, during the panning / tilting movement of a pan / tilt video camera, EIS is typically switched off. Therefore, there is a need to enable image stabilization of a video stream captured using a pan / tilt video camera that is panning and / or tilting during the capture of the video camera.
Summary of the Invention
[0004] In view of the above, it is an object of the present invention to provide image stabilization of a video stream captured by a pan / tilt video camera that is panning and / or tilting during the capture of the video camera.
[0005] The objective is to mitigate, alleviate, or eliminate one or more of the defects and disadvantages identified above in the art, either individually or in any combination, and to solve at least the problems described above.
[0006] According to a first aspect, a method is provided for image stabilization of a video stream captured by a pannable and / or tiltable video camera. The method involves generating a motor position signal Y1 of the pan / tilt motor of the video camera; generating a gyro signal Y2 of the gyroscope sensor of the video camera; generating a reference signal from a predetermined movement curve of the pan / tilt motor of the video camera, the reference signal being a reference to how the pan / tilt operation of the video camera is performed without camera shake; and generating a combined signal Y from the motor position signal Y1 and the gyro signal Y2, where Y = F1 * Y1+F2 * This includes generating a combined signal Y such that, according to Y2, F1 is a low-pass filter and F2 is a high-pass filter; and performing image stabilization on the video stream based on the difference between the combined signal Y and a reference signal.
[0007] This method offers the potential for improving image stabilization of image frames in a video stream captured during panning and / or tilting of a video camera. This is because a low-pass filter filters the motor position signal and a high-pass filter filters the gyro signal, achieving a low-noise combined signal that indicates the shake of the video camera during panning and / or tilting. Furthermore, a reference signal can be used to indicate the panning and / or tilting movement of the video camera. This method enables image stabilization induced by both the shake of the video camera and the recoil effect induced by the motor.
[0008] For example, a unipolar infinite impulse response (IIR) filter can be used. Such a filter can be used to perform both low-pass and high-pass filtering.
[0009] A high-pass filter can be considered complementary to a low-pass filter, such that F2 = 1 - F1.
[0010] According to a second embodiment, a non-temporary computer-readable storage medium is provided. The non-temporary computer-readable storage medium stores instructions for carrying out the method according to the first embodiment when executed on a video camera having processing capabilities.
[0011] According to a third embodiment, a video camera is provided. The video camera comprises: an image sensor configured to capture image frames of a video stream; a pan / tilt motor configured to pan and / or tilt the video camera; a gyroscope sensor configured to detect camera shake; and a control circuit. The control circuit comprises: a motor position signal generation function configured to generate a motor position signal Y1 of the pan / tilt motor; a gyro signal generation function configured to generate a gyro signal Y2 relating to camera shake detected by the gyroscope sensor; a reference signal generation function configured to generate a reference signal from a predetermined movement curve of the pan / tilt motor, wherein the reference signal is a reference relating to how the pan / tilt operation of the video camera is performed without camera shake; and a combined signal Y from the motor position signal Y1 and the gyro signal Y2, where Y = F1 * Y1+F2 *A signal combination function configured to generate a combined signal Y, where F1 is a low-pass filter and F2 is a high-pass filter, according to Y2; and an image stabilization function configured to perform image stabilization on the image frames of the video stream based on the difference between the combined signal Y and the reference signal.
[0012] The features of the method described above also apply to the second and third embodiments, where applicable. References are made above to avoid unnecessary repetition.
[0013] Further scope of applicability of the present invention will become apparent from the detailed description below. However, since various changes and modifications within the scope of the present invention will become apparent to those skilled in the art from this detailed description, it should be understood that the detailed description and specific examples are provided for illustrative purposes only, while illustrating preferred embodiments of the present invention.
[0014] Therefore, it is understood that the present invention is not limited to specific component parts of the device described or specific actions of the method described, because such devices and methods may vary. It is also understood that the terms used herein are merely for describing specific embodiments and are not intended to be restrictive. It should be noted that, as used herein and in the appended claims, the articles “a,” “an,” “the,” and “said” are intended to mean that there is one or more elements unless the context explicitly indicates otherwise. For example, a reference to “a unit” or “the unit” could include several devices and similar ones. Furthermore, the phrases “comprising,” “including,” “containing,” and similar phrases do not exclude other elements or steps.
[0015] The above and other aspects of the present invention are described here in more detail with reference to the accompanying drawings. The drawings should not be considered restrictive, but rather used for illustrative and understanding purposes.
[0016] As shown in the figure, the sizes of the layers and regions are exaggerated for illustrative purposes and are therefore provided to illustrate the general structure. Similar reference numerals refer to similar elements throughout. [Brief explanation of the drawing]
[0017] [Figure 1] This is a diagram of a video camera. [Figure 2] This is a block scheme for a method of image stabilization of a video stream captured by a pannable and / or tiltable video camera. [Modes for carrying out the invention]
[0018] The present invention will be described more fully hereafter with reference to the accompanying drawings illustrating currently preferred embodiments of the invention. However, the invention can be embodied in many different forms and should not be construed as being limited to the embodiments described herein; rather, these embodiments are provided for thoroughness and completeness and to fully convey the scope of the invention to those skilled in the art.
[0019] Figure 1 shows a video camera 100. The video camera 100 is typically a monitoring camera. The video camera 100 can be a digital video camera, specifically a networked digital video camera. The video camera 100 is configured to monitor a scene. Typically, this is done by capturing a video stream of the scene. The video stream is captured using an image sensor 102.
[0020] The video camera 100 is a pannable and / or tiltable video camera. Therefore, the video camera 100 can be panned and / or tilted while capturing a video stream of a scene. Panning and / or tilting of the video camera 100 is performed using a pan and / or tilt motor 104. The video camera 100 may have a dedicated pan motor and a dedicated tilt motor. The pan and / or tilt motor 104 may be a stepper motor. The pan and / or tilt motor 104 can be controlled by a pan / tilt control signal. To perform panning and / or tilting, a pan / tilt control signal is sent to the pan and / or tilt motor 104. The pan / tilt control signal commands the pan and / or tilt motor 104 to move according to a predetermined motion curve of the pan / tilt motor. The predetermined motion curve may be a known acceleration curve. Therefore, for a given pan / tilt movement, a target speed is set, and the pan and / or tilt motor 104 is set to accelerate to this speed according to the known acceleration curve. The pan / tilt motor 104 will accelerate as close as possible to the target speed according to the acceleration curve and target speed. However, the actual speed may differ slightly from the given motion curve. One example would be when the pan / tilt motor 104 receives a go-to target position command. The pan / tilt motor 104 is configured to first accelerate to the set speed, then maintain the set speed, and then decelerate to a smooth stop. However, sometimes undesirable deviations from the predetermined acceleration curve occur, especially when the pan / tilt motor 104 stops. When the pan / tilt motor 104 stops, the video camera 100 may begin to shake around the target position before coming to a complete stop. This is due to the "springiness" of the pan / tilt motor 104 and / or the mechanical drive system of the pan / tilt motor 104. Such shaking around the target position can be compensated for using image stabilization according to the present invention.
[0021] The video camera 100 can include one or more motor position sensors 105. The one or more motor position sensors 105 can be implemented as one or more encoders. The motor position can be measured by the one or more motor position sensors 105. This can be done particularly during panning and / or tilting of the video camera 100.
[0022] The video camera 100 can further include one or more gyroscope sensors 106. The one or more gyroscope sensors 106 are configured to detect the shake of the video camera 100. The one or more gyroscope sensors 106 can be configured to continuously detect the shake of the video camera 100.
[0023] The video camera 100 further includes a control circuit unit 110. The control circuit 110 is configured to perform overall control of the functions and operations of the video camera 100. The control circuit 110 can include a processor 111 such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 111 is configured to execute program codes stored in the memory 120 to perform the functions and operations of the video camera 100.
[0024] The memory 120 can be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, a random access memory (RAM), or another suitable device. In a typical arrangement, the memory 120 can include a non-volatile memory for long-term data storage and a volatile memory that functions as a system memory for the control circuit 110. The memory 120 can exchange data with the control circuit 110 through a data bus. There can also be accompanying control lines and address buses between the memory 120 and the control circuit 110.
[0025] The functions and operations of the video camera 100 can be embodied in the form of executable logic routines (e.g., lines of code, software programs, etc.), which are stored on a non - transient computer - readable medium (e.g., memory 120) of the video camera 100 and executed by the control circuit 110 (e.g., using the processor 111). Further, the functions and operations of the video camera 100 can be an independent software application or form part of a software application that performs additional tasks related to the video camera 100. The functions and operations described can be considered as the way in which the corresponding device is configured to execute them. Similarly, the functions and operations described can be implemented in software, but such functions can also be implemented by dedicated hardware or firmware, or a combination of hardware, firmware, and / or software.
[0026] The control circuit 110 can be configured to execute the pan / tilt function 121. The pan / tilt function 121 is configured to generate pan / tilt control signals used to control the panning and / or tilting of the video camera 100.
[0027] The control circuit 110 can be configured to execute a motor position signal generation function 122. The motor position signal generation function 122 is configured to generate a motor position signal Y1. The motor position signal Y1 is a signal relating to the position of the pan / tilt motor 104. The motor position signal Y1 allows for the logging of reference points for the video camera 100 at different times, i.e., what pan / tilt position the video camera is in during a certain period of time. The motor position signal can be measured while the video camera 100 is panning and / or tilting. The motor position signal generation function 122 can be configured to generate a motor position signal from a motor position measured by one or more motor position sensors 105. The position signal can be related to the rotation of the pan / tilt motor 104 or the rotation of the gears of the pan / tilt motor 104. In particular, if the pan / tilt motor 104 is a stepper motor, the motor position signal generation function 122 can be configured to generate a motor position signal from a pan / tilt control signal generated by a pan / tilt function 121, either alternatively or in combination. The motor position signal is a signal indicating the time-dependent pan / tilt position of the video camera 100.
[0028] The control circuit 110 can be configured to execute a gyro signal generation function 123. The gyro signal generation function 123 is configured to generate a gyro signal Y2. The gyro signal Y2 relates to the shaking of a video camera detected by one or more gyroscope sensors 106.
[0029] One or more gyroscope sensors 106 do not have a fixed reference point. The gyroscope sensors 106 measure motion relative to inertia, or "standing still." On its own, the gyroscope sensors 106 cannot answer questions such as "where are we now?". On the other hand, one or more motor position sensors 105 know their position relative to a rigid body, such as an encoder ring. Therefore, one or more motor position sensors 105 have a fixed reference point. Unlike the gyro signal, the motor position signal cannot detect when the entire video camera 100 is vibrating. The motor position signal is a measure of the movement of the video camera 100 relative to the fixed reference point.
[0030] The control circuit 110 is configured to execute a reference signal generation function 124. The reference signal generation function 124 is configured to generate a reference signal from a predetermined movement curve of the pan / tilt motor 104. The reference signal is a reference to how the pan / tilt operation of the video camera 100 is performed without shaking of the video camera 100.
[0031] The control circuit 110 is configured to execute a signal coupling function 125. The signal coupling function 125 is configured to generate a coupled signal Y from the motor position signal Y1 and the gyro signal Y2. The coupled signal Y is given by Y = F1 * Y1+F2 *The signal is generated according to Y2, where F1 is a low-pass filter and F2 is a high-pass filter. For example, a monopole infinite impulse response IIR filter can be used. Such a filter can be used to perform both low-pass and high-pass filtering. Improved image stabilization can be achieved by forming a combined signal. Motor position signals are typically disturbed by high-frequency noise. High-frequency noise may be introduced into the motor position signal due to quantization errors. Gyroscope signals are typically disturbed by low-frequency noise. Because the gyroscope signal (a measure of angular velocity) must be integrated to obtain position, and because the gyroscope signal will always contain some degree of noise, the noise sums up and becomes non-negligible over time. Gyroscope signals also typically contain a DC offset. Therefore, when no motion is present, the gyroscope signal will give a small DC signal due to the DC offset. The DC offset is not constant and will change slowly over time, for example, depending on the temperature. This will also contribute to the low-frequency noise. Therefore, low-frequency noise can be introduced into the gyro signal due to errors caused by sensor imperfections and temperature changes, which are integrated over time. By applying a low-pass filter to the motor position signal and a high-pass filter to the gyro signal and combining the two signals, the noise in each can be filtered out at least partially, resulting in a more accurate signal that shows the shake of the video camera during panning and / or tilting of the video camera.
[0032] The control circuit 110 is configured to perform an image stabilization function 126. The image stabilization function 126 is configured to perform image stabilization on image frames of a video stream. Image stabilization is based on the difference between a coupled signal Y and a reference signal, for example, the reference signal minus the coupled signal. The image stabilization is preferably electronic image stabilization (EIS). EIS is generally known to those skilled in the art and will not be discussed in detail in this disclosure.
[0033] In relation to Figure 2, a method 200 for image stabilization of a video stream captured by a pannable and / or tiltable video camera is discussed. Method 200 is based on the inventors' insight that improved image stabilization of image frames captured by the video camera 100 can be achieved by generating a combined signal from a motor position signal filtered by a low-pass filter and a gyro signal filtered by a high-pass filter, and generating a reference signal from a predetermined movement curve of a pan / tilt motor 104 used to pan and / or tilt the video camera 100.
[0034] All or part of the steps of Method 200 can be performed by the video camera 100 function described above. However, it will also be noticed that some or all of the steps of Method 200 can be performed by similar functions performed on other devices. Method 200 includes the following steps. The steps can be performed in any appropriate order.
[0035] S202 generates a motor position signal Y1 for the pan / tilt motor 104 of the video camera 100. The motor position signal Y1 is a signal relating to the position of the pan / tilt motor 104. The motor position signal Y1 allows for logging a reference point for the video camera 100 at different times, i.e., what pan / tilt position the video camera is in during a certain period of time. The motor position signal is a signal that can be measured while the video camera 100 is panning and / or tilting. The motor position signal can be generated from the motor position measured by one or more motor position sensors 105. Alternatively or in combination, the motor position signal can be generated from the pan / tilt control signal. The motor position signal is a signal indicating the time-dependent pan / tilt position of the video camera 100.
[0036] S204 generates a gyro signal Y2 from the gyroscope sensor 106 of the video camera 100. The gyro signal Y2 relates to the shaking of the video camera detected by one or more gyroscope sensors 106.
[0037] S206 generates a reference signal from a predetermined movement curve of the pan / tilt motor 104 of the video camera 100. The reference signal is a reference to how the pan / tilt operation of the video camera 100 is performed without shaking of the video camera 100.
[0038] S208 generates a combined signal Y from the motor position signal Y1 and the gyro signal Y2. The combined signal is Y = F1 * Y1+F2 * It is generated according to Y2, where F1 is a low-pass filter and F2 is a high-pass filter.
[0039] S210 performs image stabilization on the video stream based on the difference between the combined signal Y and the reference signal. The image stabilization is preferably electronic image stabilization (EIS). EIS is generally known to those skilled in the art and will not be discussed in detail in this disclosure.
[0040] Those skilled in the art will recognize that the present invention is by no means limited to the preferred embodiments described above. Conversely, many modifications and variations are possible within the scope of the appended claims.
[0041] For example, a high-pass filter can be considered complementary to a low-pass filter. Therefore, F2 can be considered equal to 1 - F1.
[0042] Furthermore, modifications to the disclosed embodiments can be understood and implemented by a person skilled in the art when carrying out the claimed invention, based on a search of the drawings, disclosures, and appended claims.
Claims
1. A method for image stabilization of a video stream, comprising: Capturing a video stream by a pannable and / or tiltable video camera (100) during pan / tilt of said video camera (100); generating a motor position signal Y1 of a pan / tilt motor (104) of the video camera (S202); generating (S204) a gyro signal Y2 of a gyroscope sensor (106) of the video camera (100), the gyro signal Y2 being an integral of measurements from the gyroscope sensor (106); generating (S206) a reference signal from a predetermined movement curve of the pan / tilt motor (104) of the video camera (100), the reference signal being a reference for how the pan / tilt movement of the video camera (100) would be performed without shaking of the video camera (100); From the motor position signal Y1 and the gyro signal Y2, Y=F1 * Y1+F2 * generating a combined signal Y according to F1 and F2 (S208), where F1 is a low-pass filter and F2 is a high-pass filter; and performing (S210) electronic image stabilization on the video stream captured by the pannable and / or tiltable video camera (100) during pan / tilt of the video camera (100) based on a difference between the combined signal Y and the reference signal; A method comprising:
2. 2. The method of claim 1, wherein the high pass filter is complementary to the low pass filter such that F2=1-F1.
3. The method of claim 1, wherein the motor position signal is a signal indicating a time-dependent pan / tilt position of the video camera.
4. A non-transitory computer readable storage medium having stored thereon instructions for performing the method of any one of claims 1 to 3 when executed on a video camera having processing capabilities.
5. A video camera, an image sensor (102) configured to capture image frames of a video stream; a pan / tilt motor (104) configured to pan and / or tilt the video camera; a gyroscope sensor (106) configured to detect shaking of the video camera; and a control circuit section (110), a motor position signal generating function (122) configured to generate a motor position signal Y1 of the pan / tilt motor (104); a gyro signal generating function (123) configured to generate a gyro signal Y2 related to a shake of the video camera detected by the gyro sensor (106), the gyro signal Y2 being an integral of a measurement from the gyro sensor (106); a reference signal generation function (124) configured to generate a reference signal from a predetermined movement curve of the pan / tilt motor (104), the reference signal being a reference for how a pan / tilt movement of the video camera would be performed without shaking of the video camera; From the motor position signal Y1 and the gyro signal Y2, Y=F1 * Y1+F2 * a signal combining function (125) configured to generate a combined signal Y according to F1, F2, F3, F4, F5, F6, F7, F8, F9, F10, F11, F12, F13, F14, F15, F16, F17, F18, F19, F20, F21, F22, F23, F24, F25, F26, F27, F28, F29, F30, F31, F32, F33, F34, F35, F36, F37, F38, F39, F40, F41, F42, F43, F44, F45, F46, F47, F48, F49, F50, F51, F52, F53, F54, F55, F56, F57, F60, F61, F62, F63, F6 an image stabilization function (126) configured to perform image stabilization on image frames of a video stream captured by the video camera during pan / tilt of the video camera based on a difference between the combined signal Y and the reference signal; The video camera is configured to:
6. 5. The video camera of claim 4, wherein the high pass filter is complementary to the low pass filter such that F2=1-F1.
7. The video camera of claim 5, wherein the motor position signal is a signal indicative of a time-dependent pan / tilt position of the video camera.
